tree 706588ae0f98a2e05a733b78c7348f2e19e0b80a
parent aab676a1acbce65072607a0baae0dab739e0cc86
author Benoit Jacob <jacob.benoit.1@gmail.com> 1783093049 -0400
committer GitHub <noreply@github.com> 1783093049 +0000
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJqR9c5CRC1aQ7uu5UhlAAAX/QQAGEIXzFt/bYkTZ2+I7Vs9lLe
 0HWB1ddpyb5x9g9QRxF4hlFfBM8uOh1jITEBnGKQwUIBPDL0puih8teUZMogJVQd
 6/SAW2hWM3O55nMDaZ7a/6+PP5WCyUOWVksFnKFMyMc/LrnAbpsgjrZNV3/g7PHx
 CzGxOcZxPganGo88Dot2cRCZMWE9ybvxfZQT52uZC+pNraDrfK7SiNIpkdHIxoq1
 OMvHdT6/7PHKSmFs8RbhtOEZO4YgLdfoEjgN2qmRmc7GGww6f7Spcb3EXAXu0Vb/
 IXRWBVLh0AlTd0tyNIFhHyJo5JKIKY/4nRtgIcAWpC/rCmniQ0ValSSfpgPbo5nE
 QR73yusfI9+8VaVaZ0anzxnGJ4pFgHqpa5VmW75MPFdnI+QHo39XDfmV3uKaNEvo
 72xpiFp8zGKQX3mydTVZIZwm+OPjBc3gSi6h9lA2OBxdTNHW72rc2HgGlFkM0Ssc
 P8eu0hberQBdTEUrUUhtb7/mpsmTl7LtQQMyt9A2AEIw/4L/LB1f7cGb/VYfsUnr
 94H15EjHsy/q0eXGBIsnlX3uM0LFJ5aVG83gpNy+m5Loel19gUo4l9QVccUqzmoM
 wEP3dHDRvZX8fcYYHqeamIpYo73+uXWZzR+uaisPara7sGx6rRnzKHzmeawB/2KX
 DvHt3TNF0fMTFAJzjzXs
 =9OLY
 -----END PGP SIGNATURE-----
 

[Codegen][CPU] Fill in the bf16 and i8 ukernel bodies + e2e tests. (#24572)

Replaces the bf16 and i8-VNNI seeds' stub bodies with real SIMD
implementations, both generic over the `intrinsics_{m,n,k}` unrolling
factors and structured like the AMDGPU C ukernel they were adapted from
(`iree_uk_amdgpu_multi_mma_mfma_i32_16x16x32_i8`): accumulators in
registers, an outer loop over the K tiles (`k_outer`), and inside it the
`(intrinsics_m, intrinsics_n, intrinsics_k)` unroll. The `intrinsics_*`
arrive as constants at the inlined call site, so the loops fully unroll
and the `acc_regs` arrays become fixed register files -- the bitcode-LTO
equivalent of a C++ template, as the README describes.

- bf16 (`MMA_X86_AVX512BF16_1x16x2_F32_BF16`): one `_mm512_dpbf16_ps`
per (m, n, k), with the LHS K-pair broadcast via `set1_ps`.
- i8 (`MMA_X86_AVX512VNNI_16x16x2_I32_I8_CASTI16`): the 16x16x2 tile is
bit-compatible with the codegen path `lowerX86Avx512Vnni16x16x2I8` --
one `vpmovsxbw` widen of each i8 panel to i16, the `vpshufd` /
`vbroadcasti32x4` fan-out, and 16 `vpdpwssd` over the block-interleaved
(rlo, chi, rhi, clo) ACC layout. The i8 ukernel needs `-mavx512bw` for
the widen, so it is added to the VNNI copts.

`LLVMCPUSelectUKernels` now only selects a ukernel when its bitcode
actually exists (via `attachUKernelBitcodeOnOp`'s bool return), so an
`MMAIntrinsic` the cost model picks but for which no seed exists -- e.g.
the M<->N-swapped `MMA_X86_AVX512BF16_16x1x2_F32_BF16` -- falls back to
codegen instead of dangling an undefined symbol.

Adds two execution/numerical tests, the first of the new C-bitcode
ukernel path: `e2e_matmul_cpu_dt_inner_tiled_llvm_ukernel_bf16_f32`
(avx512bf16) and `..._i8_i32` (avx512vnni). Each compiles a data-tiled
matmul with `--iree-llvmcpu-enable-llvm-ukernels=inner_tiled`, links the
ukernel bitcode, runs on host and checks results against a reference --
exercising the operand threading and generic `intrinsics_{m,n,k}`
unrolling that the IR-level lit tests cannot. Both were confirmed to
actually select their ukernel (not silently fall back to codegen).

Progress towards https://github.com/iree-org/iree/issues/24574.

Signed-off-by: Benoit Jacob <jacob.benoit.1@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>