)]}'
{
  "commit": "2ce6926916ed5095aa922a3cfd105f967139dd82",
  "tree": "706588ae0f98a2e05a733b78c7348f2e19e0b80a",
  "parents": [
    "aab676a1acbce65072607a0baae0dab739e0cc86"
  ],
  "author": {
    "name": "Benoit Jacob",
    "email": "jacob.benoit.1@gmail.com",
    "time": "Fri Jul 03 11:37:29 2026 -0400"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Fri Jul 03 15:37:29 2026 +0000"
  },
  "message": "[Codegen][CPU] Fill in the bf16 and i8 ukernel bodies + e2e tests. (#24572)\n\nReplaces the bf16 and i8-VNNI seeds\u0027 stub bodies with real SIMD\nimplementations, both generic over the `intrinsics_{m,n,k}` unrolling\nfactors and structured like the AMDGPU C ukernel they were adapted from\n(`iree_uk_amdgpu_multi_mma_mfma_i32_16x16x32_i8`): accumulators in\nregisters, an outer loop over the K tiles (`k_outer`), and inside it the\n`(intrinsics_m, intrinsics_n, intrinsics_k)` unroll. The `intrinsics_*`\narrive as constants at the inlined call site, so the loops fully unroll\nand the `acc_regs` arrays become fixed register files -- the bitcode-LTO\nequivalent of a C++ template, as the README describes.\n\n- bf16 (`MMA_X86_AVX512BF16_1x16x2_F32_BF16`): one `_mm512_dpbf16_ps`\nper (m, n, k), with the LHS K-pair broadcast via `set1_ps`.\n- i8 (`MMA_X86_AVX512VNNI_16x16x2_I32_I8_CASTI16`): the 16x16x2 tile is\nbit-compatible with the codegen path `lowerX86Avx512Vnni16x16x2I8` --\none `vpmovsxbw` widen of each i8 panel to i16, the `vpshufd` /\n`vbroadcasti32x4` fan-out, and 16 `vpdpwssd` over the block-interleaved\n(rlo, chi, rhi, clo) ACC layout. The i8 ukernel needs `-mavx512bw` for\nthe widen, so it is added to the VNNI copts.\n\n`LLVMCPUSelectUKernels` now only selects a ukernel when its bitcode\nactually exists (via `attachUKernelBitcodeOnOp`\u0027s bool return), so an\n`MMAIntrinsic` the cost model picks but for which no seed exists -- e.g.\nthe M\u003c-\u003eN-swapped `MMA_X86_AVX512BF16_16x1x2_F32_BF16` -- falls back to\ncodegen instead of dangling an undefined symbol.\n\nAdds two execution/numerical tests, the first of the new C-bitcode\nukernel path: `e2e_matmul_cpu_dt_inner_tiled_llvm_ukernel_bf16_f32`\n(avx512bf16) and `..._i8_i32` (avx512vnni). Each compiles a data-tiled\nmatmul with `--iree-llvmcpu-enable-llvm-ukernels\u003dinner_tiled`, links the\nukernel bitcode, runs on host and checks results against a reference --\nexercising the operand threading and generic `intrinsics_{m,n,k}`\nunrolling that the IR-level lit tests cannot. Both were confirmed to\nactually select their ukernel (not silently fall back to codegen).\n\nProgress towards https://github.com/iree-org/iree/issues/24574.\n\nSigned-off-by: Benoit Jacob \u003cjacob.benoit.1@gmail.com\u003e\nCo-authored-by: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "d99ef1863520557f743e2c15af806937796b3dc1",
      "old_mode": 33188,
      "old_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/BUILD.bazel",
      "new_id": "7b9a8e1b88bd0c05b42c5a6925d73b87c02988f2",
      "new_mode": 33188,
      "new_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/BUILD.bazel"
    },
    {
      "type": "modify",
      "old_id": "c7b3069c4f2494f603d3391013fd3a8806addfc8",
      "old_mode": 33188,
      "old_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/CMakeLists.txt",
      "new_id": "3e33a4794c82e086bda6460984fb648d21dc76a8",
      "new_mode": 33188,
      "new_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/CMakeLists.txt"
    },
    {
      "type": "modify",
      "old_id": "c7a57289b20c7dec75d2f2011594256eca2c55c2",
      "old_mode": 33188,
      "old_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/iree_uk_mma_x86_avx512bf16_1x16x2_f32_bf16.c",
      "new_id": "65d1ad747e472f555c346b987802334217afa887",
      "new_mode": 33188,
      "new_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/iree_uk_mma_x86_avx512bf16_1x16x2_f32_bf16.c"
    },
    {
      "type": "modify",
      "old_id": "9e27639cf53279dc0b2dfb4576610684d669f8f9",
      "old_mode": 33188,
      "old_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/iree_uk_mma_x86_avx512vnni_16x16x2_i32_i8_casti16.c",
      "new_id": "c13da434a01ba34edc54205ea14331f73b076650",
      "new_mode": 33188,
      "new_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/iree_uk_mma_x86_avx512vnni_16x16x2_i32_i8_casti16.c"
    },
    {
      "type": "modify",
      "old_id": "2952f41f1868443f870e47df207ab0ed10d11919",
      "old_mode": 33188,
      "old_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/test/select_ukernel.mlir",
      "new_id": "5365ba744d9f582be6c1eed972f3edcca9922fc7",
      "new_mode": 33188,
      "new_path": "compiler/plugins/target/LLVMCPU/builtins/ukernel/test/select_ukernel.mlir"
    },
    {
      "type": "modify",
      "old_id": "bb977866e7b551492de72047c307181c013248a5",
      "old_mode": 33188,
      "old_path": "compiler/src/iree/compiler/Codegen/LLVMCPU/LLVMCPUSelectUKernels.cpp",
      "new_id": "c52b714e3977a3f12af9e05aaefd71191482763c",
      "new_mode": 33188,
      "new_path": "compiler/src/iree/compiler/Codegen/LLVMCPU/LLVMCPUSelectUKernels.cpp"
    },
    {
      "type": "modify",
      "old_id": "0ae01d9384f5ecf7d90a2fb23207c9059a83f971",
      "old_mode": 33188,
      "old_path": "tests/e2e/matmul/BUILD.bazel",
      "new_id": "872298a8d993c7af9bd3f562482ec04bcea62c91",
      "new_mode": 33188,
      "new_path": "tests/e2e/matmul/BUILD.bazel"
    },
    {
      "type": "modify",
      "old_id": "58b4c0921130f32452a192c97f43222b14979262",
      "old_mode": 33188,
      "old_path": "tests/e2e/matmul/CMakeLists.txt",
      "new_id": "e31ef1195635b024f22e8370c21cfa03198d35ab",
      "new_mode": 33188,
      "new_path": "tests/e2e/matmul/CMakeLists.txt"
    }
  ]
}