tree 71a7b03495dca7b586d8a2bd8a7f56ced1cee637
parent e53a335559976a31ed23225480a1baff823d2888
author Zmicier Prybysh <zprybysh@baylibre.com> 1784722037 +0200
committer GitHub <noreply@github.com> 1784722037 +0000
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJqYLJ1CRC1aQ7uu5UhlAAARYwQAIUZkkF6KwRUQ3PJhoDNjLpt
 ssOLXBE5v0Q6ItBmK72Gsc2v/fVFWK1N2CHDDPMlzOn3NSDwhABA41h/xmbyQ1ZN
 xd0knlkm74sJHeSWqcCGvHVggX5pfqWhWv9l47sVCURiufvqxOwq5xX/d1+jZ1/8
 B/5rhBX+DrHwJX6Xf8/AML9psZHvDTRf+CX5EnLHY6hHQIu4vSv2IJePCB7GLvzF
 K+7Pz5HZuXtE9WRlI1dH2m5tLSE5ToZo95Mef9lv1bNfGdd3btE4fvgBnpPO84If
 dRxAoYss6PUJucQeYq+dxH7LQBaFNsolY6UjYSQaGJAO48VitYuciX9RJUQw9Inm
 orZXwK9gJkX6pfgTUnMm72kX0w6kNFgYlx6sbySdzdtOh/OcHtHB8G4/lCMS4xCi
 lU8ubKvZOKBDrxkcTvTB+mnm8XBdBRMw9JV/wKPDlTHc4HC97s0ew93MILz6rjts
 JpVeuzW1pREX5jNgyFwzvlVMESpuWvnVLhitB8lUY6n7cd/9B2SfeeXu8JPxOD5z
 7R7sqZRC51BUn5BP+eIqOSsnIXAu1RhcjgI3voLbCJ2rzTE2ZY7LYHdt2aV3W4BG
 5H6uDqnKwufTB61jfFmkqEDDhLlY9SwwKPCJox53bnvg1oZBC0q4V9YbuKT1AR/d
 c29ufcbkuzcAAt3Zy7s1
 =VpkU
 -----END PGP SIGNATURE-----
 

[LLVMCPU][RISCV] Add RVV bf16 mmt4d ukernel using the Zvfbfwma widening MAC (bf16*bf16->f32) (#24695)

## Summary

On RISC-V, a data-tiled bf16 matmul was computed by promoting LHS/RHS to
f32, ignoring the `Zvfbfwma` bf16 widening multiply-accumulate extension
(`vfwmaccbf16`) that LLVM already lowers to and that shipping silicon
implements. This PR adds a native `bf16 x bf16 -> f32` `mmt4d`
microkernel for RVV+`Zvfbfwma`. Naturally, this change is related to
(and was implementing looking at) the existing f32, `int8` (#23734), and
f16/`Zvfh` (#22231) ukernels.

## How the change was tested

1. Compiler e2e
A `128x256 bf16 * 256x128 bf16 -> 128x128 f32` matmul compiled for
`riscv64 +v,+zvfbfwma` with `--iree-opt-data-tiling
--iree-llvmcpu-enable-ukernels=all`:
   Before: dispatch contains `arith.extf` f32-promotion, no ukernel.
After: `Mmt4dTilingExpert` dispatch `matmul_128x128x256_bf16xbf16xf32`
with `iree_uk_mmt4d` + `iree_uk_pack`; **zero `arith.extf`**; all four
`iree_uk_mmt4d_tile_bf16bf16f32_*_riscv_64_zvfbfwma` symbols present;
generated RISC-V asm inner loop is `vfwmaccbf16.vf`.
2. Numerics under `qemu-riscv64`
A standalone harness calling the real tile functions matches an f32
reference exactly. A control run without the extension crashes,
confirming `vfwmaccbf16` is genuinely executed. I used qemu 8.2 which
exposes the extension as the experimental `-cpu max,x-zvfbfwma=true`
property.
3. Tile selection under `qemu-riscv64`.
Verified `iree_uk_mmt4d_select_tile_func_arch` (the
`entry_point.c`/`tiles.inl` wiring) returns the correct bf16 tile for
each M0 value when `cpu_data` has the `Zvfbfwma` bit, and does not
select it when the bit is clear — i.e. the feature is only enabled for
`Zvfbfwma`.
4. In-tree `mmt4d_test`.
The added case exercises the same path in CI's RISC-V cross-build (falls
back to the generic tile if the CI toolchain can't assemble `Zvfbfwma`,
i think it's the same limitation as `Zvfh` in #22303).

### A note on CI coverage

If I understand correctly, the RISC-V CI job runs under an older
`qemu-riscv64` on a stock `rv64gcv` profile that doesn't expose
`Zvfbfwma`, and `mmt4d_test` *skips* (does not fail) any case whose
feature the runtime CPU lacks. Zvfbfwma is only assembled/executed on a
toolchain + qemu new enough to support it (qemu ≥ 8.2 exposes it as the
experimental `x-zvfbfwma` property). My point is: a green CI does not
mean the bf16 kernel ran in CI — the functional validation of record is
the local `qemu-riscv64` 8.2 numerics + tile-selection runs described
above.

Disclosure: this change was assisted by Claude Code, but the change was
reviewed and thoroughly tested before being submitted.

---------

Signed-off-by: Zmicier Prybysh <zprybysh@baylibre.com>