tree e5ab847b1a4077e1439b67e06318040a678c6c7c
parent cc50107f19976cf2244ee152074ffc0c46587491
author Benoit Jacob <jacob.benoit.1@gmail.com> 1780416522 -0400
committer GitHub <noreply@github.com> 1780416522 -0400
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJqHwAKCRC1aQ7uu5UhlAAAbjUQACJ+9quX2qoM+XfJwD7aZbEL
 mr8kLSCWieaNCc0/o012dRbFU/NT5/1Gjx9f58ONsbG3VRP6T/nTbAFEmnHf8fOv
 FqhyhmgfnIwu5tjW6YbWP1MjeN4rz2trGfDvs38ZaQVdFa3F2kxYipQjmeiukwd3
 3zoXLOX3CQXrmqtweIaWSxLs1zXKtcPwS5hLGxAq5rHb2pKcjfJ/PS+Fp/raAdtP
 wtNh7iJwbdKo73c1utctKWCdjxGX49T2P468d0UZbrzdOzBwgiYc6UDOPlW2R1kE
 L6mIY08qJ9t2eaZK1mVJ7EkNrEG5InpgC4/aERKAAN/vu/4TyMDUXf7q60JDMxSD
 7GSgdpV2IbgqWVCtpjH81smyQRU9NczYZU7mKatJu6rwx56CTiYD5bA9gMESFI40
 Ly7X3qxAaYwNmHLfaWWHsxNDqEN6vSvX+9lY+OWVjX7PJE9oEFF636VhQx/OHa4A
 NMNhC9X+Wlf35YZTJTu7q/dYY66J6So5LGvaprN7fL0d5AYtAVsWZbIQ5iI5eDZZ
 p2Ugg2b4q9RwNv6bq6UTFjMoO9Djw0IruSH3HRB8GQjrkAohkdSAwbxVnVP01pRa
 +gQnLn8K69MjX7OaSnMMx5H1D7F/NUU5q0jBTbmK2Z0htLWJeGE5OJChWLUr8uy1
 elwp1IkSYIEgMADmrD+q
 =uXui
 -----END PGP SIGNATURE-----
 

[Codegen][CPU] Flatten contiguous trailing dims of transfers before unrolling. (#24517)

`VectorTransferLoweringPass` applies the MLIR transfer-lowering patterns
with `maxTransferRank=1` plus full-unroll, which fully unrolls any
rank-N>1 `vector.transfer_read`/`transfer_write` to multiple rank-1
transfers (one per index of the outer dim). For multi-dim tiles whose
trailing dims are contiguous in memory, this unrolls a single wide load
into many narrow ones, which then have to be reassembled into a wide
vector via a chain of `shufflevector`s in the hot inner loop.

Example surfacing the cost: a 4096x4096 dynamic-shape bf16xbf16->f32
matmul with `--iree-llvmcpu-enable-inner-tiled` on Zen 4 lowered to
inner_tiled with N=16, K_inner=2. The RHS for one K-step is a
`vector<16x2xbf16>` from a contiguous 64-byte slice. Unrolling to 16
separate `<2 x bfloat>` loads forced a sequence of `vpermt2d`/
`vpermt2q` per K-iteration in the inner loop to rebuild the wide RHS
register — accounting for ~3 cycles of extra work per K-step on top of
the 29 dpbf16ps doing the real work.

Apply `populateFlattenVectorTransferPatterns` *before* the
rank-reduction patterns. It rewrites a multi-dim transfer with
contiguous trailing dims into a transfer on a `memref.collapse_shape`
view + a `vector.shape_cast`, so the read ends up as a single 1-D
transfer over the collapsed view and lowers to one wide `vector.load`.
Per-fragment effect on the matmul benchmark above: 80.8 ms -> 67.1 ms
(1.20x). Combined with the m_bcst-fold broadcast routing in a sibling
commit, end-to-end gets to 53.4 ms (within 5% of the precompiled mmt4d
ukernel at 50.9 ms).

Test fallout: two pipelines now lower a per-row pack-tile load into a
single wide load over a collapsed-memref view rather than one load per
row (`aligned_unpack_generic` in pipeline_pack_unpack_tests) / write a
constant `vector<4x2xi1>` mask as a single flat `vector<8xi1>` store
(`transpose_mask` in vector_lowering). The new IR is strictly fewer ops
in both cases; updated the CHECK lines to match.

Progress towards #24515.

---------

Signed-off-by: Benoit Jacob <jacob.benoit.1@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>