commit 38bfdbae7e94e57d932ca740b42589ea40d85196
author Max191 <44243577+Max191@users.noreply.github.com> Tue Nov 07 10:13:19 2023 -0500
committer GitHub <noreply@github.com> Tue Nov 07 15:13:19 2023 +0000
tree 356a85a329e85b09e939bfbc17c33e1b949c5132
parent 1f61c88b30f58c3f8836d0b403e49ed613c55010

[GlobalOptimization] Add pattern to reassociate dequantization + matmul `linalg.gen… (#15278)

…eric` ops

Dequantization ops that are consumed by matmuls are currently only fused
into a dispatch region, but we can do even better by reassociating these
fused operations (see https://github.com/openxla/iree/issues/14951).

It is important to note that this pattern does affect precision, and is
a trade off between precision and performance. It is set to opt-in with
`--iree-global-opt-enable-quantized-matmul-reassociation`

This pattern rewrites a sequence of dequantization->matmul
`linalg.generic` ops into a new sequence of `linalg.generic` ops. The
new sequence of ops is as follows:

1. A sequence of `linalg.generic` ops that dynamically quantize the
non-quantized input to the matmul. This is very cheap in skinny matmul
cases, where the non-quantized input is small compared to the quantized
input.
2. A `linalg.generic` op that performs an integer matmul. This is the
key performance optimization here. On CPU, we want to be doing integer
matmuls where we can, but the matmul needs to be picked up by a
VectorContractCustomKernel for now. Eventually it will be better to
rewrite to `linalg.matmul` here to target ukernels.
3. A final `linalg.generic` op that performs the dequantization scale
and zero point math, as well as performing the remaining reduction of
the matmul. The matmul from 2. only reduces within quantized groups,
while this op does the reduction across groups.

This also moves the FuseDequantizationMatmul pass to GlobalOptimization