commit b6c7c62d15bffe4d218bab1df707799251835c22
author bjacob <benoitjacob@google.com> Fri Oct 13 18:36:21 2023 -0400
committer GitHub <noreply@github.com> Fri Oct 13 18:36:21 2023 -0400
tree 3e6e9f66522b7df7c118975fedaf6b2be8a7735c
parent c397258f87007840b8b66e66789400a9daef1c68

Limit the latency of e2e matmul tests  (#15180)

Now that I have an AMD workstation, I'm running e2e matmul tests with
Vulkan for the first time, and I'm horrified how slow it is. Test
latencies ~ 1 minute.

Profiling, found that that's due to the slow reference matmul code that
this is comparing against. And it's got nothing to do with GPU, it's
just that the GPU e2e matmul tests are the only ones exercising really
large sizes. It's roughly equally slow on all element types.

There isn't a perfect solution to that problem. Tests need to compare
against some reference. Golden test data is a pain to maintain.
"Reference" implementations that are actually optimized are a
combination of less reliable and more maintenance. And so on.

I do believe though that test latency is very important as it affects
everyone's productivity.

This PR adds a command-line flag to the e2e matmul test runner:

```c
IREE_FLAG(
    int32_t, max_elements_to_check, 10000,
    "Maximum number of matrix elements to check for each matmul. For larger "
    "matrices, only every n-th element will be checked for some n chosed to "
    "stay just under that threshold and to avoid being a divisor of the inner "
    "dimension size to avoid special patterns. As the check uses a slow "
    "reference implementation, this is a trade-off between test latency and "
    "coverage. The value 0 means check all elements.");
```

Timing this test command:
```
ctest -R e2e_matmul_.*_vulkan -j32
```

Before: total latency 48 seconds,  `266.22 sec*proc`.
After: total latency 3.7 seconds, `20.43 sec*proc`.

=> 13x latency shrink