tree 3e6e9f66522b7df7c118975fedaf6b2be8a7735c
parent c397258f87007840b8b66e66789400a9daef1c68
author bjacob <benoitjacob@google.com> 1697236581 -0400
committer GitHub <noreply@github.com> 1697236581 -0400
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJlKcZlCRBK7hj4Ov3rIwAA8DQIAJQ7F9UwEW4InOg7NqMrHWRy
 q7KWIiCh8LR2WjJN99RXsTB1SIJ/BaFHtxQqJBCJZSAM+gH6vBhoaiLOoxN/orW4
 FHb13dU3EfQaFOSNYXJEtWqTwzBa0DtyxD3TMKRCVJCOwaZugy9tYowUOwYdUvns
 nGo356Pvjjx1FBH8EXypcmAVPNq/ZqpoZLrgNseXq5WM/dLegt8pzukQDoIfXhrq
 Y575qM0J0quT/o3nQ8Y8rnzPZdLJU6a42iioqbC2tXAe6diAkn7ldtdIGSfg30+G
 9Sr4XGRi/vjghbJjyevKufg5ZcyLPkWDkad9WpBX1zwNZc2FV7WabJOq6qXkX50=
 =Mm4A
 -----END PGP SIGNATURE-----
 

Limit the latency of e2e matmul tests  (#15180)

Now that I have an AMD workstation, I'm running e2e matmul tests with
Vulkan for the first time, and I'm horrified how slow it is. Test
latencies ~ 1 minute.

Profiling, found that that's due to the slow reference matmul code that
this is comparing against. And it's got nothing to do with GPU, it's
just that the GPU e2e matmul tests are the only ones exercising really
large sizes. It's roughly equally slow on all element types.

There isn't a perfect solution to that problem. Tests need to compare
against some reference. Golden test data is a pain to maintain.
"Reference" implementations that are actually optimized are a
combination of less reliable and more maintenance. And so on.

I do believe though that test latency is very important as it affects
everyone's productivity.

This PR adds a command-line flag to the e2e matmul test runner:

```c
IREE_FLAG(
    int32_t, max_elements_to_check, 10000,
    "Maximum number of matrix elements to check for each matmul. For larger "
    "matrices, only every n-th element will be checked for some n chosed to "
    "stay just under that threshold and to avoid being a divisor of the inner "
    "dimension size to avoid special patterns. As the check uses a slow "
    "reference implementation, this is a trade-off between test latency and "
    "coverage. The value 0 means check all elements.");
```

Timing this test command:
```
ctest -R e2e_matmul_.*_vulkan -j32
```

Before: total latency 48 seconds,  `266.22 sec*proc`.
After: total latency 3.7 seconds, `20.43 sec*proc`.

=> 13x latency shrink