[ci] Disable RISC-V compliance tests
The RISC-V compliance tests are important to catch issues in the CPU and
the interconnect, but take a very long time to run (30-60 min for each
of the two chunks) in Verilator simulation on the Microsoft-provided
Azure Runners. Furthermore, the runners we get are rather unpredictable
in their performance, leading to huge variations in the test execution
time.
At the same time, these tests didn't flag a new issue recently which
wasn't caught somewhere else. Therefore, disabling them is unlikely to
reduce the quality of the testing much.
We still need to find a way to re-enable these tests. Running them on
the FPGA platform is likely to be a good approach, which will bring down
test times massively. Having a dedicated, powerful runner could be
another option if we want to continue running the tests in simulation.
Fixes #4128
Signed-off-by: Philipp Wagner <phw@lowrisc.org>
diff --git a/ci/run-riscv-compliance.yml b/ci/run-riscv-compliance.yml
index 9a39070..4f4d8bb 100644
--- a/ci/run-riscv-compliance.yml
+++ b/ci/run-riscv-compliance.yml
@@ -7,6 +7,9 @@
jobs:
- job: riscv_compliance_tests_${{ join('_', parameters.rvc_test_suites) }}
+ # Disable RISC-V Compliance tests due to excessive runtimes in simulation
+ # (#4128).
+ condition: false
displayName: Execute RISC-V compliance tests for ${{ join(' ', parameters.rvc_test_suites) }}
pool:
vmImage: ubuntu-18.04
