[test] Refactor system tests to use pytest as runner
System tests, living in `test/systemtest`, test the whole end-to-end
design of OpenTitan: they initialize the target (e.g. a simulation, an
FPGA, or later an ASIC), load test software as needed, and check for
outputs or other expected behavior. To do so, multiple components need
to work together. For example, a simulation process, components for data
transfer (such as OpenOCD), components which interact with (simulated or
real) external interfaces, etc. Pytest provides a very convenient way to
handle such situations through their fixtures. Our current system tests
make use of this functionality.
What we haven't been using in the past is the ability of pytest to serve
as test runner: select tests, collect their output and results, etc.
(Instead, a shell script was used.)
This PR refactors the system tests to make use of pytest as test runner,
with one main consequence: Tests are now "opinionated", they don't take
configuration parameters through command line arguments any more. Tests
expect build outputs in the distribution directory (`build-bin`) to be
in the well-known layout for this directory.
Notes:
* This commit lays the groundwork to add tests running on an FPGA target
in a similar way as the verilated tests, which should explain some of
the structuring of the code.
* The ability to run the debug "compliance test" built into OpenOCD,
which was present in the `test/systemtest/openocd_verilator_test.py`
file, has been removed temporarily. This test was unmaintained, and
will be brought back in revised form at a later time.
* Equally, the "FPGA test runner" available in
`test/systemtest/functional_fpga_test.py` has been rarely used (as a
poll on Slack showed) and is also removed.
It will be brought back in modified form in a future commit.
Signed-off-by: Philipp Wagner <phw@lowrisc.org>
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 3c611d3..79233ce 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -353,9 +353,9 @@
- template: ci/download-artifacts-template.yml
- bash: |
. util/build_consts.sh
- export VERILATED_SYSTEM_PATH="$BIN_DIR/hw/top_earlgrey/Vtop_earlgrey_verilator"
pytest --version
- ci/run_verilator_pytest.sh
+ pytest test/systemtest/earlgrey/test_sim_verilator.py \
+ --log-cli-level=DEBUG
displayName: Execute tests
- template: ci/run-riscv-compliance.yml
