commit 03aaf320a4b519469668dead50d12ef1c136fb3b
author Philipp Wagner <phw@lowrisc.org> Wed Sep 09 19:11:44 2020 +0100
committer Philipp Wagner <mail@philipp-wagner.com> Tue Oct 06 14:56:47 2020 +0100
tree 3fe241e9efcd55cc0904d10422a318e2240a50f5
parent bb0b2768312ec4219805ef952ab85dae77348611

[test] Refactor system tests to use pytest as runner

System tests, living in `test/systemtest`, test the whole end-to-end
design of OpenTitan: they initialize the target (e.g. a simulation, an
FPGA, or later an ASIC), load test software as needed, and check for
outputs or other expected behavior. To do so, multiple components need
to work together. For example, a simulation process, components for data
transfer (such as OpenOCD), components which interact with (simulated or
real) external interfaces, etc. Pytest provides a very convenient way to
handle such situations through their fixtures. Our current system tests
make use of this functionality.

What we haven't been using in the past is the ability of pytest to serve
as test runner: select tests, collect their output and results, etc.
(Instead, a shell script was used.)

This PR refactors the system tests to make use of pytest as test runner,
with one main consequence: Tests are now "opinionated", they don't take
configuration parameters through command line arguments any more. Tests
expect build outputs in the distribution directory (`build-bin`) to be
in the well-known layout for this directory.

Notes:
* This commit lays the groundwork to add tests running on an FPGA target
  in a similar way as the verilated tests, which should explain some of
  the structuring of the code.
* The ability to run the debug "compliance test" built into OpenOCD,
  which was present in the `test/systemtest/openocd_verilator_test.py`
  file, has been removed temporarily. This test was unmaintained, and
  will be brought back in revised form at a later time.
* Equally, the "FPGA test runner" available in
  `test/systemtest/functional_fpga_test.py` has been rarely used (as a
  poll on Slack showed) and is also removed.
  It will be brought back in modified form in a future commit.

Signed-off-by: Philipp Wagner <phw@lowrisc.org>