[doc] document styling rules for chip-level tests This completes the final tasks that fix #7976. Specifically, this completes the task that states: "Documentation will be added to the website to describe [sw/device/tests] subfoldering rule and naming conventions. Signed-off-by: Timothy Trippel <ttrippel@google.com>
diff --git a/sw/device/_index.md b/sw/device/_index.md index 3dda400..6f98f67 100644 --- a/sw/device/_index.md +++ b/sw/device/_index.md
@@ -16,8 +16,7 @@ There are also some other standalone firmware images in the repository, which are only used for testing. -- [`sw/device/tests`]({{< relref "sw/device/tests/README.md" >}}) contains many standalone smoke test images. - This directory also contains unit tests for software modules, which are executed on host (not device) systems using [GoogleTest](https://github.com/google/googletest). +- [`sw/device/tests`]({{< relref "sw/device/tests/index.md" >}}) contains several categories of chip-level tests, including smoke, IP integration, and system-level (use case) tests. - [`sw/device/benchmarks/coremark`]({{< relref "sw/device/benchmarks/coremark/README.md" >}}) contains infrastructure for running the [CoreMark](https://github.com/eembc/coremark) benchmark suite on the OpenTitan device. - `sw/device/riscv_compliance_support` contains infrastructure so we can run the [RISC-V Compliance](https://github.com/riscv/riscv-compliance) tests on the OpenTitan core.
diff --git a/sw/device/lib/testing/_index.md b/sw/device/lib/testing/_index.md index 28a9bfd..c0be277 100644 --- a/sw/device/lib/testing/_index.md +++ b/sw/device/lib/testing/_index.md
@@ -2,28 +2,20 @@ title: "Chip-Level Test Libraries" --- -This subtree provides libary code for writing chip-level tests. -Test library code should be considered any (reusable) code that could aid in the writing of chip-level tests. +# Overview -## Test Types -- **Chip-level Tests** - A collection of software level tests that run on OpenTitan hardware, whose main purpose is pre-silicon verification and post-silicon bringup. -These tests consist of: smoke, IP integration, and system-level tests. -These tests are all executed on-device using the [on-device test framework]({{< relref "sw/device/lib/testing/test_framework/index.md" >}}). -While most of these tests are top-level agnostic, some are not. - - **Smoke Tests** - A software level test, written in C using DIFs, that performs a minimal set of operations on a given IP block to verify that it is alive and functioning. - - **IP Integration Tests** - A software level test, written in C, that exercises some specific functionality specific to a given IP and toplevel. - The Eargrey toplevel [test plan](https://docs.opentitan.org/hw/top_earlgrey/doc/dv/#testplan) describes these tests. - - **System-level Scenario Test** - A software level test, written in C, that mimics complex system level use case scenarios. - Such a test is designed to encompass multiple pieces of functionality tested by the IP integration tests. +This subtree contains _test libary code_ that could aid in the writing of [chip-level tests]({{< relref "sw/device/tests/index.md" >}}). + +# Organization and Style Guide + +## File Naming Conventions +- Test libary code will be named: **{IP or functionality name}_testutils.{h,c}** ## Subfoldering Rules - [on-device test framework]({{< relref "sw/device/lib/testing/test_framework/index.md" >}}) code will live in: **sw/device/lib/testing/test\_framework**. - Remaining test library code will **_not_** be subfoldered. -## File Naming Conventions -- Test libary code will be named: **{IP or functionality name}_testutils.{h,c}** - -## Documentation Index +# Documentation Index {{% sectionContent %}}
diff --git a/sw/device/lib/testing/test_framework/index.md b/sw/device/lib/testing/test_framework/index.md index 1794627..dbbcc8f 100644 --- a/sw/device/lib/testing/test_framework/index.md +++ b/sw/device/lib/testing/test_framework/index.md
@@ -2,13 +2,20 @@ title: "On-Device Test Framework" --- +# Overview +  -Currently, chip-level tests are executed across three targets (DV, Verilator, and FPGA), using host-side test initiation tools, and an on-device test framework, as shown in the figure above. -On the _host_ side, two main tools are used to initiate tests on the device. For the DV simulation target, the [dvsim.py](https://cs.opensource.google/opentitan/opentitan/+/master:util/dvsim/) tool is used, while for Verilator and FPGA targets, the [systemtest (pytest)](https://cs.opensource.google/opentitan/opentitan/+/master:test/systemtest/) tool is used. -Focusing on the _device_ side, for all three targets, the [on-device test framework](https://cs.opensource.google/opentitan/opentitan/+/master:sw/device/lib/testing/test_framework/test_main.c) is used to provide a uniform execution environment for chip-level tests. -The [on-device test framework](https://cs.opensource.google/opentitan/opentitan/+/master:sw/device/lib/testing/test_framework/test_main.c) provides boilerplate setup code that configures the UART for communicating messages and test results back to the host. -To write a chip-level test that uses this framework, one must create a new C file for the test with the following boilerplate code: +[Chip-level tests]({{< relref "sw/device/tests/index.md" >}}) are designed to be executed across all OpenTitan verification targets DV simulation, Verilator simulation, FPGA, and (eventually) silicon, using host-side test initiation tools, and an on-device test framework, as shown in the figure above. +On the _host_ side, two main tools are used to initiate tests on the device. For the DV simulation target, the [dvsim.py](https://github.com/lowRISC/opentitan/blob/master/util/dvsim/dvsim.py) tool is used, while for Verilator and FPGA targets, the [systemtest (pytest)](https://github.com/lowRISC/opentitan/tree/master/test/systemtest) tool is used. +Focusing on the _device_ side, for all three targets, the [on-device test framework](https://github.com/lowRISC/opentitan/blob/master/sw/device/lib/testing/test_framework/test_main.c) is used to provide a uniform execution environment for chip-level tests. +The [on-device test framework](https://github.com/lowRISC/opentitan/blob/master/sw/device/lib/testing/test_framework/test_main.c) provides boilerplate setup code that configures the UART for communicating messages and test results back to the host. + +# Writing a Chip-Level Test +To write a chip-level test that uses this framework, one must create a new C file for the test (see [Chip-Level Tests]({{< relref "sw/device/tests/index.md">}}) for where to place this test) and follow the steps below. + +## Test Setup +Each chip-level test must contain the following boilerplate setup code: ``` #include "sw/device/lib/testing/test_framework/test_main.h" @@ -23,4 +30,19 @@ } ``` -Check out the [rv\_timer smoke test](https://cs.opensource.google/opentitan/opentitan/+/master:sw/device/tests/rv_timer_smoketest.c) for an example chip-level test that uses the current test framework. +Check out the [rv\_timer smoke test](https://github.com/lowRISC/opentitan/blob/master/sw/device/tests/rv_timer_smoketest.c) for an example chip-level test that contains this boilerplate code. + +## Signaling the end of test and self-checking mechanism +It is mandatory to invoke the target-agnostic API `test_status_set()` to explicitly signal the end of the test based on whether it passed or failed. +When invoked, the API calls `abort()` at the end to stop the core from executing any further. +Please see [`sw/device/lib/testing/test_framework/test_status.h`](https://github.com/lowRISC/opentitan/blob/master/sw/device/lib/testing/test_framework/test_status.h) for documentation and usage. + +### Non-DV Targets +In non-DV targets (Verilator simulation, FPGA, or silicon), the signal is a message written to the console. +It will output `PASS!\r\n` when the tests pass and `FAIL!\r\n` when they fail. +How these console messages are captured differs based on the testing target. +On FPGA / silicon, they will feed directly to a host machine through a physical UART connection, where the host can decipher the message correctness. +On Verilator, they will feed to a virtual UART terminal where the host can do the same. + +In DV simulation, the test status is written to a known location in the memory, which is monitored by the UVM testbench. +Based on the captured value, the testbench monitor invokes UVM methods to pass or fail the test.
diff --git a/sw/device/tests/README.md b/sw/device/tests/README.md deleted file mode 100644 index 1e7753c..0000000 --- a/sw/device/tests/README.md +++ /dev/null
@@ -1,32 +0,0 @@ ---- -title: "Software Device Tests" ---- - -# Overview - -This directory contains a suite of self-contained tests that are capable of running across all of OpenTitan's verification platforms. -These platforms include - UVM DV, Verilator DV, FPGA, and eventually silicon. - -To enable this cross-platform testing, each test is self-contained and do not require additional host or backend resources to verify correct behavior. -This suggests that any testing that would involve emulation of host capability, such as an external SPI host or an external host to encrypt/decrypt data, is not appropriate for these types of self-contained tests. -It is assumed those tests will live in a different directory as the host emulation will be different for each verification target. -For FPGA / silicon this may be a host machine, while for UVM / Verilator this may be some kind of SystemVerilog back-end. - -## Signaling the end of test and self-checking mechanism - -It is mandatory to invoke the platform-agnostic API `test_status_set()` to explicitly signal the end of the test based on whether it passed or failed. -When invoked, the API calls `abort()` at the end to stop the core from executing any further. -Please see `sw/device/lib/testing/test_status.h` for documentation and usage. - -In non-DV platforms (Verilator DV, FPGA or silicon), the signal is a message written to the console. -It will output `PASS!\r\n` when the tests pass and `FAIL!\r\n` when they fail. -How these console messages are captured differs based on the testing target. -On FPGA / silicon, they will feed directly to a host machine through a physical UART connection, where the host can decipher the message correctness. -On Verilator, they will feed to a virtual UART terminal where the host can do the same. - -In UVM DV, the test status is written to a known location in the memory, which is monitored by the testbench. -Based on the captured value, the testbench monitor invokes UVM methods to pass or fail the test. - -# List of Tests - -{{< incGenFromIpDesc "/hw/top_earlgrey/data/standalone_sw_testplan.hjson" "testplan" >}}
diff --git a/sw/device/tests/index.md b/sw/device/tests/index.md new file mode 100644 index 0000000..7d2f959 --- /dev/null +++ b/sw/device/tests/index.md
@@ -0,0 +1,58 @@ +--- +title: "Chip-Level Tests" +--- + +# Overview + +This subtree contains three types of chip-level tests that are capable of running across all OpenTitan verification targets, using the [on-device test framework]({{< relref "sw/device/lib/testing/test_framework/index.md" >}}). +These targets include: DV simulation, Verilator simulation, FPGA, and eventually silicon. + +# Test Types +- **Chip-level Tests** - A collection of software level tests that run on OpenTitan hardware, whose main purpose is pre-silicon verification and post-silicon bringup. +These tests consist of: smoke, IP integration, and system-level tests. +While most of these tests are top-level agnostic, some are not. + - **Smoke Tests** - A software level test, written in C using DIFs, that performs a minimal set of operations on a given IP block to verify that it is alive and functioning. + - **IP Integration Tests** - A software level test, written in C, that exercises some specific functionality specific to a given IP and toplevel. + - **System-level Scenario Test** - A software level test, written in C, that mimics complex system level use case scenarios. + Such a test is designed to encompass multiple pieces of functionality tested by the IP integration tests. + +# Organization and Style Guide + +## File Naming Conventions +* Smoke tests: **{IP name}\_smoketest.c** +* IP Integration tests: **{IP name}[\_{feature}]\_test.c** +* System-level tests: **{use case}\_systemtest.c** + +## Subfoldering Rules +* Smoke tests will be placed in (one per IP): + * (generic) **sw/device/tests/** + * (earlgrey-specific) **sw/device/tests/earlgrey/** + * ({toplevel}-specific) **sw/device/tests/{toplevel}/** +* IP Integration tests will be placed in the same folders as above. +* System-level tests will be placed in the same folders as above. +* IP Integration Test data (some tests, e.g. OTBN, load data files): **sw/device/tests/{IP}\_data/** +* Target-specific tests will be subfoldered by target (see below). + +### Subfoldering Target-Specific Tests +Ideally all smoke, IP integration, and system-level tests should be target agnostic. +However, some tests require emulation of host capabilities, such as an external SPI or I2C host, an external host to encrypt/decrypt data, or an external host that toggles GPIO pins. +Eventually, host-side test initiation tools and the [on-device test framework]({{< relref "sw/device/lib/testing/test_framework/index.md" >}}) will make host emulation opaque to each chip-level test. +However, util then, host emulation depends on the target (e.g., DV vs. Verilator simulation). +Therefore, chip-level tests that require external stimulation from the host, will be subfoldered by target, under the appropriate toplevel folder above. +One example of such a test is the [`sw/device/tests/sim_dv/gpio_test.c`](https://github.com/lowRISC/opentitan/blob/master/sw/device/tests/sim_dv/gpio_test.c), which is subfoldered under `../sim_dv/` to indicate it is a target-specific test. + +# Writing a Chip-Level Test +For instructions on how to write a chip-level test, refer to the [on-device test framework]({{< relref "sw/device/lib/testing/test_framework/index.md" >}}) page. + +# List of Tests + +## Smoke + +TBD + +## IP Integration +{{< incGenFromIpDesc "/hw/top_earlgrey/data/chip_testplan.hjson" "testplan" >}} + +## System-Level (Use Case) + +TBD