[dv, doc] documentation for testplanner tool - added documentation for testplanner tool - updated docgen tool documentation to indicate updates for tesplan table insertion - renamed 'add_testplan' to 'insert_testplan' in lowrisc_renderer.py
diff --git a/hw/ip/uart/dv/uart_dv_plan.md b/hw/ip/uart/dv/uart_dv_plan.md index 4bd55b5..21420a7 100644 --- a/hw/ip/uart/dv/uart_dv_plan.md +++ b/hw/ip/uart/dv/uart_dv_plan.md
@@ -104,4 +104,4 @@ ``` ## Testplan -{{% add_testplan x }} +{{% insert_testplan x }}
diff --git a/util/docgen/README.md b/util/docgen/README.md index 4e566e6..faf93c4 100644 --- a/util/docgen/README.md +++ b/util/docgen/README.md
@@ -27,6 +27,13 @@ * `{{% registers x }}` Insert the register tables at this point in the document. Must be after the regfile extension! TODO fix the need for `x` +* `{{% import_testplan testplan.hjson }}` Pointer to the testplan + hjson. This is expected to go early in the document. After this line + the testplan entries are available as markup items. + +* `{{% insert_testplan x }}` Insert the testplan table at this point in the + document. Must be after the `import_testplan` extension! TODO fix the need for `x` + * `{{% include file }}` Insert the file into the markdown document. Any other text on the same line as the include directive will be inserted, then a newline and then the included file. The
diff --git a/util/docgen/lowrisc_renderer.py b/util/docgen/lowrisc_renderer.py index afbed83..3a733c4 100644 --- a/util/docgen/lowrisc_renderer.py +++ b/util/docgen/lowrisc_renderer.py
@@ -358,7 +358,7 @@ self.testplan = testplan_utils.parse_testplan( path.join(self.basedir, token.text)) return "" - if token.type == "add_testplan": + if token.type == "insert_testplan": if self.testplan == None: return "<B>Errors parsing testplan prevents insertion.</B>" outbuf = io.StringIO()
diff --git a/util/testplanner/README.md b/util/testplanner/README.md new file mode 100644 index 0000000..16729f8 --- /dev/null +++ b/util/testplanner/README.md
@@ -0,0 +1,235 @@ +{{% toc 4 }} + +# Testplanner tool + +Testplanner is a python3 based tool for parsing testplans written in hjson +format into a data structure that can be used for: +* Expanding the testplan inline within the DV plan as a table +* Annotate the regression results with testplan entries for a document driven DV execution + +<!-- TODO : fix links --> +Please see [DV methodology](../../doc/ug/dv_methodology.md) for more details on the +rationale and motivation for writing and maintaining testplans in a machine-parsable +format (`hjson`). + +This document will focus on the anatomy of a hjson testplan, +list of features supported and some of the ways of using the tool. + +## Hjson Testplan +### Testplan entry contents +Minimally, the following items are sufficient to adequately capture the +intent of a planned test: +* **name: name of the planned test** + + This is a single `lower_snake_case` string that succinctly describes the intended + feature being tested. As an example, a basic sanity test which is typically the + first test written on a brand new testbench would be simply named `sanity`. + +* **milestone: verification milestone** + + This is one of {"`v1`", "`v2`" and "`v3`"}. This allows us concretely indicate that + all goals for a particular milestone have been achieved and we can transition + to the next. + +* **desc: description of the planned test** + + A multi-line string that briefly describes the intent of the test. It is + recommended to add a high level goal, stimulus and checking procedure so + that the reader gets the full picture of what and how the said feature is being + tested. + + Full markdown syntax is supported when writing the description. + +* **tests: list of actual written tests that maps to this planned test** + + Testplan is written very early in the V0 stage of the HW development + [life-cycle](../../doc/ug/hw_stages.md). When the DV engineer gets to actually + developing the test, it may not map 1:1 to the planned test - it may be possible + that an already written test that mapped to another planned test also satisfies + the current one; OR it may also be possible that the planned test needs to be + split into multiple smaller tests. To cater to these needs, we provide the ability + to set a list of actual written tests that maps to each planned test. This + information will then be used to map the regression results and annotate them + to the tesplan to generate the final table. This list does not have to be populated + right away. It can be updated as and when tests are written. + +If need arises, more entries can be added to this list relatively easily. + +Testplan entries are added using the `entries` key, which is a list that looks +like this: +```hjson + entries: [ + { + name: feature1 + milestone: v1 + desc: '''**Goal**: High level goal of this test + + **Stimulus**: Describe the stimulus procedure. + + **Check**: Describe the checking procedure.''' + tests: ["foo_feature1"] + } + { + name: feature2 + milestone: v2 + desc: '''**Goal**: High level goal of this test + + **Stimulus**: Describe the stimulus procedure. + + **Check**: Describe the checking procedure.''' + tests: ["foo_feature2_test1", + "foo_feature2_test2", + "foo_feature2_test3"] + } + ... + ] +``` + +### Import shared testplans +Typically, there are tests that are common to more that one testbench and can be +made a part of a 'shared' testplan that each DUT testplan can simply import. An +example of this is running the automated UVM RAL CSR tests, which applies to +almost all DUTs. This can be done using the `import_testplans` key: +```hjson + import_testplans: ["util/testplanner/examples/common_testplan.hjson", + "hw/dv/tools/csr_testplan.hjson"] +``` + +Note that the paths to common testplans are relative to $REPO_TOP. + +For the sake of discussion below, we will refer to the 'main' or DUT testplan +as 'DUT' testplan and the shared testplans it imports as 'shared' or 'imported' +testplans. + +The imported testplans actually present a problem - how can we set actual written +tests that maps to the shared testplan entry generically enough that they apply +to more than one DUTs? We currently solve this by providing wildcards, which are +single lower_case strings within braces `'{..}'`. A substitution value (or list +of values) for the wildcard string can be optionally provided in the DUT tesplan. +Here's an example: +```hjson +------- + // UART testplan: + name: uart + +------- + // Imported testplan: + { + name: csr + ... + tests: ["{name}{intf}_csr_hw_reset"] + } +``` +In the example above, `{name}` and `{intf}` are wildcards used in the shared +testplan for which substitution values are to be provided in the DUT testplan. +When the tool parses the DUT testplan along with the imported testplans, +it substitutes the wildcards with the substition values found in the DUT testplan. +If substitution is not available, then the wildcard is replaced with an empty string. +In the example above, the list of written test resolves to `["uart_csr_hw_reset"]` +after substituting `{name}` with `uart` and `{intf}` with an empty string. +As many wildcards as needed can be added to the tests in the shared testplans to support +as wide usecases as possible across different testbenches. Moreover, the substitution +value can be a list of strings, in which case, the list of written tests will resolve +to all values being substituted. See example below for illustration: +```hjson +------- + // Chip testplan: + name: chip + intf: ["", "_jtag"] + foo: ["x", "y", "z"] + +------- + // Imported testplan: + { + name: csr + ... + tests: ["{name}{intf}_csr_hw_reset_{foo}"] + } +``` +This will resolve to the following 6 tests: +``` +["chip_csr_hw_reset_x", "chip_csr_hw_reset_y", "chip_csr_hw_reset_z", + "chip_jtag_csr_hw_reset_x", "chip_jtag_csr_hw_reset_y", "chip_jtag_csr_hw_reset_z"] +``` + +### Example sources +The following examples provided at `util/testplanner/examples` can be used as +a starting point. +- **foo_testplan.hjson**: DUT testplan +- **common_testplan.hjson**: shared testplan imported within the DUT tesplan +- **foo_dv_plan.md**: DUT testplan imported within the DV plan doc in markdown + +In addition, see [UART DV Plan](../../hw/ip/uart/dv/uart_dv_plan.md) for a +real 'production' example of inline expansion of an imported testplan as a table +within the DV Plan document done using [docgen](../docgen/README.md). +The [UART tesplan](../../hw/ip/uart/dv/uart_testplan.hjson) imports the shared +testplans located at `hw/dv/tools/testplans` area. + +### Limitations +The following limitations currently hold: +* Only the DUT testplan can import shared testplans; the imported testplans cannot further + import more testplans +* All planned test names parsed from the DUT testplan and all of its imported tetsplans + need to be unique + +## Usage examples +### Standalone tool invocations +Generate the testplan table in HTML to stdout: +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson +``` + +Generate the testplan table in HTML to a file: +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson -o /tmp/foo_testplan_table.html +``` + +Generate the testplan table in HTML styled with [docgen](../docgen/README.md): +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson | ./docgen.py -c -o /tmp/foo_testplan_table.html +``` + +Generate regression results table in HTML to stdout: +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson -r testplanner/examples/foo_regr_results.hjson +``` + +Generate regression results table in HTML to a file: +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson \ + -r testplanner/examples/foo_regr_results.hjson -o /tmp/foo_regr_results.html +``` + +Generate regression results table in HTML styled with [docgen](../docgen/README.md): +```console +$ util/testplanner.py testplanner/examples/foo_testplan.hjson \ + -r testplanner/examples/foo_regr_results.hjson | ./docgen.py -c -o /tmp/foo_regr_results.html +``` + +### APIs for external tools +The [docgen](../docgen/README.md) invokes the testplanner utility functions +directly to parse the hjson testplan and insert a HTML table within the DV +plan document. This is done by invoking: +```console +Example 1: +$ util/docgen.py -c ../hw/ip/uart/dv/uart_dv_plan.md -o /tmp/uart_dv_plan.html + +Example 2: +$ util/docgen.py -c testplanner/examples/foo_dv_plan.md -o /tmp/foo_dv_plan.html +``` +See following snippet of code for the APIs in use: +```python +from testplanner import class_defs, testplan_utils + + # hjson_testplan_path: a string pointing to the path to hjson tesplan + # outbuf: file buffer opened for writing + testplan = testplan_utils.parse_testplan(hjson_testplan_path) + testplan_utils.gen_html_testplan_table(testplan, outbuf) +``` + +## Future work +* Allow DUT and imported testplans have the same name for the planned test as + long as they are in separate files + * If the same name exists, then append the list of tests together +* Split the regression results table generation into a separate `dashboard_gen` + script which will also cater to generating results table for `lint` and `fpv`
diff --git a/util/testplanner/examples/foo_dv_plan.md b/util/testplanner/examples/foo_dv_plan.md index ce63de6..2524a0f 100644 --- a/util/testplanner/examples/foo_dv_plan.md +++ b/util/testplanner/examples/foo_dv_plan.md
@@ -2,4 +2,4 @@ {{% import_testplan foo_testplan.hjson }} ## Testplan -{{% add_testplan x }} +{{% insert_testplan x }}