docs/developers/developing_iree/testing_guide.md

# Testing Guide

Like the IREE project in general, IREE tests are divided into a few different
components and use different tooling depending on the needs of that component.

| Test type       | Test                                            | Build system | Supported platforms |
|:--------------  | :-----------------                              | -----------  | -------------       |
| Compiler tests  | iree_lit_test                                   | Bazel/CMake  | Host                |
| Runtime tests   | iree_cc_test                                    | Bazel/CMake  | Host/Device         |
|                 | iree_native_test                                | Bazel/CMake  | Host/Device         |
|                 | iree_hal_cts_test_suite                         | CMake        | Host/Device         |
| Core E2E tests  | iree_check_test                                 | Bazel/CMake  | Host/Device         |
|                 | iree_trace_runner_test                          | Bazel/CMake  | Host/Device         |
|                 | iree_single_backend_generated_trace_runner_test | Bazel/CMake  | Host/Device         |
|                 | iree_generated_trace_runner_test                | Bazel/CMake  | Host/Device         |
|                 | iree_static_linker_test                         | CMake        | Host/Device         |

There are also more `*_test_suite` targets that groups test targets with the
same configuration together.

## Compiler Tests

Tests for the IREE compilation pipeline are written as lit tests in the same
style as MLIR.

By convention, IREE includes tests for printing and parsing of MLIR ops in
`.../IR/test/{OP_CATEGORY}_ops.mlir` files, tests for folding and
canonicalization in `.../IR/test/{OP_CATEGORY}_folding.mlir` files, and tests
for compiler passes and pipelines in other `.../test/*.mlir` files.

### Running a Test

For the test
https://github.com/openxla/iree/blob/main/compiler/src/iree/compiler/Dialect/VM/Conversion/MathToVM/test/arithmetic_ops.mlir

With CMake, run this from the build directory:

```shell
$ ctest -R iree/compiler/Dialect/VM/Conversion/MathToVM/test/arithmetic_ops.mlir.test
```

With Bazel, run this from the repo root:

```shell
$ bazel test //compiler/src/iree/compiler/Dialect/VM/Conversion/MathToVM/test:arithmetic_ops.mlir.test
```

### Writing a Test

For advice on writing MLIR compiler tests, see the
[MLIR testing guide](https://mlir.llvm.org/getting_started/TestingGuide/). Tests
should be `.mlir` files in `test` directory adjacent to the functionality they
are testing. Instead of `mlir-opt`, use `iree-opt`, which registers IREE
dialects and passes and doesn't register some unnecessary core ones.

As with most parts of the IREE compiler, these should not have a dependency on
the runtime.

### Configuring the Build System

In the Bazel BUILD file, create a `iree_lit_test_suite` rule. We usually create
a single suite that globs all `.mlir` files in the directory and is called
"lit".

```bzl
load("//iree/build_tools/bazel:iree_lit_test.bzl", "iree_lit_test_suite")

iree_lit_test_suite(
    name = "lit",
    srcs = glob(["*.mlir"]),
    tools = [
        "@llvm-project//llvm:FileCheck",
        "//tools:iree-opt",
    ],
)
```

There is a corresponding CMake function, calls to which will be generated by our
[Bazel to CMake Converter](https://github.com/openxla/iree/tree/main/build_tools/bazel_to_cmake/bazel_to_cmake.py).

```cmake
iree_lit_test_suite(
  NAME
    lit
  SRCS
    "arithmetic_ops.mlir"
  DATA
    FileCheck
    iree-opt
)
```

You can also create a test for a single file with `iree_lit_test`.

## Runtime Tests

Tests for the runtime C++ code use the
[Google Test](https://github.com/google/googletest) testing framework. They
should generally follow the style and best practices of that framework.

### Running a Test

For the test
[`/runtime/src/iree/base/bitfield_test.cc`](https://github.com/openxla/iree/blob/main/runtime/src/iree/base/bitfield_test.cc):

With CMake, run this from the build directory:

```shell
$ ctest -R iree/base/bitfield_test
```

With Bazel, run this from the repo root:

```shell
$ bazel test //runtime/src/iree/base:arena_test
```

### Setting test environments

Parallel testing for `ctest` can be enabled via the `CTEST_PARALLEL_LEVEL`
environment variable. For example:

```shell
$ export CTEST_PARALLEL_LEVEL=$(nproc)
```

To use the Vulkan backend as test driver, you may need to select between a
Vulkan implementation from SwiftShader and multiple Vulkan-capable hardware
devices. This can be done via environment variables. See the
[generic Vulkan setup](../get_started/vulkan_environment_setup.md#useful-environment-variables)
page for details regarding these variables.

For Bazel, you can persist the configuration in `user.bazelrc` to save typing.
For example:

```shell
test:vkswiftshader --test_env="LD_LIBRARY_PATH=..."
test:vkswiftshader --test_env="VK_LAYER_PATH=..."
test:vknative --test_env="LD_LIBRARY_PATH=..."
test:vknative --test_env="VK_LAYER_PATH=..."
```

Then you can use `bazel test --config=vkswiftshader` to select SwiftShader as
the Vulkan implementation. Similarly for other implementations.

### Writing a Test

For advice on writing tests in the Googletest framework, see the
[Googletest primer](https://github.com/google/googletest/blob/main/docs/primer.md).
Test files for source file `foo.cc` with build target `foo` should live in the
same directory with source file `foo_test.cc` and build target `foo_test`. You
should `#include` `iree/testing/gtest.h` instead of any of the gtest or gmock
headers.

As with all parts of the IREE runtime, these should not have a dependency on the
compiler.

### Configuring the Build System

In the Bazel BUILD file, create a `cc_test` target with your test file as the
source and any necessary dependencies. Usually, you can link in a standard gtest
main function. Use `iree/testing:gtest_main` instead of the `gtest_main` that
comes with gtest.

```bzl
cc_test(
    name = "arena_test",
    srcs = ["arena_test.cc"],
    deps = [
        ":arena",
        "//iree/testing:gtest_main",
    ],
)
```

We have created a corresponding CMake function `iree_cc_test` that mirrors the
Bazel rule's behavior. Our
[Bazel to CMake converter](https://github.com/openxla/iree/tree/main/build_tools/bazel_to_cmake/bazel_to_cmake.py)
should generally derive the `CMakeLists.txt` file from the BUILD file:

```cmake
iree_cc_test(
  NAME
    arena_test
  SRCS
    "arena_test.cc"
  DEPS
    ::arena
    iree::testing::gtest_main
)
```

There are other more specific test targets, such as `iree_hal_cts_test_suite`,
which are designed to test specific runtime support with template configuration
and is not supported by Bazel rules.

## IREE Core End-to-End Tests

Here "End-to-End" means from the input accepted by the IREE core compiler
(dialects like TOSA, MHLO, Linalg, and Arithmetic) to execution using the
IREE runtime components. It does not include tests of the integrations with ML
frameworks (e.g. TensorFlow, TensorFlow Lite) or bindings to other languages
(e.g. Python).

We avoid using the more traditional `lit` tests used elsewhere in the compiler
for runtime execution tests. Lit tests require running the compiler tools on
the test platform through shell or python scripts that act on files from a local
file system. On platforms like Android, the web, and embedded systems, each of
these features is either not available or is severely limited.

Instead, to test these flows we use a custom framework called `check`. The check
framework compiles test programs on the host machine into standalone test binary
files that can be pushed to test devices (such as Android phones) where they
run with gtest style assertions (e.g. `check.expect_almost_eq(lhs, rhs)`).

### Building e2e tests

The files needed by these tests are not built by default with CMake. You'll
need to build the special `iree-test-deps` target to generate test files prior
to running CTest (from the build directory):

```shell
$ cmake --build . --target iree-test-deps
```

### Running a Test

For the test
https://github.com/openxla/iree/tree/main/tests/e2e/xla_ops/floor.mlir
compiled for the VMVX target backend and running on the VMVX driver (here they
match exactly, but in principle there's a many-to-many mapping from backends to
drivers).

With CMake, run this from the build directory:

```shell
$ ctest -R tests/e2e/xla_ops/check_vmvx_local-task_floor.mlir
```

With Bazel, run this from the repo root:

```shell
$ bazel test tests/e2e/xla_ops:check_vmvx_local-task_floor.mlir
```

### Setting test environments

Similarly, you can use environment variables to select Vulkan implementations
for running tests as explained in the [Runtime Tests](#runtime-tests) section.

### Writing a Test

These tests live in `tests/e2e`. A single test consists of a `.mlir` source
file specifying an IREE module where each exported function takes no inputs and
returns no results and corresponds to a single test case.

As an example, here are some tests for the MHLO floor operation:

```mlir
func.func @tensor() {
  %input = util.unfoldable_constant dense<[0.0, 1.1, 2.5, 4.9]> : tensor<4xf32>
  %result = "mhlo.floor"(%input) : (tensor<4xf32>) -> tensor<4xf32>
  check.expect_almost_eq_const(%result, dense<[0.0, 1.0, 2.0, 4.0]> : tensor<4xf32>): tensor<4xf32>
  return
}

func.func @scalar() {
  %input = util.unfoldable_constant dense<101.3> : tensor<f32>
  %result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
  check.expect_almost_eq_const(%result, dense<101.0> : tensor<f32>): tensor<f32>
  return
}

func.func @negative() {
  %input = util.unfoldable_constant dense<-1.1> : tensor<f32>
  %result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
  check.expect_almost_eq_const(%result, dense<-2.0> : tensor<f32>): tensor<f32>
  return
}
```

Test cases are created in gtest for each public function exported by the module.

Note the use of `util.unfoldable_constant` to specify test constants. If we were
to use a regular constant the compiler would fold away everything at compile
time and our test would not actually test the runtime. `unfoldable_constant`
adds a barrier that prevents folding. To prevent folding/constant propagate on
an arbitrary SSA-value you can use `util.optimization_barrier`.

Next we use this input constant to exercise the runtime feature under test (in
this case, just a single floor operation). Finally, we use a check dialect
operation to make an assertion about the output. There are a few different
[assertion operations](https://github.com/openxla/iree/tree/main/compiler/src/iree/compiler/Modules/Check).
Here we use the `expect_almost_eq_const` op: *almost* because we are comparing
floats and want to allow for floating-point imprecision, and *const* because we
want to compare it to a constant value. This last part is just syntactic sugar
around

```mlir
%expected = arith.constant dense<101.0> : tensor<f32>
check.expect_almost_eq(%result, %expected) : tensor<f32>
```

The output of running this test looks like:

```txt
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from module
[ RUN      ] module.tensor
[       OK ] module.tensor (76 ms)
[ RUN      ] module.scalar
[       OK ] module.scalar (79 ms)
[ RUN      ] module.double
[       OK ] module.double (55 ms)
[ RUN      ] module.negative
[       OK ] module.negative (54 ms)
[----------] 4 tests from module (264 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (264 ms total)
[  PASSED  ] 4 tests.
```

The "module" name for the test suite comes from the default name for an implicit
MLIR module. To give the test suite a more descriptive name, use an explicit
named top-level module in this file.

### Configuring the Build System

A single `.mlir` source file can be turned into a test target with the
`iree_check_test` Bazel macro (and corresponding CMake function).

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test")

iree_check_test(
    name = "check_vmvx_local-task_floor.mlir",
    src = "floor.mlir",
    driver = "local-task",
    target_backend = "vmvx",
)
```

The target naming convention is "check_backend_driver_src". The generated test
will automatically be tagged with a "driver=vmvx" tag, which can help filter
tests by backend (especially when many tests are generated, as below).

Usually we want to create a suite of tests across many backends and drivers.
This can be accomplished with additional macros. For a single backend/driver
pair:

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_single_backend_test_suite")

iree_check_single_backend_test_suite(
    name = "check_vmvx_local-task",
    srcs = glob(["*.mlir"]),
    driver = "local-task",
    target_backend = "vmvx",
)
```

This will generate a separate test target for each file in `srcs` with a name
following the convention above as well as a Bazel
[test_suite](https://docs.bazel.build/versions/master/be/general.html#test_suite)
called "check_vmvx_local-task" that will run all the generated tests.

You can also generate suites across multiple pairs:

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test_suite")

iree_check_test_suite(
    name = "check",
    srcs = ["success.mlir"],
    # Leave this argument off to run on all supported backend/driver pairs.
    target_backends_and_drivers = [
        ("vmvx", "local-task"),
        ("vulkan-spirv", "vulkan"),
    ],
)
```

This will create a test per source file and backend/driver pair, a test suite
per backend/driver pair, and a test suite, "check", that will run all the tests.

The CMake functions follow a similar pattern. The calls to them are generated in
our `CMakeLists.txt` file by
[bazel_to_cmake](https://github.com/openxla/iree/tree/main/build_tools/bazel_to_cmake/bazel_to_cmake.py).

There are other test targets that generate tests based on template configuraton
and platform detection, such as `iree_static_linker_test`. Those targets are
not supported by Bazel rules at this point.

## Binding Tests

TODO(laurenzo): Explain binding test setup.

## Integration Tests

TODO(silvasean): Explain integration test setup.