docs/testing_guide.md

# Testing Guide

Like the IREE project in general, IREE tests are divided into a few different
components and use different tooling depending on the needs of that component.

## Runtime Tests

Tests for the runtime C++ code use the
[Google Test](https://github.com/google/googletest) testing framework. They
should generally follow the style and best practices of that framework.

### Running a Test

For the test https://github.com/google/iree/tree/master/iree/base/arena_test.cc

With CMake, run this from the build directory:

```shell
$ ctest -R iree/base:arena_test
```

With Bazel, run this from the repo root:

```shell
$ bazel test iree/base:arena_test
```

### Setting test environments

To use the Vulkan backend as test driver, you may need to select between a
Vulkan implementation from SwiftShader and multiple Vulkan-capable hardware
devices. This can be done via environment variables. See the
[generic Vulkan setup](GetStarted/generic_vulkan_env_setup.md#useful-environment-variables)
page for details regarding these variables.

For Bazel, you can persist the configuration in `user.bazelrc` to save typing.
For example:

```shell
test:vkswiftshader --test_env="LD_LIBRARY_PATH=..."
test:vkswiftshader --test_env="VK_LAYER_PATH=..."
test:vknative --test_env="LD_LIBRARY_PATH=..."
test:vknative --test_env="VK_LAYER_PATH=..."
```

Then you can use `bazel test --config=vkswiftshader` to select SwiftShader as
the Vulkan implementation. Similarly for other implementations.

### Writing a Test

For advice on writing tests in the Googletest framework, see the
[Googletest primer](https://github.com/google/googletest/blob/master/googletest/docs/primer.md).
Test files for source file `foo.cc` with build target `foo` should live in the
same directory with source file `foo_test.cc` and build target `foo_test`. You
should `#include` `iree/testing/gtest.h` instead of any of the gtest or gmock
headers.

As with all parts of the IREE runtime, these should not have a dependency on the
compiler.

### Configuring the Build System

In the Bazel BUILD file, create a `cc_test` target with your test file as the
source and any necessary dependencies. Usually, you can link in a standard gtest
main function. Use `iree/testing:gtest_main` instead of the `gtest_main` that
comes with gtest.

```bzl
cc_test(
    name = "arena_test",
    srcs = ["arena_test.cc"],
    deps = [
        ":arena",
        "//iree/testing:gtest_main",
    ],
)
```

We have created a corresponding CMake function `iree_cc_test` that mirrors the
Bazel rule's behavior. Our
[Bazel to CMake converter](https://github.com/google/iree/tree/master/build_tools/bazel_to_cmake/bazel_to_cmake.py)
should generally derive the `CMakeLists.txt` file from the BUILD file:

```cmake
iree_cc_test(
  NAME
    arena_test
  SRCS
    "arena_test.cc"
  DEPS
    ::arena
    iree::testing::gtest_main
)
```

## Compiler Tests

Tests for the IREE compilation pipeline are written as lit tests in the same
style as MLIR.

### Running a Test

For the test
https://github.com/google/iree/tree/master/iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test/math_ops.mlir

With CMake, run this from the build directory:

```shell
$ ctest -R iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test:math_ops.mlir.test
```

With Bazel, run this from the repo root:

```shell
$ bazel test iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test:math_ops.mlir.test
```

### Writing a Test

For advice on writing MLIR compiler tests, see the
[MLIR testing guide](https://mlir.llvm.org/getting_started/TestingGuide/). Tests
should be `.mlir` files in `test` directory adjacent to the functionality they
are testing. Instead of `mlir-opt`, use `iree-opt`, which registers IREE
dialects and passes and doesn't register some unnecessary core ones. Instead of
`FileCheck`, use
[`IreeFileCheck`](https://github.com/google/iree/tree/master/iree/tools/IreeFileCheck.sh),
a shell-script wrapper around FileCheck that passes it a few
`--do-the-right-thing` flags.

As with all parts of the IREE compiler, these should not have a dependency on
the runtime.

### Configuring the Build System

In the Bazel BUILD file, create a `iree_lit_test_suite` rule. We usually create
a single suite that globs all `.mlir` files in the directory and is called
"lit".

```bzl
load("//iree:lit_test.bzl", "iree_lit_test_suite")

iree_lit_test_suite(
    name = "lit",
    srcs = glob(["*.mlir"]),
    data = [
        "//iree/tools:IreeFileCheck",
        "//iree/tools:iree-opt",
    ],
)
```

There is a corresponding CMake function, calls to which will be generated by our
[Bazel to CMake Converter](https://github.com/google/iree/tree/master/build_tools/bazel_to_cmake/bazel_to_cmake.py).

```cmake
file(GLOB _GLOB_X_MLIR LIST_DIRECTORIES false RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS *.mlir)
iree_lit_test_suite(
  NAME
    lit
  SRCS
    "${_GLOB_X_MLIR}"
  DATA
    iree::tools::IreeFileCheck
    iree::tools::iree-opt
)
```

You can also create a test for a single file with `iree_lit_test`.

## IREE Core End-to-End Tests

Here "End-to-End" means from the input accepted by the IREE core compiler (the
XLA HLO MLIR dialect) to execution using the IREE runtime components. It does
not include tests of the integrations with ML frameworks (e.g. TF) or bindings
to other languages (e.g. Python).

To test these flows we use a custom framework called `check`.

> Note:<br>
> &nbsp;&nbsp;&nbsp;&nbsp;IREE end-to-end tests historically used `iree-run-mlir`.
> We are in the process of transitioning them to use the check framework, but
> that migration is incomplete, so some tests still use `iree-run-mlir`.

### Running a Test

For the test
https://github.com/google/iree/tree/master/iree/test/e2e/xla_ops/floor.mlir
compiled for the VMLA target backend and running on the VMLA driver (here they
match exactly, but in principle there's a many-to-many mapping from backends to
drivers).

With CMake, run this from the build directory:

```shell
$ ctest -R iree/test/e2e/xla_ops:check_vmla_vmla_floor.mlir
```

With Bazel, run this from the repo root:

```shell
$ bazel test iree/test/e2e/xla_ops:check_vmla_vmla_floor.mlir
```

### Setting test environments

Similarly, you can use environment variables to select Vulkan implementations
for running tests as explained in the [Runtime Tests](#runtime-tests) section.

### Writing a Test

These tests live in `iree/test/e2e`. A single test consists of a `.mlir` source
file specifying an IREE module where each exported function takes no inputs and
returns no results and corresponds to a single test case.

As an example, here are some tests for the XLA HLO floor operation:

```mlir
func @tensor() attributes { iree.module.export } {
  %input = iree.unfoldable_constant dense<[0.0, 1.1, 2.5, 4.9]> : tensor<4xf32>
  %result = "xla_hlo.floor"(%input) : (tensor<4xf32>) -> tensor<4xf32>
  check.expect_almost_eq_const(%result, dense<[0.0, 1.0, 2.0, 4.0]> : tensor<4xf32>): tensor<4xf32>
  return
}

func @scalar() attributes { iree.module.export } {
  %input = iree.unfoldable_constant dense<101.3> : tensor<f32>
  %result = "xla_hlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
  check.expect_almost_eq_const(%result, dense<101.0> : tensor<f32>): tensor<f32>
  return
}

func @double() attributes { iree.module.export } {
  %input = iree.unfoldable_constant dense<11.2> : tensor<f64>
  %result = "xla_hlo.floor"(%input) : (tensor<f64>) -> tensor<f64>
  check.expect_almost_eq_const(%result, dense<11.0> : tensor<f64>): tensor<f64>
  return
}

func @negative() attributes { iree.module.export } {
  %input = iree.unfoldable_constant dense<-1.1> : tensor<f32>
  %result = "xla_hlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
  check.expect_almost_eq_const(%result, dense<-2.0> : tensor<f32>): tensor<f32>
  return
}
```

The test case functions are exported using the `iree.module.export` attribute.
Each of these exported functions will be used to create a test case in gtest.

Note the use of
[`iree.unfoldable_constant`](https://google.github.io/iree/Dialects/IREEDialect#ireeunfoldable_constant-ireeunfoldableconstantop)
to specify test constants. If we were to use a regular constant, the compiler
would "helpfully" fold away everything at compile time and our test would not
actually test the runtime. `unfoldable_constant` hides the value of the constant
from the compiler so it cannot use it at compile time. To hide an arbitrary
SSA-value, you can use
[`iree.do_not_optimize`](https://google.github.io/iree/Dialects/IREEDialect#ireedo_not_optimize-ireedonotoptimizeop).
This wraps any value in an unoptimizable identity function.

Next we use this input constant to exercise the runtime feature under test (in
this case, just a single floor operation). Finally, we use a check dialect
operation to make an assertion about the output. There are a few different
[assertion operations](https://google.github.io/iree/Dialects/CheckDialect).
Here we use the `expect_almost_eq_const` op: *almost* because we are comparing
floats and want to allow for floating-point imprecision, and *const* because we
want to compare it to a constant value. This last part is just syntactic sugar
around

```mlir
%expected = constant dense<101.0> : tensor<f32>
check.expect_almost_eq(%result, %expected) : tensor<f32>
```

The output of running this test looks like:

```txt
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from module
[ RUN      ] module.tensor
[       OK ] module.tensor (76 ms)
[ RUN      ] module.scalar
[       OK ] module.scalar (79 ms)
[ RUN      ] module.double
[       OK ] module.double (55 ms)
[ RUN      ] module.negative
[       OK ] module.negative (54 ms)
[----------] 4 tests from module (264 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (264 ms total)
[  PASSED  ] 4 tests.
```

The "module" name for the test suite comes from the default name for an implicit
MLIR module. To give the test suite a more descriptive name, use an explicit
named top-level module in this file.

### Dynamic Shapes

Constants with dynamic shape are not yet supported. See
https://github.com/google/iree/issues/1601. For now, these tests have to use
`iree-run-mlir` lit tests and input arguments.

### Configuring the Build System

A single `.mlir` source file can be turned into a test target with the
`iree_check_test` Bazel macro (and corresponding CMake function).

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test")

iree_check_test(
    name = "check_vmla_vmla_floor.mlir",
    src = "floor.mlir",
    driver = "vmla",
    target_backend = "vmla",
)
```

The target naming convention is "check_backend_driver_src". The generated test
will automatically be tagged with a "driver=vmla" tag, which can help filter
tests by backend (especially when many tests are generated, as below).

Usually we want to create a suite of tests across many backends and drivers.
This can be accomplished with additional macros. For a single backend/driver
pair:

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_single_backend_test_suite")

iree_check_single_backend_test_suite(
    name = "check_vmla_vmla",
    srcs = glob(["*.mlir"]),
    driver = "vmla",
    target_backend = "vmla",
)
```

This will generate a separate test target for each file in `srcs` with a name
following the convention above as well as a Bazel
[test_suite](https://docs.bazel.build/versions/master/be/general.html#test_suite)
called "check_vmla_vmla" that will run all the generated tests.

You can also generate suites across multiple pairs:

```bzl
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test_suite")

iree_check_test_suite(
    name = "check",
    srcs = ["success.mlir"],
    # Leave this argument off to run on all supported backend/driver pairs.
    target_backends_and_drivers = [
        ("vmla", "vmla"),
        ("vulkan-spirv", "vulkan"),
    ],
)
```

This will create a test per source file and backend/driver pair, a test suite
per backend/driver pair, and a test suite, "check", that will run all the tests.

The CMake functions follow a similar pattern. The calls to them are generated in
our `CMakeLists.txt` file by
[bazel_to_cmake](https://github.com/google/iree/tree/master/build_tools/bazel_to_cmake/bazel_to_cmake.py).

## Binding Tests

TODO(laurenzo): Explain binding test setup.

## Integration Tests

TODO(silvasean): Explain integration test setup.