Like the IREE project in general, IREE tests are divided into a few different components and use different tooling depending on the needs of that component.
Tests for the runtime C++ code use the Google Test testing framework. They should generally follow the style and best practices of that framework.
For the test iree/base/arena_test.cc
With CMake, run this from the build directory:
$ ctest -R iree/base/arena_test
With Bazel, run this from the repo root:
$ bazel test iree/base:arena_test
Parallel testing for ctest can be enabled via the CTEST_PARALLEL_LEVEL environment variable. For example:
$ export CTEST_PARALLEL_LEVEL=$(nproc)
To use the Vulkan backend as test driver, you may need to select between a Vulkan implementation from SwiftShader and multiple Vulkan-capable hardware devices. This can be done via environment variables. See the generic Vulkan setup page for details regarding these variables.
For Bazel, you can persist the configuration in user.bazelrc to save typing. For example:
test:vkswiftshader --test_env="LD_LIBRARY_PATH=..." test:vkswiftshader --test_env="VK_LAYER_PATH=..." test:vknative --test_env="LD_LIBRARY_PATH=..." test:vknative --test_env="VK_LAYER_PATH=..."
Then you can use bazel test --config=vkswiftshader to select SwiftShader as the Vulkan implementation. Similarly for other implementations.
For advice on writing tests in the Googletest framework, see the Googletest primer. Test files for source file foo.cc with build target foo should live in the same directory with source file foo_test.cc and build target foo_test. You should #include iree/testing/gtest.h instead of any of the gtest or gmock headers.
As with all parts of the IREE runtime, these should not have a dependency on the compiler.
In the Bazel BUILD file, create a cc_test target with your test file as the source and any necessary dependencies. Usually, you can link in a standard gtest main function. Use iree/testing:gtest_main instead of the gtest_main that comes with gtest.
cc_test( name = "arena_test", srcs = ["arena_test.cc"], deps = [ ":arena", "//iree/testing:gtest_main", ], )
We have created a corresponding CMake function iree_cc_test that mirrors the Bazel rule's behavior. Our Bazel to CMake converter should generally derive the CMakeLists.txt file from the BUILD file:
iree_cc_test( NAME arena_test SRCS "arena_test.cc" DEPS ::arena iree::testing::gtest_main )
Tests for the IREE compilation pipeline are written as lit tests in the same style as MLIR.
By convention, IREE includes tests for printing and parsing of MLIR ops in .../IR/test/{OP_CATEGORY}_ops.mlir files, tests for folding and canonicalization in .../IR/test/{OP_CATEGORY}_folding.mlir files, and tests for compiler passes and pipelines in other .../test/*.mlir files.
For the test https://github.com/google/iree/tree/main/iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test/math_ops.mlir
With CMake, run this from the build directory:
$ ctest -R iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test/math_ops.mlir.test
With Bazel, run this from the repo root:
$ bazel test iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test:math_ops.mlir.test
For advice on writing MLIR compiler tests, see the MLIR testing guide. Tests should be .mlir files in test directory adjacent to the functionality they are testing. Instead of mlir-opt, use iree-opt, which registers IREE dialects and passes and doesn't register some unnecessary core ones. Instead of FileCheck, use IreeFileCheck, a shell-script wrapper around FileCheck that passes it a few --do-the-right-thing flags.
As with most parts of the IREE compiler, these should not have a dependency on the runtime.
In the Bazel BUILD file, create a iree_lit_test_suite rule. We usually create a single suite that globs all .mlir files in the directory and is called “lit”.
load("//iree:lit_test.bzl", "iree_lit_test_suite") iree_lit_test_suite( name = "lit", srcs = glob(["*.mlir"]), data = [ "//iree/tools:IreeFileCheck", "//iree/tools:iree-opt", ], )
There is a corresponding CMake function, calls to which will be generated by our Bazel to CMake Converter.
file(GLOB _GLOB_X_MLIR LIST_DIRECTORIES false RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS *.mlir) iree_lit_test_suite( NAME lit SRCS "${_GLOB_X_MLIR}" DATA iree::tools::IreeFileCheck iree::tools::iree-opt )
You can also create a test for a single file with iree_lit_test.
Here “End-to-End” means from the input accepted by the IREE core compiler (the XLA HLO MLIR dialect) to execution using the IREE runtime components. It does not include tests of the integrations with ML frameworks (e.g. TF) or bindings to other languages (e.g. Python).
To test these flows we use a custom framework called check.
Note:
IREE end-to-end tests historically usediree-run-mlir. We are in the process of transitioning them to use the check framework, but that migration is incomplete, so some tests still useiree-run-mlir.
For the test https://github.com/google/iree/tree/main/iree/test/e2e/xla_ops/floor.mlir compiled for the VMLA target backend and running on the VMLA driver (here they match exactly, but in principle there's a many-to-many mapping from backends to drivers).
With CMake, run this from the build directory:
$ ctest -R iree/test/e2e/xla_ops/check_vmvx_vmvx_floor.mlir
With Bazel, run this from the repo root:
$ bazel test iree/test/e2e/xla_ops:check_vmvx_vmvx_floor.mlir
Similarly, you can use environment variables to select Vulkan implementations for running tests as explained in the Runtime Tests section.
These tests live in iree/test/e2e. A single test consists of a .mlir source file specifying an IREE module where each exported function takes no inputs and returns no results and corresponds to a single test case.
As an example, here are some tests for the XLA HLO floor operation:
func @tensor() { %input = util.unfoldable_constant dense<[0.0, 1.1, 2.5, 4.9]> : tensor<4xf32> %result = "mhlo.floor"(%input) : (tensor<4xf32>) -> tensor<4xf32> check.expect_almost_eq_const(%result, dense<[0.0, 1.0, 2.0, 4.0]> : tensor<4xf32>): tensor<4xf32> return } func @scalar() { %input = util.unfoldable_constant dense<101.3> : tensor<f32> %result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32> check.expect_almost_eq_const(%result, dense<101.0> : tensor<f32>): tensor<f32> return } func @double() { %input = util.unfoldable_constant dense<11.2> : tensor<f64> %result = "mhlo.floor"(%input) : (tensor<f64>) -> tensor<f64> check.expect_almost_eq_const(%result, dense<11.0> : tensor<f64>): tensor<f64> return } func @negative() { %input = util.unfoldable_constant dense<-1.1> : tensor<f32> %result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32> check.expect_almost_eq_const(%result, dense<-2.0> : tensor<f32>): tensor<f32> return }
Test cases are created in gtest for each public function exported by the module.
Note the use of util.unfoldable_constant to specify test constants. If we were to use a regular constant, the compiler would “helpfully” fold away everything at compile time and our test would not actually test the runtime. unfoldable_constant hides the value of the constant from the compiler so it cannot use it at compile time. To hide an arbitrary SSA-value, you can use util.do_not_optimize. This wraps any value in an unoptimizable identity function.
Next we use this input constant to exercise the runtime feature under test (in this case, just a single floor operation). Finally, we use a check dialect operation to make an assertion about the output. There are a few different assertion operations. Here we use the expect_almost_eq_const op: almost because we are comparing floats and want to allow for floating-point imprecision, and const because we want to compare it to a constant value. This last part is just syntactic sugar around
%expected = arith.constant dense<101.0> : tensor<f32> check.expect_almost_eq(%result, %expected) : tensor<f32>
The output of running this test looks like:
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from module
[ RUN ] module.tensor
[ OK ] module.tensor (76 ms)
[ RUN ] module.scalar
[ OK ] module.scalar (79 ms)
[ RUN ] module.double
[ OK ] module.double (55 ms)
[ RUN ] module.negative
[ OK ] module.negative (54 ms)
[----------] 4 tests from module (264 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (264 ms total)
[ PASSED ] 4 tests.
The “module” name for the test suite comes from the default name for an implicit MLIR module. To give the test suite a more descriptive name, use an explicit named top-level module in this file.
Constants with dynamic shape are not yet supported. See https://github.com/google/iree/issues/1601. For now, these tests have to use iree-run-mlir lit tests and input arguments.
A single .mlir source file can be turned into a test target with the iree_check_test Bazel macro (and corresponding CMake function).
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test") iree_check_test( name = "check_vmvx_vmvx_floor.mlir", src = "floor.mlir", driver = "vmvx", target_backend = "vmvx", )
The target naming convention is “check_backend_driver_src”. The generated test will automatically be tagged with a “driver=vmvx” tag, which can help filter tests by backend (especially when many tests are generated, as below).
Usually we want to create a suite of tests across many backends and drivers. This can be accomplished with additional macros. For a single backend/driver pair:
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_single_backend_test_suite") iree_check_single_backend_test_suite( name = "check_vmvx_vmvx", srcs = glob(["*.mlir"]), driver = "vmvx", target_backend = "vmvx", )
This will generate a separate test target for each file in srcs with a name following the convention above as well as a Bazel test_suite called “check_vmvx_vmvx” that will run all the generated tests.
You can also generate suites across multiple pairs:
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_test_suite") iree_check_test_suite( name = "check", srcs = ["success.mlir"], # Leave this argument off to run on all supported backend/driver pairs. target_backends_and_drivers = [ ("vmvx", "vmvx"), ("vulkan-spirv", "vulkan"), ], )
This will create a test per source file and backend/driver pair, a test suite per backend/driver pair, and a test suite, “check”, that will run all the tests.
The CMake functions follow a similar pattern. The calls to them are generated in our CMakeLists.txt file by bazel_to_cmake.
TODO(laurenzo): Explain binding test setup.
TODO(silvasean): Explain integration test setup.