Documentation updates for testing and build scripts. (#8720)
* Refresh testing guide with rationale behind the check framework
* see https://github.com/google/iree/pull/8700#discussion_r839064429
* Update some references from "translate_tool" to "compile_tool"
* Note that some still remain, like https://github.com/google/iree/blob/393948d565566a69ad02949e5f73a9116432177b/build_tools/cmake/iree_bytecode_module.cmake#L17-L18 since some uses use "iree-compile" while others use "iree-translate" or equivalent: https://github.com/google/iree/blob/393948d565566a69ad02949e5f73a9116432177b/iree/samples/custom_modules/BUILD#L47
diff --git a/docs/developers/developing_iree/testing_guide.md b/docs/developers/developing_iree/testing_guide.md
index 8351092..ed9f758 100644
--- a/docs/developers/developing_iree/testing_guide.md
+++ b/docs/developers/developing_iree/testing_guide.md
@@ -56,7 +56,7 @@
### Writing a Test
For advice on writing tests in the Googletest framework, see the
-[Googletest primer](https://github.com/google/googletest/blob/master/googletest/docs/primer.md).
+[Googletest primer](https://github.com/google/googletest/blob/main/docs/primer.md).
Test files for source file `foo.cc` with build target `foo` should live in the
same directory with source file `foo_test.cc` and build target `foo_test`. You
should `#include` `iree/testing/gtest.h` instead of any of the gtest or gmock
@@ -113,18 +113,18 @@
### Running a Test
For the test
-https://github.com/google/iree/tree/main/iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test/math_ops.mlir
+https://github.com/google/iree/blob/main/iree/compiler/Dialect/VM/Conversion/MathToVM/test/arithmetic_ops.mlir
With CMake, run this from the build directory:
```shell
-$ ctest -R iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test/math_ops.mlir.test
+$ ctest -R iree/compiler/Dialect/VM/Conversion/MathToVM/test/arithmetic_ops.mlir.test
```
With Bazel, run this from the repo root:
```shell
-$ bazel test iree/compiler/Dialect/VMLA/Conversion/HLOToVMLA/test:math_ops.mlir.test
+$ bazel test iree/compiler/Dialect/VM/Conversion/MathToVM/test:arithmetic_ops.mlir.test
```
### Writing a Test
@@ -161,12 +161,11 @@
[Bazel to CMake Converter](https://github.com/google/iree/tree/main/build_tools/bazel_to_cmake/bazel_to_cmake.py).
```cmake
-file(GLOB _GLOB_X_MLIR LIST_DIRECTORIES false RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS *.mlir)
iree_lit_test_suite(
NAME
lit
SRCS
- "${_GLOB_X_MLIR}"
+ "arithmetic_ops.mlir"
DATA
FileCheck
iree::tools::iree-opt
@@ -177,23 +176,43 @@
## IREE Core End-to-End Tests
-Here "End-to-End" means from the input accepted by the IREE core compiler (the
-XLA HLO MLIR dialect) to execution using the IREE runtime components. It does
-not include tests of the integrations with ML frameworks (e.g. TF) or bindings
-to other languages (e.g. Python).
+Here "End-to-End" means from the input accepted by the IREE core compiler
+(dialects like TOSA, MHLO, Linalg, and Arithmetic) to execution using the
+IREE runtime components. It does not include tests of the integrations with ML
+frameworks (e.g. TensorFlow, TensorFlow Lite) or bindings to other languages
+(e.g. Python).
-To test these flows we use a custom framework called `check`.
+We avoid using the more traditional `lit` tests used elsewhere in the compiler
+for runtime execution tests. Lit tests require running the compiler tools on
+the test platform through shell or python scripts that act on files from a local
+file system. On platforms like Android, the web, and embedded systems, each of
+these features is either not available or is severely limited.
+
+Instead, to test these flows we use a custom framework called `check`. The check
+framework compiles test programs on the host machine into standalone test binary
+files that can be pushed to test devices (such as Android phones) where they
+run with gtest style assertions (e.g. `check.expect_almost_eq(lhs, rhs)`).
> Note:<br>
> IREE end-to-end tests historically used `iree-run-mlir`.
> We are in the process of transitioning them to use the check framework, but
> that migration is incomplete, so some tests still use `iree-run-mlir`.
+### Building e2e tests
+
+The files needed by these tests are not built by default with CMake. You'll
+need to build the special `iree-test-deps` target to generate test files prior
+to running CTest (from the build directory):
+
+```shell
+$ cmake --build . --target iree-test-deps
+```
+
### Running a Test
For the test
https://github.com/google/iree/tree/main/iree/test/e2e/xla_ops/floor.mlir
-compiled for the VMLA target backend and running on the VMLA driver (here they
+compiled for the VMVX target backend and running on the VMVX driver (here they
match exactly, but in principle there's a many-to-many mapping from backends to
drivers).
@@ -220,31 +239,24 @@
file specifying an IREE module where each exported function takes no inputs and
returns no results and corresponds to a single test case.
-As an example, here are some tests for the XLA HLO floor operation:
+As an example, here are some tests for the MHLO floor operation:
```mlir
-func @tensor() {
+func.func @tensor() {
%input = util.unfoldable_constant dense<[0.0, 1.1, 2.5, 4.9]> : tensor<4xf32>
%result = "mhlo.floor"(%input) : (tensor<4xf32>) -> tensor<4xf32>
check.expect_almost_eq_const(%result, dense<[0.0, 1.0, 2.0, 4.0]> : tensor<4xf32>): tensor<4xf32>
return
}
-func @scalar() {
+func.func @scalar() {
%input = util.unfoldable_constant dense<101.3> : tensor<f32>
%result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
check.expect_almost_eq_const(%result, dense<101.0> : tensor<f32>): tensor<f32>
return
}
-func @double() {
- %input = util.unfoldable_constant dense<11.2> : tensor<f64>
- %result = "mhlo.floor"(%input) : (tensor<f64>) -> tensor<f64>
- check.expect_almost_eq_const(%result, dense<11.0> : tensor<f64>): tensor<f64>
- return
-}
-
-func @negative() {
+func.func @negative() {
%input = util.unfoldable_constant dense<-1.1> : tensor<f32>
%result = "mhlo.floor"(%input) : (tensor<f32>) -> tensor<f32>
check.expect_almost_eq_const(%result, dense<-2.0> : tensor<f32>): tensor<f32>
@@ -301,12 +313,6 @@
MLIR module. To give the test suite a more descriptive name, use an explicit
named top-level module in this file.
-### Dynamic Shapes
-
-Constants with dynamic shape are not yet supported. See
-https://github.com/google/iree/issues/1601. For now, these tests have to use
-`iree-run-mlir` lit tests and input arguments.
-
### Configuring the Build System
A single `.mlir` source file can be turned into a test target with the
diff --git a/experimental/web/sample_dynamic/build_sample.sh b/experimental/web/sample_dynamic/build_sample.sh
index 1847215..6fdc93e 100755
--- a/experimental/web/sample_dynamic/build_sample.sh
+++ b/experimental/web/sample_dynamic/build_sample.sh
@@ -43,12 +43,11 @@
# Compile from .mlir input to portable .vmfb file using host tools #
###############################################################################
-TRANSLATE_TOOL="${INSTALL_ROOT?}/bin/iree-compile"
-EMBED_DATA_TOOL="${INSTALL_ROOT?}/bin/generate_embed_data"
+COMPILE_TOOL="${INSTALL_ROOT?}/bin/iree-compile"
-translate_sample() {
- echo " Translating '$1' sample..."
- ${TRANSLATE_TOOL?} $2 \
+compile_sample() {
+ echo " Compiling '$1' sample..."
+ ${COMPILE_TOOL?} $2 \
--iree-mlir-to-vm-bytecode-module \
--iree-input-type=mhlo \
--iree-hal-target-backends=llvm \
@@ -57,10 +56,10 @@
--o ${BINARY_DIR}/$1.vmfb
}
-echo "=== Translating sample MLIR files to VM flatbuffer outputs (.vmfb) ==="
-translate_sample "simple_abs" "${ROOT_DIR?}/iree/samples/models/simple_abs.mlir"
-translate_sample "fullyconnected" "${ROOT_DIR?}/iree/test/e2e/models/fullyconnected.mlir"
-translate_sample "collatz" "${ROOT_DIR?}/iree/test/e2e/models/collatz.mlir"
+echo "=== Compiling sample MLIR files to VM flatbuffer outputs (.vmfb) ==="
+compile_sample "simple_abs" "${ROOT_DIR?}/iree/samples/models/simple_abs.mlir"
+compile_sample "fullyconnected" "${ROOT_DIR?}/iree/test/e2e/models/fullyconnected.mlir"
+compile_sample "collatz" "${ROOT_DIR?}/iree/test/e2e/models/collatz.mlir"
###############################################################################
# Build the web artifacts using Emscripten #
diff --git a/experimental/web/sample_static/build_sample.sh b/experimental/web/sample_static/build_sample.sh
index fc8fa99..db2cc03 100755
--- a/experimental/web/sample_static/build_sample.sh
+++ b/experimental/web/sample_static/build_sample.sh
@@ -43,13 +43,13 @@
# Compile from .mlir input to static C source files using host tools #
###############################################################################
-TRANSLATE_TOOL="${INSTALL_ROOT?}/bin/iree-compile"
+COMPILE_TOOL="${INSTALL_ROOT?}/bin/iree-compile"
EMBED_DATA_TOOL="${INSTALL_ROOT?}/bin/generate_embed_data"
INPUT_NAME="mnist"
INPUT_PATH="${ROOT_DIR?}/iree/samples/models/mnist.mlir"
-echo "=== Translating MLIR to static library output (.vmfb, .h, .o) ==="
-${TRANSLATE_TOOL?} ${INPUT_PATH} \
+echo "=== Compiling MLIR to static library output (.vmfb, .h, .o) ==="
+${COMPILE_TOOL?} ${INPUT_PATH} \
--iree-mlir-to-vm-bytecode-module \
--iree-input-type=mhlo \
--iree-hal-target-backends=llvm \
diff --git a/iree/samples/dynamic_shapes/test.sh b/iree/samples/dynamic_shapes/test.sh
index cbc496f..dacac27 100755
--- a/iree/samples/dynamic_shapes/test.sh
+++ b/iree/samples/dynamic_shapes/test.sh
@@ -21,7 +21,7 @@
${ROOT_DIR}/iree/samples/dynamic_shapes/dynamic_shapes.ipynb
test -f ${ARTIFACTS_DIR}/dynamic_shapes.mlir && echo "dynamic_shapes.mlir exists"
-# 2. Compile the `iree-compile` tool.
+# 2. Build the `iree-compile` tool.
cmake -B ${BUILD_DIR} -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo ${ROOT_DIR}
cmake --build ${BUILD_DIR} --target iree_tools_iree-compile -- -k 0
diff --git a/iree/samples/static_library/CMakeLists.txt b/iree/samples/static_library/CMakeLists.txt
index cd8120e..804b41b 100644
--- a/iree/samples/static_library/CMakeLists.txt
+++ b/iree/samples/static_library/CMakeLists.txt
@@ -11,19 +11,19 @@
endif()
# Set iree-compile binary.
-set(_TRANSLATE_TOOL_EXECUTABLE $<TARGET_FILE:iree_tools_iree-compile>)
+set(_COMPILE_TOOL_EXECUTABLE $<TARGET_FILE:iree_tools_iree-compile>)
## Example with VM bytecode module.
# Setup args for iree-compile.
-set(_TRANSLATE_ARGS)
-list(APPEND _TRANSLATE_ARGS "-iree-mlir-to-vm-bytecode-module")
-list(APPEND _TRANSLATE_ARGS "-iree-hal-target-backends=dylib-llvm-aot")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-link-embedded=false")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-link-static")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-static-library-output-path=simple_mul.o")
-list(APPEND _TRANSLATE_ARGS "${CMAKE_CURRENT_SOURCE_DIR}/simple_mul.mlir")
-list(APPEND _TRANSLATE_ARGS "-o")
-list(APPEND _TRANSLATE_ARGS "simple_mul.vmfb")
+set(_COMPILE_ARGS)
+list(APPEND _COMPILE_ARGS "-iree-mlir-to-vm-bytecode-module")
+list(APPEND _COMPILE_ARGS "-iree-hal-target-backends=dylib-llvm-aot")
+list(APPEND _COMPILE_ARGS "-iree-llvm-link-embedded=false")
+list(APPEND _COMPILE_ARGS "-iree-llvm-link-static")
+list(APPEND _COMPILE_ARGS "-iree-llvm-static-library-output-path=simple_mul.o")
+list(APPEND _COMPILE_ARGS "${CMAKE_CURRENT_SOURCE_DIR}/simple_mul.mlir")
+list(APPEND _COMPILE_ARGS "-o")
+list(APPEND _COMPILE_ARGS "simple_mul.vmfb")
# Custom command for iree-compile to generate static library and bytecode.
add_custom_command(
@@ -31,8 +31,8 @@
${CMAKE_CURRENT_BINARY_DIR}/simple_mul.h
${CMAKE_CURRENT_BINARY_DIR}/simple_mul.o
${CMAKE_CURRENT_BINARY_DIR}/simple_mul.vmfb
- COMMAND ${_TRANSLATE_TOOL_EXECUTABLE} ${_TRANSLATE_ARGS}
- DEPENDS ${_TRANSLATE_TOOL_EXECUTABLE} "simple_mul.mlir"
+ COMMAND ${_COMPILE_TOOL_EXECUTABLE} ${_COMPILE_ARGS}
+ DEPENDS ${_COMPILE_TOOL_EXECUTABLE} "simple_mul.mlir"
)
# Tell cmake about the simple_mul library so it will link it.
@@ -101,15 +101,15 @@
## Example with VM C module.
# Setup args for iree-compile.
-set(_TRANSLATE_ARGS)
-list(APPEND _TRANSLATE_ARGS "-iree-mlir-to-vm-c-module")
-list(APPEND _TRANSLATE_ARGS "-iree-hal-target-backends=dylib-llvm-aot")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-link-embedded=false")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-link-static")
-list(APPEND _TRANSLATE_ARGS "-iree-llvm-static-library-output-path=simple_mul_c_module.o")
-list(APPEND _TRANSLATE_ARGS "${CMAKE_CURRENT_SOURCE_DIR}/simple_mul.mlir")
-list(APPEND _TRANSLATE_ARGS "-o")
-list(APPEND _TRANSLATE_ARGS "simple_mul_emitc.h")
+set(_COMPILE_ARGS)
+list(APPEND _COMPILE_ARGS "-iree-mlir-to-vm-c-module")
+list(APPEND _COMPILE_ARGS "-iree-hal-target-backends=dylib-llvm-aot")
+list(APPEND _COMPILE_ARGS "-iree-llvm-link-embedded=false")
+list(APPEND _COMPILE_ARGS "-iree-llvm-link-static")
+list(APPEND _COMPILE_ARGS "-iree-llvm-static-library-output-path=simple_mul_c_module.o")
+list(APPEND _COMPILE_ARGS "${CMAKE_CURRENT_SOURCE_DIR}/simple_mul.mlir")
+list(APPEND _COMPILE_ARGS "-o")
+list(APPEND _COMPILE_ARGS "simple_mul_emitc.h")
# Custom command for iree-compile to generate static library and C module.
add_custom_command(
@@ -117,8 +117,8 @@
${CMAKE_CURRENT_BINARY_DIR}/simple_mul_c_module.h
${CMAKE_CURRENT_BINARY_DIR}/simple_mul_c_module.o
${CMAKE_CURRENT_BINARY_DIR}/simple_mul_emitc.h
- COMMAND ${_TRANSLATE_TOOL_EXECUTABLE} ${_TRANSLATE_ARGS}
- DEPENDS ${_TRANSLATE_TOOL_EXECUTABLE} "simple_mul.mlir"
+ COMMAND ${_COMPILE_TOOL_EXECUTABLE} ${_COMPILE_ARGS}
+ DEPENDS ${_COMPILE_TOOL_EXECUTABLE} "simple_mul.mlir"
)
# TODO(marbre): Cleanup custom targets and libraries.
diff --git a/iree/samples/static_library/README.md b/iree/samples/static_library/README.md
index 92288e6..3451e00 100644
--- a/iree/samples/static_library/README.md
+++ b/iree/samples/static_library/README.md
@@ -50,7 +50,7 @@
```
2. Build the `static_library_demo` CMake target to create the static demo. This
-target has several dependencies that will translate `simple_mul.mlir` into a
+target has several dependencies that will compile `simple_mul.mlir` into a
static library (`simple_mul.h` & `simple_mul.c`) as well as a bytecode module
(`simple_mul.vmfb`) which are finally built into the demo binary:
diff --git a/iree/samples/variables_and_state/test.sh b/iree/samples/variables_and_state/test.sh
index 72ed455..f78db1b 100755
--- a/iree/samples/variables_and_state/test.sh
+++ b/iree/samples/variables_and_state/test.sh
@@ -22,7 +22,7 @@
test -f ${ARTIFACTS_DIR}/counter.mlir && echo "counter.mlir exists"
test -f ${ARTIFACTS_DIR}/counter_vmvx.vmfb && echo "counter_vmvx.vmfb exists"
-# 2. Compile the `iree_samples_variables_and_state` CMake target.
+# 2. Build the `iree_samples_variables_and_state` CMake target.
cmake -B ${BUILD_DIR} -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo ${ROOT_DIR}
cmake --build ${BUILD_DIR} --target iree_samples_variables_and_state -- -k 0
diff --git a/iree/test/e2e/tensor_ops/BUILD b/iree/test/e2e/tensor_ops/BUILD
index 6c8f78c..77402d9 100644
--- a/iree/test/e2e/tensor_ops/BUILD
+++ b/iree/test/e2e/tensor_ops/BUILD
@@ -4,12 +4,6 @@
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-# Tests of end-to-end IREE support for individual ops in the XLA HLO dialect.
-# Each test file should have a name matching the corresponding XLA HLO op and test only the
-# functionality of that op (though may make use of other ops where necessary). Tests should be
-# written using the IREE Check framework and should always pass on the reference VMLA backend.
-# See https://github.com/google/iree/blob/main/docs/developers/developing_iree/testing_guide.md#iree-core-end-to-end-tests.
-
load("//build_tools/bazel:enforce_glob.bzl", "enforce_glob")
load("//build_tools/bazel:iree_lit_test.bzl", "iree_lit_test_suite")
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_single_backend_test_suite")
diff --git a/iree/test/e2e/xla_ops/BUILD b/iree/test/e2e/xla_ops/BUILD
index 62aca10..8f47290 100644
--- a/iree/test/e2e/xla_ops/BUILD
+++ b/iree/test/e2e/xla_ops/BUILD
@@ -7,7 +7,7 @@
# Tests of end-to-end IREE support for individual ops in the XLA HLO dialect.
# Each test file should have a name matching the corresponding XLA HLO op and test only the
# functionality of that op (though may make use of other ops where necessary). Tests should be
-# written using the IREE Check framework and should always pass on the reference VMLA backend.
+# written using the IREE Check framework and should always pass on the reference VMVX backend.
# See https://github.com/google/iree/blob/main/docs/developers/developing_iree/testing_guide.md#iree-core-end-to-end-tests.
load("//build_tools/bazel:enforce_glob.bzl", "enforce_glob")
diff --git a/iree/test/e2e/xla_ops/partial/BUILD b/iree/test/e2e/xla_ops/partial/BUILD
index c0a0a65..0dd287f 100644
--- a/iree/test/e2e/xla_ops/partial/BUILD
+++ b/iree/test/e2e/xla_ops/partial/BUILD
@@ -12,7 +12,7 @@
# corresponding XLA HLO op with a suffix indicating the subset it tests. Only
# the functionality of that op should be tessted (though it may make use of
# other ops where necessary). Tests should be written using the IREE Check
-# framework and should always pass on the reference VMLA backend. See
+# framework and should always pass on the reference VMVX backend. See
# https://github.com/google/iree/blob/main/docs/developers/developing_iree/testing_guide.md#iree-core-end-to-end-tests.
load("//build_tools/bazel:iree_check_test.bzl", "iree_check_single_backend_test_suite")
