icon: octicons/bug-16

Integration test debugging

This document includes tips for triaging integration test correctness issues. Feel free to reach out to @hanhanW or @newling, or ask questions on Discord for more help.

General tips

Narrow down reproducers

Models themselves can be large, and IREE breaks models into dispatches/kernels and then launches those individually. Program outputs could diverge starting from any individual launch. To get a smaller reproducer, you can use --iree-flow-trace-dispatch-tensors, especially powerful when combined with debug sink callbacks, as described here.
You can compare the logs between builds/backends to get an idea about which dispatch results in wrong outputs. The dumped inputs can be reused in a flagfile.

Once a suspicious dispatch is identified, we can create a test case based on the dispatch function. The dispatch function can be derived after the OutlineDispatchRegions pass. The function signatures have to be modified manually. You'll have to put iree_tensor_ext.dispatch.tensor.load variables to function arguments, and replace iree_tensor_ext.dispatch.tensor.store with return op.

Note: This only works when dispatch formation logics are identical between runs.

iree-experimental repository tests

Follow README to run the model. The MLIR files will be generated. You'll find the saved file from log. E.g.,

[ RUN      ] MobilenetV2Int8Test.test_compile_tflite
I0401 17:27:04.084272 140182373025024 test_util.py:119] Setting up for IREE
I0401 17:27:04.085064 140182373025024 binaries.py:218] Invoke IREE Pipeline:
  /tmp/iree-experimental/iree-experimental.venv/lib/python3.11/site-packages/iree/tools/tflite/iree-import-tflite
    /tmp/iree-experimental/tflitehub/tmp/mobilenet_v2_int8_test.py/model.tflite
    --mlir-print-debuginfo
    --save-temp-tfl-input=/tmp/iree-experimental/tflitehub/tmp/mobilenet_v2_int8_test.py/tflite.mlir
    --save-temp-iree-input=/tmp/iree-experimental/tflitehub/tmp/mobilenet_v2_int8_test.py/tosa.mlir

Unfortunately, the artifacts are not dumped in the runs. There is an issue for tracking this. A workaround can be found in the issue.

TensorFlow integration tests

These are steps to reproduce/address failures in TF/TFLite integration tests. These instructions are most stable on Linux, though they may work with a few tweaks on Windows and macOS.

All steps here assume starting from the IREE root directory.

First create a Python virtual environment to install packages into:

python -m venv iree-tf.venv
source iree-tf.venv/bin/activate

# Install test requirements
python -m pip install -r ./integrations/tensorflow/test/requirements.txt

Install IREE's tools and Python bindings or build them from source

Install distributed packages

# Install packages from nightly pre-releases
# This should work for most cases, as the importers change infrequently
python -m pip install --pre \
  iree-base-compiler iree-base-runtime iree-tools-tf iree-tools-tflite \
  --find-links https://iree.dev/pip-release-links.html

OR build from source

# Build Python bindings from source
cmake -G Ninja -B ../iree-build/ -DIREE_BUILD_PYTHON_BINDINGS=ON .
cmake --build ../iree-build/

# Add IREE built-from-source Python packages to PYTHONPATH
source .env

# Install IREE TF/TFLite Python packages
python -m pip install integrations/tensorflow/python_projects/iree_tf
python -m pip install integrations/tensorflow/python_projects/iree_tflite

Run the python test command line

The command can be obtained from the run file. For example, if iree_tfl_tests/llvmcpu_posenet_i8.run failed,

cd integrations/tensorflow/test/
cat iree_tfl_tests/llvmcpu_posenet_i8.run

# REQUIRES: llvmcpu
# RUN: %PYTHON -m iree_tfl_tests.posenet_i8_test --target_backend=llvmcpu --artifacts_dir=%t

cd python/
python -m iree_tfl_tests.posenet_i8_test --target_backend=llvmcpu --artifacts_dir=/tmp/posenet_i8_failure

Note that the command can only be run under integrations/tensorflow/test/python directory.

Extract intermediate files and use with native tools
The test will create an iree_input.mlir in the temp directory specified. Those can then be fed into iree-compile (built locally to reproduce the error)
```
iree-compile \
  --iree-hal-target-device=local \
  --iree-hal-local-target-device-backends=llvm-cpu \
  --iree-input-type=stablehlo \
  iree_input.mlir
```