This directory contains the tools to run IREE benchmark suites and generate reports. More information about benchmark suites can be found here.
Currently we have run_benchmarks_on_android.py and run_benchmarks_on_linux.py scripts to run benchmark suites on Android devices (with adb) and Linux machines.
The available arguments can be shown with --help. Some common usages are listed below. Here we assume:
IREE_BUILD_DIR="/path/to/IREE build root dir". It should contain the "benchmark_suites" directory built with the target "iree-benchmark-suites". IREE_NORMAL_TOOL_DIR="/path/to/IREE tool dir". It is usually "$IREE_BUILD_DIR/tools". IREE_TRACED_TOOL_DIR="/path/to/IREE tool dir built with IREE_ENABLE_RUNTIME_TRACING=ON".
See details about IREE_ENABLE_RUNTIME_TRACING here.
Run all benchmarks
./run_benchmarks_on_linux.py \ --normal_benchmark_tool_dir=$IREE_NORMAL_TOOL_DIR \ --output=results.json $IREE_BUILD_DIR
Run all benchmarks and perform the Tracy captures
./run_benchmarks_on_linux.py \ --normal_benchmark_tool_dir=$IREE_NORMAL_TOOL_DIR \ --traced_benchmark_tool_dir=$IREE_TRACED_TOOL_DIR \ --trace_capture_tool=/path/to/iree-tracy-capture \ --capture_tarball=captured_tracy_files.tar.gz --output=results.json $IREE_BUILD_DIR
Run selected benchmarks with the filters
./run_benchmarks_on_linux.py \ --normal_benchmark_tool_dir=$IREE_NORMAL_TOOL_DIR \ --model_name_regex="MobileBertSquad" \ --driver_filter_regex="local-task" \ --mode_regex="4-threads" \ --output=results.json $IREE_BUILD_DIR
The tools here are mainly designed for benchmark automation pipelines. The post_benchmarks_as_pr_comment.py and upload_benchmarks_to_dashboard.py scripts are used to upload and post reports to pull requests or the dashboard.
If you want to generate a comparison report locally, you can use diff_local_benchmarks.py script to compare two result json files and generate the report. For example:
./diff_local_benchmarks.py --base before.json --target after.json > report.md