Make benchmark artifact generation a bit more flexible. (#8846)
Here's my use case:
* I'm working on Windows, where we don't support building the TensorFlow integration tools (notably `iree-import-tflite`)
* I want to use the .mlir files that we feed into our benchmarking for local development (more specifically I want _some_ set of real programs, but the benchmark suite offers a set that we continuously build/test and have metrics for already)
---
Now, a sensible solution would be to extend
https://github.com/google/iree/blob/1ddd9170bd6cc2701e2d669754a8fff248739233/build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml#L9-L14
to also zip up the imported .mlir files so anyone could download them from Buildkite.
---
I, however, had the more indirect idea to write a Colab notebook ([work in progress here](https://colab.research.google.com/gist/ScottTodd/4b27c6b44f90d239d4676dacddb87895/iree-benchmarks-import-demo.ipynb)) that installs the latest IREE release along with the project source and then builds `iree-benchmark-suites` using the installed tools. That _almost_ worked, except for the few issues that this PR patches over:
* `$<TARGET_FILE:iree::tools::iree-compile>` only works when building with `IREE_BUILD_COMPILER`, but it's also useful to point at already built tools using `IREE_HOST_BINARY_ROOT`. The `iree_get_executable_path` function handles switching between those modes easily
* Compiling imported files for the full list of benchmark configurations can be slow. If a developer just wants to import .mlir files then skipping those following steps would be useful. I added a new `iree-benchmark-import-models` target that does that.
(I also considered the _even more indirect_ idea to parse the artifact paths from [benchmarks/TFLite/CMakeLists.txt](https://github.com/google/iree/blob/main/benchmarks/TFLite/CMakeLists.txt) and run `iree-import-tflite` manually, but the `iree-benchmark-suites` target already does that natively :P)
---
I'm open to any workflow that solves my original use case, but these changes seem generally good to have anyways.
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8a39a47..78109eb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -558,7 +558,16 @@
#-------------------------------------------------------------------------------
if(${IREE_BUILD_BENCHMARKS})
- # Add a top-level custom target to drive generating benchmark suites.
+ # Add top-level custom targets to drive generating benchmark suites.
+
+ # iree-benchmark-import-models imports benchmark models from their source
+ # formats, such as .tflite flatbuffers, to IREE-compatible .mlir files.
+ add_custom_target(iree-benchmark-import-models)
+
+ # iree-benchmark-suites fully prepares benchmark models for benchmarking:
+ # * importing from source formats to IREE-compatible .mlir files
+ # * compiling from .mlir files to benchmark-ready .vmfb files
+ # * generating flagfiles for executing the benchmark .vmfb files
add_custom_target(iree-benchmark-suites)
endif()
diff --git a/build_tools/cmake/iree_benchmark_suite.cmake b/build_tools/cmake/iree_benchmark_suite.cmake
index a830c5e..34280a2 100644
--- a/build_tools/cmake/iree_benchmark_suite.cmake
+++ b/build_tools/cmake/iree_benchmark_suite.cmake
@@ -97,6 +97,7 @@
get_filename_component(_CATEGORY "${CMAKE_CURRENT_SOURCE_DIR}" NAME)
set(_ROOT_ARTIFACTS_DIR "${IREE_BINARY_DIR}/benchmark_suites/${_CATEGORY}")
set(_VMFB_ARTIFACTS_DIR "${_ROOT_ARTIFACTS_DIR}/vmfb")
+ file(MAKE_DIRECTORY ${_VMFB_ARTIFACTS_DIR})
# The name of any custom target that drives creation of the final source
# MLIR file. Depending on the format of the source, this will get updated.
@@ -159,6 +160,7 @@
COMMENT
"Importing ${_TFLITE_FILE_BASENAME} into MLIR"
)
+ add_dependencies(iree-benchmark-import-models "${_TFLITE_IMPORT_TARGET}")
endif()
set(_MODULE_SOURCE_TARGET "${_TFLITE_IMPORT_TARGET}")
endif()
@@ -205,17 +207,18 @@
"${PACKAGE_NAME}_iree-generate-benchmark-artifact-${_MODULE_SOURCE_BASENAME}-${_VMFB_HASH}"
)
if(NOT TARGET "${_TRANSLATION_TARGET_NAME}")
+ iree_get_executable_path(_COMPILE_TOOL_EXECUTABLE "iree-compile")
add_custom_command(
OUTPUT "${_VMFB_FILE}"
COMMAND
- "$<TARGET_FILE:iree::tools::iree-compile>"
+ ${_COMPILE_TOOL_EXECUTABLE}
${_TRANSLATION_ARGS}
"--mlir-print-op-on-diagnostic=false"
"${_MODULE_SOURCE}"
-o "${_VMFB_FILE}"
WORKING_DIRECTORY "${_VMFB_ARTIFACTS_DIR}"
DEPENDS
- iree::tools::iree-compile
+ ${_COMPILE_TOOL_EXECUTABLE}
"${_MODULE_SOURCE_TARGET}"
COMMENT "Generating VMFB for ${_COMMON_NAME_SEGMENTS}"
)