Refreshing VM performance by separating verification and tweaking buffer access. (#12426)
This moves much of the interpreter checks to an ahead-of-time bytecode
verifier. This allows us to share the same verification with JITs and
disable it entirely for code size reasons using the
`-DIREE_VM_BYTECODE_VERIFICATION_ENABLE=0` compiler flag. Verification
is pretty exhaustive but may still need some additions. It's
significantly better than before, though, so even if not the final form
it's a good step. Simpler verification (and dispatch) will come with a
pending bytecode shuffling into instruction classes.
Since this required breaking binary compatibility I did some deferred
changes that had been sitting in
https://github.com/openxla/iree/projects/32, namely requirement bits and
fixing vm.buffer.fill from bytes to elements.
The requirements give us much nicer error messages by supporting
per-module and per-function bitfields indicating features required to
execute the bytecode they contain, e.g.:
```
D:\Dev\iree\runtime\src\iree\vm\bytecode\module.c:309: INVALID_ARGUMENT; required module features [EXT_F32] are not available in this runtime configuration; have [] while module requires [EXT_F32]; while invoking native function hal.executable.create; while calling import;
[ 1] native hal.executable.create:0 -
[ 0] bytecode module.__init:446 D:\Dev\iree/tests/e2e/models/unidirectional_lstm.mlir:0:0
```
Splitting out the verification from dispatch is good for code
reuse/optionality but also now lets us dispatch without verifying.
Between removing the inlined verification/register masking/etc and
streamlining buffer access we get a near 2x speedup of compute-heavy
VMVX workloads. resnet50 for example on my ryzen system (with
`--iree-vm-target-index-bits=64`, which I need to make default):
```
before:
1 core: BM_predict/process_time/real_time 343774 ms 343766 ms 1 items_per_second=2.90888m/s
8 core: BM_predict/process_time/real_time 48306 ms 361156 ms 1 items_per_second=0.0207012/s
32 core: BM_predict/process_time/real_time 18943 ms 408922 ms 2 items_per_second=0.0527891/s
after:
1 core: BM_predict/process_time/real_time 147856 ms 147859 ms 1 items_per_second=6.76332m/s
8 core: BM_predict/process_time/real_time 21569 ms 158781 ms 1 items_per_second=0.0463637/s
32 core: BM_predict/process_time/real_time 8962 ms 186276 ms 3 items_per_second=0.111579/s
```
About ~20-30% of the remaining runtime is spent in bytecode dispatch
which needs a larger op table reworking to make better. That'll also
reduce code size quite a bit as today we have a lot of duplicate
decoding work. The most expensive ops remaining are buffer loads/stores
and short of JIT or scatter/gather such as #8477 there's not much to do
besides less work. Today codegen is producing some phenomenally bad code
and we're executing ~100x+ more instructions than required
(https://gist.github.com/benvanik/e2b45891e02baf8318109b60189a1b12 for
example - that's _a lot_ of loop arithmetic, a useless fill that should
be removed or at least turned into a util.buffer.fill, and unfused
writebacks) - even without op table shuffling or microkernels we should
be well under 900ms instead of 9000ms. We've also got to parameterize
our workgroup distribution - today we don't tend to use more than 4-16
cores so we don't see the latency improvement we'd expect going 8->32
cores (16 cores is nearly identical).
These issues also impact emitc paths as the C compiler downstream of
that is dealing with all this difficult to analyze output and can't do
much. I thought an emitc resnet with inline VMVX would be a good
approximation of what a JIT could do and it's not good: 448154ms vs the
147856ms of the interpreter! It's also 2x larger in size on disk (380KB
x86_64 vs 200KB bytecode) and that's prior to optimization of the
bytecode. We should definitely be able to do 2-4x faster with a naïve
JIT - if not 10x!
Fixes #5732.
Fixes #12373.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2295e77..c34cc8c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -205,8 +205,6 @@
# Attempt to restore from cache unconditionally.
# Note: this will first try to grab a cache entry for this exact commit
# then it will fall back to the latest for any commit.
- # We're pretty limited on repository cache space (10GB) relative to cache
- # entry size (~2.5GB), so this usage is pretty optimistic.
- name: "Fetching cache (CMake/ccache)"
uses: actions/cache/restore@58c146cc91c5b9e778e71775dfe9bf1442ad9a12 # v3.2.3
with:
@@ -238,9 +236,15 @@
IREE_READ_LOCAL_CCACHE: 1
IREE_WRITE_LOCAL_CCACHE: ${{ needs.setup.outputs.write-caches }}
CCACHE_DIR: ${{ github.workspace }}/.ccache
- # A full build is around 2.5GB, but uploads/downloads are slow and
- # we have a limit of 10GB per repository
- CCACHE_MAXSIZE: 3G
+ # Cache size and compression level settings are a delicate balance.
+ # * A full build cache is around 2-5GB depending on compression level
+ # * Upload/download is slow (double compression may or may not help)
+ # * We have a limit of 10GB across all cached files per repository
+ # * Cache misses are quite costly:
+ # * 99% cache hits -> ~5 minutes to build
+ # * 20% cache hits -> ~15-20 minutes to build
+ CCACHE_MAXSIZE: 4G
+ CCACHE_COMPRESSLEVEL: 5
run: ./build_tools/cmake/build_all.sh "${BUILD_DIR}"
- name: "Testing IREE"
run: ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}"
diff --git a/build_tools/bazel/iree.bazelrc b/build_tools/bazel/iree.bazelrc
index 3bf77da..827036a 100644
--- a/build_tools/bazel/iree.bazelrc
+++ b/build_tools/bazel/iree.bazelrc
@@ -5,14 +5,6 @@
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
###############################################################################
-# Short-hand config settings for IREE specific features.
-###############################################################################
-
-# Equivalent to CMake flag IREE_ENABLE_RUNTIME_TRACING.
-# Builds the runtime with tracing support enabled.
-build --flag_alias=iree_enable_runtime_tracing=@tracy_client//:runtime_enable
-
-###############################################################################
# Common flags that apply to all configurations.
# Use sparingly for things common to all compilers and platforms.
###############################################################################
@@ -45,6 +37,10 @@
# Build settings flag aliases. See https://bazel.build/rules/config
###############################################################################
build --flag_alias=iree_drivers=//runtime/src/iree/hal/drivers:enabled_drivers
+build --flag_alias=iree_link_compiler_shared=//compiler/src/iree/compiler/API2:link_shared
+# Equivalent to CMake flag IREE_ENABLE_RUNTIME_TRACING.
+# Builds the runtime with tracing support enabled.
+build --flag_alias=iree_enable_runtime_tracing=@tracy_client//:runtime_enable
###############################################################################
# Options for "generic_gcc" builds
diff --git a/build_tools/bazel_to_cmake/bazel_to_cmake_converter.py b/build_tools/bazel_to_cmake/bazel_to_cmake_converter.py
index 798d64f..ba805a5 100644
--- a/build_tools/bazel_to_cmake/bazel_to_cmake_converter.py
+++ b/build_tools/bazel_to_cmake/bazel_to_cmake_converter.py
@@ -49,6 +49,12 @@
}
+def _should_skip_target(tags=None, **kwargs):
+ if tags and "skip-bazel_to_cmake" in tags:
+ return True
+ return False
+
+
def _convert_timeout_arg_block(name, value):
if value is None:
return ""
@@ -221,6 +227,9 @@
def alias(self, *args, **kwargs):
pass
+ def bool_flag(self, *args, **kwargs):
+ pass
+
def load(self, *args, **kwargs):
pass
@@ -255,6 +264,8 @@
self._convert_unimplemented_function("filegroup", name)
def sh_binary(self, name, **kwargs):
+ if _should_skip_target(**kwargs):
+ return
self._convert_unimplemented_function("sh_binary", name)
def enforce_glob(self, files, **kwargs):
@@ -299,9 +310,11 @@
def platform_trampoline_deps(self, basename, path="base"):
return [f"//{path}/internal:{basename}_internal"]
- def select(self, d):
- self._convert_unimplemented_function("select", str(d))
- return d["//conditions:default"]
+ def select(self, selector):
+ default_value = selector.get("//conditions:default")
+ if default_value is None:
+ raise ValueError("bazel_to_cmake can only convert selects with a default")
+ return default_value
def cc_library(self,
name,
@@ -316,6 +329,8 @@
linkopts=None,
includes=None,
**kwargs):
+ if _should_skip_target(**kwargs):
+ return
if linkopts:
self._convert_unimplemented_function("linkopts")
name_block = _convert_string_arg_block("NAME", name, quote=False)
@@ -363,6 +378,8 @@
tags=None,
includes=None,
**kwargs):
+ if _should_skip_target(tags=tags, **kwargs):
+ return
name_block = _convert_string_arg_block("NAME", name, quote=False)
hdrs_block = _convert_string_list_block("HDRS", hdrs, sort=True)
srcs_block = _convert_srcs_block(srcs)
@@ -406,6 +423,8 @@
testonly=None,
includes=None,
**kwargs):
+ if _should_skip_target(**kwargs):
+ return
if linkopts:
self._convert_unimplemented_function("linkopts")
name_block = _convert_string_arg_block("NAME", name, quote=False)
@@ -442,6 +461,8 @@
identifier=None,
deps=None,
**kwargs):
+ if _should_skip_target(**kwargs):
+ return
name_block = _convert_string_arg_block("NAME", name, quote=False)
srcs_block = _convert_srcs_block(srcs)
c_file_output_block = _convert_string_arg_block("C_FILE_OUTPUT",
@@ -565,6 +586,8 @@
f")\n\n")
def iree_gentbl_cc_library(self, **kwargs):
+ if _should_skip_target(**kwargs):
+ return
# The bazel version of this rule adds some include directories and defs
# that are implicitly handled by the cmake version.
self.gentbl_cc_library(**kwargs)
@@ -598,6 +621,8 @@
tags=None,
timeout=None,
**kwargs):
+ if _should_skip_target(tags=tags, **kwargs):
+ return
name_block = _convert_string_arg_block("NAME", name, quote=False)
srcs_block = _convert_srcs_block(srcs)
tools_block = _convert_target_list_block("TOOLS", tools)
@@ -626,6 +651,8 @@
target_cpu_features=None,
timeout=None,
**kwargs):
+ if _should_skip_target(tags=tags, **kwargs):
+ return
name_block = _convert_string_arg_block("NAME", name, quote=False)
srcs_block = _convert_srcs_block(srcs)
target_backend_block = _convert_string_arg_block("TARGET_BACKEND",
@@ -661,6 +688,8 @@
target_cpu_features_variants=None,
timeout=None,
**kwargs):
+ if _should_skip_target(tags=tags, **kwargs):
+ return
target_backends = None
drivers = None
if target_backends_and_drivers is not None:
@@ -703,6 +732,8 @@
tags=None,
target_cpu_features_variants=None,
**kwargs):
+ if _should_skip_target(tags=tags, **kwargs):
+ return
target_backends = None
drivers = None
if target_backends_and_drivers is not None:
@@ -749,6 +780,8 @@
data=None,
tags=None,
timeout=None):
+ if _should_skip_target(tags=tags):
+ return
if data is not None:
self._convert_unimplemented_function("native_test", name + " has data")
@@ -779,7 +812,8 @@
# unused
size="small",
timeout=None):
-
+ if _should_skip_target(tags=tags):
+ return
name_block = _convert_string_arg_block("NAME", name, quote=False)
srcs_block = _convert_srcs_block(srcs)
data_block = _convert_target_list_block("DATA", data)
diff --git a/compiler/bindings/c/BUILD b/compiler/bindings/c/BUILD
index a5cdac9..6f0144a 100644
--- a/compiler/bindings/c/BUILD
+++ b/compiler/bindings/c/BUILD
@@ -33,4 +33,20 @@
],
)
-# TODO: Support loader_test in Bazel.
+cc_test(
+ name = "loader_test",
+ srcs = [
+ "iree/compiler/loader/loader_test.c",
+ ],
+ args = [
+ "lib/libIREECompiler.so",
+ ],
+ data = [
+ "//lib:libIREECompiler.so",
+ ],
+ tags = ["skip-bazel_to_cmake"],
+ deps = [
+ ":headers",
+ ":loader",
+ ],
+)
diff --git a/compiler/src/iree/compiler/API2/BUILD b/compiler/src/iree/compiler/API2/BUILD
index eabcec4..e44f131 100644
--- a/compiler/src/iree/compiler/API2/BUILD
+++ b/compiler/src/iree/compiler/API2/BUILD
@@ -4,6 +4,7 @@
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
load("//build_tools/bazel:build_defs.oss.bzl", "iree_compiler_cc_library")
package(
@@ -49,12 +50,30 @@
],
)
-# Bazel does not have a well-defined mechanism for managing
-# shared libraries in a cross-platform way. Therefore, we
-# only support static linking of tools. Separately, we do
-# build an actual shared library for platforms that support
-# it, but this is only done to support stub use-cases.
+iree_compiler_cc_library(
+ name = "SharedImpl",
+ srcs = [
+ "//lib:libIREECompiler.so",
+ ],
+ tags = ["skip-bazel_to_cmake"],
+)
+
+bool_flag(
+ name = "link_shared",
+ build_setting_default = False,
+)
+
+config_setting(
+ name = "link_shared_config",
+ flag_values = {
+ ":link_shared": "True",
+ },
+)
+
alias(
name = "Impl",
- actual = ":StaticImpl",
+ actual = select({
+ ":link_shared_config": ":SharedImpl",
+ "//conditions:default": ":StaticImpl",
+ }),
)
diff --git a/compiler/src/iree/compiler/Codegen/Common/BUILD b/compiler/src/iree/compiler/Codegen/Common/BUILD
index cb4597a..78bad29 100644
--- a/compiler/src/iree/compiler/Codegen/Common/BUILD
+++ b/compiler/src/iree/compiler/Codegen/Common/BUILD
@@ -132,6 +132,7 @@
"TestExecutablePreprocessing.cpp",
"TestPartitionableLoopsInterface.cpp",
"TileAndDistributeToWorkgroupsPass.cpp",
+ "TileDispatchUsingInterface.cpp",
"TypePropagationPass.cpp",
"VectorizePackUnPackOps.cpp",
"VectorizePad.cpp",
@@ -216,9 +217,6 @@
"MemrefCopyToLinalg.cpp",
"PadDynamicAlloc.cpp",
"RemoveTrivialLoops.cpp",
- "TestPartitionableLoopsInterface.cpp",
- "TileAndDistributeToWorkgroupsPass.cpp",
- "TileDispatchUsingInterface.cpp",
"UserConfig.cpp",
"VectorReductionToGPU.cpp",
"WorkGroupSwizzle.cpp",
diff --git a/compiler/src/iree/compiler/Codegen/Common/CMakeLists.txt b/compiler/src/iree/compiler/Codegen/Common/CMakeLists.txt
index e887349..1b0e4e3 100644
--- a/compiler/src/iree/compiler/Codegen/Common/CMakeLists.txt
+++ b/compiler/src/iree/compiler/Codegen/Common/CMakeLists.txt
@@ -107,6 +107,7 @@
"TestExecutablePreprocessing.cpp"
"TestPartitionableLoopsInterface.cpp"
"TileAndDistributeToWorkgroupsPass.cpp"
+ "TileDispatchUsingInterface.cpp"
"TypePropagationPass.cpp"
"VectorizePackUnPackOps.cpp"
"VectorizePad.cpp"
@@ -189,9 +190,6 @@
"MemrefCopyToLinalg.cpp"
"PadDynamicAlloc.cpp"
"RemoveTrivialLoops.cpp"
- "TestPartitionableLoopsInterface.cpp"
- "TileAndDistributeToWorkgroupsPass.cpp"
- "TileDispatchUsingInterface.cpp"
"UserConfig.cpp"
"VectorReductionToGPU.cpp"
"WorkGroupSwizzle.cpp"
diff --git a/compiler/src/iree/compiler/API2/Shlib/Posix/BUILD.bazel b/lib/BUILD
similarity index 64%
rename from compiler/src/iree/compiler/API2/Shlib/Posix/BUILD.bazel
rename to lib/BUILD
index 9923488..daf2d17 100644
--- a/compiler/src/iree/compiler/API2/Shlib/Posix/BUILD.bazel
+++ b/lib/BUILD
@@ -1,16 +1,15 @@
-# Copyright 2022 The IREE Authors
+# Copyright 2023 The IREE Authors
#
# Licensed under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-# bazel-to-cmake: skip
+package(
+ default_visibility = ["//visibility:public"],
+ features = ["layering_check"],
+ licenses = ["notice"], # Apache 2.0
+)
-# This is problematic on anything non-Linux but Bazel has failed
-# for years to provide the flexibility to make it any better. Yolo.
-# We really only support this for some very narrow use cases that
-# need to dynamically load the compiler as a shared library, and other
-# uses are served by the cross-platform CMake build.
cc_binary(
name = "libIREECompiler.so",
srcs = [
diff --git a/runtime/src/iree/builtins/device/BUILD b/runtime/src/iree/builtins/device/BUILD
index 29e48e6..531d733 100644
--- a/runtime/src/iree/builtins/device/BUILD
+++ b/runtime/src/iree/builtins/device/BUILD
@@ -42,7 +42,7 @@
iree_cmake_extra_content(
content = """
-if(NOT IREE_BUILD_COMPILER)
+if(NOT IREE_BUILD_COMPILER OR NOT IREE_TARGET_BACKEND_LLVM_CPU)
return()
endif()
""",
diff --git a/runtime/src/iree/builtins/device/CMakeLists.txt b/runtime/src/iree/builtins/device/CMakeLists.txt
index ca2bd01..0b01c73 100644
--- a/runtime/src/iree/builtins/device/CMakeLists.txt
+++ b/runtime/src/iree/builtins/device/CMakeLists.txt
@@ -22,7 +22,7 @@
PUBLIC
)
-if(NOT IREE_BUILD_COMPILER)
+if(NOT IREE_BUILD_COMPILER OR NOT IREE_TARGET_BACKEND_LLVM_CPU)
return()
endif()
