Use RBE to cache TF build results (#6867)
TF makes it a total PITA to build it with remote execution, but we *can*
get a bunch of the benefits (and with much less configuration) with
remote caching. This has actions executed locally but cached remotely.
Note that we need to ensure that the machines reading and writing from
the cache are ~identical, so this should only be executed inside of a
docker container and the docker image digest is used as the cache key.
Not really sure why I didn't think of this earlier.
Tested: observe that in the integrations build of the second commit
here, the Bazel part of the build took 30 seconds.
diff --git a/build_tools/bazel/iree.bazelrc b/build_tools/bazel/iree.bazelrc
index 226e4fd..ec3ab83 100644
--- a/build_tools/bazel/iree.bazelrc
+++ b/build_tools/bazel/iree.bazelrc
@@ -230,22 +230,43 @@
build:x86opt --copt=-mavx2
###############################################################################
-# Configuration for building remotely using Remote Build Execution (RBE)
-# https://cloud.google.com/remote-build-execution/
+# Configuration for building or caching remotely using Remote Build Execution
+# (RBE) https://cloud.google.com/remote-build-execution/
# Based on https://github.com/bazelbuild/bazel-toolchains/blob/master/bazelrc/bazel-1.0.0.bazelrc
-# Currently in development only usable by CI.
+# Usage of remote executors requires permissions in the cloud project (ask gcmn).
+# Local execution with remote caching is limited to the CI to ensure a stable
+# environment.
###############################################################################
-# Options for connecting to the IREE GCP remote build project. These are
-# repeated separately in each config to avoid warnings about configs being
-# expanded more than once.
+# Options for connecting to the IREE GCP remote build project.
# Enable authentication. This will pick up application default credentials by
# default. You can use --google_credentials=some_file.json to use a service
# account credential instead.
-build:rbe --google_default_credentials=true
+build:_rbe_base --google_default_credentials=true
# Point to the remote instance constructed in the iree-oss project
-build:rbe --remote_instance_name=projects/iree-oss/instances/default_instance
+build:_rbe_base --remote_instance_name=projects/iree-oss/instances/default_instance
+# Local disk cache is incompatible with remote execution or caching.
+build:_rbe_base --disk_cache=""
+
+### Remote Caching ###
+# Config for executing locally and caching the results to the remote cache.
+# This requires permissions to write to the remote cache, which only the CI
+# machines have.
+build:_remote_cache_base --config=_rbe_base
+build:_remote_cache_base --remote_cache=grpcs://remotebuildexecution.googleapis.com
+
+# --config=remote_cache_tf_integrations
+build:remote_cache_tf_integrations --config=_remote_cache_base
+# The cache key specifies the environment, which in this case points to the
+# specific docker container the TF build is run in. The image URL is included
+# for clarity and so that this reference is automatically updated by
+# manage_images.py
+build:remote_cache_tf_integrations --host_platform_remote_properties_override='properties:{name:"cache-silo-key" value:"gcr.io/iree-oss/cmake-bazel-frontends-swiftshader@sha256:103676490242311b9fad841294689a7ce1c755b935a21d8d898c25cfe3ec15e8"}'
+
+### Remote Execution ###
+# --config=rbe Execute and cache builds remotely.
+build:rbe --config=_rbe_base
# Depending on how many machines are in the remote execution instance, setting
# this higher can make builds faster by allowing more jobs to run in parallel.
@@ -287,9 +308,6 @@
# Set a higher timeout value, just in case.
build:rbe --remote_timeout=3600
-# Local disk cache is incompatible with remote execution (for obvious reasons).
-build:rbe --disk_cache=""
-
###############################################################################
# Configuration for uploading build results to Result Store UI
# https://cloud.google.com/remote-build-execution/docs/results-ui/getting-started-results-ui
@@ -297,14 +315,11 @@
###############################################################################
# Options for connecting to the IREE GCP remote build project. These are
-# repeated separately in each config to avoid warnings about configs being
-# expanded more than once.
+# repeated separately from _rbe_base to avoid warnings about configs being
+# expanded more than once when a remote execution/caching config is used
+# together with result store.
-# Enable authentication. This will pick up application default credentials by
-# default. You can use --google_credentials=some_file.json to use a service
-# account credential instead.
build:rs --google_default_credentials=true
-# Point to the remote instance constructed in the iree-oss project
build:rs --remote_instance_name=projects/iree-oss/instances/default_instance
build:rs --bes_backend="buildeventservice.googleapis.com"
diff --git a/build_tools/kokoro/gcp_ubuntu/cmake-bazel/linux/x86-swiftshader/build.sh b/build_tools/kokoro/gcp_ubuntu/cmake-bazel/linux/x86-swiftshader/build.sh
index 5966d91..d1869ac 100755
--- a/build_tools/kokoro/gcp_ubuntu/cmake-bazel/linux/x86-swiftshader/build.sh
+++ b/build_tools/kokoro/gcp_ubuntu/cmake-bazel/linux/x86-swiftshader/build.sh
@@ -36,7 +36,9 @@
BAZEL_CMD=(bazel --noworkspace_rc --bazelrc=build_tools/bazel/iree-tf.bazelrc)
BAZEL_BINDIR="$(${BAZEL_CMD[@]?} info bazel-bin)"
"${BAZEL_CMD[@]?}" query //iree_tf_compiler/... | \
- xargs "${BAZEL_CMD[@]?}" test --config=generic_clang \
+ xargs "${BAZEL_CMD[@]?}" test \
+ --config=remote_cache_tf_integrations \
+ --config=generic_clang \
--test_tag_filters="-nokokoro" \
--build_tag_filters="-nokokoro"
popd
