Google Git
Sign in
opensecura/3p/openxla/iree/46e47e3458ec9a88cce75d7d3e49eea13eb0e06e^!/.
commit
46e47e3458ec9a88cce75d7d3e49eea13eb0e06e
[log]
author
Geoffrey Martin-Noble <gcmn@google.com>
Tue Nov 29 16:39:45 2022 -0800
committer
GitHub <noreply@github.com>
Tue Nov 29 16:39:45 2022 -0800
tree
1977b4ea0da66a5099a9b9632cc04c1d9a01a4c9
parent
30356ad7c51a945981a3207d3ef604a3f231c273 [diff]
Use ccache in CI (#11311)

This connects our CMake builds to a [ccache](https://ccache.dev/)
hosted in a GCS bucket. `ccache` newly (ish) supports using remote
storage for the cache! Currently it only supports Redis, FTP, and HTTP.
HTTPS is *not* supported right now, but there are plans to add an HTTPS
backend, as well as potentially a direct GCS backend (see
https://github.com/ccache/ccache/issues/1214).

I think this adds a little bit of overhead for the network requests,
potentially increasing the time for building with a completely cold 
cache. 

An example `build_all` job with a completely cold cache took 13.2
minutes for the entire job, 10 minutes for just the build step, of
which 6.1 minutes was spent in the actual `cmake --build` command (not
including builds of the `install` or `iree-test-deps` targets, which
don't involve building C++):
https://github.com/iree-org/iree/actions/runs/3562697821/jobs/5984663663

Going through that commit's ancestors on the main branch, this looks
like it's adding about 30±30 seconds to the build, using the
statistical technique of "eyeballing".

We get wins on the flip side though, where with a fully cached build,
the times are 6.3m, 3.8m, 1.6m.

The impact is even bigger with asan, where we see the same ~50%
improvement on the already-slower build.

Unfortunately, since ccache is a language-specific cache, we can't
do the same trick with all the test artifacts.

The lack of HTTPS support does present somewhat of a problem because
GCP doesn't allow using unsecured HTTP for many API access scopes. I
ran into trouble with this when trying to get things to work locally
because the local gcloud credentials for a user account usually have
very broad scope (see discussion in
https://github.com/ccache/ccache/pull/1001). But it *does* work fine on
our GCP VMs since those service accounts have much more limited
permissions. Luckily, we don't actually want users writing to the
cache, so this mostly just impacted me setting it up.

I also tried
[sccache](https://github.com/mozilla/sccache), which has a GCS backend,
but configuring the backend locally was pretty janky (see
https://github.com/mozilla/sccache/issues/144#issuecomment-1307997283).
I ultimately went with ccache since it's the much more established
project and it seems like there's quite a bit of design work going in
to making it work well.

ccache also supports two caching layers (indeed this is the standard
setup), so devs could make use of the remote cache by setting a
single config/env variable to point at it and continue using their
local ccache as well. This will of course only work as long as their
local machine is sufficiently similar to the docker containers or they
choose to build within docker containers.

Co-authored-by: Scott Todd <scotttodd@google.com>
diff --git a/.github/workflows/android_tflite_oneshot_build.yml b/.github/workflows/android_tflite_oneshot_build.yml
index 1ea7511..3152eaf 100644
--- a/.github/workflows/android_tflite_oneshot_build.yml
+++ b/.github/workflows/android_tflite_oneshot_build.yml

@@ -9,7 +9,7 @@
   build_android_with_docker:
     runs-on: ubuntu-20.04
     env:
-      ANDROID_CONTAINER: "gcr.io/iree-oss/gradle-android@sha256:868038ec2c27b33a13ceee831fdb0cb0ac6e9035205305cc03c4e27b043228de"
+      ANDROID_CONTAINER: "gcr.io/iree-oss/gradle-android@sha256:c4c3450ee634a3fc6b75fa73b24f0d97d8b295e0cf599aab63dd5f12a8746333"
     steps:
       - uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
         with:

diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml
index 8f607f7..477a763 100644
--- a/.github/workflows/benchmarks.yml
+++ b/.github/workflows/benchmarks.yml

@@ -96,8 +96,8 @@
           build_tools/github_actions/docker_run.sh \
             --env "IREE_TF_BINARIES_DIR=${TF_BINARIES_DIR}" \
             --env "IREE_HOST_BINARY_ROOT=${BUILD_DIR}/install" \
-            --env "BUILD_BENCHMARKS_DIR=${BENCHMARKS_OUTPUT_DIR}" \
-            gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 \
+            --env "IREE_BUILD_BENCHMARKS_DIR=${BENCHMARKS_OUTPUT_DIR}" \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             build_tools/cmake/build_benchmarks.sh
           echo "::set-output name=benchmarks-dir::${BENCHMARKS_OUTPUT_DIR}"
       - name: "Creating archive of benchmarks"
@@ -135,7 +135,7 @@
         target:
           - platform: "riscv"
             architecture: "rv64"
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
     outputs:
       benchmark-tools-gcs-artifacts: ${{ toJSON(steps.upload.outputs) }}
     env:

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f13403a..52423d1 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml

@@ -115,9 +115,14 @@
         with:
           submodules: true
       - name: "Building IREE"
+        env:
+          IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 \
+            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
+            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
+            --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206" \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             ./build_tools/cmake/build_all.sh \
             "${BUILD_DIR}"
       # The archive step below doesn't include these files. Remove them first to
@@ -200,11 +205,11 @@
           submodules: true
       - name: "Building with Bazel"
         env:
-          IREE_BAZEL_WRITE_REMOTE_CACHE: ${{ needs.setup.outputs.write-caches }}
+          IREE_WRITE_REMOTE_BAZEL_CACHE: ${{ needs.setup.outputs.write-caches }}
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            --env "IREE_BAZEL_WRITE_REMOTE_CACHE=${IREE_BAZEL_WRITE_REMOTE_CACHE}" \
-            gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:50301f9ccc0bf63c4d498c225444cc5585218612d15b83d06e4f4672ab2dfc03 \
+            --env "IREE_WRITE_REMOTE_BAZEL_CACHE=${IREE_WRITE_REMOTE_BAZEL_CACHE}" \
+            gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:c22afc61198e14a98e06e5261149de74c629acd2bc61b82e57ec90e5461b69be \
             ./build_tools/bazel/build_core.sh
 
   test_all:
@@ -233,7 +238,7 @@
         run: |
           ./build_tools/github_actions/docker_run.sh \
             --env IREE_CUDA_DISABLE=1 \
-            gcr.io/iree-oss/swiftshader@sha256:484720d57fa816280d9a7455d16e1f9f3d90cc3efdb744646016503ea5ab2b30 \
+            gcr.io/iree-oss/swiftshader@sha256:27b843417fd548fab1b636d2b65d2b8b8c09f42a65cd5dde7ecfb9881e4332ac \
             ./build_tools/cmake/ctest_all.sh \
             "${BUILD_DIR}"
 
@@ -271,7 +276,7 @@
             --env CTEST_PARALLEL_LEVEL=2 \
             --gpus all \
             --env NVIDIA_DRIVER_CAPABILITIES=all \
-            gcr.io/iree-oss/nvidia@sha256:74867181ab29bc890775bf3e2da717225431eaa022422e2dc865dc04d37289a4 \
+            gcr.io/iree-oss/nvidia@sha256:b0751e9f2fcb104d9d3d56fab6c6e79405bdcd2e503e53f2bf4f2b66d13cd88b \
             bash -euo pipefail -c \
               "./build_tools/scripts/check_cuda.sh
               ./build_tools/scripts/check_vulkan.sh
@@ -292,16 +297,18 @@
       - name: "Checking out runtime submodules"
         run: ./build_tools/scripts/git/update_runtime_submodules.sh
       - name: "Building runtime"
+        # Note ccache is read-only here since this job runs on a GitHub-hosted runner
+        # and GitHub runners don't have write access to GCS
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             ./build_tools/cmake/build_runtime.sh \
             "${BUILD_DIR}"
       - name: "Testing runtime"
         run: |
           ./build_tools/github_actions/docker_run.sh \
             --env IREE_VULKAN_DISABLE=1 \
-            gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             ./build_tools/cmake/ctest_all.sh \
             "${BUILD_DIR}"
 
@@ -352,12 +359,12 @@
         id: build
         env:
           IREE_TF_BINARIES_OUTPUT_DIR: iree-tf-binaries
-          IREE_BAZEL_WRITE_REMOTE_CACHE: ${{ needs.setup.outputs.write-caches }}
+          IREE_WRITE_REMOTE_BAZEL_CACHE: ${{ needs.setup.outputs.write-caches }}
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            --env "IREE_BAZEL_WRITE_REMOTE_CACHE=${IREE_BAZEL_WRITE_REMOTE_CACHE}" \
+            --env "IREE_WRITE_REMOTE_BAZEL_CACHE=${IREE_WRITE_REMOTE_BAZEL_CACHE}" \
             --env "IREE_TF_BINARIES_OUTPUT_DIR=${IREE_TF_BINARIES_OUTPUT_DIR}" \
-            gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd \
+            gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d \
             build_tools/cmake/build_tf_binaries.sh
           echo "::set-output name=binaries-dir::${IREE_TF_BINARIES_OUTPUT_DIR}"
       - name: "Creating archive of binaries"
@@ -412,7 +419,7 @@
       - name: "Running TF integrations tests"
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd \
+            gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d \
             build_tools/cmake/run_tf_tests.sh \
             "${BUILD_DIR}"
 
@@ -454,7 +461,7 @@
             --env IREE_LLVM_CPU_DISABLE=1 \
             --gpus all \
             --env NVIDIA_DRIVER_CAPABILITIES=all \
-            gcr.io/iree-oss/frontends-nvidia@sha256:f9256c8f752179c72fe4fafaeac5d31c3b63ec30c8c35a5ab2c5ae4d0db1a631 \
+            gcr.io/iree-oss/frontends-nvidia@sha256:3e3965b6b9e431be296900440e5bc66ed46a8f6d6fa80ead7c7a75866f3bd416 \
             bash -euo pipefail -c \
               "./build_tools/scripts/check_cuda.sh
               ./build_tools/scripts/check_vulkan.sh
@@ -536,12 +543,17 @@
         with:
           submodules: true
       - name: "Building and testing with AddressSanitizer"
+        env:
+          IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
         run: |
           # Note that this uses the latest version of the clang compiler, etc.
           # This gives us access to the latest features and validates that IREE
           # builds using the latest versions.
           ./build_tools/github_actions/docker_run.sh \
-            gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:50301f9ccc0bf63c4d498c225444cc5585218612d15b83d06e4f4672ab2dfc03 \
+            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
+            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
+            --env "CCACHE_NAMESPACE=swiftshader-bleeding-edge@sha256:c22afc61198e14a98e06e5261149de74c629acd2bc61b82e57ec90e5461b69be" \
+            gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:c22afc61198e14a98e06e5261149de74c629acd2bc61b82e57ec90e5461b69be \
             ./build_tools/cmake/build_and_test_asan.sh
 
   tsan:
@@ -559,9 +571,14 @@
         with:
           submodules: true
       - name: "Building and testing with ThreadSanitizer"
+        env:
+          IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
         run: |
           ./build_tools/github_actions/docker_run.sh \
-            gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 \
+            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
+            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
+            --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206" \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             ./build_tools/cmake/build_and_test_tsan.sh
 
   benchmarks:
@@ -620,7 +637,7 @@
           build_tools/github_actions/docker_run.sh \
             --env "IREE_TF_BINARIES_DIR=${TF_BINARIES_DIR}" \
             --env "IREE_HOST_BINARY_ROOT=${HOST_BUILD_DIR}/install" \
-            gcr.io/iree-oss/base@sha256:7c3027c48b94fc38e64488987fc7893c100526c57308d25cef0c6b76a2dfe117 \
+            gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 \
             build_tools/cmake/build_e2e_test_artifacts.sh \
             "${BUILD_E2E_TEST_ARTIFACTS_DIR}"
           echo "::set-output name=e2e-test-artifacts-dir::${BUILD_E2E_TEST_ARTIFACTS_DIR}/e2e_test_artifacts"
@@ -659,22 +676,22 @@
           - platform: android
             architecture: arm64-v8a
             abi: arm64-v8a
-            docker_image: "gcr.io/iree-oss/android@sha256:740f87664b2f4d9288549c95d4d355d5b4a543d55257dcff0308b0c2920bd229"
+            docker_image: "gcr.io/iree-oss/android@sha256:e5f45760ab2dc8a1d72dc6e9d90056719bedac5678ee90e6d9f9b958172b4834"
             # No test_script
           - platform: riscv
             architecture: rv64
             abi: lp64d
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
             test_script: "./build_tools/cmake/test_riscv.sh"
           - platform: riscv
             architecture: rv32-linux
             abi: ilp32d
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
             test_script: "./build_tools/cmake/test_riscv.sh"
           - platform: riscv
             architecture: rv32-baremetal
             abi: ilp32
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
             test_script: "./tests/riscv32/smoke.sh"
     env:
       PLATFORM: ${{ matrix.target.platform }}
@@ -686,6 +703,7 @@
       HOST_BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
       HOST_BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
       HOST_BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
+      IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
     steps:
       - name: "Checking out repository"
         uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
@@ -698,6 +716,9 @@
       - name: "Build cross-compiling target"
         run: |
           ./build_tools/github_actions/docker_run.sh \
+            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
+            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
+            --env "CCACHE_NAMESPACE=${DOCKER_IMAGE}" \
             --env "${PLATFORM^^}_ARCH=${ARCHITECTURE}" \
             --env "${PLATFORM^^}_ABI=${ABI}" \
             --env "BUILD_PRESET=test" \
@@ -741,7 +762,7 @@
         run: |
           build_tools/github_actions/docker_run.sh \
             --env "IREE_HOST_BINARY_ROOT=${HOST_BUILD_DIR}/install" \
-            gcr.io/iree-oss/emscripten@sha256:6e75298c786a5a7d0a076578a210527bcc753d0a228c635ad3b75efd5421632a \
+            gcr.io/iree-oss/emscripten@sha256:20847a0a9503852d9594f3c4e7e633b913b450240b097d378dcd42d22ac0948e \
             build_tools/cmake/build_runtime_emscripten.sh
 
   # TODO(#10391): Update the benchmark dependency with the new benchmark
@@ -761,15 +782,15 @@
         target:
           - platform: riscv
             architecture: rv64
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
             run_scripts: "./build_tools/cmake/build_riscv.sh && ./build_tools/cmake/test_riscv.sh"
           - platform: riscv
             architecture: rv32-linux
-            docker_image: "gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451"
+            docker_image: "gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04"
             run_scripts: "./build_tools/cmake/build_riscv.sh && ./build_tools/cmake/test_riscv.sh"
           - platform: host
             architecture: x86_64
-            docker_image: "gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4"
+            docker_image: "gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206"
             run_scripts: "./build_tools/cmake/test_benchmark_suites_host.sh"
     env:
       PLATFORM: ${{ matrix.target.platform }}

diff --git a/build_tools/bazel/build_core.sh b/build_tools/bazel/build_core.sh
index d5c4798..85f0e60 100755
--- a/build_tools/bazel/build_core.sh
+++ b/build_tools/bazel/build_core.sh

@@ -23,12 +23,12 @@
 set -xeuo pipefail
 
 
-IREE_BAZEL_READ_REMOTE_CACHE="${IREE_BAZEL_READ_REMOTE_CACHE:-1}"
-IREE_BAZEL_WRITE_REMOTE_CACHE="${IREE_BAZEL_WRITE_REMOTE_CACHE:-0}"
+IREE_READ_REMOTE_BAZEL_CACHE="${IREE_READ_REMOTE_BAZEL_CACHE:-1}"
+IREE_WRITE_REMOTE_BAZEL_CACHE="${IREE_WRITE_REMOTE_BAZEL_CACHE:-0}"
 BAZEL_BIN="${BAZEL_BIN:-$(which bazel)}"
 
-if (( ${IREE_BAZEL_WRITE_REMOTE_CACHE} == 1 && ${IREE_BAZEL_READ_REMOTE_CACHE} != 1 )); then
-  echo "Can't have 'IREE_BAZEL_WRITE_REMOTE_CACHE' (${IREE_BAZEL_WRITE_REMOTE_CACHE}) set without 'IREE_BAZEL_READ_REMOTE_CACHE' (${IREE_BAZEL_READ_REMOTE_CACHE})"
+if (( ${IREE_WRITE_REMOTE_BAZEL_CACHE} == 1 && ${IREE_READ_REMOTE_BAZEL_CACHE} != 1 )); then
+  echo "Can't have 'IREE_WRITE_REMOTE_BAZEL_CACHE' (${IREE_WRITE_REMOTE_BAZEL_CACHE}) set without 'IREE_READ_REMOTE_BAZEL_CACHE' (${IREE_READ_REMOTE_BAZEL_CACHE})"
 fi
 
 # Use user-environment variables if set, otherwise use CI-friendly defaults.
@@ -101,11 +101,11 @@
   BAZEL_TEST_CMD+=(--sandbox_base="${SANDBOX_BASE}")
 fi
 
-if (( IREE_BAZEL_READ_REMOTE_CACHE == 1 )); then
+if (( IREE_READ_REMOTE_BAZEL_CACHE == 1 )); then
   BAZEL_TEST_CMD+=(--config=remote_cache_bazel_ci)
 fi
 
-if (( IREE_BAZEL_WRITE_REMOTE_CACHE != 1 )); then
+if (( IREE_WRITE_REMOTE_BAZEL_CACHE != 1 )); then
   BAZEL_TEST_CMD+=(--noremote_upload_local_results)
 fi
 

diff --git a/build_tools/bazel/iree.bazelrc b/build_tools/bazel/iree.bazelrc
index 1de93d0..b95b039 100644
--- a/build_tools/bazel/iree.bazelrc
+++ b/build_tools/bazel/iree.bazelrc

@@ -271,8 +271,8 @@
 # specific docker container the CI Bazel builds are run in. The image URL is
 # included for clarity and so that this reference is automatically updated by
 # manage_images.py
-build:remote_cache_bazel_ci --host_platform_remote_properties_override='properties:{name:"cache-silo-key" value:"gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:50301f9ccc0bf63c4d498c225444cc5585218612d15b83d06e4f4672ab2dfc03"}'
-build:remote_cache_bazel_tf_ci --host_platform_remote_properties_override='properties:{name:"cache-silo-key" value:"gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd"}'
+build:remote_cache_bazel_ci --host_platform_remote_properties_override='properties:{name:"cache-silo-key" value:"gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:c22afc61198e14a98e06e5261149de74c629acd2bc61b82e57ec90e5461b69be"}'
+build:remote_cache_bazel_tf_ci --host_platform_remote_properties_override='properties:{name:"cache-silo-key" value:"gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d"}'
 
 
 ###############################################################################

diff --git a/build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml b/build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml
index 5b58b65..590393e 100644
--- a/build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml
+++ b/build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml

@@ -9,7 +9,7 @@
 steps:
   - label: "Build"
     commands:
-      - "docker run --user=$(id -u):$(id -g) --volume=\\${HOME?}:\\${HOME?} --volume=/etc/passwd:/etc/passwd:ro --volume=/etc/group:/etc/group:ro --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5 build_tools/cmake/build_android_benchmark.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\${HOME?}:\\${HOME?} --volume=/etc/passwd:/etc/passwd:ro --volume=/etc/group:/etc/group:ro --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030 build_tools/cmake/build_android_benchmark.sh"
       - "tar --exclude='*.tar.gz' --exclude='*.tgz' --exclude='*.mlir' --exclude='*.tflite' --exclude='*tf-model' -czvf benchmark-suites-${BUILDKITE_BUILD_NUMBER}.tgz build-host/benchmark_suites"
       - "find build-host/benchmark_suites -name '*.mlir' | tar -czvf source-mlir-models-${BUILDKITE_BUILD_NUMBER}.tgz -T -"
       - "tar -czvf iree-android-tools-${BUILDKITE_BUILD_NUMBER}.tgz build-android/tools/iree-benchmark-module build-android-trace/tools/iree-benchmark-module build-android/tools/build_config.txt"

diff --git a/build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml b/build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml
index c5832e0..4a1929e 100644
--- a/build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml
+++ b/build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml

@@ -8,7 +8,7 @@
   - label: "build"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/android@sha256:740f87664b2f4d9288549c95d4d355d5b4a543d55257dcff0308b0c2920bd229 build_tools/cmake/build_host_and_android.sh arm64-v8a"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/android@sha256:e5f45760ab2dc8a1d72dc6e9d90056719bedac5678ee90e6d9f9b958172b4834 build_tools/cmake/build_host_and_android.sh arm64-v8a"
       - "tar --exclude='*.o' --exclude='*.a' -czvf build-artifacts.tgz build-android"
     agents:
       - "queue=build"

diff --git a/build_tools/buildkite/cmake/build_configurations.yml b/build_tools/buildkite/cmake/build_configurations.yml
index 4e1a113..9e39b5e 100644
--- a/build_tools/buildkite/cmake/build_configurations.yml
+++ b/build_tools/buildkite/cmake/build_configurations.yml

@@ -8,7 +8,7 @@
   - label: ":zap: Build with tracing enabled"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 ./build_tools/cmake/build_tracing.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 ./build_tools/cmake/build_tracing.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:
@@ -17,7 +17,7 @@
   - label: ":hammer_and_wrench: Build the runtime only"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 ./build_tools/cmake/build_runtime.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 ./build_tools/cmake/build_runtime.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:
@@ -26,7 +26,7 @@
   - label: ":pinching_hand: Build the size-optimized runtime only"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 ./build_tools/cmake/build_runtime_small.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 ./build_tools/cmake/build_runtime_small.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:
@@ -36,7 +36,7 @@
     key: "build-gcc"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --env CC=/usr/bin/gcc-9 --env CXX=/usr/bin/g++-9 --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 ./build_tools/cmake/clean_build.sh"
+      - "docker run --env CC=/usr/bin/gcc-9 --env CXX=/usr/bin/g++-9 --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 ./build_tools/cmake/clean_build.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:

diff --git a/build_tools/buildkite/cmake/linux/pipeline.yml b/build_tools/buildkite/cmake/linux/pipeline.yml
index 791412a..f95efdb 100644
--- a/build_tools/buildkite/cmake/linux/pipeline.yml
+++ b/build_tools/buildkite/cmake/linux/pipeline.yml

@@ -5,7 +5,7 @@
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 env:
-  DOCKER_IMAGE: "gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5"
+  DOCKER_IMAGE: "gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030"
   IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
 
 steps:

diff --git a/build_tools/buildkite/cmake/linux/x86_64/benchmark.yml b/build_tools/buildkite/cmake/linux/x86_64/benchmark.yml
index d5af609..166f19f 100644
--- a/build_tools/buildkite/cmake/linux/x86_64/benchmark.yml
+++ b/build_tools/buildkite/cmake/linux/x86_64/benchmark.yml

@@ -15,7 +15,7 @@
         --volume="$$PWD:$$IREE_DOCKER_WORKDIR" \
         --workdir="$$IREE_DOCKER_WORKDIR" \
         --rm \
-        gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5 \
+        gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030 \
         build_tools/cmake/build_linux_benchmark.sh
       tar --exclude="*.tar.gz" \
         --exclude="*.tgz" \

diff --git a/build_tools/buildkite/samples.yml b/build_tools/buildkite/samples.yml
index b0850ad..d006501 100644
--- a/build_tools/buildkite/samples.yml
+++ b/build_tools/buildkite/samples.yml

@@ -7,7 +7,7 @@
 steps:
   - label: "Test Colab notebooks"
     commands:
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/samples@sha256:946c51d6ccc22656540ed1a73eed10f32591f9455fc190cdd076eade3a848422 python3 ./samples/colab/test_notebooks.py"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/samples@sha256:291d87d125b41e1539dbb96c7c4f734cc98fc158a36d18496cda360658f00bde python3 ./samples/colab/test_notebooks.py"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:
@@ -15,7 +15,7 @@
 
   - label: "Test Samples"
     commands:
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/samples@sha256:946c51d6ccc22656540ed1a73eed10f32591f9455fc190cdd076eade3a848422 ./build_tools/testing/test_samples.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/samples@sha256:291d87d125b41e1539dbb96c7c4f734cc98fc158a36d18496cda360658f00bde ./build_tools/testing/test_samples.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:
@@ -29,7 +29,7 @@
     key: "build-host-install"
     commands:
       - "git submodule sync && git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 ./build_tools/cmake/build_host_tools.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 ./build_tools/cmake/build_host_tools.sh"
       - "tar -czvf build-artifacts.tgz build-host/install"
     artifact_paths: "build-artifacts.tgz"
     env:
@@ -43,7 +43,7 @@
       - "buildkite-agent artifact download --step build-host-install build-artifacts.tgz ./"
       - "tar xzf build-artifacts.tgz"
       - "git submodule update --init --jobs 8 --depth 1"
-      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/emscripten@sha256:6e75298c786a5a7d0a076578a210527bcc753d0a228c635ad3b75efd5421632a ./experimental/web/test_samples.sh"
+      - "docker run --user=$(id -u):$(id -g) --volume=\\$PWD:\\$IREE_DOCKER_WORKDIR --workdir=\\$IREE_DOCKER_WORKDIR --rm gcr.io/iree-oss/emscripten@sha256:20847a0a9503852d9594f3c4e7e633b913b450240b097d378dcd42d22ac0948e ./experimental/web/test_samples.sh"
     env:
       IREE_DOCKER_WORKDIR: "/usr/src/github/iree"
     agents:

diff --git a/build_tools/cmake/build_all.sh b/build_tools/cmake/build_all.sh
index 9501100..bb51ae6 100755
--- a/build_tools/cmake/build_all.sh
+++ b/build_tools/cmake/build_all.sh

@@ -21,6 +21,7 @@
 IREE_PYTHON3_EXECUTABLE="${IREE_PYTHON3_EXECUTABLE:-$(which python3)}"
 
 source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 declare -a CMAKE_ARGS=(
   "-G" "Ninja"
@@ -64,3 +65,7 @@
 echo "Building test deps"
 echo "------------------"
 "$CMAKE_BIN" --build "${BUILD_DIR}" --target iree-test-deps -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_and_test_asan.sh b/build_tools/cmake/build_and_test_asan.sh
index 73118fe..d9b60f6 100755
--- a/build_tools/cmake/build_and_test_asan.sh
+++ b/build_tools/cmake/build_and_test_asan.sh

@@ -23,7 +23,7 @@
 IREE_ENABLE_ASSERTIONS="${IREE_ENABLE_ASSERTIONS:-ON}"
 
 source build_tools/cmake/setup_build.sh
-
+source build_tools/cmake/setup_ccache.sh
 
 CMAKE_ARGS=(
   "-G" "Ninja"
@@ -58,6 +58,10 @@
 echo "------------------"
 "${CMAKE_BIN?}" --build "${BUILD_DIR?}" --target iree-microbenchmark-suites -- -k 0
 
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi
+
 # Respect the user setting, but default to as many jobs as we have cores.
 export CTEST_PARALLEL_LEVEL=${CTEST_PARALLEL_LEVEL:-$(nproc)}
 

diff --git a/build_tools/cmake/build_and_test_tsan.sh b/build_tools/cmake/build_and_test_tsan.sh
index c251f46..1517471 100755
--- a/build_tools/cmake/build_and_test_tsan.sh
+++ b/build_tools/cmake/build_and_test_tsan.sh

@@ -20,6 +20,7 @@
 BUILD_DIR="${1:-${IREE_TSAN_BUILD_DIR:-build-tsan}}"
 
 source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 CMAKE_ARGS=(
   "-G" "Ninja"
@@ -65,6 +66,10 @@
 echo "------------------"
 "$CMAKE_BIN" --build "${BUILD_DIR}" --target iree-test-deps -- -k 0
 
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi
+
 # Disable actually running GPU tests. This tends to yield TSan reports that are
 # specific to one's particular GPU driver and therefore hard to reproduce across
 # machines and often un-actionable anyway.

diff --git a/build_tools/cmake/build_android.sh b/build_tools/cmake/build_android.sh
index b9bbc4d..ded43aa 100755
--- a/build_tools/cmake/build_android.sh
+++ b/build_tools/cmake/build_android.sh

@@ -29,6 +29,7 @@
 BUILD_PRESET="${BUILD_PRESET:-test}"
 
 source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 declare -a args=(
   -G Ninja
@@ -85,3 +86,7 @@
 echo "Building test deps for device"
 echo "------------------"
 "${CMAKE_BIN}" --build "${BUILD_DIR}" --target iree-test-deps -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_host_tools.sh b/build_tools/cmake/build_host_tools.sh
index fe42483..d1f4bb5 100755
--- a/build_tools/cmake/build_host_tools.sh
+++ b/build_tools/cmake/build_host_tools.sh

@@ -21,6 +21,7 @@
 IREE_ENABLE_ASSERTIONS="${IREE_ENABLE_ASSERTIONS:-OFF}"
 
 source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 mkdir -p "${INSTALL_DIR}"
 
@@ -48,3 +49,7 @@
 
 "${CMAKE_BIN}" "${CMAKE_ARGS[@]}"
 "${CMAKE_BIN}" --build "${BUILD_DIR}" --target install -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_runtime.sh b/build_tools/cmake/build_runtime.sh
index a67d296..9899a34 100755
--- a/build_tools/cmake/build_runtime.sh
+++ b/build_tools/cmake/build_runtime.sh

@@ -18,8 +18,13 @@
 BUILD_DIR="${1:-${IREE_RUNTIME_BUILD_DIR:-build-runtime}}"
 
 source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 "${CMAKE_BIN}" -B "${BUILD_DIR}" -G Ninja . \
   -DCMAKE_BUILD_TYPE=RelWithDebInfo \
   -DIREE_BUILD_COMPILER=OFF
 "${CMAKE_BIN}" --build "${BUILD_DIR}" -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_runtime_emscripten.sh b/build_tools/cmake/build_runtime_emscripten.sh
index 0231785..1433a22 100755
--- a/build_tools/cmake/build_runtime_emscripten.sh
+++ b/build_tools/cmake/build_runtime_emscripten.sh

@@ -21,12 +21,11 @@
     exit 1
 fi
 
-ROOT_DIR=$(git rev-parse --show-toplevel)
 BUILD_DIR="build-emscripten"
 IREE_HOST_BINARY_ROOT="$(realpath ${IREE_HOST_BINARY_ROOT})"
 
-cd "${ROOT_DIR}"
-source "${ROOT_DIR}/build_tools/cmake/setup_build.sh"
+source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 cd "${BUILD_DIR}"
 
@@ -48,3 +47,7 @@
 echo "Building test deps"
 echo "------------------"
 "${CMAKE_BIN?}" --build . --target iree-test-deps -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_runtime_small.sh b/build_tools/cmake/build_runtime_small.sh
index a358839..e59df0c 100755
--- a/build_tools/cmake/build_runtime_small.sh
+++ b/build_tools/cmake/build_runtime_small.sh

@@ -11,11 +11,10 @@
 
 set -xeuo pipefail
 
-ROOT_DIR=$(git rev-parse --show-toplevel)
 BUILD_DIR="${1:-${IREE_RUNTIME_SMALL_BUILD_DIR:-build-runtime-small}}"
 
-cd "${ROOT_DIR}"
-source "${ROOT_DIR}/build_tools/cmake/setup_build.sh"
+source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 "${CMAKE_BIN?}" -B "${BUILD_DIR}" \
   -G Ninja . \
@@ -23,3 +22,7 @@
   -DIREE_SIZE_OPTIMIZED=ON \
   -DIREE_BUILD_COMPILER=OFF
 "${CMAKE_BIN?}" --build "${BUILD_DIR}" -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/build_tf_binaries.sh b/build_tools/cmake/build_tf_binaries.sh
index 7f31499..9175f57 100755
--- a/build_tools/cmake/build_tf_binaries.sh
+++ b/build_tools/cmake/build_tf_binaries.sh

@@ -15,13 +15,13 @@
 
 ROOT_DIR="${ROOT_DIR:-$(git rev-parse --show-toplevel)}"
 IREE_USE_WORKSPACE_RC="${IREE_USE_WORKSPACE_RC:-0}"
-IREE_BAZEL_READ_REMOTE_CACHE="${IREE_BAZEL_READ_REMOTE_CACHE:-1}"
-IREE_BAZEL_WRITE_REMOTE_CACHE="${IREE_BAZEL_WRITE_REMOTE_CACHE:-0}"
+IREE_READ_REMOTE_BAZEL_CACHE="${IREE_READ_REMOTE_BAZEL_CACHE:-1}"
+IREE_WRITE_REMOTE_BAZEL_CACHE="${IREE_WRITE_REMOTE_BAZEL_CACHE:-0}"
 IREE_TF_BINARIES_OUTPUT_DIR="${IREE_TF_BINARIES_OUTPUT_DIR:-}"
 INTEGRATIONS_DIR="${ROOT_DIR}/integrations/tensorflow"
 
-if (( ${IREE_BAZEL_WRITE_REMOTE_CACHE} == 1 && ${IREE_BAZEL_READ_REMOTE_CACHE} != 1 )); then
-  echo "Can't have 'IREE_BAZEL_WRITE_REMOTE_CACHE' (${IREE_BAZEL_WRITE_REMOTE_CACHE}) set without 'IREE_BAZEL_READ_REMOTE_CACHE' (${IREE_BAZEL_READ_REMOTE_CACHE})"
+if (( ${IREE_WRITE_REMOTE_BAZEL_CACHE} == 1 && ${IREE_READ_REMOTE_BAZEL_CACHE} != 1 )); then
+  echo "Can't have 'IREE_WRITE_REMOTE_BAZEL_CACHE' (${IREE_WRITE_REMOTE_BAZEL_CACHE}) set without 'IREE_READ_REMOTE_BAZEL_CACHE' (${IREE_READ_REMOTE_BAZEL_CACHE})"
 fi
 
 # We want to get back to wherever we were called from and output to the output
@@ -43,11 +43,11 @@
 
 BAZEL_TEST_CMD=("${BAZEL_STARTUP_CMD[@]}" test)
 
-if (( IREE_BAZEL_READ_REMOTE_CACHE == 1 )); then
+if (( IREE_READ_REMOTE_BAZEL_CACHE == 1 )); then
   BAZEL_TEST_CMD+=(--config=remote_cache_bazel_tf_ci)
 fi
 
-if (( IREE_BAZEL_WRITE_REMOTE_CACHE != 1 )); then
+if (( IREE_WRITE_REMOTE_BAZEL_CACHE != 1 )); then
   BAZEL_TEST_CMD+=(--noremote_upload_local_results)
 fi
 

diff --git a/build_tools/cmake/build_tracing.sh b/build_tools/cmake/build_tracing.sh
index 219563a..90f2861 100755
--- a/build_tools/cmake/build_tracing.sh
+++ b/build_tools/cmake/build_tracing.sh

@@ -12,11 +12,10 @@
 set -e
 set -x
 
-ROOT_DIR=$(git rev-parse --show-toplevel)
 BUILD_DIR="${1:-${IREE_TRACING_BUILD_DIR:-build-tracing}}"
 
-cd "${ROOT_DIR}"
-source "${ROOT_DIR}/build_tools/cmake/setup_build.sh"
+source build_tools/cmake/setup_build.sh
+source build_tools/cmake/setup_ccache.sh
 
 # Note: https://github.com/iree-org/iree/issues/6404 prevents us from building
 # tests with these other settings. Many tests invoke the compiler tools with
@@ -27,3 +26,7 @@
   -DIREE_ENABLE_RUNTIME_TRACING=ON \
   -DIREE_BUILD_COMPILER=OFF
 "${CMAKE_BIN?}" --build "${BUILD_DIR}" -- -k 0
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  ccache --show-stats
+fi

diff --git a/build_tools/cmake/setup_ccache.sh b/build_tools/cmake/setup_ccache.sh
new file mode 100644
index 0000000..558ce74
--- /dev/null
+++ b/build_tools/cmake/setup_ccache.sh

@@ -0,0 +1,30 @@
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# This file must be *sourced* not executed.
+
+set -euo pipefail
+
+IREE_READ_REMOTE_CCACHE="${IREE_READ_REMOTE_CCACHE:-1}"
+IREE_WRITE_REMOTE_CCACHE="${IREE_WRITE_REMOTE_CCACHE:-0}"
+if (( ${IREE_WRITE_REMOTE_CCACHE} == 1 && ${IREE_READ_REMOTE_CCACHE} != 1 )); then
+  echo "Can't have 'IREE_WRITE_REMOTE_CCACHE' (${IREE_WRITE_REMOTE_CCACHE})" \
+       " set without 'IREE_READ_REMOTE_CCACHE' (${IREE_READ_REMOTE_CCACHE})"
+fi
+
+if (( IREE_READ_REMOTE_CCACHE == 1 )); then
+  export CCACHE_REMOTE_STORAGE="http://storage.googleapis.com/iree-sccache/ccache"
+  export CCACHE_REMOTE_ONLY=1
+  export CMAKE_C_COMPILER_LAUNCHER=ccache
+  export CMAKE_CXX_COMPILER_LAUNCHER=ccache
+fi
+if (( IREE_WRITE_REMOTE_CCACHE == 1 )); then
+  set +x # Don't leak the token (even though it's short-lived)
+  export CCACHE_REMOTE_STORAGE="${CCACHE_REMOTE_STORAGE}|bearer-token=${IREE_CCACHE_GCP_TOKEN}"
+  set -x
+else
+  export CCACHE_REMOTE_STORAGE="${CCACHE_REMOTE_STORAGE}|read-only"
+fi

diff --git a/build_tools/docker/context/install_ccache.sh b/build_tools/docker/context/install_ccache.sh
new file mode 100755
index 0000000..cc4102c
--- /dev/null
+++ b/build_tools/docker/context/install_ccache.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+CCACHE_VERSION=4.7.4
+
+curl --silent --show-error --fail --location \
+    "https://github.com/ccache/ccache/releases/download/v${CCACHE_VERSION}/ccache-${CCACHE_VERSION}-linux-x86_64.tar.xz" \
+    --output ccache.tar.xz \
+    "https://github.com/ccache/ccache/releases/download/v${CCACHE_VERSION}/ccache-${CCACHE_VERSION}-linux-x86_64.tar.xz.asc" \
+    --output ccache.tar.xz.asc \
+    "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x5a939a71a46792cf57866a51996dda075594adb8" \
+    --output JOEL_PUBKEY.asc
+
+gpg --import JOEL_PUBKEY.asc
+gpg --verify ccache.tar.xz.asc
+tar -xvf ccache.tar.xz --strip-components=1
+cp ccache /usr/bin/

diff --git a/build_tools/docker/docker_run.sh b/build_tools/docker/docker_run.sh
index 8ef2db9..df6f4b9 100755
--- a/build_tools/docker/docker_run.sh
+++ b/build_tools/docker/docker_run.sh

@@ -24,6 +24,7 @@
     DOCKER_RUN_ARGS=(
       --mount="type=bind,source=${DOCKER_HOST_WORKDIR},dst=${DOCKER_CONTAINER_WORKDIR}"
       --workdir="${DOCKER_CONTAINER_WORKDIR}"
+      --env "CCACHE_BASE_DIR=${DOCKER_CONTAINER_WORKDIR}"
     )
 
     # Delete the container after the run is complete.
@@ -33,6 +34,9 @@
     # Run as the current user and group. If only it were this simple...
     DOCKER_RUN_ARGS+=(--user="$(id -u):$(id -g)")
 
+    # Use the host network stack. Improves network performance and makes it
+    # possible for the container to talk to localhost.
+    DOCKER_RUN_ARGS+=(--network="host")
 
     # The Docker container doesn't know about the users and groups of the host
     # system. We have to tell it. This is just a mapping of IDs to names though.
@@ -97,7 +101,9 @@
     fi
 
     # Give the container a ramdisk and set the Bazel sandbox base to point to
-    # it. This helps a lot with Bazel getting IO bound.
+    # it. This helps a lot with Bazel getting IO bound. Note that SANDBOX_BASE
+    # is a custom environment variable we translate into the corresponding Bazel
+    # option.
     DOCKER_RUN_ARGS+=(
       --mount="type=tmpfs,dst=/dev/shm"
       --env SANDBOX_BASE=/dev/shm

diff --git a/build_tools/docker/dockerfiles/android.Dockerfile b/build_tools/docker/dockerfiles/android.Dockerfile
index 95d9e16..77fa81d 100644
--- a/build_tools/docker/dockerfiles/android.Dockerfile
+++ b/build_tools/docker/dockerfiles/android.Dockerfile

@@ -6,7 +6,7 @@
 
 # An image for cross-compiling IREE towards Android.
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206
 ARG NDK_VERSION=r25b
 WORKDIR /install-ndk
 

diff --git a/build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile b/build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile
index 4f2aa75..82a53a5 100644
--- a/build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile
+++ b/build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile

@@ -44,6 +44,13 @@
   && ./install_iree_deps.sh "${LLVM_VERSION}" \
   && rm -rf /install-basics
 
+######## ccache ########
+
+WORKDIR /install-ccache
+
+COPY build_tools/docker/context/install_ccache.sh ./
+RUN ./install_ccache.sh && rm -rf /install-ccache
+
 ######## CMake ########
 WORKDIR /install-cmake
 

diff --git a/build_tools/docker/dockerfiles/base.Dockerfile b/build_tools/docker/dockerfiles/base.Dockerfile
index 4c4d76a..3f8d1bf 100644
--- a/build_tools/docker/dockerfiles/base.Dockerfile
+++ b/build_tools/docker/dockerfiles/base.Dockerfile

@@ -38,6 +38,13 @@
 RUN ./install_iree_deps.sh "${LLVM_VERSION}" \
   && rm -rf /install-basics
 
+######## ccache ########
+
+WORKDIR /install-ccache
+
+COPY build_tools/docker/context/install_ccache.sh ./
+RUN ./install_ccache.sh && rm -rf /install-ccache
+
 ######## CMake ########
 WORKDIR /install-cmake
 

diff --git a/build_tools/docker/dockerfiles/emscripten.Dockerfile b/build_tools/docker/dockerfiles/emscripten.Dockerfile
index 98114b1..7fb379e 100644
--- a/build_tools/docker/dockerfiles/emscripten.Dockerfile
+++ b/build_tools/docker/dockerfiles/emscripten.Dockerfile

@@ -6,7 +6,7 @@
 
 # An image for building IREE through Emscripten.
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206
 
 # See also
 #   * https://github.com/emscripten-core/emsdk/blob/main/docker/Dockerfile

diff --git a/build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile b/build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile
index 2da9c8e..443d88f 100644
--- a/build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile
+++ b/build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile

@@ -8,7 +8,7 @@
 # The NVidia drivers need to *exactly* match between the host machine and the
 # docker image.
 
-FROM gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5
+FROM gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030
 
 # We use .deb files that we host because we have to pin the version exactly to
 # match the host machine and packages routinely dissapear from the Ubuntu

diff --git a/build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile b/build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile
index b3cbfb4..df4b5aa 100644
--- a/build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile
+++ b/build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile

@@ -4,8 +4,8 @@
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-FROM gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5
-COPY --from=gcr.io/iree-oss/swiftshader@sha256:484720d57fa816280d9a7455d16e1f9f3d90cc3efdb744646016503ea5ab2b30 \
+FROM gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030
+COPY --from=gcr.io/iree-oss/swiftshader@sha256:27b843417fd548fab1b636d2b65d2b8b8c09f42a65cd5dde7ecfb9881e4332ac \
   /swiftshader /swiftshader
 
 # Set VK_ICD_FILENAMES so Vulkan loader can find the SwiftShader ICD.

diff --git a/build_tools/docker/dockerfiles/frontends.Dockerfile b/build_tools/docker/dockerfiles/frontends.Dockerfile
index ec82606..06e14cd 100644
--- a/build_tools/docker/dockerfiles/frontends.Dockerfile
+++ b/build_tools/docker/dockerfiles/frontends.Dockerfile

@@ -4,7 +4,7 @@
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-FROM gcr.io/iree-oss/android@sha256:740f87664b2f4d9288549c95d4d355d5b4a543d55257dcff0308b0c2920bd229
+FROM gcr.io/iree-oss/android@sha256:e5f45760ab2dc8a1d72dc6e9d90056719bedac5678ee90e6d9f9b958172b4834
 
 WORKDIR /pip-install
 

diff --git a/build_tools/docker/dockerfiles/gradle-android.Dockerfile b/build_tools/docker/dockerfiles/gradle-android.Dockerfile
index 49c772c..13112b4 100644
--- a/build_tools/docker/dockerfiles/gradle-android.Dockerfile
+++ b/build_tools/docker/dockerfiles/gradle-android.Dockerfile

@@ -7,7 +7,7 @@
 # An image for cross-compiling IREE's TFLite Java Bindings with Gradle and
 # CMake.
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206
 
 ### Java ###
 WORKDIR /install-jdk

diff --git a/build_tools/docker/dockerfiles/nvidia.Dockerfile b/build_tools/docker/dockerfiles/nvidia.Dockerfile
index 35407b5..fb6b9aa 100644
--- a/build_tools/docker/dockerfiles/nvidia.Dockerfile
+++ b/build_tools/docker/dockerfiles/nvidia.Dockerfile

@@ -16,7 +16,7 @@
 ARG NVIDIA_COMMON_DEB="libnvidia-common-460_460.39-0ubuntu0.18.04.1_all.deb"
 
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 AS fetch-nvidia
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 AS fetch-nvidia
 ARG NVIDIA_COMMON_DEB
 ARG NVIDIA_GL_DEB
 ARG NVIDIA_COMPUTE_DEB
@@ -36,7 +36,7 @@
 #      does not support Ubuntu 18.04.
 # This allows to share configuration with base CMake, but it also means we need
 # to MATCH the driver version between the host machine and the docker image.
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 AS final
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 AS final
 ARG NVIDIA_COMMON_DEB
 ARG NVIDIA_GL_DEB
 ARG NVIDIA_COMPUTE_DEB

diff --git a/build_tools/docker/dockerfiles/riscv.Dockerfile b/build_tools/docker/dockerfiles/riscv.Dockerfile
index 036c9ef..6172cee 100644
--- a/build_tools/docker/dockerfiles/riscv.Dockerfile
+++ b/build_tools/docker/dockerfiles/riscv.Dockerfile

@@ -6,7 +6,7 @@
 
 # An image for cross-compiling IREE towards RISCV using CMake.
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 AS install-riscv
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 AS install-riscv
 WORKDIR /install-riscv
 RUN wget --no-verbose "https://storage.googleapis.com/iree-shared-files/toolchain_iree_20220918.tar.gz"
 RUN tar -xf "toolchain_iree_20220918.tar.gz" -C /usr/src/
@@ -15,7 +15,7 @@
 RUN wget --no-verbose "https://storage.googleapis.com/iree-shared-files/qemu-riscv.tar.gz"
 RUN tar -xf "qemu-riscv.tar.gz" -C /usr/src/
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 AS final
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 AS final
 COPY --from=install-riscv "/usr/src/toolchain_iree" "/usr/src/toolchain_iree"
 COPY --from=install-riscv "/usr/src/toolchain_iree_rv32imf" "/usr/src/toolchain_iree_rv32imf"
 COPY --from=install-riscv "/usr/src/qemu-riscv" "/usr/src/qemu-riscv"

diff --git a/build_tools/docker/dockerfiles/samples.Dockerfile b/build_tools/docker/dockerfiles/samples.Dockerfile
index ba0363a..78c2a43 100644
--- a/build_tools/docker/dockerfiles/samples.Dockerfile
+++ b/build_tools/docker/dockerfiles/samples.Dockerfile

@@ -9,7 +9,7 @@
 #   * Vulkan (using SwiftShader)
 #   * Python (including `venv` and common pip packages needed for Colab)
 
-FROM gcr.io/iree-oss/swiftshader@sha256:484720d57fa816280d9a7455d16e1f9f3d90cc3efdb744646016503ea5ab2b30
+FROM gcr.io/iree-oss/swiftshader@sha256:27b843417fd548fab1b636d2b65d2b8b8c09f42a65cd5dde7ecfb9881e4332ac
 
 # Install additional packages often used in notebooks.
 # Installing these at the system level helps with caching, since venvs can

diff --git a/build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile b/build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile
index 0f4c26c..31d4b63 100644
--- a/build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile
+++ b/build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile

@@ -4,7 +4,7 @@
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-FROM gcr.io/iree-oss/base-bleeding-edge@sha256:99a5d6324b464bac19f78cd7fe50c8d5068dde9bf96af18686e91f1356ed586a AS install-swiftshader
+FROM gcr.io/iree-oss/base-bleeding-edge@sha256:479eefb76447c865cf58c5be7ca9fe33f48584b474a1da3dfaa125aad2510463 AS install-swiftshader
 WORKDIR /install-swiftshader
 
 COPY build_tools/third_party/swiftshader/build_vk_swiftshader.sh ./

diff --git a/build_tools/docker/dockerfiles/swiftshader.Dockerfile b/build_tools/docker/dockerfiles/swiftshader.Dockerfile
index 103cb9d..3cc2dd7 100644
--- a/build_tools/docker/dockerfiles/swiftshader.Dockerfile
+++ b/build_tools/docker/dockerfiles/swiftshader.Dockerfile

@@ -4,7 +4,7 @@
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-FROM gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4 AS install-swiftshader
+FROM gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206 AS install-swiftshader
 WORKDIR /install-swiftshader
 
 COPY build_tools/third_party/swiftshader/build_vk_swiftshader.sh ./

diff --git a/build_tools/docker/prod_digests.txt b/build_tools/docker/prod_digests.txt
index 6cdf1ff..f39ed10 100644
--- a/build_tools/docker/prod_digests.txt
+++ b/build_tools/docker/prod_digests.txt

@@ -1,16 +1,16 @@
-gcr.io/iree-oss/base@sha256:22c43975179265296e016d15eb6f65d18abd5f9d4d3a5fa5e478ca4862bb61c4
-gcr.io/iree-oss/swiftshader@sha256:484720d57fa816280d9a7455d16e1f9f3d90cc3efdb744646016503ea5ab2b30
-gcr.io/iree-oss/samples@sha256:946c51d6ccc22656540ed1a73eed10f32591f9455fc190cdd076eade3a848422
-gcr.io/iree-oss/frontends@sha256:4b8f97bccf8443f0314d25c2737a46c979d9acffd28ad52f7d0ce5b879449da5
-gcr.io/iree-oss/frontends-nvidia@sha256:f9256c8f752179c72fe4fafaeac5d31c3b63ec30c8c35a5ab2c5ae4d0db1a631
-gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd
-gcr.io/iree-oss/gradle-android@sha256:868038ec2c27b33a13ceee831fdb0cb0ac6e9035205305cc03c4e27b043228de
-gcr.io/iree-oss/riscv@sha256:e11710158fd09a04ae7accc275a38c95cd59f83fd15a0630a083eb5623807451
-gcr.io/iree-oss/nvidia@sha256:74867181ab29bc890775bf3e2da717225431eaa022422e2dc865dc04d37289a4
-gcr.io/iree-oss/emscripten@sha256:6e75298c786a5a7d0a076578a210527bcc753d0a228c635ad3b75efd5421632a
-gcr.io/iree-oss/android@sha256:740f87664b2f4d9288549c95d4d355d5b4a543d55257dcff0308b0c2920bd229
+gcr.io/iree-oss/base@sha256:605d86ccf4197e978a24867fabb7fc100334c926b067ee0518e46d0a4396e206
+gcr.io/iree-oss/swiftshader@sha256:27b843417fd548fab1b636d2b65d2b8b8c09f42a65cd5dde7ecfb9881e4332ac
+gcr.io/iree-oss/samples@sha256:291d87d125b41e1539dbb96c7c4f734cc98fc158a36d18496cda360658f00bde
+gcr.io/iree-oss/frontends@sha256:e70c8a13ce02fed19fd9407517d8f945070ecd2e2a97657d4d3117ac6d7e0030
+gcr.io/iree-oss/frontends-nvidia@sha256:3e3965b6b9e431be296900440e5bc66ed46a8f6d6fa80ead7c7a75866f3bd416
+gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d
+gcr.io/iree-oss/gradle-android@sha256:c4c3450ee634a3fc6b75fa73b24f0d97d8b295e0cf599aab63dd5f12a8746333
+gcr.io/iree-oss/riscv@sha256:5034a40068f14d91d5caf07d615f323561416caee8afda32ae622746a4dd8b04
+gcr.io/iree-oss/nvidia@sha256:b0751e9f2fcb104d9d3d56fab6c6e79405bdcd2e503e53f2bf4f2b66d13cd88b
+gcr.io/iree-oss/emscripten@sha256:20847a0a9503852d9594f3c4e7e633b913b450240b097d378dcd42d22ac0948e
+gcr.io/iree-oss/android@sha256:e5f45760ab2dc8a1d72dc6e9d90056719bedac5678ee90e6d9f9b958172b4834
 gcr.io/iree-oss/manylinux2014_x86_64-release@sha256:b3b096e4b96746c3ae4cc52e880000d91038e79441334a7cfce9914cbbec1312
 gcr.io/iree-oss/shark@sha256:c72ef54dcb6ec485e8a96b0dfc43307875f4c4c7619f7fdc60bf5220a5672259
-gcr.io/iree-oss/base-bleeding-edge@sha256:99a5d6324b464bac19f78cd7fe50c8d5068dde9bf96af18686e91f1356ed586a
-gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:50301f9ccc0bf63c4d498c225444cc5585218612d15b83d06e4f4672ab2dfc03
+gcr.io/iree-oss/base-bleeding-edge@sha256:479eefb76447c865cf58c5be7ca9fe33f48584b474a1da3dfaa125aad2510463
+gcr.io/iree-oss/swiftshader-bleeding-edge@sha256:c22afc61198e14a98e06e5261149de74c629acd2bc61b82e57ec90e5461b69be
 gcr.io/iree-oss/mmperf@sha256:7a7f66eb4ff1f0c36b01fb2b5c99c9c01814fc18754537f4140e86081439cd8c

diff --git a/build_tools/github_actions/runner/gcp/image_setup.sh b/build_tools/github_actions/runner/gcp/image_setup.sh
index c4cabbe..2062001 100644
--- a/build_tools/github_actions/runner/gcp/image_setup.sh
+++ b/build_tools/github_actions/runner/gcp/image_setup.sh

@@ -276,8 +276,8 @@
           bash -c "${script_dir}/check_cuda.sh && ${script_dir}/check_vulkan.sh"
     }
 
-    check_docker gcr.io/iree-oss/nvidia@sha256:74867181ab29bc890775bf3e2da717225431eaa022422e2dc865dc04d37289a4
-    check_docker gcr.io/iree-oss/frontends-nvidia@sha256:f9256c8f752179c72fe4fafaeac5d31c3b63ec30c8c35a5ab2c5ae4d0db1a631
+    check_docker gcr.io/iree-oss/nvidia@sha256:b0751e9f2fcb104d9d3d56fab6c6e79405bdcd2e503e53f2bf4f2b66d13cd88b
+    check_docker gcr.io/iree-oss/frontends-nvidia@sha256:3e3965b6b9e431be296900440e5bc66ed46a8f6d6fa80ead7c7a75866f3bd416
 
     # Remove the docker images we've fetched. We might want to pre-fetch Docker
     # images into the VM image, but that should be a separate decision.

diff --git a/build_tools/scripts/integrate/README.md b/build_tools/scripts/integrate/README.md
index 107695e..28b8bdc 100644
--- a/build_tools/scripts/integrate/README.md
+++ b/build_tools/scripts/integrate/README.md

@@ -358,8 +358,8 @@
 An example from a log:
 
 ```
-[18:30:23 UTC] docker run --volume=/tmpfs/src/github/iree:/tmpfs/src/github/iree --workdir=/tmpfs/src/github/iree --rm --user=1003:1004 --volume=/tmpfs/fake_etc/group:/etc/group:ro --volume=/tmpfs/fake_etc/passwd:/etc/passwd:ro --volume=/tmpfs/fake_home:/home/kbuilder --volume=/home/kbuilder/.config/gcloud:/home/kbuilder/.config/gcloud:ro gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/core/build.sh
-Unable to find image 'gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd' locally
+[18:30:23 UTC] docker run --volume=/tmpfs/src/github/iree:/tmpfs/src/github/iree --workdir=/tmpfs/src/github/iree --rm --user=1003:1004 --volume=/tmpfs/fake_etc/group:/etc/group:ro --volume=/tmpfs/fake_etc/passwd:/etc/passwd:ro --volume=/tmpfs/fake_home:/home/kbuilder --volume=/home/kbuilder/.config/gcloud:/home/kbuilder/.config/gcloud:ro gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/core/build.sh
+Unable to find image 'gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d' locally
 sha256:aeb8de9fb7af3913d385ec6b274320197d61aa7bc51a6e8bc0deba644da3e405: Pulling from iree-oss/frontends-swiftshader
 ```
 
@@ -367,7 +367,7 @@
 you have the enviroment as same as CI bot and requires less local setup.
 
 ```
-docker run --interactive --tty --rm --volume=$PWD:/src/iree --workdir=/src/iree gcr.io/iree-oss/frontends-swiftshader@sha256:1de463291cfb337089ae4c3beead3c382ea64e53502f57e35457944b6ae152cd
+docker run --interactive --tty --rm --volume=$PWD:/src/iree --workdir=/src/iree gcr.io/iree-oss/frontends-swiftshader@sha256:8dbbfe9d0b9d72e8af591f9390ef8865536cc67bd262285335b7a83b2218548d
 ```
 
 To repro failures in `iree/e2e/`: