)]}'
{
  "commit": "46e47e3458ec9a88cce75d7d3e49eea13eb0e06e",
  "tree": "1977b4ea0da66a5099a9b9632cc04c1d9a01a4c9",
  "parents": [
    "30356ad7c51a945981a3207d3ef604a3f231c273"
  ],
  "author": {
    "name": "Geoffrey Martin-Noble",
    "email": "gcmn@google.com",
    "time": "Tue Nov 29 16:39:45 2022 -0800"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Tue Nov 29 16:39:45 2022 -0800"
  },
  "message": "Use ccache in CI (#11311)\n\nThis connects our CMake builds to a [ccache](https://ccache.dev/)\r\nhosted in a GCS bucket. `ccache` newly (ish) supports using remote\r\nstorage for the cache! Currently it only supports Redis, FTP, and HTTP.\r\nHTTPS is *not* supported right now, but there are plans to add an HTTPS\r\nbackend, as well as potentially a direct GCS backend (see\r\nhttps://github.com/ccache/ccache/issues/1214).\r\n\r\nI think this adds a little bit of overhead for the network requests,\r\npotentially increasing the time for building with a completely cold \r\ncache. \r\n\r\nAn example `build_all` job with a completely cold cache took 13.2\r\nminutes for the entire job, 10 minutes for just the build step, of\r\nwhich 6.1 minutes was spent in the actual `cmake --build` command (not\r\nincluding builds of the `install` or `iree-test-deps` targets, which\r\ndon\u0027t involve building C++):\r\nhttps://github.com/iree-org/iree/actions/runs/3562697821/jobs/5984663663\r\n\r\nGoing through that commit\u0027s ancestors on the main branch, this looks\r\nlike it\u0027s adding about 30±30 seconds to the build, using the\r\nstatistical technique of \"eyeballing\".\r\n\r\nWe get wins on the flip side though, where with a fully cached build,\r\nthe times are 6.3m, 3.8m, 1.6m.\r\n\r\nThe impact is even bigger with asan, where we see the same ~50%\r\nimprovement on the already-slower build.\r\n\r\nUnfortunately, since ccache is a language-specific cache, we can\u0027t\r\ndo the same trick with all the test artifacts.\r\n\r\nThe lack of HTTPS support does present somewhat of a problem because\r\nGCP doesn\u0027t allow using unsecured HTTP for many API access scopes. I\r\nran into trouble with this when trying to get things to work locally\r\nbecause the local gcloud credentials for a user account usually have\r\nvery broad scope (see discussion in\r\nhttps://github.com/ccache/ccache/pull/1001). But it *does* work fine on\r\nour GCP VMs since those service accounts have much more limited\r\npermissions. Luckily, we don\u0027t actually want users writing to the\r\ncache, so this mostly just impacted me setting it up.\r\n\r\nI also tried\r\n[sccache](https://github.com/mozilla/sccache), which has a GCS backend,\r\nbut configuring the backend locally was pretty janky (see\r\nhttps://github.com/mozilla/sccache/issues/144#issuecomment-1307997283).\r\nI ultimately went with ccache since it\u0027s the much more established\r\nproject and it seems like there\u0027s quite a bit of design work going in\r\nto making it work well.\r\n\r\nccache also supports two caching layers (indeed this is the standard\r\nsetup), so devs could make use of the remote cache by setting a\r\nsingle config/env variable to point at it and continue using their\r\nlocal ccache as well. This will of course only work as long as their\r\nlocal machine is sufficiently similar to the docker containers or they\r\nchoose to build within docker containers.\r\n\r\nCo-authored-by: Scott Todd \u003cscotttodd@google.com\u003e",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "1ea751178881a6639d602423018a996f1c7e7b8e",
      "old_mode": 33188,
      "old_path": ".github/workflows/android_tflite_oneshot_build.yml",
      "new_id": "3152eafcbc9b679b372fac30257896ec5bbaac60",
      "new_mode": 33188,
      "new_path": ".github/workflows/android_tflite_oneshot_build.yml"
    },
    {
      "type": "modify",
      "old_id": "8f607f7647f13d547fd7fb4238cf0f56649ded6e",
      "old_mode": 33188,
      "old_path": ".github/workflows/benchmarks.yml",
      "new_id": "477a7631747e9d07291193e532448d25af26b4c9",
      "new_mode": 33188,
      "new_path": ".github/workflows/benchmarks.yml"
    },
    {
      "type": "modify",
      "old_id": "f13403a1f4e8d01829a9f7953d75549ae097b093",
      "old_mode": 33188,
      "old_path": ".github/workflows/ci.yml",
      "new_id": "52423d1aea8edafa42c5fc43a3e734ced0e477c7",
      "new_mode": 33188,
      "new_path": ".github/workflows/ci.yml"
    },
    {
      "type": "modify",
      "old_id": "d5c4798d2f9febf1117aecc8b870287e41d68be8",
      "old_mode": 33261,
      "old_path": "build_tools/bazel/build_core.sh",
      "new_id": "85f0e6052c0bb963cb2a382fd4bdeb474ed08b83",
      "new_mode": 33261,
      "new_path": "build_tools/bazel/build_core.sh"
    },
    {
      "type": "modify",
      "old_id": "1de93d0e1ce9bd230d2c157276e031f0caa39729",
      "old_mode": 33188,
      "old_path": "build_tools/bazel/iree.bazelrc",
      "new_id": "b95b039a65ce9692e54364ff23154223ababebfa",
      "new_mode": 33188,
      "new_path": "build_tools/bazel/iree.bazelrc"
    },
    {
      "type": "modify",
      "old_id": "5b58b65dbe6988ddf2050084968bb3a9b6f67acf",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml",
      "new_id": "590393e6874bca480436c558e573bfda9c5d72a1",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/cmake/android/arm64-v8a/benchmark2.yml"
    },
    {
      "type": "modify",
      "old_id": "c5832e03929ecf8313ee6023f8770e61aa828744",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml",
      "new_id": "4a1929e1e84cf50531f95b5e16b64461de77c479",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/cmake/android/arm64-v8a/pipeline.yml"
    },
    {
      "type": "modify",
      "old_id": "4e1a113b9034b29cf899628bb9ebde263cb7fff9",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/cmake/build_configurations.yml",
      "new_id": "9e39b5e529fef62fb747e98416b72f01d4a573d2",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/cmake/build_configurations.yml"
    },
    {
      "type": "modify",
      "old_id": "791412a2c0826ace36fd6b41653b8ee30ac2aa6b",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/cmake/linux/pipeline.yml",
      "new_id": "f95efdb1f927c52187004c206f52b9175eec0aaf",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/cmake/linux/pipeline.yml"
    },
    {
      "type": "modify",
      "old_id": "d5af6098013be67ec9f70fcc81200ecd36f05ba6",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/cmake/linux/x86_64/benchmark.yml",
      "new_id": "166f19f6aacf6cba9fe0eb51bb5b0ee8fda25a69",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/cmake/linux/x86_64/benchmark.yml"
    },
    {
      "type": "modify",
      "old_id": "b0850addd9f3d2b5972b2d784fb03a96f9f91f31",
      "old_mode": 33188,
      "old_path": "build_tools/buildkite/samples.yml",
      "new_id": "d006501fb38f54f377c37dfa94e242a14897db2d",
      "new_mode": 33188,
      "new_path": "build_tools/buildkite/samples.yml"
    },
    {
      "type": "modify",
      "old_id": "9501100f35f4d0f1eb17587143f21b9f9853deab",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_all.sh",
      "new_id": "bb51ae64518c1b17e92c2cf76450b8f555976109",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_all.sh"
    },
    {
      "type": "modify",
      "old_id": "73118fe7d282ba91242c2c8e374c092ac0a13f3c",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_and_test_asan.sh",
      "new_id": "d9b60f65ffadad4573470b1d308dd47140c398b0",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_and_test_asan.sh"
    },
    {
      "type": "modify",
      "old_id": "c251f466f1da85a8e057a9eabcd92c1faf173dbc",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_and_test_tsan.sh",
      "new_id": "15174716220bfad028af7ae11979461b78335044",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_and_test_tsan.sh"
    },
    {
      "type": "modify",
      "old_id": "b9bbc4d5a05c4e680be030b17bcaaaef0410bb1e",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_android.sh",
      "new_id": "ded43aa588ec1d40f0df63569b987348705f07db",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_android.sh"
    },
    {
      "type": "modify",
      "old_id": "fe424833659191190c0bf29ec662254e45c7fcf4",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_host_tools.sh",
      "new_id": "d1f4bb55b6214d0ce5dc2fcb5a030581a4029a7c",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_host_tools.sh"
    },
    {
      "type": "modify",
      "old_id": "a67d29660edc236f38164e098239aacae56e8e9c",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_runtime.sh",
      "new_id": "9899a34e624a945a1e14a4056038671a35da1e29",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_runtime.sh"
    },
    {
      "type": "modify",
      "old_id": "02317851aef4bd5332979d388dba5540605eeaad",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_runtime_emscripten.sh",
      "new_id": "1433a224087d9b18e9d1b8bea78518d598bdd6c1",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_runtime_emscripten.sh"
    },
    {
      "type": "modify",
      "old_id": "a358839f7c1da53e59e190e47e52129c3b3e5f06",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_runtime_small.sh",
      "new_id": "e59df0ce972e084a51dc79039f80a565576c8f73",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_runtime_small.sh"
    },
    {
      "type": "modify",
      "old_id": "7f31499c4e2501e96c0d37291784fc2f3917a74c",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_tf_binaries.sh",
      "new_id": "9175f57e4338ee9aa0b11d4935cc194de32b2934",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_tf_binaries.sh"
    },
    {
      "type": "modify",
      "old_id": "219563aa46d11867c392785e4ad96ef733a2ba68",
      "old_mode": 33261,
      "old_path": "build_tools/cmake/build_tracing.sh",
      "new_id": "90f2861f40885304c521b5eccf60318a68a4e7f2",
      "new_mode": 33261,
      "new_path": "build_tools/cmake/build_tracing.sh"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "558ce749438e8ccc0c9cc6ab0f22e4e280372a2d",
      "new_mode": 33188,
      "new_path": "build_tools/cmake/setup_ccache.sh"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "cc4102c973c420656cd6a462b8abc0b5dd2ba69a",
      "new_mode": 33261,
      "new_path": "build_tools/docker/context/install_ccache.sh"
    },
    {
      "type": "modify",
      "old_id": "8ef2db916364d642b41af920b18172e3e9ae2416",
      "old_mode": 33261,
      "old_path": "build_tools/docker/docker_run.sh",
      "new_id": "df6f4b997be941e2ed43c2a8079a88c1cc1c05c3",
      "new_mode": 33261,
      "new_path": "build_tools/docker/docker_run.sh"
    },
    {
      "type": "modify",
      "old_id": "95d9e16b2acd94ce2d7cde2448b7779cf56b172e",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/android.Dockerfile",
      "new_id": "77fa81d84312cbec1059bc6bef8238f48f0b4826",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/android.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "4f2aa75cc43bb1bd861cda99bc2cb1dcec27b3e7",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile",
      "new_id": "82a53a5527d84cd5f8be0a6ccbc1a5f28390b732",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/base-bleeding-edge.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "4c4d76adcda23eb8abd739b67e5d266729ec93c9",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/base.Dockerfile",
      "new_id": "3f8d1bfcede65bc7fd8a846be826c0ba69f6306f",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/base.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "98114b15e87d394bba9f88183cda29aaa852871d",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/emscripten.Dockerfile",
      "new_id": "7fb379e14de064bd9d13aadbfbe8ceaf8c0a8576",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/emscripten.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "2da9c8e20fbbc26ccc23c3456a1ccb59e1c570ba",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile",
      "new_id": "443d88f1e492823fae2e1001cf133c10cbef6778",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/frontends-nvidia.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "b3cbfb487a0b6d73be5dc0dc03b0ce7cbda23664",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile",
      "new_id": "df4b5aa6efdb7995d855a0c4df95e3c3eb991c56",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/frontends-swiftshader.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "ec826067e689f49dbd26b7dc642788274128d35c",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/frontends.Dockerfile",
      "new_id": "06e14cd24bc3cd793bb48e66fc457ed9dc3f4e4f",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/frontends.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "49c772c4d1aeb09d6aa64131f949a03ae8ef2fb4",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/gradle-android.Dockerfile",
      "new_id": "13112b4206f84c8d003786388bfb21998e388a01",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/gradle-android.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "35407b5834c0652f218685e169780649b2372132",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/nvidia.Dockerfile",
      "new_id": "fb6b9aa9859c2d0e9a17bbc707ad316eb1525bd0",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/nvidia.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "036c9ef58fa8e2d92a6d9fb2d32a0598d56aa171",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/riscv.Dockerfile",
      "new_id": "6172cee1c6a8184a0cfbfce6131b41f285b0c8be",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/riscv.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "ba0363aab5943b1918fa226991ac01749bf2792c",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/samples.Dockerfile",
      "new_id": "78c2a437d6f84f022f167044ca7bc71394a52ad6",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/samples.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "0f4c26c1894050821636dfd1003dcb02a0987bf9",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile",
      "new_id": "31d4b6381b75bd830dbf69dbe370b7a035ba1328",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/swiftshader-bleeding-edge.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "103cb9d82a2c0f847305169cbb12ed5cba87a1a7",
      "old_mode": 33188,
      "old_path": "build_tools/docker/dockerfiles/swiftshader.Dockerfile",
      "new_id": "3cc2dd7c5ee40f0cebcbf03a23c0e34195247989",
      "new_mode": 33188,
      "new_path": "build_tools/docker/dockerfiles/swiftshader.Dockerfile"
    },
    {
      "type": "modify",
      "old_id": "6cdf1ff1edb09af56d4fdda395189eaed30b6013",
      "old_mode": 33188,
      "old_path": "build_tools/docker/prod_digests.txt",
      "new_id": "f39ed10c15711c0c960fe5e21279d01065eb840e",
      "new_mode": 33188,
      "new_path": "build_tools/docker/prod_digests.txt"
    },
    {
      "type": "modify",
      "old_id": "c4cabbef856a2992f8d15dccb4810716ed06630c",
      "old_mode": 33188,
      "old_path": "build_tools/github_actions/runner/gcp/image_setup.sh",
      "new_id": "206200115c9cd3cbf7077e41d88e04b09364e716",
      "new_mode": 33188,
      "new_path": "build_tools/github_actions/runner/gcp/image_setup.sh"
    },
    {
      "type": "modify",
      "old_id": "107695e4687e91c51e719e2bd0de39f4b4693bcf",
      "old_mode": 33188,
      "old_path": "build_tools/scripts/integrate/README.md",
      "new_id": "28b8bdc5fd2a57d93c8cdd9ccb73db520a40232a",
      "new_mode": 33188,
      "new_path": "build_tools/scripts/integrate/README.md"
    }
  ]
}