| commit | 46e47e3458ec9a88cce75d7d3e49eea13eb0e06e | [log] [tgz] |
|---|---|---|
| author | Geoffrey Martin-Noble <gcmn@google.com> | Tue Nov 29 16:39:45 2022 -0800 |
| committer | GitHub <noreply@github.com> | Tue Nov 29 16:39:45 2022 -0800 |
| tree | 1977b4ea0da66a5099a9b9632cc04c1d9a01a4c9 | |
| parent | 30356ad7c51a945981a3207d3ef604a3f231c273 [diff] |
Use ccache in CI (#11311) This connects our CMake builds to a [ccache](https://ccache.dev/) hosted in a GCS bucket. `ccache` newly (ish) supports using remote storage for the cache! Currently it only supports Redis, FTP, and HTTP. HTTPS is *not* supported right now, but there are plans to add an HTTPS backend, as well as potentially a direct GCS backend (see https://github.com/ccache/ccache/issues/1214). I think this adds a little bit of overhead for the network requests, potentially increasing the time for building with a completely cold cache. An example `build_all` job with a completely cold cache took 13.2 minutes for the entire job, 10 minutes for just the build step, of which 6.1 minutes was spent in the actual `cmake --build` command (not including builds of the `install` or `iree-test-deps` targets, which don't involve building C++): https://github.com/iree-org/iree/actions/runs/3562697821/jobs/5984663663 Going through that commit's ancestors on the main branch, this looks like it's adding about 30±30 seconds to the build, using the statistical technique of "eyeballing". We get wins on the flip side though, where with a fully cached build, the times are 6.3m, 3.8m, 1.6m. The impact is even bigger with asan, where we see the same ~50% improvement on the already-slower build. Unfortunately, since ccache is a language-specific cache, we can't do the same trick with all the test artifacts. The lack of HTTPS support does present somewhat of a problem because GCP doesn't allow using unsecured HTTP for many API access scopes. I ran into trouble with this when trying to get things to work locally because the local gcloud credentials for a user account usually have very broad scope (see discussion in https://github.com/ccache/ccache/pull/1001). But it *does* work fine on our GCP VMs since those service accounts have much more limited permissions. Luckily, we don't actually want users writing to the cache, so this mostly just impacted me setting it up. I also tried [sccache](https://github.com/mozilla/sccache), which has a GCS backend, but configuring the backend locally was pretty janky (see https://github.com/mozilla/sccache/issues/144#issuecomment-1307997283). I ultimately went with ccache since it's the much more established project and it seems like there's quite a bit of design work going in to making it work well. ccache also supports two caching layers (indeed this is the standard setup), so devs could make use of the remote cache by setting a single config/env variable to point at it and continue using their local ccache as well. This will of course only work as long as their local machine is sufficiently similar to the docker containers or they choose to build within docker containers. Co-authored-by: Scott Todd <scotttodd@google.com>
IREE (Intermediate Representation Execution Environment, pronounced as “eerie”) is an MLIR-based end-to-end compiler and runtime that lowers Machine Learning (ML) models to a unified IR that scales up to meet the needs of the datacenter and down to satisfy the constraints and special considerations of mobile and edge deployments.
See our website for project details, user guides, and instructions on building from source.
IREE is still in its early phase. We have settled down on the overarching infrastructure and are actively improving various software components as well as project logistics. It is still quite far from ready for everyday use and is made available without any support at the moment. With that said, we welcome any kind of feedback on any communication channels!
See our website for more information.
IREE is licensed under the terms of the Apache 2.0 License with LLVM Exceptions. See LICENSE for more information.