Adds Dockerfiles for building with bazel and updates GitHub CI workflows
- Adds Dockerfiles that define images for our bazel and bazel with tensorflow builds.
- Adds a script for updating these images on the IREE-OSS Google container registry (GCR).
- These images are hosted as `gcr.io/iree-oss/bazel` and `gcr.io/iree-oss/bazel-tensorflow`
- Updates the GitHub CI workflows to use these images.
This change adds Dockerfiles specifying the dependencies of IREE's core bazel build and IREE's integrations/bindings builds. This allows us to run CI tests in GitHub Actions without having to install the dependencies for each commit to master.
We are hosting the Docker images that these files build on the IREE-OSS GCP project. Currently, this means that every time there is a change to one of the Docker images, someone who is a storage admin for IREE-OSS must run the `build_tools/docker/update_gcr_images.sh` script. It should be possible to automate this process whenever a change is made to these files on GitHub in a future PR.
This change was tested by creating another branch based off of this one, changing the workflows to run on push to that branch and then pushing that change. Links to each test:
- [Bazel Build - Bindings](https://github.com/phoenix-meadowlark/iree/actions/runs/100891735)
- [Bazel Build - Fallthrough](https://github.com/phoenix-meadowlark/iree/actions/runs/100891736)
- [Bazel Build - Integrations](https://github.com/phoenix-meadowlark/iree/actions/runs/100891738) (Failed due to the build being broken at the time of testing).
- [Bazel Build -Core](https://github.com/phoenix-meadowlark/iree/actions/runs/100891741)
Since the integrations build failed due to the state of the repo at the time it was run, I ran the test again on my machine on a commit with a working build as a stand in and it passed.
Closes https://github.com/google/iree/pull/1874
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/iree/pull/1874 from phoenix-meadowlark:docker-containers b280481b306a1a2ba76758437930a235ef6cf4f7
PiperOrigin-RevId: 311007783
diff --git a/.github/workflows/bazel_build_bindings.yml b/.github/workflows/bazel_build_bindings.yml
index 30129c9..30787f1 100644
--- a/.github/workflows/bazel_build_bindings.yml
+++ b/.github/workflows/bazel_build_bindings.yml
@@ -23,38 +23,13 @@
jobs:
linux:
runs-on: ubuntu-18.04
- env:
- CXX: clang++
- CC: clang
- PYTHON_BIN: /usr/bin/python3
- # Install the version of Bazel that is required by our .bazelversion file.
- BAZEL_VERSION: 2.1.0
+ container: gcr.io/iree-oss/bazel-tensorflow
steps:
- name: Printing environment
run: |
"$CXX" --version
"$CC" --version
"$PYTHON_BIN" --version
- - name: Installing bazel
- run: |
- # https://docs.bazel.build/versions/master/install-ubuntu.html
- sudo apt-get install unzip zip
- wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION?}/bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- chmod +x bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- ./bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh --user
- rm bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- echo "::add-path::${HOME?}/bin"
- - name: Installing dependencies
- run: |
- sudo apt-get update
- sudo apt-get install clang libsdl2-dev python3 python3-pip python3-setuptools
- # Update pip before using it so it picks up the latest versions of our dependencies.
- sudo python3 -m pip install --upgrade pip
- sudo python3 -m pip install numpy tf-nightly
- # Install python2 numpy. Temporary fix for issue #1737.
- sudo apt-get install -y python-pip
- python -m pip install --upgrade pip
- python -m pip install numpy
- name: Checking out repository
uses: actions/checkout@v2
- name: Initializing submodules
diff --git a/.github/workflows/bazel_build_core.yml b/.github/workflows/bazel_build_core.yml
index 24ee158..05d8dcc 100644
--- a/.github/workflows/bazel_build_core.yml
+++ b/.github/workflows/bazel_build_core.yml
@@ -23,28 +23,8 @@
jobs:
linux:
runs-on: ubuntu-18.04
- container: l.gcr.io/google/bazel:2.1.0
- env:
- CXX: clang++
- CC: clang
- PYTHON_BIN: /usr/bin/python3
+ container: gcr.io/iree-oss/bazel
steps:
- - name: Installing IREE dependencies
- run: |
- apt-get update
- apt-get install -y clang libsdl2-dev
- - name: Installing actions/checkout@v2 dependencies
- run: |
- # Update git to version >= 2.18 for actions/checkout@v2.
- add-apt-repository ppa:git-core/ppa
- apt-get update
- apt-get install -y git
- - name: Installing tensorflow dependencies
- run: |
- # Install python2 numpy. Temporary fix for issue #1737.
- apt-get install -y python-pip
- python -m pip install --upgrade pip
- python -m pip install numpy
- name: Checking out repository
uses: actions/checkout@v2
- name: Initializing submodules
diff --git a/.github/workflows/bazel_build_fallthrough.yml b/.github/workflows/bazel_build_fallthrough.yml
index 7871ba1..0f319e7 100644
--- a/.github/workflows/bazel_build_fallthrough.yml
+++ b/.github/workflows/bazel_build_fallthrough.yml
@@ -23,33 +23,8 @@
jobs:
linux:
runs-on: ubuntu-18.04
- env:
- CXX: clang++
- CC: clang
- PYTHON_BIN: /usr/bin/python3
- # Install the version of Bazel that is required by our .bazelversion file.
- BAZEL_VERSION: 2.1.0
+ container: gcr.io/iree-oss/bazel
steps:
- - name: Installing bazel
- run: |
- # https://docs.bazel.build/versions/master/install-ubuntu.html
- sudo apt-get install unzip zip
- wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION?}/bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- chmod +x bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- ./bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh --user
- rm bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- echo "::add-path::${HOME?}/bin"
- - name: Installing dependencies
- run: |
- sudo apt-get update
- sudo apt-get install clang libsdl2-dev python3 python3-pip python3-setuptools
- # Update pip before using it so it picks up the latest versions of our dependencies.
- sudo python3 -m pip install --upgrade pip
- sudo python3 -m pip install numpy tf-nightly
- # Install python2 numpy. Temporary fix for issue #1737.
- sudo apt-get install -y python-pip
- python -m pip install --upgrade pip
- python -m pip install numpy
- name: Checking out repository
uses: actions/checkout@v2
- name: Initializing submodules
diff --git a/.github/workflows/bazel_build_integrations.yml b/.github/workflows/bazel_build_integrations.yml
index 3cff5e2..dda584b 100644
--- a/.github/workflows/bazel_build_integrations.yml
+++ b/.github/workflows/bazel_build_integrations.yml
@@ -23,33 +23,8 @@
jobs:
linux:
runs-on: ubuntu-18.04
- env:
- CXX: clang++
- CC: clang
- PYTHON_BIN: /usr/bin/python3
- # Install the version of Bazel that is required by our .bazelversion file.
- BAZEL_VERSION: 2.1.0
+ container: gcr.io/iree-oss/bazel-tensorflow
steps:
- - name: Installing bazel
- run: |
- # https://docs.bazel.build/versions/master/install-ubuntu.html
- sudo apt-get install unzip zip
- wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION?}/bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- chmod +x bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- ./bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh --user
- rm bazel-${BAZEL_VERSION?}-installer-linux-x86_64.sh
- echo "::add-path::${HOME?}/bin"
- - name: Installing dependencies
- run: |
- sudo apt-get update
- sudo apt-get install clang libsdl2-dev python3 python3-pip python3-setuptools
- # Update pip before using it so it picks up the latest versions of our dependencies.
- sudo python3 -m pip install --upgrade pip
- sudo python3 -m pip install numpy tf-nightly
- # Install python2 numpy. Temporary fix for issue #1737.
- sudo apt-get install -y python-pip
- python -m pip install --upgrade pip
- python -m pip install numpy
- name: Checking out repository
uses: actions/checkout@v2
- name: Initializing submodules
diff --git a/build_tools/docker/bazel/Dockerfile b/build_tools/docker/bazel/Dockerfile
new file mode 100644
index 0000000..a8d6415
--- /dev/null
+++ b/build_tools/docker/bazel/Dockerfile
@@ -0,0 +1,40 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Builds an image for running IREE on bazel.
+
+# Set up the image and working directory.
+FROM l.gcr.io/google/bazel:2.1.0
+WORKDIR /usr/src/git
+
+# Set environment variables.
+ENV CXX clang++
+ENV CC clang
+ENV PYTHON_BIN /usr/bin/python3
+
+# Install the newest version of git. This is necessary because the checkout
+# action (https://github.com/actions/checkout) requires `git >= 2.18` for this
+# action to work with git submodules.
+RUN add-apt-repository ppa:git-core/ppa
+RUN apt-get update
+RUN apt-get install -y git
+
+# Install core IREE dependencies.
+RUN apt-get install -y clang libsdl2-dev
+
+# Install python2 numpy. Temporary fix for issue #1737:
+# https://github.com/google/iree/issues/1737
+RUN apt-get install -y python-pip
+RUN python -m pip install --upgrade pip
+RUN python -m pip install numpy
diff --git a/build_tools/docker/bazel_tensorflow/Dockerfile b/build_tools/docker/bazel_tensorflow/Dockerfile
new file mode 100644
index 0000000..19c3d76
--- /dev/null
+++ b/build_tools/docker/bazel_tensorflow/Dockerfile
@@ -0,0 +1,42 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Builds an image for running IREE on bazel with python bindings and tensorflow
+# integrations. Requires that the bazel image has already been built. Build
+# using:
+# docker build --tag bazel-tensorflow build_tools/docker/bazel_tensorflow/
+
+# Set up the image and working directory.
+FROM bazel
+WORKDIR /usr/src/git
+
+# Update python3 to 3.6
+RUN add-apt-repository ppa:deadsnakes/ppa
+RUN apt-get update
+RUN apt-get install -y python3.6 python3.6-dev
+
+# Replace with the block below with
+# ENV PYTHON_BIN /usr/bin/python3.6
+# once PYTHON_BIN also controls Bazel's runtime interpreter (issue:
+# https://github.com/google/iree/issues/1875).
+# For now, create a symlink from correct python version to the system's python3
+# (which PYTHON_BIN currently points to).
+# This makes many things unhappy (e.g. add-apt-repository).
+RUN rm /usr/bin/python3
+RUN ln -s /usr/bin/python3.6 /usr/bin/python3
+
+# Install tensorflow and numpy
+RUN apt-get install -y python3-pip python3-setuptools
+RUN python3 -m pip install --upgrade pip
+RUN python3 -m pip install numpy tf-nightly
diff --git a/build_tools/docker/update_gcr_images.sh b/build_tools/docker/update_gcr_images.sh
new file mode 100755
index 0000000..ce34a2a
--- /dev/null
+++ b/build_tools/docker/update_gcr_images.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Builds and pushes bazel and bazel-tensorflow images to gcr.io/iree-oss/ from
+# within the GCP IREE-OSS project.
+
+set -x
+set -e
+
+# Ensure correct authorization.
+gcloud auth configure-docker
+
+# Build and push the bazel image.
+docker build --tag bazel build_tools/docker/bazel/
+docker tag bazel gcr.io/iree-oss/bazel
+docker push gcr.io/iree-oss/bazel
+
+# Build and push the bazel-tensorflow image.
+docker build --tag bazel-tensorflow build_tools/docker/bazel_tensorflow/
+docker tag bazel-tensorflow gcr.io/iree-oss/bazel-tensorflow
+docker push gcr.io/iree-oss/bazel-tensorflow
