Build and publish Windows Python releases. (#14575)
Fixes https://github.com/openxla/iree/issues/13484 (hopefully,
#test-in-prod)
* New setup script `build_tools/python_deploy/install_windows_deps.ps1`
can be used to install Python versions. GitHub Actions could use
https://github.com/actions/setup-python, but I wanted to match what we
have for macOS (`install_macos_deps.sh`). I tried writing this script in
bash, but couldn't run the Python installer with `/quiet` for some
reason, so it's written in powershell instead. Tested locally and on CI
but could be brittle on other systems since Windows is Windows.
* New build script `build_tools/python_deploy/build_windows_packages.sh`
is forked from matching Linux and macOS scripts. Also tested this
locally and on CI.
These scripts can build for multiple Python versions, but I'm starting
with just 3.11, which might be enough.
Tested at
https://github.com/openxla/iree/actions/runs/5765425296/job/15631341317,
and I sanity checked that I could install and use locally built wheels.
diff --git a/.github/workflows/build_package.yml b/.github/workflows/build_package.yml
index 1dfa2fd..97ac2a4 100644
--- a/.github/workflows/build_package.yml
+++ b/.github/workflows/build_package.yml
@@ -1,12 +1,16 @@
-# Builds packages for native (non cross-compiled) targets on supported
-# platforms.
-#
-# For these mainline distributions, we use cibuildwheel and drive the
-# packaging through python, extracting native assets. While this may seem
-# hopelessly round-about, it lets us leverage a lot of what cibuildwheel
-# does for free and get python packages to boot.
-name: Build Native Release Packages
+# Builds release packages on supported platforms:
+# * main-dist-linux (CMake 'install')
+# * py-compiler-pkg (`iree-compiler` Python package)
+# * Linux, macOS, Windows
+# * All supported Python versions (e.g. 3.8, 3.9, 3.10, 3.11)
+# * py-runtime-pkg (`iree-runtime[-instrumented]` Python package)
+# * Linux, macOS, Windows
+# * All supported Python versions (e.g. 3.8, 3.9, 3.10, 3.11)
+# * py-tf-compiler-tools-pkg (`iree-tools-[tf, tflite]`, pure Python packages)
+name: Build Release Packages
+
+# Note: this workflow is typically invoked by schedule_candidate_release.yml.
on:
workflow_dispatch:
inputs:
@@ -26,7 +30,7 @@
default: ""
jobs:
- build_core:
+ build_packages:
name: "${{ matrix.build-family }} :: Build ${{ matrix.build-package }} Package"
runs-on: ${{ matrix.runs-on }}
continue-on-error: ${{ matrix.experimental }}
@@ -67,13 +71,25 @@
build-family: macos
build-package: py-runtime-pkg
experimental: true
+
+ # Windows packages.
+ - runs-on:
+ - ${{ github.repository == 'openxla/iree' && 'windows-2022-64core' || 'windows-2022'}}
+ build-family: windows
+ build-package: py-compiler-pkg
+ experimental: true
+ - runs-on: windows-2022
+ build-family: windows
+ build-package: py-runtime-pkg
+ experimental: true
+
env:
MANYLINUX_X86_64_IMAGE: gcr.io/iree-oss/manylinux2014_x86_64-release@sha256:e83893d35be4ce3558c989e9d5ccc4ff88d058bc3e74a83181059cc76e2cf1f8
steps:
- uses: actions/checkout@8f4b7f84864484a7bf31766abe9204da3cbe65b3 # v3.5.0
with:
- path: "main_checkout"
+ path: "c" # Windows can hit path length limits, so use a short path.
submodules: true
ref: ${{ github.event.inputs.commit }}
@@ -81,6 +97,14 @@
# OS specific setup
##########################################################################
+ - name: Install dependencies (Windows)
+ if: "matrix.build-family == 'windows'"
+ shell: powershell
+ run: ./c/build_tools/python_deploy/install_windows_deps.ps1
+ - name: "Configure MSVC (Windows)"
+ if: "matrix.build-family == 'windows'"
+ uses: ilammy/msvc-dev-cmd@7315a94840631165970262a99c72cfb48a65d25d # v1.12.0
+
##########################################################################
# Write version_info.json
# Various tools will read this in order to embed release information.
@@ -88,14 +112,14 @@
- name: Write version info (release)
shell: bash
run: |
- cat << EOF > ./main_checkout/version_info.json
+ cat << EOF > ./c/version_info.json
{
"package-suffix": "${{ github.event.inputs.package_suffix }}",
"package-version": "${{ github.event.inputs.package_version }}",
- "iree-revision": "$(cd ./main_checkout && git rev-parse HEAD)"
+ "iree-revision": "$(cd ./c && git rev-parse HEAD)"
}
EOF
- cat ./main_checkout/version_info.json
+ cat ./c/version_info.json
##########################################################################
# Build the main distribution tarball.
@@ -116,7 +140,7 @@
docker run --rm -w=/work \
-v $PWD:/work \
"${MANYLINUX_X86_64_IMAGE}" \
- bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./main_checkout/build_tools/github_actions/build_dist.py main-dist'
+ bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./c/build_tools/github_actions/build_dist.py main-dist'
##########################################################################
# py-runtime-pkg
@@ -132,7 +156,7 @@
packages: "iree-runtime iree-runtime-instrumented"
output_dir: "${{ github.workspace }}/bindist"
run: |
- ./main_checkout/build_tools/python_deploy/build_linux_packages.sh
+ ./c/build_tools/python_deploy/build_linux_packages.sh
- name: Build runtime wheels (MacOS)
if: "matrix.build-package == 'py-runtime-pkg' && matrix.build-family == 'macos'"
@@ -143,11 +167,22 @@
output_dir: "${{ github.workspace }}/bindist"
override_python_versions: "3.11"
run: |
- ./main_checkout/build_tools/python_deploy/build_macos_packages.sh
+ ./c/build_tools/python_deploy/build_macos_packages.sh
+
+ - name: Build runtime wheels (Windows)
+ if: "matrix.build-package == 'py-runtime-pkg' && matrix.build-family == 'windows'"
+ shell: bash
+ env:
+ package_suffix: ${{ github.event.inputs.package_suffix }}
+ packages: "iree-runtime iree-runtime-instrumented"
+ output_dir: "${{ github.workspace }}/bindist"
+ override_python_versions: "3.11"
+ run: |
+ ./c/build_tools/python_deploy/build_windows_packages.sh
##########################################################################
# py-compiler-pkg
- # Builds the iree-compielr wheel.
+ # Builds the iree-compiler wheel.
# One step per OS.
##########################################################################
- name: Build compiler wheels (Linux)
@@ -158,7 +193,7 @@
packages: "iree-compiler"
output_dir: "${{ github.workspace }}/bindist"
run: |
- ./main_checkout/build_tools/python_deploy/build_linux_packages.sh
+ ./c/build_tools/python_deploy/build_linux_packages.sh
- name: Build compiler wheels (MacOS)
if: "matrix.build-package == 'py-compiler-pkg' && matrix.build-family == 'macos'"
@@ -169,13 +204,23 @@
output_dir: "${{ github.workspace }}/bindist"
override_python_versions: "3.11"
run: |
- ./main_checkout/build_tools/python_deploy/build_macos_packages.sh
+ ./c/build_tools/python_deploy/build_macos_packages.sh
+
+ - name: Build compiler wheels (Windows)
+ if: "matrix.build-package == 'py-compiler-pkg' && matrix.build-family == 'windows'"
+ shell: bash
+ env:
+ package_suffix: ${{ github.event.inputs.package_suffix }}
+ packages: "iree-compiler"
+ output_dir: "${{ github.workspace }}/bindist"
+ override_python_versions: "3.11"
+ run: |
+ ./c/build_tools/python_deploy/build_windows_packages.sh
##########################################################################
# TF Compiler Tools
- # Compiler tools wheels are not python version specific, so just build
- # for one examplar python version.
- # TODO: This currently only builds on Linux and is hardcoded to do so.
+ # These wheels are not Python version specific and include only Python
+ # code (no C/C++), so just build for one examplar python version on Linux.
##########################################################################
- name: Build TF Compiler Tools wheels
@@ -185,7 +230,7 @@
docker run --rm -w=/work \
-v $PWD:/work \
"${MANYLINUX_X86_64_IMAGE}" \
- bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./main_checkout/build_tools/github_actions/build_dist.py py-tf-compiler-tools-pkg'
+ bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./c/build_tools/github_actions/build_dist.py py-tf-compiler-tools-pkg'
- uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
with:
@@ -207,8 +252,8 @@
assets_path: ./bindist/iree*.*
validate_and_publish:
- name: "Invoke workflow to validate and publish release"
- needs: build_core
+ name: "Trigger validate and publish release"
+ needs: build_packages
if: github.event.inputs.release_id != ''
runs-on: ubuntu-20.04
steps:
diff --git a/build_tools/github_actions/build_dist.py b/build_tools/github_actions/build_dist.py
index 19dbcc6..be49b60 100644
--- a/build_tools/github_actions/build_dist.py
+++ b/build_tools/github_actions/build_dist.py
@@ -17,9 +17,9 @@
This is expected to be run from the project directory, containing the
following sub-directories:
- - main_checkout/ : Main IREE repository checkout.
+ - c/ : Main IREE repository checkout.
- bindist/ : Directory where binary distribution artifacts are written.
- - main_checkout/version_info.json : Version config information.
+ - c/version_info.json : Version config information.
Within the build environment (which may be the naked runner or a docker image):
- iree-build/ : The build tree.
@@ -33,13 +33,13 @@
It is not recommended to run cibuildwheel locally. However, this script can
be executed as if running within such an environment. To do so, create
a directory and:
- ln -s /path/to/iree main_checkout
+ ln -s /path/to/iree c
python -m venv .venv
source .venv/bin/activate
- python ./main_checkout/build_tools/github_actions/build_dist.py main-dist
- python ./main_checkout/build_tools/github_actions/build_dist.py py-tflite-compiler-tools-pkg
- python ./main_checkout/build_tools/github_actions/build_dist.py py-tf-compiler-tools-pkg
+ python ./c/build_tools/github_actions/build_dist.py main-dist
+ python ./c/build_tools/github_actions/build_dist.py py-tflite-compiler-tools-pkg
+ python ./c/build_tools/github_actions/build_dist.py py-tf-compiler-tools-pkg
That is not a perfect approximation but is close.
@@ -58,7 +58,7 @@
WORK_DIR = os.path.realpath(os.path.curdir)
BUILD_DIR = os.path.join(WORK_DIR, "iree-build")
INSTALL_DIR = os.path.join(WORK_DIR, "iree-install")
-IREESRC_DIR = os.path.join(WORK_DIR, "main_checkout")
+IREESRC_DIR = os.path.join(WORK_DIR, "c")
TF_INTEGRATIONS_DIR = os.path.join(IREESRC_DIR, "integrations/tensorflow")
BINDIST_DIR = os.environ.get("BINDIST_DIR")
if BINDIST_DIR is None:
@@ -184,7 +184,7 @@
def build_py_tf_compiler_tools_pkg():
- """Builds the iree-install/python_packages/iree_tools_tf package."""
+ """Builds iree-install/python_packages/iree_tools_[tf, tflite] packages."""
install_python_requirements()
configure_bazel()
diff --git a/build_tools/python_deploy/build_windows_packages.sh b/build_tools/python_deploy/build_windows_packages.sh
new file mode 100644
index 0000000..7cc1d7a
--- /dev/null
+++ b/build_tools/python_deploy/build_windows_packages.sh
@@ -0,0 +1,97 @@
+#!/bin/bash
+# Copyright 2023 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# build_windows_packages.sh
+# One stop build of IREE Python packages for Windows. This presumes that
+# dependencies are installed from install_windows_deps.ps1.
+#
+# Valid packages:
+# iree-runtime
+# iree-runtime-instrumented
+# iree-compiler
+
+set -eu -o errtrace
+
+this_dir="$(cd $(dirname $0) && pwd)"
+repo_root="$(cd $this_dir/../../ && pwd)"
+python_versions="${override_python_versions:-3.11}"
+output_dir="${output_dir:-${this_dir}/wheelhouse}"
+packages="${packages:-iree-runtime iree-runtime-instrumented iree-compiler}"
+
+# Canonicalize paths.
+mkdir -p "$output_dir"
+output_dir="$(cd $output_dir && pwd)"
+
+function run() {
+ echo "Using python versions: ${python_versions}"
+
+ local orig_path="$PATH"
+
+ # Build phase.
+ for package in $packages; do
+ echo "******************** BUILDING PACKAGE ${package} ********************"
+ for python_version in $python_versions; do
+ if [[ $(py --list) != *${python_version}* ]]; then
+ echo "ERROR: Could not find python version: ${python_version}"
+ continue
+ fi
+
+ echo ":::: Version: $(py -${python_version} --version)"
+ case "$package" in
+ iree-runtime)
+ clean_wheels iree_runtime $python_version
+ build_iree_runtime $python_version
+ ;;
+ iree-runtime-instrumented)
+ clean_wheels iree_runtime_instrumented $python_version
+ build_iree_runtime_instrumented $python_version
+ ;;
+ iree-compiler)
+ clean_wheels iree_compiler $python_version
+ build_iree_compiler $python_version
+ ;;
+ *)
+ echo "Unrecognized package '$package'"
+ exit 1
+ ;;
+ esac
+ done
+ done
+
+ echo "******************** BUILD COMPLETE ********************"
+ echo "Generated binaries:"
+ ls -l $output_dir
+}
+
+function build_iree_runtime() {
+ local python_version="$1"
+ IREE_HAL_DRIVER_VULKAN=ON \
+ py -${python_version} -m pip wheel -v -w $output_dir $repo_root/runtime/
+}
+
+function build_iree_runtime_instrumented() {
+ local python_version="$1"
+ IREE_HAL_DRIVER_VULKAN=ON IREE_ENABLE_RUNTIME_TRACING=ON \
+ IREE_RUNTIME_CUSTOM_PACKAGE_SUFFIX="-instrumented" \
+ py -${python_version} -m pip wheel -v -w $output_dir $repo_root/runtime/
+}
+
+function build_iree_compiler() {
+ local python_version="$1"
+ py -${python_version} -m pip wheel -v -w $output_dir $repo_root/compiler/
+}
+
+function clean_wheels() {
+ local wheel_basename="$1"
+ local python_version="$2"
+ echo ":::: Clean wheels $wheel_basename $python_version"
+ # python_version is something like "3.11", but we'd want something like "cp311".
+ local cpython_version_string="cp${python_version%.*}${python_version#*.}"
+ rm -f -v ${output_dir}/${wheel_basename}-*-${cpython_version_string}-*.whl
+}
+
+run
diff --git a/build_tools/python_deploy/install_macos_deps.sh b/build_tools/python_deploy/install_macos_deps.sh
index 9e01333..ec51cde 100755
--- a/build_tools/python_deploy/install_macos_deps.sh
+++ b/build_tools/python_deploy/install_macos_deps.sh
@@ -18,19 +18,13 @@
exit 1
fi
-PYTHON_INSTALLER_URLS=(
- "https://www.python.org/ftp/python/3.11.2/python-3.11.2-macos11.pkg"
- "https://www.python.org/ftp/python/3.10.10/python-3.10.10-macos11.pkg"
- "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg"
-)
-
PYTHON_SPECS=(
3.11@https://www.python.org/ftp/python/3.11.2/python-3.11.2-macos11.pkg
3.10@https://www.python.org/ftp/python/3.10.5/python-3.10.5-macos11.pkg
3.9@https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg
)
-for python_spec in $PYTHON_SPECS; do
+for python_spec in "${PYTHON_SPECS[@]}"; do
python_version="${python_spec%%@*}"
url="${python_spec##*@}"
echo "-- Installing Python $python_version from $url"
diff --git a/build_tools/python_deploy/install_windows_deps.ps1 b/build_tools/python_deploy/install_windows_deps.ps1
new file mode 100644
index 0000000..c3d9187
--- /dev/null
+++ b/build_tools/python_deploy/install_windows_deps.ps1
@@ -0,0 +1,67 @@
+# Copyright 2023 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Installs dependencies on Windows necessary to build IREE Python wheels.
+
+$PYTHON_VERSIONS = @(
+ "3.11" #,
+ # "3.10",
+ # "3.9"
+)
+
+$PYTHON_VERSIONS_NO_DOT = @(
+ "311" #,
+ # "310",
+ # "39"
+)
+
+$PYTHON_INSTALLER_URLS = @(
+ "https://www.python.org/ftp/python/3.11.2/python-3.11.2-amd64.exe" #,
+ # "https://www.python.org/ftp/python/3.10.5/python-3.10.5-amd64.exe",
+ # "https://www.python.org/ftp/python/3.9.13/python-3.9.13-amd64.exe"
+)
+
+# Multiple Python install locations are valid, so we use the `py` helper to
+# check for versions and call into them. Some valid install locations are:
+# C:\Python39\python.exe
+# C:\Program Files\Python39\python.exe
+# C:\Users\[NAME]\AppData\Local\Programs\Python\Python39\python.exe
+# See https://docs.python.org/3/using/windows.html#python-launcher-for-windows.
+$INSTALLED_VERSIONS_OUTPUT = py --list | Out-String
+
+for($i=0 ; $i -lt $PYTHON_VERSIONS.Length; $i++) {
+ $PYTHON_VERSION = $PYTHON_VERSIONS[$i]
+ $PYTHON_VERSION_NO_DOT = $PYTHON_VERSIONS_NO_DOT[$i]
+ $PYTHON_INSTALLER_URL = $PYTHON_INSTALLER_URLS[$i]
+ Write-Host "-- Installing Python ${PYTHON_VERSION} from ${PYTHON_INSTALLER_URL}"
+
+ if ("${INSTALLED_VERSIONS_OUTPUT}" -like "*${PYTHON_VERSION}*") {
+ Write-Host ":: Python version already installed. Not reinstalling."
+ } else {
+ $DOWNLOAD_ROOT = "$env:TEMP/iree_python_install"
+ $DOWNLOAD_FILENAME = $PYTHON_INSTALLER_URL.Substring($PYTHON_INSTALLER_URL.LastIndexOf("/") + 1)
+ $DOWNLOAD_PATH = "${DOWNLOAD_ROOT}/$DOWNLOAD_FILENAME"
+
+ # Create download folder as needed.
+ md -Force ${DOWNLOAD_ROOT} | Out-Null
+
+ Write-Host ":: Downloading $PYTHON_INSTALLER_URL -> $DOWNLOAD_PATH"
+ curl $PYTHON_INSTALLER_URL -o $DOWNLOAD_PATH
+
+ Write-Host ":: Running installer: $DOWNLOAD_PATH"
+ # https://docs.python.org/3/using/windows.html#installing-without-ui
+ & "$DOWNLOAD_PATH" /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
+ }
+
+ Write-Host ":: Python version $PYTHON_VERSION installed:"
+ & py -${PYTHON_VERSION} --version
+ & py -${PYTHON_VERSION} -m pip --version
+
+ Write-Host ":: Installing system pip packages"
+ & py -${PYTHON_VERSION} -m pip install --upgrade pip
+}
+
+Write-Host "*** All done ***"
diff --git a/compiler/setup.py b/compiler/setup.py
index 5a7f41e..eab5112 100644
--- a/compiler/setup.py
+++ b/compiler/setup.py
@@ -75,8 +75,10 @@
# We must do the intermediate installation to a fixed location that agrees
# between what we pass to setup() and cmake. So hard-code it here.
# Note that setup() needs a relative path (to the setup.py file).
+# We keep the path short ('i' instead of 'install') for platforms like Windows
+# that have file length limits.
SETUPPY_DIR = os.path.realpath(os.path.dirname(__file__))
-CMAKE_INSTALL_DIR_REL = os.path.join("build", "cmake_install")
+CMAKE_INSTALL_DIR_REL = os.path.join("build", "i")
CMAKE_INSTALL_DIR_ABS = os.path.join(SETUPPY_DIR, CMAKE_INSTALL_DIR_REL)
IS_CONFIGURED = CONFIGURED_SOURCE_DIR[0] != "@"
@@ -96,7 +98,7 @@
# Note that setuptools always builds into a "build" directory that
# is a sibling of setup.py, so we just colonize a sub-directory of that
# by default.
- IREE_BINARY_DIR = os.path.join(SETUPPY_DIR, "build", "cmake_build")
+ IREE_BINARY_DIR = os.path.join(SETUPPY_DIR, "build", "b")
print(
f"Running setup.py from source tree: "
f"SOURCE_DIR = {IREE_SOURCE_DIR} "
diff --git a/docs/developers/debugging/releases.md b/docs/developers/debugging/releases.md
index 2b4a15b..3142ed9 100644
--- a/docs/developers/debugging/releases.md
+++ b/docs/developers/debugging/releases.md
@@ -5,9 +5,9 @@
* `.github/workflows/build_package.yml`: Release packaging jobs
* `build_tools/github_actions/build_dist.py`: Main script to build various
release packages (for all platforms). We usually use this when reproing to
- approximate exactly what the CI does. Assumes a subdirectory of `main_checkout`
+ approximate exactly what the CI does. Assumes a subdirectory of `c`
and writes builds to `iree-build` and `iree-install` as a peer of it. To use
- locally, just symlink your source dir as `main_checkout` in an empty
+ locally, just symlink your source dir as `c` in an empty
directory (versus checking out).
## Manylinux releases
@@ -21,11 +21,11 @@
Get a docker shell (see exact docker image in build_package.yml workflow):
```shell
-docker run --rm -it -v $(pwd):/work/main_checkout stellaraccident/manylinux2014_x86_64-bazel-4.2.2:latest /bin/bash
+docker run --rm -it -v $(pwd):/work/c stellaraccident/manylinux2014_x86_64-bazel-4.2.2:latest /bin/bash
```
Remember that docker runs as root unless if you take steps otherwise. Don't
-touch write files in the `/work/main_checkout` directory to avoid scattering
+touch write files in the `/work/c` directory to avoid scattering
root owned files on your workstation.
The default system Python is 2.x, so you must select one of the more modern
@@ -41,7 +41,7 @@
```shell
# (from within docker)
cd /work
-python ./main_checkout/build_tools/github_actions/build_dist.py main-dist
+python ./c/build_tools/github_actions/build_dist.py main-dist
# Also supports:
# main-dist
diff --git a/runtime/setup.py b/runtime/setup.py
index f5077f9..c3eb7ed 100644
--- a/runtime/setup.py
+++ b/runtime/setup.py
@@ -79,8 +79,10 @@
# We must do the intermediate installation to a fixed location that agrees
# between what we pass to setup() and cmake. So hard-code it here.
# Note that setup() needs a relative path (to the setup.py file).
+# We keep the path short ('i' instead of 'install') for platforms like Windows
+# that have file length limits.
SETUPPY_DIR = os.path.realpath(os.path.dirname(__file__))
-CMAKE_INSTALL_DIR_REL = os.path.join("build", "cmake_install")
+CMAKE_INSTALL_DIR_REL = os.path.join("build", "i")
CMAKE_INSTALL_DIR_ABS = os.path.join(SETUPPY_DIR, CMAKE_INSTALL_DIR_REL)
IS_CONFIGURED = CONFIGURED_SOURCE_DIR[0] != "@"
@@ -100,7 +102,7 @@
# Note that setuptools always builds into a "build" directory that
# is a sibling of setup.py, so we just colonize a sub-directory of that
# by default.
- IREE_BINARY_DIR = os.path.join(SETUPPY_DIR, "build", "cmake_build")
+ IREE_BINARY_DIR = os.path.join(SETUPPY_DIR, "build", "b")
print(
f"Running setup.py from source tree: "
f"SOURCE_DIR = {IREE_SOURCE_DIR} "
