Add LLVM integrate scripts to the main repository.
These are now under build_tools/scripts/integrate.
Ported the llvm_version_bump.md into an expanded README in this directory.
diff --git a/docs/developers/developing_iree/llvm_version_bump.md b/build_tools/scripts/integrate/README.md
similarity index 88%
rename from docs/developers/developing_iree/llvm_version_bump.md
rename to build_tools/scripts/integrate/README.md
index 43a6173..6ff9909 100644
--- a/docs/developers/developing_iree/llvm_version_bump.md
+++ b/build_tools/scripts/integrate/README.md
@@ -1,17 +1,37 @@
-# Notes on integrating LLVM from the OSS side
+# Scripts for Integrating Changes from Third Party Dependencies
+
+This directory contains scripts for managing some of our more storied third
+party dependencies (which are submodules in the project):
+
+* llvm-project
+* mlir-hlo
+
+Depending on your activity, please refer to the appropriate script,
+which has comments at the top on how to use it:
+
+* `bump_llvm.py` : Bumping LLVM and related sub-projects to a new commit
+* `patch_module.py` : Push local patches (cherry-picks) from a submodule to
+ a mirror for use by others.
+
+NOTE: If needing to make changes to these scripts, DO NOT make changes while
+using them to perform one of the above tasks (i.e. your changes will just
+get bundled into an unrelated patch unless if very careful). In this rare
+case, just copy the contents of this directory to a temporary location and
+edit/run from there. When done, submit the changes to main.
+
+What follows are some rambly notes on how to do an LLVM integrate.
+
+TODO: Refactor these based on the common procedure we actually use.
+
+## Notes on integrating LLVM from the OSS side
This is a work in progress guide on how to bump versions of LLVM and related
dependencies. In the recent past, we did this in a different system and this
is just to get us by until we get it better scripted/automated.
-Note that scripts referenced in this guide are temporarily hosted in the
-[iree-samples repository](https://github.com/iree-org/iree-samples/tree/main/scripts/integrate).
-This is because it is very non-user friendly to have branch and submodule
-management scripts in the repository being managed, and we don't have an
-immediately better place. In this guide, we reference this location as
-`$SCRIPTS`.
+In this guide, we reference this directory as `$SCRIPTS`.
-## Advancing the mainline branch in forks
+### Advancing the mainline branch in forks
The IREE team maintains fork repositories for both llvm-project and mlir-hlo,
allowing them to be patched out of band. These repositories are:
@@ -20,16 +40,16 @@
* https://github.com/iree-org/iree-mhlo-fork (`master` branch)
* https://github.com/iree-org/iree-tf-fork (`master` branch)
-By the time you read this, they may be on a cron to advance automatically, but
-even so, it is a good idea to advance them prior to any integrate activities
-so that you have freshest commits available. Iree repository has an
+Iree repository has an
action named [Advance Upstream Forks](https://github.com/iree-org/iree/actions/workflows/advance_upstream_forks.yml)
to update the forks. Just select `Run Workflow` on that action and give it a
-minute. You should see the fork repository mainline branch move forward.
+minute. You should see the fork repository mainline branch move forward. This
+action runs hourly. If needing up to the minute changes, you may need to trigger
+it manually.
-## Bumping LLVM and Dependent Projects
+### Bumping LLVM and Dependent Projects
-### Strategy 1: Bump third_party/llvm-project in isolation
+#### Strategy 1: Bump third_party/llvm-project in isolation
It is very common to only bump llvm-project and not sync to new versions of
mlir-hlo and tensorflow. However, as we need to periodically integrate those
@@ -102,7 +122,7 @@
Good luck!
-### Strategy 2: Sync everything to a Google/TensorFlow commit
+#### Strategy 2: Sync everything to a Google/TensorFlow commit
TODO: Add a script for this. Also note that there is a forked copy of
`iree-dialects` in integrations/tensorflow. When bumping that dependency,
@@ -228,7 +248,7 @@
git push UPSTREAM_AUTOMATION bump-llvm-...
```
-## Cherry-picking
+### Cherry-picking
We support cherry-picking specific commits in to both llvm-project and mlir-hlo.
This should only ever be done to incorporate patches that enable further
@@ -333,3 +353,4 @@
cd integrations/tensorflow
bazel test ...
```
+
diff --git a/build_tools/scripts/integrate/bump_llvm.py b/build_tools/scripts/integrate/bump_llvm.py
new file mode 100755
index 0000000..687d0be
--- /dev/null
+++ b/build_tools/scripts/integrate/bump_llvm.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Creates a new branch that bumps the llvm-project commit.
+# Typical usage (from the iree/ repository):
+# /path/to/here/bump_llvm.py
+#
+# In the default configuration, it will create a new branch
+# "bump-llvm-YYYYMMDD"
+# This will fail if the branch already exists, in which case, you can:
+# * Specify an explicit branch name with --branch-name=my-integrate
+# * Pass "--reuse-branch" if you are sure that you want to lose the current
+# branch state. This is largely meant for developing this script or YOLO
+# use.
+#
+# In order to not interfere with your personal preferences, a remote named
+# 'UPSTREAM_AUTOMATION' is used (and setup if needed). Sorry if you use that
+# name for your daily work.
+#
+# This then reverts any changes to the llvm-project submodule setup (i.e.
+# resets it to llvm-project's repository and disables branch tracking) and
+# resets the submodule to the curren HEAD commit, generating a nice commit
+# message.
+#
+# The branch is then pushed to the main repository. GitHub will print the usual
+# message to create a pull request, which you should do to kick off pre-merge
+# checks. You should land changes to this branch until green (or get other
+# people to do so).
+#
+# When satisfied, Squash and Merge, opting to delete the branch to keep things
+# tidy.
+
+import argparse
+from datetime import date
+import os
+import sys
+
+import iree_modules
+import iree_utils
+
+
+def main(args):
+ if not args.disable_setup_remote:
+ iree_utils.git_setup_remote(args.upstream_remote,
+ args.upstream_repository)
+
+ iree_utils.git_check_porcelain()
+ print(f"Fetching remote repository: {args.upstream_remote}")
+ iree_utils.git_fetch(repository=args.upstream_remote)
+
+ # If re-using a branch, make sure we are not on that branch.
+ if args.reuse_branch:
+ iree_utils.git_checkout("main")
+
+ # Create branch.
+ branch_name = args.branch_name
+ if not branch_name:
+ branch_name = f"bump-llvm-{date.today().strftime('%Y%m%d')}"
+ print(f"Creating branch {branch_name} (override with --branch-name=)")
+ iree_utils.git_create_branch(branch_name,
+ checkout=True,
+ ref=f"{args.upstream_remote}/main",
+ force=args.reuse_branch)
+
+ # Reset the llvm-project submodule to track upstream.
+ # This will discard any cherrypicks that may have been committed locally,
+ # but the assumption is that if doing a main llvm version bump, the
+ # cherrypicks will be incorporated at the new commit. If not, well, ymmv
+ # and you will find out.
+ iree_utils.git_submodule_set_origin(
+ "third_party/llvm-project",
+ url="https://github.com/iree-org/iree-llvm-fork.git",
+ branch="--default")
+
+ # Remove the branch pin file, reverting us to pure upstream.
+ branch_pin_file = os.path.join(
+ iree_utils.get_repo_root(),
+ iree_modules.MODULE_INFOS["llvm-project"].branch_pin_file)
+ if os.path.exists(branch_pin_file):
+ os.remove(branch_pin_file)
+
+ # Update the LLVM submodule.
+ llvm_commit = args.llvm_commit
+ print(f"Updating LLVM submodule to {llvm_commit}")
+ llvm_root = iree_utils.get_submodule_root("llvm-project")
+ iree_utils.git_fetch(repository="origin",
+ ref="refs/heads/main", repo_dir=llvm_root)
+ if llvm_commit == "HEAD":
+ llvm_commit = "origin/main"
+ iree_utils.git_reset(llvm_commit, repo_dir=llvm_root)
+ llvm_commit, llvm_summary = iree_utils.git_current_commit(
+ repo_dir=llvm_root)
+ print(f"LLVM submodule reset to:\n {llvm_summary}\n")
+
+ # Create a commit.
+ print("Create commit...")
+ iree_utils.git_create_commit(
+ message=(f"Integrate llvm-project at {llvm_commit}\n\n"
+ f"* Reset third_party/llvm-project: {llvm_summary}"),
+ add_all=True)
+
+ # Push.
+ print("Pushing...")
+ iree_utils.git_push_branch(args.upstream_remote, branch_name)
+
+
+def parse_arguments(argv):
+ parser = argparse.ArgumentParser(description="IREE LLVM-bump-inator")
+ parser.add_argument("--upstream-remote",
+ help="Upstream remote",
+ default="UPSTREAM_AUTOMATION")
+ parser.add_argument("--upstream-repository",
+ help="Upstream repository URL",
+ default="git@github.com:iree-org/iree.git")
+ parser.add_argument("--disable-setup-remote",
+ help="Disable remote setup",
+ action="store_true",
+ default=False)
+ parser.add_argument("--llvm-commit",
+ help="LLVM commit sha",
+ default="HEAD")
+ parser.add_argument("--branch-name",
+ help="Integrate branch to create",
+ default=None)
+ parser.add_argument("--reuse-branch",
+ help="Allow re-use of an existing branch",
+ action="store_true",
+ default=False)
+ args = parser.parse_args(argv)
+ return args
+
+
+if __name__ == "__main__":
+ main(parse_arguments(sys.argv[1:]))
diff --git a/build_tools/scripts/integrate/iree_modules.py b/build_tools/scripts/integrate/iree_modules.py
new file mode 100644
index 0000000..3d9cf04
--- /dev/null
+++ b/build_tools/scripts/integrate/iree_modules.py
@@ -0,0 +1,43 @@
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+class ModuleInfo:
+
+ def __init__(self, *, name: str, path: str, branch_pin_file: str,
+ default_repository_url: str, fork_repository_push: str,
+ fork_repository_pull: str, branch_prefix: str):
+ self.name = name
+ self.path = path
+ self.branch_pin_file = branch_pin_file
+ self.default_repository_url = default_repository_url
+ self.fork_repository_push = fork_repository_push
+ self.fork_repository_pull = fork_repository_pull
+ self.branch_prefix = branch_prefix
+
+
+MODULE_INFOS = {
+ "llvm-project":
+ ModuleInfo(
+ name="llvm-project",
+ path="third_party/llvm-project",
+ branch_pin_file="third_party/llvm-project.branch-pin",
+ default_repository_url="https://github.com/iree-org/iree-llvm-fork.git",
+ fork_repository_push="git@github.com:iree-org/iree-llvm-fork.git",
+ fork_repository_pull="https://github.com/iree-org/iree-llvm-fork.git",
+ branch_prefix="patched-llvm-project-",
+ ),
+ "mlir-hlo":
+ ModuleInfo(
+ name="mlir-hlo",
+ path="third_party/mlir-hlo",
+ branch_pin_file="third_party/mlir-hlo.branch-pin",
+ default_repository_url="https://github.com/iree-org/iree-mhlo-fork.git",
+ fork_repository_push="git@github.com:iree-org/iree-mhlo-fork.git",
+ fork_repository_pull="https://github.com/iree-org/iree-mhlo-fork.git",
+ branch_prefix="patched-mlir-hlo-",
+ )
+}
diff --git a/build_tools/scripts/integrate/iree_utils.py b/build_tools/scripts/integrate/iree_utils.py
new file mode 100644
index 0000000..ef83ac5
--- /dev/null
+++ b/build_tools/scripts/integrate/iree_utils.py
@@ -0,0 +1,203 @@
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import os
+import re
+import shlex
+import shutil
+import subprocess
+from typing import Tuple
+
+_repo_root = None
+
+
+def get_repo_root() -> str:
+ global _repo_root
+ if _repo_root is None:
+ _repo_root = os.getcwd()
+ _validate_repo_root()
+ return _repo_root
+
+
+def get_submodule_root(submodule) -> str:
+ path = os.path.join(get_repo_root(), "third_party", submodule)
+ if not os.path.isdir(path):
+ raise SystemExit(f"Could not find submodule: {path}")
+ return path
+
+
+def _validate_repo_root():
+ # Look for something we know is there.
+ known_dir = os.path.join(_repo_root, "compiler")
+ if not os.path.isdir(known_dir):
+ raise SystemExit(f"ERROR: Must run from the iree repository root. "
+ f"Actually in: {_repo_root}")
+
+
+def git_setup_remote(remote_alias, url, *, repo_dir=None):
+ needs_create = False
+ try:
+ existing_url = git_exec(["remote", "get-url", remote_alias],
+ capture_output=True,
+ repo_dir=repo_dir,
+ quiet=True)
+ existing_url = existing_url.strip()
+ if existing_url == url:
+ return
+ except subprocess.CalledProcessError:
+ # Does not exist.
+ needs_create = True
+
+ if needs_create:
+ git_exec(["remote", "add", "--no-tags", remote_alias, url],
+ repo_dir=repo_dir)
+ else:
+ git_exec(["remote", "set-url", remote_alias, url], repo_dir=repo_dir)
+
+
+def git_is_porcelain(*, repo_dir=None):
+ output = git_exec(["status", "--porcelain", "--untracked-files=no"],
+ capture_output=True,
+ quiet=True,
+ repo_dir=repo_dir).strip()
+ return not bool(output)
+
+
+def git_check_porcelain(*, repo_dir=None):
+ output = git_exec(["status", "--porcelain", "--untracked-files=no"],
+ capture_output=True,
+ quiet=True,
+ repo_dir=repo_dir).strip()
+ if output:
+ actual_repo_dir = get_repo_root() if repo_dir is None else repo_dir
+ raise SystemExit(
+ f"ERROR: git directory {actual_repo_dir} is not clean. "
+ f"Please stash changes:\n{output}")
+
+
+def git_fetch(*, repository=None, ref=None, repo_dir=None):
+ args = ["fetch"]
+ if repository:
+ args.append(repository)
+ if ref is not None:
+ args.append(ref)
+ git_exec(args, repo_dir=repo_dir)
+
+
+def git_checkout(ref, *, repo_dir=None):
+ git_exec(["checkout", ref], repo_dir=repo_dir)
+
+
+def git_create_branch(branch_name,
+ *,
+ checkout=True,
+ ref=None,
+ force=False,
+ repo_dir=None):
+ branch_args = ["branch"]
+ if force:
+ branch_args.append("-f")
+ branch_args.append(branch_name)
+ if ref is not None:
+ branch_args.append(ref)
+ git_exec(branch_args, repo_dir=repo_dir)
+
+ if checkout:
+ git_exec(["checkout", branch_name], repo_dir=repo_dir)
+
+
+def git_push_branch(repository, branch_name, *, force=False, repo_dir=None):
+ push_args = ["push", "--set-upstream"]
+ if force:
+ push_args.append("-f")
+ push_args.append(repository)
+ push_args.append(f"{branch_name}:{branch_name}")
+ git_exec(push_args, repo_dir=repo_dir)
+
+
+def git_branch_exists(branch_name, *, repo_dir=None):
+ output = git_exec(["branch", "-l", branch_name],
+ repo_dir=repo_dir,
+ quiet=True,
+ capture_output=True).strip()
+ return bool(output)
+
+
+def git_submodule_set_origin(path, *, url=None, branch=None, repo_dir=None):
+ if url is not None:
+ git_exec(["submodule", "set-url", "--", path, url], repo_dir=repo_dir)
+
+ if branch is not None:
+ try:
+ if branch == "--default":
+ git_exec(["submodule", "set-branch", "--default", "--", path],
+ repo_dir=repo_dir)
+ else:
+ git_exec([
+ "submodule", "set-branch", "--branch", branch, "--", path
+ ],
+ repo_dir=repo_dir)
+ except subprocess.CalledProcessError:
+ # The set-branch command returns 0 on change and !0 on no change.
+ # This is a bit unfortunate.
+ ...
+
+
+def git_reset(ref, *, hard=True, repo_dir=None):
+ args = ["reset"]
+ if hard:
+ args.append("--hard")
+ args.append(ref)
+ git_exec(args, repo_dir=repo_dir)
+
+
+def git_current_commit(*, repo_dir=None) -> Tuple[str, str]:
+ output = git_exec(["log", "-n", "1", "--pretty=format:%H (%ci): %s"],
+ capture_output=True,
+ repo_dir=repo_dir,
+ quiet=True)
+ output = output.strip()
+ parts = output.split(" ")
+ # Return commit, full_summary
+ return parts[0], output
+
+
+def git_create_commit(*, message, add_all=False, repo_dir=None):
+ if add_all:
+ git_exec(["add", "-A"], repo_dir=repo_dir)
+ git_exec(["commit", "-m", message])
+
+
+def git_ls_remote_branches(repository_url, *, filter=None, repo_dir=None):
+ args = ["ls-remote", "-h", repository_url]
+ if filter:
+ args.extend(filter)
+ output = git_exec(args, quiet=True, capture_output=True)
+ lines = output.strip().splitlines(keepends=False)
+
+ # Format is <commit> refs/heads/branch_name
+ def extract_branch(line):
+ parts = re.split("\\s+", line)
+ ref = parts[1]
+ prefix = "refs/heads/"
+ if ref.startswith(prefix):
+ ref = ref[len(prefix):]
+ return ref
+
+ return [extract_branch(l) for l in lines]
+
+
+def git_exec(args, *, repo_dir=None, quiet=False, capture_output=False):
+ full_args = ["git"] + args
+ full_args_quoted = [shlex.quote(a) for a in full_args]
+ if not repo_dir:
+ repo_dir = get_repo_root()
+ if not quiet:
+ print(f" ++ EXEC: (cd {repo_dir} && {' '.join(full_args_quoted)})")
+ if capture_output:
+ return subprocess.check_output(full_args, cwd=repo_dir).decode("utf-8")
+ else:
+ subprocess.check_call(full_args, cwd=repo_dir)
diff --git a/build_tools/scripts/integrate/patch_module.py b/build_tools/scripts/integrate/patch_module.py
new file mode 100755
index 0000000..7b77a75
--- /dev/null
+++ b/build_tools/scripts/integrate/patch_module.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Pushes module patches to an appropriate repository.
+#
+# Typical steps:
+# 1. Advance the main branches on our forks to get all upstream commits
+# (if you forget to do this, your cherry-pick will likely complain about
+# a bad ref).
+# 2. In your submodule:
+# git fetch
+# git cherry-pick <some commit>
+# 3. Run this script from the main IREE repository (one of the following):
+# patch_module.py --module=llvm-project
+# patch_module.py --module=mlir-hlo
+# 4. Send a PR on the main IREE repo to bump the submodule. Be sure to include
+# the name of the patch branch for posterity.
+
+import argparse
+from datetime import date
+import os
+import sys
+
+import iree_utils
+import iree_modules
+
+PATCH_REMOTE_ALIAS = "patched"
+
+
+def main(args):
+ module_info = iree_modules.MODULE_INFOS.get(args.module)
+ if not module_info:
+ raise SystemExit(f"ERROR: Bad value for --module. Must be one of: "
+ f"{', '.join(iree_modules.MODULE_INFOS.keys())}")
+
+ if args.command == "patch":
+ main_patch(args, module_info)
+ else:
+ raise SystemExit(
+ f"ERROR: Unrecognized --command. Must be one of: patch, unpatch")
+
+
+def main_patch(args, module_info: iree_modules.ModuleInfo):
+ module_root = os.path.join(iree_utils.get_repo_root(), module_info.path)
+ setup_module_remotes(module_root, module_info)
+
+ branch_name = find_unused_branch_name(module_info)
+ print(f"Allocated branch: {branch_name}")
+ current_commit, summary = iree_utils.git_current_commit(
+ repo_dir=module_root)
+ print(f"Module is currently at: {summary}")
+ print(
+ f"*** Pushing branch {branch_name} to {module_info.fork_repository_push} ***"
+ )
+ print(f"(Please ignore any messages below about creating a PR)\n")
+ iree_utils.git_exec([
+ "push", PATCH_REMOTE_ALIAS,
+ f"{current_commit}:refs/heads/{branch_name}"
+ ],
+ repo_dir=module_root)
+ print(f"*** Branch {branch_name} pushed ***")
+
+ print(f"******* Congratulations *******")
+ print(f"You have pushed your commits to {branch_name} on {module_info.fork_repository_push}.")
+ print(f"Your main repository should now show that the submodule has been edited.")
+ print(f"Make a commit, referencing the above branch cherry-picks and ")
+ print(f"land the resulting PR.")
+ print(f"You can push more commits to this module's patch branch via:")
+ print(f" (cd {module_info.path} && git push {PATCH_REMOTE_ALIAS} HEAD:{branch_name})")
+
+
+def setup_module_remotes(module_root: str,
+ module_info: iree_modules.ModuleInfo):
+ iree_utils.git_setup_remote(PATCH_REMOTE_ALIAS,
+ url=module_info.fork_repository_push,
+ repo_dir=module_root)
+
+
+def find_unused_branch_name(module_info: iree_modules.ModuleInfo):
+ branch_base = f"{module_info.branch_prefix}{date.today().strftime('%Y%m%d')}"
+ branch_name = branch_base
+ existing_branches = iree_utils.git_ls_remote_branches(
+ module_info.fork_repository_pull,
+ filter=[f"refs/heads/{module_info.branch_prefix}*"])
+ i = 1
+ while branch_name in existing_branches:
+ branch_name = f"{branch_base}.{i}"
+ i += 1
+ return branch_name
+
+
+def parse_arguments(argv):
+ parser = argparse.ArgumentParser(description="IREE Submodule Patcher")
+ parser.add_argument("--module",
+ help="Submodule to operate on",
+ default=None)
+ parser.add_argument("--command",
+ help="Command to execute",
+ default="patch")
+ args = parser.parse_args(argv)
+ return args
+
+
+if __name__ == "__main__":
+ main(parse_arguments(sys.argv[1:]))
