[Dev Container] - Adding VSCode Dev Container Capability (#12921)
This PR aims to add support for VS Code development container -
https://github.com/openxla/iree/issues/12907
Full Documentation: .devcontainer/README.md
---------
Co-authored-by: Scott Todd <scotttodd@google.com>
Co-authored-by: Geoffrey Martin-Noble <gcmn@google.com>
diff --git a/.devcontainer/README.md b/.devcontainer/README.md
new file mode 100644
index 0000000..12f4f1d
--- /dev/null
+++ b/.devcontainer/README.md
@@ -0,0 +1,125 @@
+# Getting Started with IREE Development Containers
+
+VS Code Development Container is a feature of Visual Studio Code that allows
+developers to create a consistent and isolated development environment for their
+projects. It leverages Docker containers to provide an environment that is
+identical to the production environment, which means developers can develop and
+test their code in an environment that closely mimics the production
+environment.
+
+With VS Code Development Container, developers can avoid the hassle of setting
+up a local development environment, installing dependencies, and managing
+different versions of software packages. Instead, they can use a pre-configured
+container that contains all the required dependencies and tools to develop their
+project. This makes it easier to share the development environment with other
+team members and ensures that everyone is using the same environment.
+
+In addition, VS Code Development Container provides a seamless development
+experience by allowing developers to work directly within the container using
+Visual Studio Code. They can edit code, run tests, and debug their applications
+as if they were working on their local machine.
+
+Overall, VS Code Development Container is a powerful tool for developers who
+want to streamline their development process and ensure consistency across their
+team's development environment.
+
+## A. Prerequisites
+
+Please follow the following steps to prepare your working environment:
+
+1. **Install Docker:** Before you can use VS Code Development Containers, you
+ need to have Docker installed on your machine. You can download and install
+ Docker for your operating system from the official Docker website.
+
+2. **[Optional] Install the NVIDIA-Docker runtime:** If you have an NVIDIA GPU
+ and want to use it for accelerated computing in your container, you can
+ install the NVIDIA-Docker runtime. Follow the instructions on the
+ [NVIDIA-Docker GitHub page](https://github.com/NVIDIA/nvidia-docker) or on
+ [NVIDIA Developer Documentation](https://developer.nvidia.com/nvidia-container-runtime)
+ to install the runtime for your operating system.
+
+3. **Install VS Code:** Naturally you will need VS Code to be installed on your
+ machine: https://code.visualstudio.com/download
+
+4. **Install Dev Container VS Code Extension:** Once VS Code installed, you will
+ need *Microsoft's Dev Containers extension:*
+ [ms-vscode-remote.remote-containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
+
+5. **[Optional] Install Remote Development VS Code Extension Pack:** If you wish
+ to setup the `Dev Container` on a remote machine over SSH, we recommend
+ installing *Microsoft's Remote Development extension pack:* :
+ [ms-vscode-remote.vscode-remote-extensionpack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack).
+
+
+## B. Configuring your Development Container.
+
+IREE Project provides multiple possible development containers which corresponds
+to different usecases (e.g CPU, NVIDIA GPUs, etc.).
+
+More containers will be integrated in the future, contributions are welcome.
+
+In order to setup the environment, please execute this script:
+
+```sh
+python3 .devcontainer/configure.py
+
+# This question is optional and will only appear if your environment was
+# previously configured. It aims to avoid overriding an existing configuration
+# inadvertently.
+>>> A `docker-compose.yml` already exists. Are you certain you want to overwrite it [y/N]?
+
+# Google provides prebuilt container images hosted on `gcr.io`. We invite you
+# and recommend using them. However, for expert users you can use locally built
+# containers by answering `N`.
+>>> Do you wish to use the official prebuild development containers [Y/n]?
+
+# We offer the option to mount a host directory to use as persistant local
+# CCache cache. The directory needs to exist in order to be mounted.
+>>> [OPTIONAL] Enter the the directory path to mount as CCache Cache.
+ It will be mounted in the container under: `/home/iree/.cache/ccache`. [Default: None]:
+
+# We offer the option to mount a host directory to use as persistant local
+# build directory. The directory needs to exist in order to be mounted.
+>>> [OPTIONAL] Enter the the directory path to mount as Build Directory.
+ It will mounted in the container under: `/home/iree/build` [Default: None]:
+
+# If your environment has any NVIDIA GPU with properly installed drivers and
+# NVIDIA Docker Runtime, you will be asked if you wish to use an NVIDIA GPU
+# ready container.
+>>> [OPTIONAL] Do you wish to use NVIDIA GPU [y/N]?
+
+# Finally a success message will be printed.
+>>> ================================================================================
+>>>
+>>> Success! Wrote Docker Compose file to `/path/to/iree/.devcontainer/docker-compose.yml`.
+```
+
+## C. (Building and) Launching the Development
+
+We can now build or download, if necessary, and launch the development
+container. In order to do so, we need to open the VS Code "Command Palette" with
+the shortcut: `Ctrl + Shift + P`.
+
+We need to select and click `Dev Containers: Rebuild and Reopen in Container`.
+
+
+
+A message will appear signifying that the environment is being setup (this step
+might take some time if you need to download or build the container, you can
+click on `show log` to see the details and watch the progress):
+
+
+
+The window will refresh once the environment is ready, and the container will be
+started (You can click on `show log` to watch the progress, this might take a
+few minutes).
+
+
+
+Once the container is started, you will be greeted by the container terminal
+(notice the username `iree`):
+
+
+
+**Once done, you are ready to start developing, building and contributing to
+IREE 🎉🎉🎉.**
\ No newline at end of file
diff --git a/.devcontainer/configure.py b/.devcontainer/configure.py
new file mode 100644
index 0000000..cf44952
--- /dev/null
+++ b/.devcontainer/configure.py
@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+
+# Copyright 2023 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Configure script to setup VSCode DevContainer Environment."""
+
+import json
+import os
+import pathlib
+import shlex
+import subprocess
+import sys
+
+from importlib.machinery import SourceFileLoader
+from shutil import which
+
+CURRENT_DIR = pathlib.Path(__file__).resolve().parent
+
+DOCKER_IMAGE_SHORTNAME_DICT = {
+ "base": "base-bleeding-edge",
+ "nvidia": "nvidia-bleeding-edge",
+}
+
+
+def run_shell(cmd):
+ cmd = shlex.split(cmd)
+ return subprocess.run(cmd, check=True, text=True,
+ stdout=subprocess.PIPE).stdout
+
+
+def is_nvidia_gpu_available():
+ """This function verifies NVIDIA Docker runtime is installed and
+ available. It also checks NVIDIA GPU are available and the driver
+ is installed."""
+
+ command_str = f"'{which('docker')}' info --format '{{{{json .}}}}'"
+ data = json.loads(run_shell(command_str))
+
+ if "nvidia" not in data["Runtimes"].keys():
+ return (
+ False, "NVIDIA Docker Runtime is not available. Please see: "
+ "https://developer.nvidia.com/nvidia-container-runtime for additional "
+ "information.")
+
+ nvidia_smi_executable = which('nvidia-smi')
+ if nvidia_smi_executable is None:
+ return False, "NVIDIA Driver is not installed or not in the user path."
+
+ command_str = f"{nvidia_smi_executable} --query-gpu=gpu_name --format=csv"
+
+ if run_shell(command_str).splitlines()[1:]: # Skip header
+ return True, None
+ else:
+ return False, "No NVIDIA GPU was found on this system."
+
+
+def _get_input(question, default_answer):
+ try:
+ answer = input(f"\n- {question} ")
+ except EOFError:
+ answer = default_answer
+
+ return (answer or default_answer).strip().lower()
+
+
+def get_input(question, default_answer='', accepted_answers=None):
+ if accepted_answers is None:
+ raise RuntimeError("Argument `accepted_answers` is None.")
+
+ accepted_answers = [x.strip().lower() for x in accepted_answers]
+
+ while True:
+ answer = _get_input(question, default_answer)
+ if answer not in accepted_answers:
+ print(f"\tERROR: Unsupported answer received: {answer}."
+ f"Expected: {accepted_answers}")
+ continue
+ break
+
+ return answer
+
+
+def get_input_path(question):
+
+ while True:
+ answer = _get_input(question, default_answer="")
+ if answer != "" and not os.path.isdir(answer):
+ print(f"\tERROR: Received path does not exist: `{answer}`")
+ continue
+ break
+
+ return answer
+
+
+if __name__ == "__main__":
+
+ # ------------------------------------------------------------------------- #
+ # Environment Verifications #
+ # ------------------------------------------------------------------------- #
+
+ docker_executable = which('docker')
+ if docker_executable is None:
+ raise RuntimeError(
+ "Docker is not installed or in the user path. Please refer to: "
+ "https://docs.docker.com/desktop/")
+
+ try:
+ run_shell(f"'{docker_executable}' compose version")
+ except subprocess.CalledProcessError as e:
+ raise RuntimeError(
+ "Docker Compose must be installed in order to use IREE VS Code "
+ "Development Container. Please refer to: "
+ "https://docs.docker.com/compose/") from e
+
+ docker_image_key = "base"
+
+ # ------------------------------------------------------------------------- #
+ # Mandatory Steps #
+ # ------------------------------------------------------------------------- #
+
+ # STEP 1: Verify the user doesn't have a pre-existing `docker-compose.yml`.
+ # If yes, confirm they want to overwrite it.
+ if os.path.isfile(CURRENT_DIR / "docker-compose.yml"):
+ if get_input(
+ "A `docker-compose.yml` already exists. Are you certain you want to overwrite it [y/N]?",
+ default_answer="n",
+ accepted_answers=["y", "n"]) == "n":
+ sys.exit(1)
+
+ # STEP 2: Read the template configuration file
+ with open(CURRENT_DIR / "docker-compose.base.yml") as f:
+ docker_config = json.load(f)
+
+ # STEP 3: Prebuilt vs Locally Built Containers
+ use_official_img = get_input(
+ "Do you wish to use the official prebuild development containers [Y/n]?",
+ default_answer="y",
+ accepted_answers=["y", "n"]) == "y"
+
+ # ------------------------------------------------------------------------- #
+ # Optional Mounting Points - Build & Cache #
+ # ------------------------------------------------------------------------- #
+
+ # STEP 4 [OPTIONAL]: Does the user want to mount a directory for CCACHE ?
+ ccache_mount_cache = get_input_path(
+ "[OPTIONAL] Enter the the directory path to mount as CCache Cache.\n"
+ " It will be mounted in the container under: `/home/iree/.cache/ccache` [Default: None]:"
+ )
+ if ccache_mount_cache:
+ docker_config["services"]["iree-dev"]["volumes"].append(
+ f"{ccache_mount_cache}:/home/iree/.cache/ccache:cached")
+
+ # STEP 5 [OPTIONAL]: Does the user want to mount a directory for BUILD ?
+ build_mount_cache = get_input_path(
+ "[OPTIONAL] Enter the the directory path to mount as Build Directory.\n"
+ " It will mounted in the container under: `/home/iree/build` [Default: None]:"
+ )
+ if build_mount_cache:
+ docker_config["services"]["iree-dev"]["volumes"].append(
+ f"{build_mount_cache}:/home/iree/build:cached")
+
+ # ------------------------------------------------------------------------- #
+ # Optional Deep Learning Accelerator Support #
+ # ------------------------------------------------------------------------- #
+
+ # STEP 6 [OPTIONAL]: Does the user want to use NVIDIA GPUs ?
+ nvidia_gpu_available, err_msg = is_nvidia_gpu_available()
+
+ if nvidia_gpu_available:
+ if get_input("[OPTIONAL] Do you wish to use NVIDIA GPU [y/N]?",
+ default_answer="n",
+ accepted_answers=["y", "n"]) == "y":
+ docker_image_key = "nvidia"
+
+ else:
+ print(f"[INFO] NVIDIA GPUs are not available for use: {err_msg}")
+
+ if docker_image_key != "nvidia":
+ del docker_config["services"]["iree-dev"]["deploy"]
+
+ # ------------------------------------------------------------------------- #
+ # Setting the right docker image / container to build #
+ # ------------------------------------------------------------------------- #
+
+ docker_img_shortname = DOCKER_IMAGE_SHORTNAME_DICT[docker_image_key]
+
+ if use_official_img:
+ del docker_config["services"]["iree-dev"]["build"]
+
+ docker_iree_registry = SourceFileLoader(
+ "docker_iree_registry",
+ str(CURRENT_DIR /
+ "../build_tools/docker/get_image_name.py")).load_module()
+
+ image_name = docker_iree_registry.find_image_by_name(docker_img_shortname)
+ docker_config["services"]["iree-dev"]["image"] = image_name
+
+ else:
+ del docker_config["services"]["iree-dev"]["image"]
+
+ dockerfile = f"build_tools/docker/dockerfiles/{docker_img_shortname}.Dockerfile"
+ docker_config["services"]["iree-dev"]["build"]["dockerfile"] = dockerfile
+
+ if (not os.path.isfile(CURRENT_DIR / ".." / dockerfile)):
+ raise FileNotFoundError(f"The file `{dockerfile}` does not exist.")
+
+ docker_compose_filepath = os.path.join(CURRENT_DIR, "docker-compose.yml")
+ with open(docker_compose_filepath, "w") as f:
+ json.dump(docker_config, f, indent=2)
+
+ # ------------------------------------------------------------------------- #
+ # SUCCESS #
+ # ------------------------------------------------------------------------- #
+
+ print("\n" + "=" * 80)
+ print(f"\nSuccess! Wrote Docker Compose file to `{docker_compose_filepath}`.")
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
new file mode 100644
index 0000000..e9c762e
--- /dev/null
+++ b/.devcontainer/devcontainer.json
@@ -0,0 +1,45 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/docker-existing-dockerfile
+{
+ "name": "IREE Development Container",
+ "dockerComposeFile": [
+ "docker-compose.yml"
+ ],
+ "service": "iree-dev",
+ "remoteUser": "iree",
+ "workspaceFolder": "/workspaces/iree",
+ "customizations": {
+ "vscode": {
+ "extensions": [
+ // LLVM & MLIR
+ "llvm-vs-code-extensions.vscode-clangd",
+ "llvm-vs-code-extensions.vscode-mlir",
+ "jakob-erzar.llvm-tablegen",
+ // CPP
+ "ms-vscode.cpptools-extension-pack",
+ // Python
+ "ms-python.python",
+ // Build Tools
+ "ms-azuretools.vscode-docker",
+ "ms-vscode.makefile-tools",
+ "ms-vscode.cmake-tools",
+ // Git & Github
+ "GitHub.vscode-pull-request-github"
+ ]
+ }
+ },
+ "features": {
+ "ghcr.io/devcontainers/features/common-utils:2": {
+ "username": "iree",
+ "uid": "automatic",
+ "gid": "automatic",
+ "installZsh": true,
+ "installOhMyZsh": true,
+ "configureZshAsDefaultShell": false,
+ "upgradePackages": true
+ },
+ "ghcr.io/devcontainers/features/git:1": {
+ "version": "latest"
+ }
+ }
+}
diff --git a/.devcontainer/docker-compose.base.yml b/.devcontainer/docker-compose.base.yml
new file mode 100644
index 0000000..ba6b5da2
--- /dev/null
+++ b/.devcontainer/docker-compose.base.yml
@@ -0,0 +1,42 @@
+{
+ "version": "3.9",
+ "services": {
+ "iree-dev": {
+ "image": null,
+ "build": {
+ "context": "../",
+ "network": "host",
+ "dockerfile": null
+ },
+ "network_mode": "host",
+ "ipc": "host",
+ "cap_add": [
+ "SYS_PTRACE",
+ "SYS_ADMIN"
+ ],
+ "security_opt": [
+ "seccomp:unconfined"
+ ],
+ "deploy": {
+ "resources": {
+ "reservations": {
+ "devices": [
+ {
+ "driver": "nvidia",
+ "count": 1,
+ "capabilities": [
+ "gpu"
+ ]
+ }
+ ]
+ }
+ }
+ },
+ "volumes": [
+ "..:/workspaces/iree:cached",
+ "~/.gitconfig:/etc/gitconfig:cached"
+ ],
+ "command": "/bin/sh -c \"while sleep 1000; do :; done\""
+ }
+ }
+}
diff --git a/.devcontainer/docs_data/command_palette.png b/.devcontainer/docs_data/command_palette.png
new file mode 100644
index 0000000..b787ec9
--- /dev/null
+++ b/.devcontainer/docs_data/command_palette.png
Binary files differ
diff --git a/.devcontainer/docs_data/dev_container_setup.png b/.devcontainer/docs_data/dev_container_setup.png
new file mode 100644
index 0000000..e4f3302
--- /dev/null
+++ b/.devcontainer/docs_data/dev_container_setup.png
Binary files differ
diff --git a/.devcontainer/docs_data/dev_container_start.png b/.devcontainer/docs_data/dev_container_start.png
new file mode 100644
index 0000000..c0c260e
--- /dev/null
+++ b/.devcontainer/docs_data/dev_container_start.png
Binary files differ
diff --git a/.devcontainer/docs_data/dev_container_terminal.png b/.devcontainer/docs_data/dev_container_terminal.png
new file mode 100644
index 0000000..f7969d8
--- /dev/null
+++ b/.devcontainer/docs_data/dev_container_terminal.png
Binary files differ
diff --git a/.gitignore b/.gitignore
index d2056f0..569d140 100644
--- a/.gitignore
+++ b/.gitignore
@@ -72,5 +72,8 @@
*.tar
*.tar.*
+# VS Code DevContainer
+.devcontainer/docker-compose.yml
+
# Local cache files
llvm-external-projects/iree-dialects/.cache
diff --git a/build_tools/docker/get_image_name.py b/build_tools/docker/get_image_name.py
index bf99339..ae8e180 100755
--- a/build_tools/docker/get_image_name.py
+++ b/build_tools/docker/get_image_name.py
@@ -20,19 +20,24 @@
from pathlib import Path
import sys
+
+def find_image_by_name(img_name):
+ this_dir = Path(__file__).resolve().parent
+
+ with open(this_dir / "prod_digests.txt", "rt") as f:
+ for line in f.readlines():
+ line = line.strip()
+ if line.startswith(f"gcr.io/iree-oss/{img_name}@"):
+ return line
+ else:
+ raise ValueError(
+ f"ERROR: Image name {img_name} not found in prod_digests.txt")
+
+
if __name__ == "__main__":
if len(sys.argv) != 2:
print("ERROR: Expected image short name", file=sys.stderr)
sys.exit(1)
short_name = sys.argv[1]
- this_dir = Path(__file__).resolve().parent
- with open(this_dir / "prod_digests.txt", "rt") as f:
- for line in f.readlines():
- line = line.strip()
- if line.startswith(f"gcr.io/iree-oss/{short_name}@"):
- print(line)
- break
- else:
- print(f"ERROR: Image name {short_name} not found in prod_digests.txt",
- file=sys.stderr)
- sys.exit(1)
+ image_name = find_image_by_name(short_name)
+ print(image_name)
