Migrate GPU test jobs to pkgci. (#18007)
Progress on https://github.com/iree-org/iree/issues/16203. Depends on
https://github.com/iree-org/iree/pull/18000.
These jobs used to use the 3.2GB install directory produced by `cmake
--build full-build-dir --target install` in the `build_all` job. Now
they use the 73MB Python packages produced by `python -m pip wheel
runtime/` and `python -m pip wheel compiler/` in the `build_packages`
job. Python packages are what we expect users to consume, so test jobs
should use them too.
* Note that the Python packages will be larger once we enable asserts
and/or debug symbols in them. These tests may also fail with less useful
error messages and callstacks as a result of this change until that is
fixed.
I tried to keep changes to the workflow jobs minimal for now. Once the
migrations are further along we can cut out some of the remaining layers
of scripts / Dockerfiles. As before, these jobs are all opt-in on
presubmit (always running on LLVM integrate PRs or PRs affecting
NVGPU/AMDGPU code). Diffs between previous jobs and new jobs to confirm
how similar they are:
Job name | Logs before | Logs after | Notes
-- | -- | -- | --
`test_nvidia_t4` | [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102664951/job/27939423899)
| [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102841435/job/27939725841?pr=18007)
| 433 tests -> 430 tests<br>skipping
`samples/custom_dispatch/vulkan/shaders`<br>`IREE
custom_dispatch/vulkan/shaders ignored -- glslc not found`<br>(no longer
running under Docker)
`test_amd_mi250` | [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102664951/job/27939423747)
| [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102841435/job/27939725347?pr=18007)
| 138 tests before/after
`test_amd_mi300` | [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102664951/job/27939424223)
| [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102841435/job/27939725525?pr=18007)
| 141 tests before/after
`test_amd_w7900` | [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102664951/job/27939424084)
| [workflow
logs](https://github.com/iree-org/iree/actions/runs/10102841435/job/27939725679?pr=18007)
| 289 tests before/after
Each job is now included from its own standalone workflow file, allowing
for testing of individual workflows using `workflow_dispatch` triggers.
I have some other ideas for further decoupling these optional jobs from
the core workflow(s).
ci-extra: test_amd_mi250, test_amd_mi300, test_amd_w7900, test_nvidia_t4
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 8be9d44..cd82af4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -89,70 +89,6 @@
gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \
./build_tools/bazel/build_test_all.sh
- test_nvidia_gpu:
- needs: [setup, build_all]
- if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_nvidia_gpu')
- env:
- BUILD_DIR: build-tests
- INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }}
- INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }}
- INSTALL_DIR_GCS_URL: ${{ needs.build_all.outputs.install-dir-gcs-url }}
- IREE_CPU_DISABLE: 1
- IREE_VULKAN_DISABLE: 0
- IREE_CUDA_DISABLE: 0
- IREE_HIP_DISABLE: 1
- runs-on:
- - self-hosted # must come first
- - runner-group=${{ needs.setup.outputs.runner-group }}
- - environment=${{ needs.setup.outputs.runner-env }}
- - gpu
- - os-family=Linux
- steps:
- - name: "Checking out repository"
- uses: actions/checkout@v4.1.7
- - name: "Checking out runtime submodules"
- run: ./build_tools/scripts/git/update_runtime_submodules.sh
- - name: Querying GPU information
- run: |
- ./build_tools/scripts/check_cuda.sh
- ./build_tools/scripts/check_vulkan.sh
- - name: "Downloading install dir archive"
- run: wget "${INSTALL_DIR_GCS_URL}" -O "${INSTALL_DIR_ARCHIVE}"
- - name: "Extracting install directory"
- run: tar -xf "${INSTALL_DIR_ARCHIVE}"
- - name: "Building tests"
- run: |
- ./build_tools/github_actions/docker_run.sh \
- --env IREE_CPU_DISABLE \
- --env IREE_VULKAN_DISABLE \
- --env IREE_CUDA_DISABLE \
- --env IREE_HIP_DISABLE \
- gcr.io/iree-oss/nvidia@sha256:433e072f075f90fdf07471673626b17091d8d8e2395626f1e6ac6c98803c8807 \
- ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}
- - name: "Running GPU tests"
- env:
- IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=cuda$
- IREE_NVIDIA_SM80_TESTS_DISABLE: 1
- IREE_MULTI_DEVICE_TESTS_DISABLE: 1
- run: |
- ./build_tools/github_actions/docker_run.sh \
- --env IREE_VULKAN_DISABLE \
- --env IREE_CUDA_DISABLE \
- --env IREE_HIP_DISABLE \
- --env IREE_CTEST_LABEL_REGEX \
- --env IREE_NVIDIA_SM80_TESTS_DISABLE \
- --env IREE_MULTI_DEVICE_TESTS_DISABLE \
- --env IREE_VULKAN_F16_DISABLE=0 \
- --env IREE_NVIDIA_GPU_TESTS_DISABLE=0 \
- --env CTEST_PARALLEL_LEVEL=2 \
- --env NVIDIA_DRIVER_CAPABILITIES=all \
- --gpus all \
- gcr.io/iree-oss/nvidia@sha256:433e072f075f90fdf07471673626b17091d8d8e2395626f1e6ac6c98803c8807 \
- bash -euo pipefail -c \
- "./build_tools/scripts/check_cuda.sh
- ./build_tools/scripts/check_vulkan.sh
- ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}"
-
# Disabled while runner is offline.
# test_nvidia_a100:
# needs: [setup, build_all]
@@ -193,7 +129,7 @@
# --env IREE_CUDA_DISABLE \
# --env IREE_HIP_DISABLE \
# gcr.io/iree-oss/nvidia@sha256:433e072f075f90fdf07471673626b17091d8d8e2395626f1e6ac6c98803c8807 \
- # ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}
+ # ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}/bin
# - name: "Running GPU tests"
# env:
# IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80|^requires-gpu|^driver=vulkan$|^driver=cuda$
@@ -218,129 +154,6 @@
# ./build_tools/scripts/check_vulkan.sh
# ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}"
- test_amd_mi250:
- needs: [setup, build_all]
- if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi250')
- env:
- BUILD_DIR: build-tests
- INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }}
- INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }}
- INSTALL_DIR_GCS_URL: ${{ needs.build_all.outputs.install-dir-gcs-url }}
- IREE_CPU_DISABLE: 1
- IREE_VULKAN_DISABLE: 1
- IREE_CUDA_DISABLE: 1
- IREE_HIP_DISABLE: 0
- IREE_HIP_TEST_TARGET_CHIP: "gfx90a"
- runs-on: nodai-amdgpu-mi250-x86-64
- steps:
- - name: "Checking out repository"
- uses: actions/checkout@v4.1.7
- - name: "Checking out runtime submodules"
- run: ./build_tools/scripts/git/update_runtime_submodules.sh
- - name: "Downloading install dir archive"
- run: wget "${INSTALL_DIR_GCS_URL}" -O "${INSTALL_DIR_ARCHIVE}"
- - name: "Extracting install directory"
- run: tar -xf "${INSTALL_DIR_ARCHIVE}"
- - name: "Building tests"
- run: |
- ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}
- - name: "Running GPU tests"
- env:
- IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$
- IREE_NVIDIA_SM80_TESTS_DISABLE: 1
- IREE_MULTI_DEVICE_TESTS_DISABLE: 0
- IREE_AMD_RDNA3_TESTS_DISABLE: 1
- IREE_NVIDIA_GPU_TESTS_DISABLE: 0
- IREE_CUDA_DISABLE: 1
- IREE_CPU_DISABLE: 1
- IREE_HIP_DISABLE: 0
- run: |
- ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
-
- test_amd_mi300:
- needs: [setup, build_all]
- if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi300')
- env:
- BUILD_DIR: build-tests
- INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }}
- INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }}
- INSTALL_DIR_GCS_URL: ${{ needs.build_all.outputs.install-dir-gcs-url }}
- IREE_CPU_DISABLE: 1
- IREE_VULKAN_DISABLE: 1
- IREE_CUDA_DISABLE: 1
- IREE_HIP_DISABLE: 0
- IREE_HIP_TEST_TARGET_CHIP: "gfx942"
- LD_LIBRARY_PATH: /home/esaimana/Python-3.11.9
- runs-on: nodai-amdgpu-mi300-x86-64
- steps:
- - name: Pre Checkout MI300 Step
- if: contains(matrix.name, 'gfx942')
- run: |
- sudo chmod -R 777 ~/actions-runner/_work
- - name: "Checking out repository"
- uses: actions/checkout@v4.1.7
- - name: "Checking out runtime submodules"
- run: ./build_tools/scripts/git/update_runtime_submodules.sh
- - name: "Downloading install dir archive"
- run: wget "${INSTALL_DIR_GCS_URL}" -O "${INSTALL_DIR_ARCHIVE}"
- - name: "Extracting install directory"
- run: tar -xf "${INSTALL_DIR_ARCHIVE}"
- - name: "Building tests"
- run: |
- ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}
- - name: "Running GPU tests"
- env:
- IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$
- IREE_NVIDIA_SM80_TESTS_DISABLE: 1
- IREE_MULTI_DEVICE_TESTS_DISABLE: 0
- IREE_AMD_RDNA3_TESTS_DISABLE: 1
- IREE_NVIDIA_GPU_TESTS_DISABLE: 0
- IREE_CUDA_DISABLE: 1
- IREE_CPU_DISABLE: 1
- IREE_HIP_DISABLE: 0
- run: |
- ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
-
- test_amd_w7900:
- needs: [setup, build_all]
- if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_w7900')
- env:
- BUILD_DIR: build-tests
- INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }}
- INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }}
- INSTALL_DIR_GCS_URL: ${{ needs.build_all.outputs.install-dir-gcs-url }}
- IREE_CPU_DISABLE: 1
- IREE_VULKAN_DISABLE: 0
- IREE_CUDA_DISABLE: 1
- IREE_HIP_DISABLE: 0
- IREE_HIP_TEST_TARGET_CHIP: "gfx1100"
- runs-on: nodai-amdgpu-w7900-x86-64
- steps:
- - name: "Checking out repository"
- uses: actions/checkout@v4.1.7
- - name: "Checking out runtime submodules"
- run: ./build_tools/scripts/git/update_runtime_submodules.sh
- - name: "Downloading install dir archive"
- run: wget "${INSTALL_DIR_GCS_URL}" -O "${INSTALL_DIR_ARCHIVE}"
- - name: "Extracting install directory"
- run: tar -xf "${INSTALL_DIR_ARCHIVE}"
- - name: "Building tests"
- run: |
- ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}
- - name: "Running GPU tests"
- env:
- CTEST_PARALLEL_LEVEL: 1
- IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=hip$
- IREE_AMD_RDNA3_TESTS_DISABLE: 0
- IREE_NVIDIA_GPU_TESTS_DISABLE: 0
- IREE_NVIDIA_SM80_TESTS_DISABLE: 1
- IREE_MULTI_DEVICE_TESTS_DISABLE: 0
- IREE_CUDA_DISABLE: 1
- IREE_CPU_DISABLE: 1
- IREE_HIP_DISABLE: 0
- run: |
- ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
-
############################### Configurations ###############################
# Jobs that build IREE in some non-default configuration
##############################################################################
@@ -606,11 +419,7 @@
- build_test_all_bazel
# Accelerators
- - test_nvidia_gpu
# - test_nvidia_a100
- - test_amd_mi250
- - test_amd_mi300
- - test_amd_w7900
# Configurations
- build_test_runtime
diff --git a/.github/workflows/pkgci.yml b/.github/workflows/pkgci.yml
index da427fa..b4827f6 100644
--- a/.github/workflows/pkgci.yml
+++ b/.github/workflows/pkgci.yml
@@ -49,6 +49,30 @@
if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'regression_test')
uses: ./.github/workflows/pkgci_regression_test.yml
+ test_amd_mi250:
+ name: Test AMD MI250
+ needs: [setup, build_packages]
+ if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi250')
+ uses: ./.github/workflows/pkgci_test_amd_mi250.yml
+
+ test_amd_mi300:
+ name: Test AMD MI300
+ needs: [setup, build_packages]
+ if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi300')
+ uses: ./.github/workflows/pkgci_test_amd_mi300.yml
+
+ test_amd_w7900:
+ name: Test AMD W7900
+ needs: [setup, build_packages]
+ if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_w7900')
+ uses: ./.github/workflows/pkgci_test_amd_w7900.yml
+
+ test_nvidia_t4:
+ name: Test NVIDIA T4
+ needs: [setup, build_packages]
+ if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_nvidia_t4')
+ uses: ./.github/workflows/pkgci_test_nvidia_t4.yml
+
test_tensorflow_cpu:
name: Test TensorFlow CPU
needs: [setup, build_packages]
diff --git a/.github/workflows/pkgci_test_amd_mi250.yml b/.github/workflows/pkgci_test_amd_mi250.yml
new file mode 100644
index 0000000..07a3bec
--- /dev/null
+++ b/.github/workflows/pkgci_test_amd_mi250.yml
@@ -0,0 +1,62 @@
+# Copyright 2024 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+name: PkgCI Test AMD MI250
+on:
+ workflow_call:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+ workflow_dispatch:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+
+jobs:
+ test:
+ runs-on: nodai-amdgpu-mi250-x86-64
+ env:
+ PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages
+ BUILD_DIR: build-tests
+ VENV_DIR: ${{ github.workspace }}/.venv
+ IREE_CPU_DISABLE: 1
+ IREE_VULKAN_DISABLE: 1
+ IREE_CUDA_DISABLE: 1
+ IREE_HIP_DISABLE: 0
+ IREE_HIP_TEST_TARGET_CHIP: "gfx90a"
+ steps:
+ - name: Checking out repository
+ uses: actions/checkout@v4.1.7
+ with:
+ submodules: false
+ - name: "Checking out runtime submodules"
+ run: ./build_tools/scripts/git/update_runtime_submodules.sh
+ - uses: actions/setup-python@v5.1.0
+ with:
+ # Must match the subset of versions built in pkgci_build_packages.
+ python-version: "3.11"
+ - uses: actions/download-artifact@v4.1.7
+ with:
+ name: linux_x86_64_release_packages
+ path: ${{ env.PACKAGE_DOWNLOAD_DIR }}
+ - name: Setup base venv
+ run: |
+ ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \
+ --artifact-path=${PACKAGE_DOWNLOAD_DIR} \
+ --fetch-gh-workflow=${{ inputs.artifact_run_id }}
+ - name: "Building tests"
+ run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin
+ - name: "Running GPU tests"
+ env:
+ CTEST_PARALLEL_LEVEL: 2
+ IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$
+ IREE_AMD_RDNA3_TESTS_DISABLE: 1
+ IREE_NVIDIA_GPU_TESTS_DISABLE: 0
+ IREE_NVIDIA_SM80_TESTS_DISABLE: 1
+ IREE_MULTI_DEVICE_TESTS_DISABLE: 0
+ run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
diff --git a/.github/workflows/pkgci_test_amd_mi300.yml b/.github/workflows/pkgci_test_amd_mi300.yml
new file mode 100644
index 0000000..e3bb23a
--- /dev/null
+++ b/.github/workflows/pkgci_test_amd_mi300.yml
@@ -0,0 +1,67 @@
+# Copyright 2024 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+name: PkgCI Test AMD MI300
+on:
+ workflow_call:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+ workflow_dispatch:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+
+jobs:
+ test:
+ runs-on: nodai-amdgpu-mi300-x86-64
+ env:
+ PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages
+ BUILD_DIR: build-tests
+ VENV_DIR: ${{ github.workspace }}/.venv
+ IREE_CPU_DISABLE: 1
+ IREE_VULKAN_DISABLE: 1
+ IREE_CUDA_DISABLE: 1
+ IREE_HIP_DISABLE: 0
+ IREE_HIP_TEST_TARGET_CHIP: "gfx942"
+ LD_LIBRARY_PATH: /home/esaimana/Python-3.11.9
+ steps:
+ - name: Pre Checkout MI300 Step
+ if: contains(matrix.name, 'gfx942')
+ run: |
+ sudo chmod -R 777 ~/actions-runner/_work
+ - name: Checking out repository
+ uses: actions/checkout@v4.1.7
+ with:
+ submodules: false
+ - name: "Checking out runtime submodules"
+ run: ./build_tools/scripts/git/update_runtime_submodules.sh
+ - uses: actions/setup-python@v5.1.0
+ with:
+ # Must match the subset of versions built in pkgci_build_packages.
+ python-version: "3.11"
+ - uses: actions/download-artifact@v4.1.7
+ with:
+ name: linux_x86_64_release_packages
+ path: ${{ env.PACKAGE_DOWNLOAD_DIR }}
+ - name: Setup base venv
+ run: |
+ ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \
+ --artifact-path=${PACKAGE_DOWNLOAD_DIR} \
+ --fetch-gh-workflow=${{ inputs.artifact_run_id }}
+ - name: "Building tests"
+ run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin
+ - name: "Running GPU tests"
+ env:
+ CTEST_PARALLEL_LEVEL: 2
+ IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$
+ IREE_AMD_RDNA3_TESTS_DISABLE: 1
+ IREE_NVIDIA_GPU_TESTS_DISABLE: 0
+ IREE_NVIDIA_SM80_TESTS_DISABLE: 1
+ IREE_MULTI_DEVICE_TESTS_DISABLE: 0
+ run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
diff --git a/.github/workflows/pkgci_test_amd_w7900.yml b/.github/workflows/pkgci_test_amd_w7900.yml
new file mode 100644
index 0000000..3f6024a
--- /dev/null
+++ b/.github/workflows/pkgci_test_amd_w7900.yml
@@ -0,0 +1,62 @@
+# Copyright 2024 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+name: PkgCI Test AMD W7900
+on:
+ workflow_call:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+ workflow_dispatch:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+
+jobs:
+ test:
+ runs-on: nodai-amdgpu-w7900-x86-64
+ env:
+ PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages
+ BUILD_DIR: build-tests
+ VENV_DIR: ${{ github.workspace }}/.venv
+ IREE_CPU_DISABLE: 1
+ IREE_VULKAN_DISABLE: 0
+ IREE_CUDA_DISABLE: 1
+ IREE_HIP_DISABLE: 0
+ IREE_HIP_TEST_TARGET_CHIP: "gfx1100"
+ steps:
+ - name: Checking out repository
+ uses: actions/checkout@v4.1.7
+ with:
+ submodules: false
+ - name: "Checking out runtime submodules"
+ run: ./build_tools/scripts/git/update_runtime_submodules.sh
+ - uses: actions/setup-python@v5.1.0
+ with:
+ # Must match the subset of versions built in pkgci_build_packages.
+ python-version: "3.11"
+ - uses: actions/download-artifact@v4.1.7
+ with:
+ name: linux_x86_64_release_packages
+ path: ${{ env.PACKAGE_DOWNLOAD_DIR }}
+ - name: Setup base venv
+ run: |
+ ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \
+ --artifact-path=${PACKAGE_DOWNLOAD_DIR} \
+ --fetch-gh-workflow=${{ inputs.artifact_run_id }}
+ - name: "Building tests"
+ run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin
+ - name: "Running GPU tests"
+ env:
+ CTEST_PARALLEL_LEVEL: 1
+ IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=hip$
+ IREE_AMD_RDNA3_TESTS_DISABLE: 0
+ IREE_NVIDIA_GPU_TESTS_DISABLE: 0
+ IREE_NVIDIA_SM80_TESTS_DISABLE: 1
+ IREE_MULTI_DEVICE_TESTS_DISABLE: 0
+ run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
diff --git a/.github/workflows/pkgci_test_nvidia_t4.yml b/.github/workflows/pkgci_test_nvidia_t4.yml
new file mode 100644
index 0000000..fa8d7e6
--- /dev/null
+++ b/.github/workflows/pkgci_test_nvidia_t4.yml
@@ -0,0 +1,76 @@
+# Copyright 2024 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+name: PkgCI Test NVIDIA T4
+on:
+ workflow_call:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+ workflow_dispatch:
+ inputs:
+ artifact_run_id:
+ type: string
+ default: ""
+
+jobs:
+ test:
+ runs-on:
+ - self-hosted # must come first
+ - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}
+ - environment=prod
+ - gpu
+ - os-family=Linux
+ env:
+ PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages
+ BUILD_DIR: build-tests
+ VENV_DIR: ${{ github.workspace }}/.venv
+ IREE_CPU_DISABLE: 1
+ IREE_VULKAN_DISABLE: 0
+ IREE_CUDA_DISABLE: 0
+ IREE_HIP_DISABLE: 1
+ steps:
+ - name: Checking out repository
+ uses: actions/checkout@v4.1.7
+ with:
+ submodules: false
+ - name: "Checking out runtime submodules"
+ run: ./build_tools/scripts/git/update_runtime_submodules.sh
+ - name: Install dependencies
+ run: |
+ sudo apt update
+ sudo apt install -y cmake clang ninja-build
+ export CC=clang
+ export CXX=clang
+ - name: Querying GPU information
+ run: |
+ ./build_tools/scripts/check_cuda.sh
+ ./build_tools/scripts/check_vulkan.sh
+ - uses: actions/setup-python@v5.1.0
+ with:
+ # Must match the subset of versions built in pkgci_build_packages.
+ python-version: "3.11"
+ - uses: actions/download-artifact@v4.1.7
+ with:
+ name: linux_x86_64_release_packages
+ path: ${{ env.PACKAGE_DOWNLOAD_DIR }}
+ - name: Setup base venv
+ run: |
+ ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \
+ --artifact-path=${PACKAGE_DOWNLOAD_DIR} \
+ --fetch-gh-workflow=${{ inputs.artifact_run_id }}
+ - name: "Building tests"
+ run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin
+ - name: "Running GPU tests"
+ env:
+ CTEST_PARALLEL_LEVEL: 2
+ IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=cuda$
+ IREE_NVIDIA_GPU_TESTS_DISABLE: 0
+ IREE_NVIDIA_SM80_TESTS_DISABLE: 1
+ IREE_MULTI_DEVICE_TESTS_DISABLE: 1
+ IREE_VULKAN_F16_DISABLE: 0
+ run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}
diff --git a/build_tools/cmake/ctest_all.sh b/build_tools/cmake/ctest_all.sh
index 85894d4..13df378 100755
--- a/build_tools/cmake/ctest_all.sh
+++ b/build_tools/cmake/ctest_all.sh
@@ -134,7 +134,7 @@
# TODO(#12305): figure out how to run samples with custom binary outputs
# on the CI. $IREE_BINARY_DIR may not be setup right or the object files may
-# not be getting deployed to the test_all/test_nvidia_gpu bots.
+# not be getting deployed to the test bots.
excluded_tests+=(
"iree/samples/custom_dispatch/cpu/embedded/example_hal.mlir.test"
"iree/samples/custom_dispatch/cpu/embedded/example_stream.mlir.test"
diff --git a/build_tools/cmake/iree_check_test.cmake b/build_tools/cmake/iree_check_test.cmake
index fd89caf..aa37760 100644
--- a/build_tools/cmake/iree_check_test.cmake
+++ b/build_tools/cmake/iree_check_test.cmake
@@ -211,6 +211,7 @@
"--module={{${_MODULE_FILE_NAME}}}"
${_RULE_RUNNER_ARGS}
LABELS
+ "test-type=check-test"
${_RULE_LABELS}
TIMEOUT
${_RULE_TIMEOUT}
diff --git a/build_tools/cmake/iree_lit_test.cmake b/build_tools/cmake/iree_lit_test.cmake
index f93d84d..46f8622 100644
--- a/build_tools/cmake/iree_lit_test.cmake
+++ b/build_tools/cmake/iree_lit_test.cmake
@@ -18,8 +18,8 @@
# TOOLS: Tools that should be included on the PATH
# DATA: Additional data dependencies invoked by the test (e.g. binaries
# called in the RUN line)
-# LABELS: Additional labels to apply to the test. The package path is added
-# automatically.
+# LABELS: Additional labels to apply to the test. Package path and
+# "test-type=lit-test" labels are added automatically.
function(iree_lit_test)
if(NOT IREE_BUILD_TESTS)
return()
@@ -75,6 +75,7 @@
)
list(APPEND _RULE_LABELS "${_PACKAGE_PATH}")
+ list(APPEND _RULE_LABELS "test-type=lit-test")
set_property(TEST ${_NAME_PATH} PROPERTY LABELS "${_RULE_LABELS}")
set_property(TEST ${_NAME_PATH} PROPERTY REQUIRED_FILES "${_TEST_FILE_PATH}")
set_property(TEST ${_NAME_PATH} PROPERTY ENVIRONMENT
diff --git a/build_tools/cmake/iree_run_module_test.cmake b/build_tools/cmake/iree_run_module_test.cmake
index be1037d..b16e2db 100644
--- a/build_tools/cmake/iree_run_module_test.cmake
+++ b/build_tools/cmake/iree_run_module_test.cmake
@@ -27,8 +27,9 @@
# EXPECTED_OUTPUT: A string representing the expected output from executing
# the module in the format accepted by `iree-run-module` or a file
# containing the same. Can also be an HTTPS URL to download large npy.
-# LABELS: Additional labels to apply to the test. The package path and
-# "driver=${DRIVER}" are added automatically.
+# LABELS: Additional labels to apply to the test. The package path,
+# "test-type=run-module-test", and "driver=${DRIVER}" labels are added
+# automatically.
# XFAIL_PLATFORMS: List of platforms (see iree_get_platform) for which the
# test is expected to fail. The test pass/fail status is inverted.
# UNSUPPORTED_PLATFORMS: List of platforms (see iree_get_platform) for which
diff --git a/build_tools/github_actions/configure_ci.py b/build_tools/github_actions/configure_ci.py
index 6bb7fce..b728dac 100755
--- a/build_tools/github_actions/configure_ci.py
+++ b/build_tools/github_actions/configure_ci.py
@@ -124,7 +124,7 @@
# They may also run on presubmit only under certain conditions.
DEFAULT_POSTSUBMIT_ONLY_JOBS = frozenset(
[
- "test_nvidia_gpu",
+ "test_nvidia_t4",
# "test_nvidia_a100", # Currently disabled
"test_amd_mi250",
"test_amd_mi300",
@@ -155,7 +155,7 @@
PRESUBMIT_TOUCH_ONLY_JOBS = [
# The runners with GPUs for these jobs can be unstable or in short supply,
# so limit jobs to only code paths most likely to affect the tests.
- ("test_nvidia_gpu", NVGPU_PATHS),
+ ("test_nvidia_t4", NVGPU_PATHS),
# Due to the outstock of A100, only run this test in postsubmit.
# ("test_nvidia_a100", NVGPU_PATHS),
("test_amd_mi250", AMDGPU_PATHS),
diff --git a/build_tools/pkgci/build_tests_using_package.sh b/build_tools/pkgci/build_tests_using_package.sh
index 522c0c3..e30075d 100755
--- a/build_tools/pkgci/build_tests_using_package.sh
+++ b/build_tools/pkgci/build_tests_using_package.sh
@@ -27,7 +27,7 @@
set -xeuo pipefail
-PACKAGE_DIR="$1"
+BINARY_DIR="$1"
BUILD_DIR="${BUILD_DIR:-build-tests}"
source build_tools/scripts/install_lit.sh
@@ -76,7 +76,7 @@
"-DIREE_BUILD_PYTHON_BINDINGS=OFF"
"-DIREE_BUILD_COMPILER=OFF"
"-DIREE_BUILD_ALL_CHECK_TEST_MODULES=OFF"
- "-DIREE_HOST_BIN_DIR=${PACKAGE_DIR?}/bin"
+ "-DIREE_HOST_BIN_DIR=${BINARY_DIR?}"
"-DLLVM_EXTERNAL_LIT=${LLVM_EXTERNAL_LIT?}"
)
cmake_args+=(${cmake_config_options[@]})
