commit a8d17fa96e445e3b93c70b0e6aaa24a58a7777f0
author Scott Todd <scott.todd0@gmail.com> Fri Dec 06 09:56:27 2024 -0800
committer GitHub <noreply@github.com> Fri Dec 06 17:56:27 2024 +0000
tree 6a9f4ee8ce9e2c0f2435a2fabdd0e0d10b04fe22
parent 1c73358fdc244e91b4c8e27fa0adc74c7c6c1fe3

Include prereleases from downstream projects in pip release links page. (#19391)

This fixes https://github.com/iree-org/iree/issues/19193.

Nightly releases of packages from
https://github.com/iree-org/iree-turbine will now be available from
https://iree.dev/pip-release-links.html with (for example)
```bash
pip install --pre --find-links https://iree.dev/pip-release-links.html iree-turbine
```

Previously, this relied on its own index page at
https://github.com/iree-org/iree-turbine/releases/expanded_assets/dev-wheels
. While this link will continue to work, the consolidated page is more
convenient for users.

## Projects included

I'm starting with just iree-turbine. See the notes below about other
packages.

We could still switch to a [PEP 503-compliant package
index](https://peps.python.org/pep-0503/) (usable with
`--extra-index-url`, _not_ `-f, --find-links`), like what PyTorch
[provides](https://pytorch.org/get-started/locally/) at
https://download.pytorch.org/whl/cpu. Note the 50+ packages listed
there, included mirrors of a few versions of all of PyTorch's
dependencies.

## Testing

* Publish action:
https://github.com/ScottTodd/iree/actions/runs/12188757072/job/34002585332
* Pushed to GitHub pages:
https://scotttodd.github.io/iree/pip-release-links.html
* Installed using the release links:

  ```bash
  python3.11 -m venv 3.11.venv
  source 3.11.venv/bin/activate
  
pip install --pre --find-links
https://scotttodd.github.io/iree/pip-release-links.html iree-turbine
--no-deps
# Looking in links:
https://scotttodd.github.io/iree/pip-release-links.html
  # Collecting iree-turbine
# Downloading
https://github.com/iree-org/iree-turbine/releases/download/dev-wheels/iree_turbine-3.1.0rc20241205-py3-none-any.whl
(292 kB)
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.3/292.3 kB 8.1 MB/s eta
0:00:00
  # Installing collected packages: iree-turbine
  # Successfully installed iree-turbine-3.1.0rc20241205
  ```

## Other considerations

We may want to also include downstream packages like
https://github.com/nod-ai/shark-ai (release index page:
https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels).
I'm currently trying to simplify workflows for that project ([example
here](https://github.com/nod-ai/shark-ai/blob/1763a82a5ebc002d47edb5bc53106f9b312069b9/.github/workflows/ci-shark-ai.yml#L48-L69)).

I left some notes in the workflow file too, but I could see us extending
this to also include torch-mlir and stablehlo packages. The extra
release scraping doesn't cost us much CI time, git storage on the
`gh-pages` branch, or network bandwidth on the website. I also don't
foresee any issues with making it easier to install nightly versions of
those packages. I would pause a bit if the list grows too much beyond
that though.

### Note on the `shark-ai` package

The `shark-ai` meta package (https://pypi.org/project/shark-ai/,
https://github.com/nod-ai/shark-ai/tree/main/shark-ai) pins dependencies
in ways that isn't currently compatible with this package installation
flow. Specifically, the latest nightly releases pin `shortfin==3.1.0`
which can't be resolved by a release candidate like `3.1.0rc20241205`,
so this doesn't work as one might expect:

```bash
pip install --pre --find-links https://iree.dev/pip-release-links.html shark-ai
# Successfully installed shark-ai-3.0.0

# Test with an explicit version to demonstrate the issue:
pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html \
  shark-ai==3.1.0rc20241205
# ERROR: Could not find a version that satisfies the requirement shortfin==3.1.0 (from shark-ai) (from versions: 2.9.0rc20241108, 2.9.0rc20241109, 2.9.0rc20241110, 2.9.0rc20241111, 2.9.0, 2.9.1rc20241112, 2.9.1rc20241113, 2.9.1rc20241114, 2.9.1, 2.9.2rc20241115, 2.9.2rc20241116, 2.9.2rc20241117, 2.9.2rc20241118, 2.9.2, 3.0.0rc20241118, 3.0.0, 3.1.0rc20241119, 3.1.0rc20241120, 3.1.0rc20241121, 3.1.0rc20241122, 3.1.0rc20241123, 3.1.0rc20241124, 3.1.0rc20241125, 3.1.0rc20241126, 3.1.0rc20241127, 3.1.0rc20241128, 3.1.0rc20241129, 3.1.0rc20241130, 3.1.0rc20241201, 3.1.0rc20241202, 3.1.0rc20241203, 3.1.0rc20241204, 3.1.0rc20241205)
# ERROR: No matching distribution found for shortfin==3.1.0
```