Add MLIR dialects to website "Reference" section. (#14117)
Progress on https://github.com/openxla/iree/issues/5477
This adds a new subsection to the "Reference" section of our website for
MLIR dialect documentation:
Preview: https://scotttodd.github.io/iree/reference/mlir-dialects/
## Rationale / background
We have a diverse set of "users" in our website's target audience.
Historically, I've considered MLIR to be an implementation detail of the
IREE compiler. However, we're seeing more collaboration with developers
throughout the compiler ecosystem, and MLIR is the common language used
in that ecosystem. The newly added "Reference" section of the website
gives pages with no direct relationship with a user journey like these a
natural place to live.
## CI details
Generating markdown files from tablegen (.td) files requires fetching
the LLVM submodule and running CMake. I've added a script to do that and
included some notes in the files about where execution time is spent.
Here is a trial run on my fork:
https://github.com/ScottTodd/iree/actions/runs/5271864329/jobs/9533338273
(45s getting submodules, 30s installing deps, 3m30s building
`iree-tblgen`).
Website generation will succeed without running the script, but mkdocs
will warn about 404s / broken links.
diff --git a/docs/website/README.md b/docs/website/README.md
index 0cb3b7b..6f777b2 100644
--- a/docs/website/README.md
+++ b/docs/website/README.md
@@ -11,9 +11,20 @@
Follow <https://squidfunk.github.io/mkdocs-material/getting-started/> and read
<https://www.mkdocs.org/>.
-Develop (from this folder):
+All steps below are from this docs/website/ folder.
+
+Setup (as needed):
```shell
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+Develop:
+
+```shell
+./generate_extra_files.sh
mkdocs serve
```
@@ -21,6 +32,8 @@
* This force pushes to `gh-pages` on `<your remote>`. Please don't push to the
main repository :)
+* The `publish_website.yml` workflow takes care of publishing from the central
+ repository automatically
```shell
mkdocs gh-deploy --remote-name <your remote>
diff --git a/docs/website/docs/reference/index.md b/docs/website/docs/reference/index.md
index 09fa6e4..2c20a04 100644
--- a/docs/website/docs/reference/index.md
+++ b/docs/website/docs/reference/index.md
@@ -1,6 +1,6 @@
# Reference pages
-## API Bindings
+## API bindings
!!! info ""
@@ -16,6 +16,13 @@
* [Runtime TensorFlow Lite bindings](./bindings/tensorflow-lite.md)
+## MLIR dialects
+
+Automatically generated documentation for the MLIR dialects defined in the IREE
+repository.
+
+* [Index page](./mlir-dialects/index.md)
+
## Other topics
* [Glossary](./glossary.md)
diff --git a/docs/website/docs/reference/mlir-dialects/.gitignore b/docs/website/docs/reference/mlir-dialects/.gitignore
new file mode 100644
index 0000000..0595efe
--- /dev/null
+++ b/docs/website/docs/reference/mlir-dialects/.gitignore
@@ -0,0 +1,5 @@
+# Default to excluding the generated dialect .md files
+*.md
+
+# Include the hand-authored index page.
+!index.md
diff --git a/docs/website/docs/reference/mlir-dialects/index.md b/docs/website/docs/reference/mlir-dialects/index.md
new file mode 100644
index 0000000..c37f06c
--- /dev/null
+++ b/docs/website/docs/reference/mlir-dialects/index.md
@@ -0,0 +1,20 @@
+# MLIR dialects
+
+These pages contain automatically generated documentation for the MLIR dialects
+defined in the IREE repository. IREE also makes extensive use of dialects from
+the upstream MLIR repository, which are documented at
+[https://mlir.llvm.org/docs/Dialects/](https://mlir.llvm.org/docs/Dialects/).
+
+Dialect | Description
+--------------------------- | -----------
+[Check](./Check.md) | Defines assertions for IREE tests
+[Flow](./Flow.md) | Models execution data flow and partitioning
+[HAL](./HAL.md) | Represents operations against the IREE HAL[^1]
+[HALInline](./HALInline.md) | Inline HAL interop runtime module dialect
+[HALLoader](./HALLoader.md) | HAL inline executable loader runtime module dialect
+[Stream](./Stream.md) | Model execution partitioning and scheduling
+[Util](./Util.md) | Types and ops common across IREE subdialects
+[VM](./VM.md) | Represents operations against an abstract virtual machine
+[VMVX](./VMVX.md) | Virtual Machine Vector Extensions
+
+[^1]: Hardware Abstraction Layer
diff --git a/docs/website/generate_extra_files.sh b/docs/website/generate_extra_files.sh
new file mode 100755
index 0000000..6005093
--- /dev/null
+++ b/docs/website/generate_extra_files.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+# Copyright 2023 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Generates documentation files that require extra steps outside of mkdocs.
+#
+# Typical usage:
+# docs/website$ ./generate_extra_files.sh
+# docs/website$ ./mkdocs serve
+#
+# This script should be re-run whenever the generated files change, e.g. when
+# updating MLIR dialect .td files.
+
+set -xeuo pipefail
+
+THIS_DIR="$(cd $(dirname $0) && pwd)"
+REPO_ROOT="$(cd $THIS_DIR/../../ && pwd)"
+BUILD_DIR="${BUILD_DIR:-${REPO_ROOT}/build-docs}"
+
+source ${REPO_ROOT}/build_tools/cmake/setup_build.sh
+source ${REPO_ROOT}/build_tools/cmake/setup_ccache.sh
+
+# Build `iree-doc` CMake target. This requires the LLVM submodule and can take
+# several minutes, as it builds `iree-tblgen`.
+cmake -G Ninja \
+ -B "${BUILD_DIR}" "${REPO_ROOT}" \
+ -DIREE_BUILD_DOCS=ON
+cmake --build "${BUILD_DIR}" --target iree-doc
+
+# Copy from build directory -> source directory (files are .gitignore'd).
+cp -r \
+ "${BUILD_DIR}/doc/Dialects/." \
+ "${THIS_DIR}/docs/reference/mlir-dialects/"
+
+# Delete sample dialects.
+rm "${THIS_DIR}/docs/reference/mlir-dialects/SimpleIODialect.md"
+
+# Trim "Dialect" suffix from file names, e.g. FlowDialect.md -> Flow.md.
+for f in ${THIS_DIR}/docs/reference/mlir-dialects/*Dialect.md; do
+ mv "$f" "${f/%Dialect.md/.md}"
+done
+
+# Note: any post-processing on the .md files could go here.
diff --git a/docs/website/mkdocs.yml b/docs/website/mkdocs.yml
index 72d40af..959b8ec 100644
--- a/docs/website/mkdocs.yml
+++ b/docs/website/mkdocs.yml
@@ -104,6 +104,7 @@
- tables
- toc: # Table of Contents
permalink: 'link' # Use Material font's "link" icon; see openxla.css
+ toc_depth: 3
# Navigation with explicit ordering and nesting.
# https://www.mkdocs.org/user-guide/configuration/#nav
@@ -137,6 +138,18 @@
- Runtime C API: "reference/bindings/c-api.md"
- Python: "reference/bindings/python.md"
- TensorFlow Lite: "reference/bindings/tensorflow-lite.md"
+ - "MLIR dialects":
+ - "reference/mlir-dialects/index.md"
+ # Note: these files are generated by generate_extra_files.sh.
+ - Check: "reference/mlir-dialects/Check.md"
+ - Flow: "reference/mlir-dialects/Flow.md"
+ - HAL: "reference/mlir-dialects/HAL.md"
+ - HALInline: "reference/mlir-dialects/HALInline.md"
+ - HALLoader: "reference/mlir-dialects/HALLoader.md"
+ - Stream: "reference/mlir-dialects/Stream.md"
+ - Util: "reference/mlir-dialects/Util.md"
+ - VM: "reference/mlir-dialects/VM.md"
+ - VMVX: "reference/mlir-dialects/VMVX.md"
- "Other topics":
- Glossary: "reference/glossary.md"
- Optimization options: "reference/optimization-options.md"
diff --git a/docs/website/requirements.txt b/docs/website/requirements.txt
new file mode 100644
index 0000000..887d1fd
--- /dev/null
+++ b/docs/website/requirements.txt
@@ -0,0 +1,4 @@
+# TODO(scotttodd): pin versions to control how often we pull in updates?
+
+mkdocs-material
+mkdocs-redirects
