| commit | 392a4475a4dc74f8cfa72fa1a096be15daeec3af | [log] [tgz] |
|---|---|---|
| author | Ben Vanik <ben.vanik@gmail.com> | Thu Aug 18 13:49:02 2022 -0700 |
| committer | GitHub <noreply@github.com> | Thu Aug 18 13:49:02 2022 -0700 |
| tree | 6950d109953fc3cd7ba718c5251bd15e99b59b3e | |
| parent | ab47135693e1a1bc5648e8a7143435f7887de449 [diff] | |
| parent | 2696589525b82e4dd4a4e0b8d6e963cc69fe9b25 [diff] |
Merge pull request #9945 from iree-org/benvanik-hal-inline This adds two new semi-experimental execution models: `inline-static` and `inline-dynamic`. Unlike the full `async-*` models we use by default with command buffers, submission queues, and fences these execute as if the host and device are the same (local in-process shared memory CPU) and running synchronously. This change to synchronous behavior allows us to elide nearly the entirety of the HAL and only keep around a minimal surface area used for passing buffers/views across the ABI. Even when executing synchronously there are still reasons to use the full HAL (ability to sandbox, replay optimized command buffers, interop with hosting applications using async behavior, etc) but in very tiny deployments not pulling all of that in is a massive savings. The `inline-static` mode works with the `vmvx-inline` HAL target to produce a completely inlined VM program using the vmvx module for codegen translation. The compiler flow is nearly identical to the full HAL only when lowering out of the stream dialect we switch to the minimal `hal_inline` dialect and inline the codegen-produced executable contents right into the module. Example iree-compile flags for static inline vmvx (note `vmvx-inline` is the only supported backend!): ``` --iree-execution-model=inline-static --iree-hal-target-backends=vmvx-inline ``` Example `unidirectional_lstm.mlir`: https://gist.github.com/benvanik/216e3963a464d278192b6d5cbd8ee10b The `inline-dynamic` mode is mostly the same but instead of inlining the executables it leaves them as dynamically loadable executables and reuses the HAL's local executable loader infrastructure. This means any codegen target that we can load via the full HAL when running locally can also be used here, such as embedded elfs, system dynamic libraries, vmvx, static libraries, or any future things (wasm/etc). For dynamic inline codegen, using LLVM codegen for producing ELFs and then loading them at runtime: ``` --iree-execution-model=inline-dynamic --iree-hal-target-backends=llvm-cpu ``` Example `unidirectional_lstm.mlir`: https://gist.github.com/benvanik/7ee9879875c8eb1bf5be85e2b2d2cf0f More extensive testing is required as well as examples showing how to use this with emitc and the new vmvx microkernels. This initial submission is enough for prototyping as the compiler portions are mostly done. There's remaining optimization work required but most of that is done at the stream dialect level (reducing copies/etc) and not specific to this work.
IREE (Intermediate Representation Execution Environment, pronounced as “eerie”) is an MLIR-based end-to-end compiler and runtime that lowers Machine Learning (ML) models to a unified IR that scales up to meet the needs of the datacenter and down to satisfy the constraints and special considerations of mobile and edge deployments.
See our website for project details, user guides, and instructions on building from source.
IREE is still in its early phase. We have settled down on the overarching infrastructure and are actively improving various software components as well as project logistics. It is still quite far from ready for everyday use and is made available without any support at the moment. With that said, we welcome any kind of feedback on any communication channels!
See our website for more information.
IREE is licensed under the terms of the Apache 2.0 License with LLVM Exceptions. See LICENSE for more information.