| commit | f022d29ad6d9c9ba793e08e529c0472a6b43af12 | [log] [tgz] |
|---|---|---|
| author | Ben Vanik <ben.vanik@gmail.com> | Tue Aug 15 18:51:46 2023 -0700 |
| committer | GitHub <noreply@github.com> | Wed Aug 16 01:51:46 2023 +0000 |
| tree | 926ef1fba7700666224cbdf89f3aabca1f082708 | |
| parent | d4e17edccf4f4177230761ee0484f8865c415e7b [diff] |
Reworking constant upload with a HAL file API. (#14665) This changes the compiler-generated map/staging IR into file reads that are managed by the runtime HAL targets. In this initial version only one type of file is supported that wraps host memory and exposes it via the file API. For targets that don't natively support file streaming a utility for asynchronously streaming files using an iree_loop_t is available and all targets currently use that. In the future targets like CUDA/Metal/Direct3D12 can use cuFile/MTLIOCommandBuffer/DirectStorage to directly stream file contents from/to disk. Future changes will expand the iree_hal_file_t vtable with what can be reliably implemented or guarded with feature flags. When any form of asynchronous loop is implemented (such as a task-system fiber loop) we'll be able to perform overlapped copies to reduce total transfer latency by letting the CPU and GPU do their respective staging operations concurrently. Today only synchronous loops are used so it's fully serialized. Now that the compiler doesn't emit staging with iree_allocator_allocate_buffer's initial_data that can be removed in #14605 to remove the primary use of the existing buffer_transfer util. The remaining places using it can be switched to memory file streaming instead such as numpy IO and support much larger content. This is a breaking HAL change and all existing VMFBs will fail to load on newer runtimes. CPU and CUDA targets benefit from this as should all other backends though some may need to have zero-copy paths added - they'll at least benefit from streaming and bounded staging buffer sizes. Vulkan has been optimized a bit and the follow-on sparse bindings support improves it further. Vulkan dedicated GPU w/ 1GB model before this change (1024MB + 1024MB of staging, even when mmapped):  Vulkan dedicated GPU w/ 1GB model now (1024MB + ~64MB of staging used as streamed from a memory mapped file):  Vulkan integrated GPU w/ 1GB model now (1024MB of host memory imported and used directly with no copies):  Progress on #14607.
IREE (Intermediate Representation Execution Environment, pronounced as “eerie”) is an MLIR-based end-to-end compiler and runtime that lowers Machine Learning (ML) models to a unified IR that scales up to meet the needs of the datacenter and down to satisfy the constraints and special considerations of mobile and edge deployments.
See our website for project details, user guides, and instructions on building from source.
IREE is still in its early phase. We have settled down on the overarching infrastructure and are actively improving various software components as well as project logistics. It is still quite far from ready for everyday use and is made available without any support at the moment. With that said, we welcome any kind of feedback on any communication channels!
See our website for more information.
IREE is licensed under the terms of the Apache 2.0 License with LLVM Exceptions. See LICENSE for more information.