| commit | 3ef0ea1127c196029dfbd8764c94412778d592b7 | [log] [tgz] |
|---|---|---|
| author | Han-Chung Wang <hanhan0912@gmail.com> | Thu Nov 09 14:38:39 2023 -0800 |
| committer | GitHub <noreply@github.com> | Thu Nov 09 14:38:39 2023 -0800 |
| tree | 44975f7b3b1e63cf2d9ab9097e476d3a7646ab11 | |
| parent | 7be399211d324fb0028d948d161b644cf5810d48 [diff] |
[CPU] Improve distribution tile sizes selection for mmt4d ops. (#15448) Perfect distribution is a really hard problem, especially for the whole program/model. As a general rule workgroups like these should be processing thousands to tens of thousands of operations in order to amortize the scheduling overheads. Ideally each workgroup would roughly take 100us-500us; less than that and overhead starts to dominate and more than that needs to be balanced with total count: ~200 workgroups is more than enough to hide latencies and more than that adds overhead with diminishing returns. The basic heuristic we want is to spend enough time inside each workgroup to justify the overhead involved in launching it but not so much time and so few total that variance in the system can't be hidden (4 x 400ms workgroups on 3 threads will always have 400ms of work happen on one thread with the other 2 idle, or 4 x 400ms on 4 threads where 1 thread gets delayed 50ms will cause the total latency to increase 50ms, etc). In the context, `maxTileSizes` is critical because it is one of factors about number of workgroups. In IREE CPU, we've been using `defaultDistTileSize` to model tile sizes. It is used by matmul and other generic ops. In `mmt4d` semantics, some data are already tiled and moved to inner loops. To model distribution with the same factor, we should scale it down with corresponding inner tile sizes. Then the distribution of all the dispatches is modeled by the same factor. Fixes https://github.com/openxla/iree/issues/15391
IREE (Intermediate Representation Execution Environment, pronounced as “eerie”) is an MLIR-based end-to-end compiler and runtime that lowers Machine Learning (ML) models to a unified IR that scales up to meet the needs of the datacenter and down to satisfy the constraints and special considerations of mobile and edge deployments.
See our website for project details, user guides, and instructions on building from source.
IREE is still in its early phase. We have settled down on the overarching infrastructure and are actively improving various software components as well as project logistics. It is still quite far from ready for everyday use and is made available without any support at the moment. With that said, we welcome any kind of feedback on any communication channels!
See our website for more information.
IREE is licensed under the terms of the Apache 2.0 License with LLVM Exceptions. See LICENSE for more information.