tree 6038f3317f1ec66927490372889e595715b8c4a6
parent 7a27de5bdbf6a064af5f84c4fd82b7ef9a9476f7
author Ben Vanik <ben.vanik@gmail.com> 1664896410 -0700
committer Ben Vanik <ben.vanik@gmail.com> 1665594936 -0700

Implementing basic `--iree-execution-model=async-external` support.
Currently only coarse fences are supported: when the flag is specified
exported functions will take a wait and signal fence pair. Upon return to
a caller execution is not assumed to have completed and the caller can
either wait on the signal fence or chain further invocations with it.
Future invocation models will support specifying an arbitrary set of
fences that allow for up to per-I/O granularity (attention layers could
signal sooner than full decoders, etc) and specifying fences for in-place
buffers (wait until buffer is available to write in before filling).

This initial version is conservative and may include additional queue
barriers in order to signal the user-provided fence but future
improvements to timepoint elision and IPO will make that better. Nearly
all models we work with today end up becoming async with the current
heuristics.

iree-run-module/mlir has been updated to support programs compiled with
the async-external mode. iree-benchmark-module now supports pipelined
and concurrent execution via the --batch_size= and --batch_concurrency=
flags: batch_size defines how many invocations there are and
batch_concurrency defines how many of those are able to run concurrently.
Examples:
--batch_size=1 --batch_concurrency=1: default single-shot invocation
--batch_size=4 --batch_concurrency=1: 4 sequential invocations
--batch_size=4 --batch_concurrency=4: 4 concurrent invocations
--batch_size=4 --batch_concurrency=2: 2 concurrent sequences of 2 invocations