| This directory contains examples to get developers started | |
| with using IREE's HalModuleDebugSink feature with callbacks | |
| to bisect layers of multi-dispatch model, to locate numerical | |
| divergences. It contains 2 examples: | |
| ## Simple callback example: `sync_callback_log.py` | |
| A simple multi-dispatch example showing how to log statistics of | |
| all dispatch inputs and outputs. When a model is compiled with | |
| `--iree-flow-trace-dispatch-tensors`, IREE inserts trace points | |
| before and after each dispatch function. | |
| When iree-run-module executes a model with trace points, the | |
| tensor values of dispatch inputs and outputs are printed to standard | |
| output. | |
| For models with small tensors, this is useful for debugging | |
| numerical issues: By comparing tensor values before and after a | |
| numerical regression, one can often find the dispatch that | |
| introduced the numerical error. However, for large models, it is | |
| infeasible to print all tensor values to the terminal. Even saving | |
| tensor values to disk may be impractical due to size constraints. | |
| In such cases, it is often sufficient to compare simple summary | |
| statistics, such as the mean. To enable this level of flexibility, | |
| one can use HalModuleDebugSink with arbitrary callback functions. | |
| This example presents a simple callback that logs mean values. | |
| ## Two model callback example: `sync_callback_log_async.py` | |
| This example extends Example 1. It shows how to run two versions of | |
| a model in parallel. The versions differ only in one dispatch, and | |
| the goal is automatically identify the 'bad' dispatch. | |
| In practice, the two models are usually compilation artifacts of | |
| the same source model, built before and after a numerical | |
| regression. This technique is especially useful when numerical | |
| divergence appears early in execution. |