benchmarks/dashboard.md - 3p/openxla/iree - Git at Google

 # IREE Performance Dashboard

 This documentation explains IREE's performance dashboard (https://perf.iree.dev).
 A [Buildkite pipeline](https://buildkite.com/iree/iree-benchmark) runs on each
 commit to the `main` branch and posts those results to the dashboard.

 ## Benchmarking philosophy

 Benchmarking and interpreting results properly is a delicate task. We can record
 metrics from various parts of a system, but depending on what we are trying to
 evaluate, those numbers may or may not be relevant. For example, for somebody
 working solely on better kernel code generation, the end-to-end model reference
 latency is unlikely meaningful given it also includes runtime overhead. The
 environment could also vary per benchmark run in uncontrollable ways, causing
 instability in the results. This is especially true for mobile and embedded
 systems, where a tight compromise between performance and thermal/battery limits
 is made. Too many aspects can affect the benchmarking results. So before going
 into details, it's worth nothing the general guideline to IREE benchmarking as
 context.

 The overarching goal for benchmarking here is to track IREE's performance
 progress and guard against regression. So the benchmarks are meant to understand
 the performance of IREE _itself_, not the absolute capability of the exercised
 hardware. In order to fulfill the above goal, we have the following guidelines
 for benchmarking:

 * We choose representative real-world models with varying characteristics.
 * We cover different IREE backends and different modes for each backend so that
   folks working on different components can find the metrics they need.

 ## Model benchmark specification

 Each benchmark in IREE has a unique identifier with the following format:

 ```
 <model-name> `[` <model-tag>.. `]` `(` <model-source> `)` <benchmark-mode>..
 `with` <iree-driver>
 `@` <device-name> `(` <target-architecture> `)`
 ```

 The following subsections explain possible choices in each field.

 ### Model source

 This field specifies the original model source:

 * `TFLite`: Models originally in TensorFlow Lite Flatbuffer format.

 ### Model name

 This field specifies the input model:

 * `DeepLabV3` [[source](https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/default/1)]:
   Vision model for semantic image segmentation.
   Characteristics: convolution, feedforward NN.
 * `MobileBERT` [[source](https://tfhub.dev/tensorflow/lite-model/mobilebert/1/default/1)]:
   NLP for Q&A.
   Characteristics: matmul, attention, feedforward NN.
 * `MobileNetV2` [[source](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite)]:
   Vision model for image classification.
   Characteristics: convolution, feedforward NN
 * `MobileNetV3Small` [[source](https://tfhub.dev/google/imagenet/mobilenet_v3_small_100_224/classification/5)]:
   Vision model for image classification.
   Characteristics: convolution, feedforward NN.
 * `MobileSSD` [[source](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/mobile_ssd_v2_float_coco.tflite)]:
   Vision model for object detection.
   Characteristics: convolution, feedforward NN.
 * `PoseNet` [[source](https://tfhub.dev/tensorflow/lite-model/posenet/mobilenet/float/075/1/default/1)]:
   Vision model for pose estimation.
   Characteristics: convolution, feedforward NN.

 ### Model tag

 This field specifies the model variant. It depends on the model, but here are
 some examples:

 * `f32`: the model is working on float types.
 * `imagenet`: the model takes ImageNet-sized inputs (224x224x3).

 ### IREE driver

 This field specifies the IREE HAL driver:

 * [`local-task`](https://iree-org.github.io/iree/deployment-configurations/cpu/):
   For CPU via the local task system. Kernels contain CPU native instructions AOT
   compiled using LLVM. This driver issues workloads to the CPU asynchronously
   and supports multithreading.
 * [`local-sync`](https://iree-org.github.io/iree/deployment-configurations/cpu/):
   For CPU via the local 'sync' device. Kernels contain contain CPU native
   instructions AOT compiled using LLVM. This driver issues workloads to the CPU
   synchronously.
 * [`Vulkan`](https://iree-org.github.io/iree/deployment-configurations/gpu-vulkan/):
   For GPU via Vulkan. Kernels contain SPIR-V. This driver issues workload to
   the GPU via the Vulkan API.

 ### Device name and target architecture

 These two fields are tightly coupled. They specify the device and hardware
 target for executing the benchmark.

 Right now there are two Android devices:

 * `Pixel-4`: Google Pixel 4 running Android 11. The SoC is
   [Snapdragon 855](https://www.qualcomm.com/products/snapdragon-855-plus-and-860-mobile-platform),
   with 1+3+4 ARMv8.2 CPU cores and Adreno 640 GPU.
 * `Pixel-6`: Google Pixel 6 running Android 12. The SoC is
   [Google Tensor](https://blog.google/products/pixel/introducing-google-tensor/),
   with 2+2+4 ARMv8 CPU cores and Mali G78 GPU.
 * `SM-G980F`: Samsung Galaxy S20 running Android 11. The SoC is
   [Exynos 990](https://www.samsung.com/semiconductor/minisite/exynos/products/mobileprocessor/exynos-990/),
   with 2+2+4 ARMv8.2 CPU cores and Mali G77 MP11 GPU.

 Therefore the target architectures are:

 * `CPU-CPU-ARMv8.2-A`: can benchmark all CPU-based IREE backends and drivers.
 * `GPU-Adreno-640`: can benchmark IREE Vulkan with Adreno target triples.
 * `GPU-Mali-G77`: can benchmark IREE Vulkan with Mali target triples.
 * `GPU-Mali-G78`: can benchmark IREE Vulkan with Mali target triples.

 ### Benchmark mode

 This field is to further specify the benchmark variant, given the same input
 model and target architecture. It controls important aspects like:

 * `*-core`: specifies the core flavor for CPU.
 * `*-thread`: specifies the number of threads for CPU.
 * `full-inference`: measures the latency for one full inference. Note that this
   does not include the IREE system initialization time.
 * `kernel-execution`: measures only kernel execution latency for GPU. Note that
   this is only possible for feedforward NN models that can be put into one
   command buffer.

 `*-core` and `*-thread` together determines the `taskset` mask used for
 benchmarking IREE backends and drivers on CPU. For example,

 * `1-thread,big-core` would mean `taskset 80`.
 * `1-thread,little-core` would mean `taskset 08`.
 * `3-thread,big-core` would mean `taskset f0`.
 * `3-thread,little-core` would mean `taskset 0f`.
	# IREE Performance Dashboard

	This documentation explains IREE's performance dashboard (https://perf.iree.dev).
	A [Buildkite pipeline](https://buildkite.com/iree/iree-benchmark) runs on each
	commit to the `main` branch and posts those results to the dashboard.

	## Benchmarking philosophy

	Benchmarking and interpreting results properly is a delicate task. We can record
	metrics from various parts of a system, but depending on what we are trying to
	evaluate, those numbers may or may not be relevant. For example, for somebody
	working solely on better kernel code generation, the end-to-end model reference
	latency is unlikely meaningful given it also includes runtime overhead. The
	environment could also vary per benchmark run in uncontrollable ways, causing
	instability in the results. This is especially true for mobile and embedded
	systems, where a tight compromise between performance and thermal/battery limits
	is made. Too many aspects can affect the benchmarking results. So before going
	into details, it's worth nothing the general guideline to IREE benchmarking as
	context.

	The overarching goal for benchmarking here is to track IREE's performance
	progress and guard against regression. So the benchmarks are meant to understand
	the performance of IREE _itself_, not the absolute capability of the exercised
	hardware. In order to fulfill the above goal, we have the following guidelines
	for benchmarking:

	* We choose representative real-world models with varying characteristics.
	* We cover different IREE backends and different modes for each backend so that
	folks working on different components can find the metrics they need.

	## Model benchmark specification

	Each benchmark in IREE has a unique identifier with the following format:

	```
	<model-name> `[` <model-tag>.. `]` `(` <model-source> `)` <benchmark-mode>..
	`with` <iree-driver>
	`@` <device-name> `(` <target-architecture> `)`
	```

	The following subsections explain possible choices in each field.

	### Model source

	This field specifies the original model source:

	* `TFLite`: Models originally in TensorFlow Lite Flatbuffer format.

	### Model name

	This field specifies the input model:

	* `DeepLabV3` [[source](https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/default/1)]:
	Vision model for semantic image segmentation.
	Characteristics: convolution, feedforward NN.
	* `MobileBERT` [[source](https://tfhub.dev/tensorflow/lite-model/mobilebert/1/default/1)]:
	NLP for Q&A.
	Characteristics: matmul, attention, feedforward NN.
	* `MobileNetV2` [[source](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite)]:
	Vision model for image classification.
	Characteristics: convolution, feedforward NN
	* `MobileNetV3Small` [[source](https://tfhub.dev/google/imagenet/mobilenet_v3_small_100_224/classification/5)]:
	Vision model for image classification.
	Characteristics: convolution, feedforward NN.
	* `MobileSSD` [[source](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/mobile_ssd_v2_float_coco.tflite)]:
	Vision model for object detection.
	Characteristics: convolution, feedforward NN.
	* `PoseNet` [[source](https://tfhub.dev/tensorflow/lite-model/posenet/mobilenet/float/075/1/default/1)]:
	Vision model for pose estimation.
	Characteristics: convolution, feedforward NN.

	### Model tag

	This field specifies the model variant. It depends on the model, but here are
	some examples:

	* `f32`: the model is working on float types.
	* `imagenet`: the model takes ImageNet-sized inputs (224x224x3).

	### IREE driver

	This field specifies the IREE HAL driver:

	* [`local-task`](https://iree-org.github.io/iree/deployment-configurations/cpu/):
	For CPU via the local task system. Kernels contain CPU native instructions AOT
	compiled using LLVM. This driver issues workloads to the CPU asynchronously
	and supports multithreading.
	* [`local-sync`](https://iree-org.github.io/iree/deployment-configurations/cpu/):
	For CPU via the local 'sync' device. Kernels contain contain CPU native
	instructions AOT compiled using LLVM. This driver issues workloads to the CPU
	synchronously.
	* [`Vulkan`](https://iree-org.github.io/iree/deployment-configurations/gpu-vulkan/):
	For GPU via Vulkan. Kernels contain SPIR-V. This driver issues workload to
	the GPU via the Vulkan API.

	### Device name and target architecture

	These two fields are tightly coupled. They specify the device and hardware
	target for executing the benchmark.

	Right now there are two Android devices:

	* `Pixel-4`: Google Pixel 4 running Android 11. The SoC is
	[Snapdragon 855](https://www.qualcomm.com/products/snapdragon-855-plus-and-860-mobile-platform),
	with 1+3+4 ARMv8.2 CPU cores and Adreno 640 GPU.
	* `Pixel-6`: Google Pixel 6 running Android 12. The SoC is
	[Google Tensor](https://blog.google/products/pixel/introducing-google-tensor/),
	with 2+2+4 ARMv8 CPU cores and Mali G78 GPU.
	* `SM-G980F`: Samsung Galaxy S20 running Android 11. The SoC is
	[Exynos 990](https://www.samsung.com/semiconductor/minisite/exynos/products/mobileprocessor/exynos-990/),
	with 2+2+4 ARMv8.2 CPU cores and Mali G77 MP11 GPU.

	Therefore the target architectures are:

	* `CPU-CPU-ARMv8.2-A`: can benchmark all CPU-based IREE backends and drivers.
	* `GPU-Adreno-640`: can benchmark IREE Vulkan with Adreno target triples.
	* `GPU-Mali-G77`: can benchmark IREE Vulkan with Mali target triples.
	* `GPU-Mali-G78`: can benchmark IREE Vulkan with Mali target triples.

	### Benchmark mode

	This field is to further specify the benchmark variant, given the same input
	model and target architecture. It controls important aspects like:

	* `*-core`: specifies the core flavor for CPU.
	* `*-thread`: specifies the number of threads for CPU.
	* `full-inference`: measures the latency for one full inference. Note that this
	does not include the IREE system initialization time.
	* `kernel-execution`: measures only kernel execution latency for GPU. Note that
	this is only possible for feedforward NN models that can be put into one
	command buffer.

	`-core` and `-thread` together determines the `taskset` mask used for
	benchmarking IREE backends and drivers on CPU. For example,

	* `1-thread,big-core` would mean `taskset 80`.
	* `1-thread,little-core` would mean `taskset 08`.
	* `3-thread,big-core` would mean `taskset f0`.
	* `3-thread,little-core` would mean `taskset 0f`.