Arm(R) Ethos(TM)-U is a new class of machine learning processors, called a microNPU, specifically designed to accelerate ML inference in area-constrained embedded and IoT devices. This readme briefly describes how to integrate Ethos-U related hardware and software into TFLM. See also Ethos-U ML Evaluation kit examples.
To enable the Ethos-U software stack, add CO_PROCESSOR=ethos_u
to the make command. Use ETHOSU_ARCH to specify the architecture. See examples below.
The TFLM runtime will dispatch workloads to Ethos-U when it encounters an Ethos-U custom op in the tflite file. See an ASCII art example below. The Ethos-U custom op is added by a tool called Ethos-U Vela and contains information the Ethos-U hardware need to execute the workload. More info in the Vela repository.
| tensor0 | v +------------+ | ethos-u | | custom op | +------------+ + | | tensor1 | v +-----------+ | transpose | | | +----|------+ | | tensor2 | v
Note that the ethousu_init()
API of the Ethos-U driver need to be called at startup, before calling the TFLM API. More info in the Ethos-U driver repo.
For even more info regarding Vela and Ethos-U, checkout Ethos-U landing page.
In order to run a test with Ethos-U55 enabled, a platform with corresponding hardware support is required. One such platform is the fixed virtual platform (FVP) based on Arm Corstone-300 software. See Corstone-300 readme for more info.
On top of that the .tflite model needs to be modified according subchapter “Ethos-U custom operator” above.
The log level of the Ethos-U driver can be set in the build command. For example: ETHOSU_LOG_SEVERITY=ETHOSU_LOG_INFO.
See tensorflow/lite/micro/examples/network_tester/README.md for more info.
make -f tensorflow/lite/micro/tools/make/Makefile network_tester_test CO_PROCESSOR=ethos_u ETHOSU_ARCH=u55 TARGET=cortex_m_generic TARGET_ARCH=cortex-m55 microlite
For the Arm Corstone-300 target, ETHOSU_ARCH is defined in cortex_m_corstone_300_makefile.inc so it doesn't need to be defined on the command line.
make -f tensorflow/lite/micro/tools/make/Makefile network_tester_test CO_PROCESSOR=ethos_u TARGET=cortex_m_corstone_300 TARGET_ARCH=cortex-m55 test_network_tester_test NETWORK_MODEL=path/to/network_model.h INPUT_DATA=path/to/input_data.h OUTPUT_DATA=path/to/expected_output_data.h make -f tensorflow/lite/micro/tools/make/Makefile network_tester_test CO_PROCESSOR=ethos_u TARGET=cortex_m_corstone_300 TARGET_ARCH=cortex-m55 test_network_tester_test