docs/developing_iree/profiling_with_tracy.md - 3p/openxla/iree - Git at Google

 # Profiling with Tracy

 [Tracy](https://github.com/wolfpld/tracy) is a profiler that puts together in a
 single view:
 * Both CPU and GPU profiling.
 * Both sampling and instrumentation.
 * Both specifics of our own process, and whole-system profiling a la "systrace".

 Since Tracy relies on instrumentation, it requires IREE binaries to be built
 with a special flag to enable it.

 There are two components to Tracy. They communicate over a TCP socket.
 * The "client" is the program being profiled.
 * The "server" is the Tracy profiler UI.

 ## The Tracy manual

 The primary source of Tracy documentation, including for build instructions, is
 a PDF manual that's part of each numbered release. To find the latest one,
 navigate [here](https://github.com/wolfpld/tracy/releases) and search for
 `tracy.pdf`.

 ## Building the Tracy UI (the "server")

 This is explained in section 2.3 of the manual for Windows and Linux. Here we
 give some more detailed instructions for some systems.

 The IREE repository contains its own clone of the Tracy repository in
 `third_party/tracy`, so there is no need to make a separate clone of it. You can
 use one if you want, but be aware that the Tracy client/server protocol gets
 updated sometimes. Building both sides from the same
 `iree/third_party/tracy` lowers the risk of running into a protocol version
 mismatch.

 ### Linux

 Install dependencies (Debian-based distributions):
 ```
 sudo apt install libcapstone-dev libtbb-dev libglfw3-dev libfreetype6-dev libgtk-3-dev
 ```

 Build (from your `iree/` clone root directory):
 ```
 make -C third_party/tracy/profiler/build/unix -j12 release
 ```

 ### Mac

 TODO write this (Kojo?)

 ## Building IREE with Tracy instrumentation (the "client")

 IREE needs to be build with Tracy instrumentation enabled. This enables both the
 collection of data, and its streaming to the Tracy server over a socket.

 This is only supported in the CMake build system of IREE, not in Bazel.

 In the initial CMake configuration command:
 *   Set `IREE_ENABLE_RUNTIME_TRACING` to `ON`.
 *   Use the `RelWithDebInfo` build type.

 For example:

 ```shell
 $ cmake \
   -DIREE_ENABLE_RUNTIME_TRACING=ON \
   -DCMAKE_BUILD_TYPE=RelWithDebInfo \
   ... # other cmake arguments as usual
 ```
 ## Permissions issues

 The profiled application (i.e. the Tracy client) needs to have appropriate
 permissions so perform the special I/O required to collect the profile
 information. This is OS-specific.

 ### Desktop Linux

 On desktop Linux, the Tracy client must be run as root, e.g. with `sudo`.
 Otherwise, profile data will lack important components.

 ### Android

 On Android it is not necessary to run as root and in fact, Android graphical
 applications never run as root, so it's advisable to run all programs as
 non-root for consistency.

 The Android device must be prepared as follows to enable Tracy profiling.
 * The device must be rooted.
   * That means that in `adb shell`, the command `su` must succeed.
   * That does NOT mean doing `adb root`. The effect of `adb root` is to have the
     `adbd` daemon itself run as root, which causes `adb shell` to give you a
     root shell by default. If you are in that case, consider doing `adb unroot`
     to restart the `adbd` server as non-root. Not mandatory, but again, running
     anything as root on Android is a deviation from normal user conditions.
 * Execute the following commands in a root shell on the device (i.e. `adb
   shell`, then `su`, then the following commands). These are from the Tracy PDF
   manual, but hard to find there, and copy-pasting from PDF introduces unwanted
   whitespace. These settings normally persist until the next reboot of the
   device.
   * `setenforce 0`
   * `mount -o remount,hidepid=0 /proc`
   * `echo 0 > /proc/sys/kernel/perf_event_paranoid`

 ## Port forwarding

 The Tracy client and server communicate by default over port `8086`. When they
 run on different machines, e.g. with embedded/Android profiling or remote
 profiling, port forwarding must be set up.

 ### Between a computer and a local Android device connected to it by USB

 Run this command. You might need to run it again more a little frequently than
 you reboot the device. When experiencing connection issues, try that first.

 ```shell
 adb forward tcp:8086 tcp:8086
 ```

 ### Between two computers over the network

 TODO write this (`ssh` stuff...)

 ## Running the profiled program

 Run your IREE workload as you normally would: now that it's been built with
 Tracy instrumentation enabled, it should do all the right things automatically.

 The only change that you are likely to need in your command line is to set the
 `TRACY_NO_EXIT=1` environment variable. This ensures that your program does not
 exit until the Tracy server (the UI) has connected to it and finished uploading
 the profile data.

 Typically, `TRACY_NO_EXIT=1` is needed when profiling `iree-benchmark-module`.
 It wouldn't be needed when profiling a real user-facing application.

 Example:

 ```shell
 TRACY_NO_EXIT=1 /data/local/tmp/iree-benchmark-module \
   --driver=dylib \
   --function_inputs='1x384xi32,1x384xi32,1x384xi32' \
   --module_file=/data/local/tmp/android_module.fbvm \
   --entry_function=serving_default
 ```

 ## Running the Tracy profiler UI, connecting and visualizing

 While the profile program is still running (possibly thanks to
 `TRACY_NO_EXIT=1`), start the Tracy profiler UI which we had built above.
 From the IREE root directory:
 ```shell
 ./third_party/tracy/profiler/build/unix/Tracy-release
 ```

 It should show a dialog offering to connect to a client i.e. a profiled program.

 If connecting doesn't work:
 * If the profiled program is on a separate machine, make sure you've correctly
   set up port forwarding.
   * On Android, the `adb forward` many need to be run again.
 * Make sure that the profiled program is still running. Do you need
   `TRACY_NO_EXIT=1`?
 * Kill the profiled program and restart it.

 You should then start seeing a profile.

 Before going further, check that your profile has sampling data recorded, not
 just instrumentation data. Click the 'Statistics' button at the top. The window
 that it opens should show a 'Sampling' radio button next to 'Instrumentation'.
 Back to the main view, look for the 'Main thread' label on the far left. Next to
 it should be a small ghost icon. If either of these things are missing, you need
 to fix that first, because many of Tracy's interesting features rely on
 smapling. Most likely this is a permissions issue: make sure that you have
 performed the steps outlined in the above section on permissions. As an
 experiment, retry with the profiled program running as root. Look for any
 interesting `stderr` message (in the profiled program's terminal). Try
 `strace`'ing it.

 Besides sampling, let's check that you also have "systrace" working. In the
 initial main view, on the far left, you should see a list of CPU cores,
 typically labelled `CPU0`, `CPU1`, etc. To the right, a timeline view should
 show system-wide activity on all these CPUs - so for instance you can see
 context switches and you can see if anything else happened on your device that
 could have interfered with the benchmark's performance. If this "systrace"
 functionality isn't working, then again that's likely a permissions issue,
 perform the same troubleshooting as above about sampling.

 Finally, look for the part of the timeline that is of interest to you. Your area
 of interest might not be on the Main thread. In fact, it might be on a thread
 that's not visible in the initial view at all. Indeed, the initial view tends to
 be zoomed-in a lot. Either use the mouse directly to zoom out and navigate, to
 look for the 'Frame' control at the top of the Tracy window. Use the 'next
 frame' arrow button until more interesting threads appear. Typically, IREE
 generated code tends to run on a thread named `cpu0`, which is actually a thread
 name and unrelated to `CPU0` from the systrace view.

 Once you click a zone, you should see a line-by-line or
 instruction-by-instruction view annotated with the percentage of time spent. If
 you don't see the percentages, then again that would point to sampling not
 working, refer to the above steps.

 ## Configuring Tracy instrumentation

 Set IREE's `IREE_TRACING_MODE` value (defined in
 [iree/base/tracing.h](https://github.com/google/iree/blob/main/iree/base/tracing.h))
 to adjust which tracing features, such as allocation tracking and callstacks,
 are enabled.
	# Profiling with Tracy

	[Tracy](https://github.com/wolfpld/tracy) is a profiler that puts together in a
	single view:
	* Both CPU and GPU profiling.
	* Both sampling and instrumentation.
	* Both specifics of our own process, and whole-system profiling a la "systrace".

	Since Tracy relies on instrumentation, it requires IREE binaries to be built
	with a special flag to enable it.

	There are two components to Tracy. They communicate over a TCP socket.
	* The "client" is the program being profiled.
	* The "server" is the Tracy profiler UI.

	## The Tracy manual

	The primary source of Tracy documentation, including for build instructions, is
	a PDF manual that's part of each numbered release. To find the latest one,
	navigate [here](https://github.com/wolfpld/tracy/releases) and search for
	`tracy.pdf`.

	## Building the Tracy UI (the "server")

	This is explained in section 2.3 of the manual for Windows and Linux. Here we
	give some more detailed instructions for some systems.

	The IREE repository contains its own clone of the Tracy repository in
	`third_party/tracy`, so there is no need to make a separate clone of it. You can
	use one if you want, but be aware that the Tracy client/server protocol gets
	updated sometimes. Building both sides from the same
	`iree/third_party/tracy` lowers the risk of running into a protocol version
	mismatch.

	### Linux

	Install dependencies (Debian-based distributions):
	```
	sudo apt install libcapstone-dev libtbb-dev libglfw3-dev libfreetype6-dev libgtk-3-dev
	```

	Build (from your `iree/` clone root directory):
	```
	make -C third_party/tracy/profiler/build/unix -j12 release
	```

	### Mac

	TODO write this (Kojo?)

	## Building IREE with Tracy instrumentation (the "client")

	IREE needs to be build with Tracy instrumentation enabled. This enables both the
	collection of data, and its streaming to the Tracy server over a socket.

	This is only supported in the CMake build system of IREE, not in Bazel.

	In the initial CMake configuration command:
	* Set `IREE_ENABLE_RUNTIME_TRACING` to `ON`.
	* Use the `RelWithDebInfo` build type.

	For example:

	```shell
	$ cmake \
	-DIREE_ENABLE_RUNTIME_TRACING=ON \
	-DCMAKE_BUILD_TYPE=RelWithDebInfo \
	... # other cmake arguments as usual
	```
	## Permissions issues

	The profiled application (i.e. the Tracy client) needs to have appropriate
	permissions so perform the special I/O required to collect the profile
	information. This is OS-specific.

	### Desktop Linux

	On desktop Linux, the Tracy client must be run as root, e.g. with `sudo`.
	Otherwise, profile data will lack important components.

	### Android

	On Android it is not necessary to run as root and in fact, Android graphical
	applications never run as root, so it's advisable to run all programs as
	non-root for consistency.

	The Android device must be prepared as follows to enable Tracy profiling.
	* The device must be rooted.
	* That means that in `adb shell`, the command `su` must succeed.
	* That does NOT mean doing `adb root`. The effect of `adb root` is to have the
	`adbd` daemon itself run as root, which causes `adb shell` to give you a
	root shell by default. If you are in that case, consider doing `adb unroot`
	to restart the `adbd` server as non-root. Not mandatory, but again, running
	anything as root on Android is a deviation from normal user conditions.
	* Execute the following commands in a root shell on the device (i.e. `adb
	shell`, then `su`, then the following commands). These are from the Tracy PDF
	manual, but hard to find there, and copy-pasting from PDF introduces unwanted
	whitespace. These settings normally persist until the next reboot of the
	device.
	* `setenforce 0`
	* `mount -o remount,hidepid=0 /proc`
	* `echo 0 > /proc/sys/kernel/perf_event_paranoid`

	## Port forwarding

	The Tracy client and server communicate by default over port `8086`. When they
	run on different machines, e.g. with embedded/Android profiling or remote
	profiling, port forwarding must be set up.

	### Between a computer and a local Android device connected to it by USB

	Run this command. You might need to run it again more a little frequently than
	you reboot the device. When experiencing connection issues, try that first.

	```shell
	adb forward tcp:8086 tcp:8086
	```

	### Between two computers over the network

	TODO write this (`ssh` stuff...)

	## Running the profiled program

	Run your IREE workload as you normally would: now that it's been built with
	Tracy instrumentation enabled, it should do all the right things automatically.

	The only change that you are likely to need in your command line is to set the
	`TRACY_NO_EXIT=1` environment variable. This ensures that your program does not
	exit until the Tracy server (the UI) has connected to it and finished uploading
	the profile data.

	Typically, `TRACY_NO_EXIT=1` is needed when profiling `iree-benchmark-module`.
	It wouldn't be needed when profiling a real user-facing application.

	Example:

	```shell
	TRACY_NO_EXIT=1 /data/local/tmp/iree-benchmark-module \
	--driver=dylib \
	--function_inputs='1x384xi32,1x384xi32,1x384xi32' \
	--module_file=/data/local/tmp/android_module.fbvm \
	--entry_function=serving_default
	```

	## Running the Tracy profiler UI, connecting and visualizing

	While the profile program is still running (possibly thanks to
	`TRACY_NO_EXIT=1`), start the Tracy profiler UI which we had built above.
	From the IREE root directory:
	```shell
	./third_party/tracy/profiler/build/unix/Tracy-release
	```

	It should show a dialog offering to connect to a client i.e. a profiled program.

	If connecting doesn't work:
	* If the profiled program is on a separate machine, make sure you've correctly
	set up port forwarding.
	* On Android, the `adb forward` many need to be run again.
	* Make sure that the profiled program is still running. Do you need
	`TRACY_NO_EXIT=1`?
	* Kill the profiled program and restart it.

	You should then start seeing a profile.

	Before going further, check that your profile has sampling data recorded, not
	just instrumentation data. Click the 'Statistics' button at the top. The window
	that it opens should show a 'Sampling' radio button next to 'Instrumentation'.
	Back to the main view, look for the 'Main thread' label on the far left. Next to
	it should be a small ghost icon. If either of these things are missing, you need
	to fix that first, because many of Tracy's interesting features rely on
	smapling. Most likely this is a permissions issue: make sure that you have
	performed the steps outlined in the above section on permissions. As an
	experiment, retry with the profiled program running as root. Look for any
	interesting `stderr` message (in the profiled program's terminal). Try
	`strace`'ing it.

	Besides sampling, let's check that you also have "systrace" working. In the
	initial main view, on the far left, you should see a list of CPU cores,
	typically labelled `CPU0`, `CPU1`, etc. To the right, a timeline view should
	show system-wide activity on all these CPUs - so for instance you can see
	context switches and you can see if anything else happened on your device that
	could have interfered with the benchmark's performance. If this "systrace"
	functionality isn't working, then again that's likely a permissions issue,
	perform the same troubleshooting as above about sampling.

	Finally, look for the part of the timeline that is of interest to you. Your area
	of interest might not be on the Main thread. In fact, it might be on a thread
	that's not visible in the initial view at all. Indeed, the initial view tends to
	be zoomed-in a lot. Either use the mouse directly to zoom out and navigate, to
	look for the 'Frame' control at the top of the Tracy window. Use the 'next
	frame' arrow button until more interesting threads appear. Typically, IREE
	generated code tends to run on a thread named `cpu0`, which is actually a thread
	name and unrelated to `CPU0` from the systrace view.

	Once you click a zone, you should see a line-by-line or
	instruction-by-instruction view annotated with the percentage of time spent. If
	you don't see the percentages, then again that would point to sampling not
	working, refer to the above steps.

	## Configuring Tracy instrumentation

	Set IREE's `IREE_TRACING_MODE` value (defined in
	[iree/base/tracing.h](https://github.com/google/iree/blob/main/iree/base/tracing.h))
	to adjust which tracing features, such as allocation tracking and callstacks,
	are enabled.