docs/developers/get_started/vulkan_environment_setup.md - 3p/openxla/iree - Git at Google


            
            # Generic Vulkan Development Environment Setup and Troubleshooting


            [Vulkan](https://www.khronos.org/vulkan/) is a new generation graphics and

            
            compute API that provides high-efficiency, cross-platform access to modern GPUs

            
            used in a wide variety of devices from PCs and consoles to mobile phones and

            
            embedded platforms.


            This page lists steps and tips for setting up and trouble shooting a Vulkan

            
            development environment. The information here is meant to be generic.


            ## Vulkan architecture


            Vulkan adopts a layered architecture, which aims to better support extensiblity.

            
            There are four components involved in this architecture:


            *   The Vulkan Application

            
            *   [The Vulkan Loader][VulkanLoader]

            
            *   [Vulkan Layers][VulkanLayer]

            
            *   [Installable Client Drivers (ICDs)][VulkanICD]


            ![High Level View of Loader][VulkanArchPicture]


            The Vulkan loader sits between the Vulkan application, which calls Vulkan APIs,

            
            and the ICDs, which implements these Vulkan APIs. Vulkan layers agument the

            
            Vulkan system to provide optional features like validation and debugging. The

            
            Vulkan loader composes a chain of requested layers, which processes the Vulkan

            
            application's API calls one by one, and finally redirects the API calls made by

            
            the Vulkan application to one or more ICDs.


            It's highly recommned to read the

            
            [Architecture of the Vulkan Loader Interfaces Overview][VulkanArchOverview] to

            
            get a general understanding of what these components are and how they interact

            
            with one another.


            ## Vulkan development environment setup


            ### Windows


            You need to install the [Vulkan SDK][VulkanSDK] from LunarG to get the Vulkan

            
            loader.


            Typically the Vulkan SDK will be installed at `C:\VulkanSDK\<version>\ ` and

            
            there will be an environment variable `VULKAN_SDK` pointing to it. You can run

            
            the `vulkancube` executable under the `Bin\` subdirectory of the Vulkan SDK to

            
            make sure everything works properly. If not, you probably need to check whether

            
            the graphics card is Vulkan capable or update the driver.


            ### Debian/Ubuntu


            The following packages should be installed for a proper Vulkan runtime to test

            
            the runtime functions properly:


            *   [libvulkan1][PackageLibVulkan1] for the Vulkan loader `libvulkan.so`.

            
            *   [AMD] [mesa-vulkan-drivers][PackageMesaVulkan] for Mesa AMD Vulkan ICD.

            
            *   [NVIDIA] [nvidia-vulkan-icd][PackageNvidiaVulkan] for NVIDIA Vulkan ICD.


            The above packages provide the Vulkan loader and ICDs. With them an Vulkan

            
            application should be able to run. You may additionally want to install


            *   [vulkan-tools][PackageVulkanTools] for command-line tools like `vulkaninfo`

            
                (dumping available ICDs and their capabilities) and GUI application like

            
                `vulkancube` (rendering a rotating cube).


            In order to develop Vulkan applications, you additionally need the following

            
            packages:


            *   [libvulkan-dev][PackageVulkanDev] for various Vulkan header files.

            
            *   [vulkan-validationlayers][PackageVulkanValidation] for Vulkan validation

            
                layers like `VkLayer_standard_validation`.


            ### Linux


            For other Linux distros, please consult the corresponding package management

            
            tools for the packages needed. (And please feel free to update this doc

            
            regarding them.)


            You can also download and install the [Vulkan SDK][VulkanSDK] from LunarG. It

            
            packages the loader with many useful layers and other shader tools. The source

            
            code of the SDK component projects are included, allowing you to recompile the

            
            artifacts if needed.


            You can also build the Vulkan SDK component projects like

            
            [Vulkan-Loader][VulkanLoaderSource] and

            
            [Vulkan-ValidationLayers][VulkanValidationLayersSource] from source. But note

            
            that building these components separately you need to make sure they are

            
            consistent with one another (e.g., using the same version of Vulkan headers) to

            
            function together.


            If you have multiple versions of Vulkan loaders exist, you may also need to set

            
            `LD_LIBRARY_PATH` and `LD_PRELOAD` to load the desired version of the loader.

            
            For example:


            ```shell

            
            $ LD_LIBRARY_PATH={PATH_TO_VULKAN_SDK}/x86_64/lib/

            
            $ LD_PRELOAD=libvulkan.so.1

            
            ```


            This can also be done by sourcing the proper `setup-env.sh` from one of the

            
            downloaded Vulkan SDKs.


            ### Android


            Please make sure your Android device is Vulkan capable. Vulkan is supported on

            
            Android since 7, but we track newer Android versions (10+) closely and haven't

            
            set a clear min version yet.


            #### Troubleshooting Vulkan function `vkCreateInstance` not available


            Since Android 8 Oreo, Android re-architected the OS framework with

            
            [project Treble](https://source.android.com/devices/architecture#hidl).

            
            Framework libraries and

            
            [vendor libraries](https://source.android.com/devices/architecture/vndk) have a

            
            more strict and clear separation. Their dependencies are carefully scrutinized

            
            and only selected cases are allowed. This is enforced with

            
            [linker namespaces](https://source.android.com/devices/architecture/vndk/linker-namespace).


            `/data/local/tmp` is the preferred directory for automating native binary tests

            
            built using NDK toolchain. They should be allowed to access libraries like

            
            `libvulkan.so` for their functionality. However, there was an issue with fully

            
            treblized Android 10 where `/data/local/tmp` did not have access to the linker

            
            namespaces needed by `libvulkan.so`. This should be

            
            [fixed](https://android.googlesource.com/platform/system/linkerconfig/+/296da5b1eb88a3527ee76352c2d987f82f3252eb)

            
            now. But as typically in the Android system, it takes a long time to see the fix

            
            getting propagated, if ever.


            A known workaround is to symlink the vendor Vulkan implementation under

            
            `/vendor/lib[64]` as `libvulkan.so` under `/data/local/tmp` and use

            
            `LD_LIBRARY_PATH=/data/local/tmp` when invoking IREE executables.


            For Qualcomm Adreno GPUs, the vendor Vulkan implementation is at

            
            `/vendor/lib[64]/hw/vulkan.*.so`. So for example for Snapdragon 865:


            ```shell

            
            $ adb shell ln -s /vendor/lib64/hw/vulkan.kona.so /data/local/tmp/libvulkan.so

            
            ```


            For ARM Mali GPUs, there is only one monolithic driver

            
            (`/vendor/lib[64]/libGLES_mali.so`) for OpenGL and Vulkan and the Vulkan vendor

            
            driver (`/vendor/lib[64]/hw/vulkan.*.so`) is just a symlink to it. So for

            
            example:


            ```shell

            
            $ adb shell ln -s /vendor/lib64/libGLES_mali.so /data/local/tmp/libvulkan.so

            
            ```


            ## Vulkan debugging and profiling


            ### RenderDoc


            RenderDoc is an awesome tool that one can use to capture and introspect Vulkan

            
            applications. It can be downloaded from RenderDoc's

            
            [website](https://renderdoc.org/) or compiled from [source][RenderDocSource].


            ## Vulkan development environment troubleshooting


            ### Useful environment variables


            There are a few environment variables that can alter the default Vulkan loader

            
            behavior and print verbose information, notably:


            *   `VK_LOADER_DEBUG`: enable loader debug messages. Setting it to `all` will

            
                enable the most verbose logging from the loader. This is especially useful

            
                when trying to see what layers/ICDs are searched and used.

            
            *   `VK_ICD_FILENAMES`: force the loader to use a specific ICD. This is

            
                especially useful when you have multiple Vulkan capable devices and want to

            
                select which one to use manually.

            
            *   `VK_INSTANCE_LAYERS`: force the loader to enable the given layers. For

            
                example, You can force enable `VK_LAYER_LUNARG_api_dump` to have a detailed

            
                dump of all Vulkan API calls made by the application. You can force enable

            
                `VK_LAYER_LUNARG_core_validation` to validate the API calls made by the

            
                application.

            
            *   `VK_LAYER_PATH`: override the loader's standard layer library search folders.


            Please see the [Vulkan loader's documentation][VulkanLoaderEnvVars] for detailed

            
            explanation for these variables.


            ### Setting environment variables for Bazel test


            Bazel runs tests in a sandbox and environment variables must be passed through

            
            to the test runner. Consider putting environment setup in a `user.bazelrc` to

            
            save typing. For example:


            ```

            
            test --test_env="LD_LIBRARY_PATH=/absolute/path/to/vulkan/sdk/x86_64/lib/"

            
            test --test_env="LD_PRELOAD=libvulkan.so.1"

            
            test --test_env="VK_LAYER_PATH=/absolute/path/to/additional/layers/:$VK_LAYER_PATH"

            
            ```


            ### SSH on Linux and X forwarding


            Physical devices enumerated on NVIDIA drivers can be affected by the `DISPLAY`

            
            environment variable. If you are running under an SSH session to Linux or using

            
            chrome remote desktop and have problems with physical device enumeration, you

            
            probably want to check the `DISPLAY` environment and set it to point to a

            
            display at the server side, for example:


            ```

            
            export DISPLAY=:0

            
            ```


            [VulkanArchOverview]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#overview

            
            [VulkanArchPicture]: https://raw.githubusercontent.com/KhronosGroup/Vulkan-Loader/master/docs/images/high_level_loader.png

            
            [VulkanICD]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers

            
            [VulkanLayer]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#layers

            
            [VulkanLoader]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#the-loader

            
            [VulkanLoaderEnvVars]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#table-of-debug-environment-variables

            
            [VulkanLoaderSource]: https://github.com/KhronosGroup/Vulkan-Loader

            
            [VulkanSDK]: https://www.lunarg.com/vulkan-sdk/

            
            [VulkanValidationLayersSource]: https://github.com/KhronosGroup/Vulkan-ValidationLayers

            
            [PackageLibVulkan1]: https://packages.ubuntu.com/focal/libvulkan1

            
            [PackageMesaVulkan]: https://packages.ubuntu.com/focal/mesa-vulkan-drivers

            
            [PackageNvidiaVulkan]: https://packages.debian.org/buster/nvidia-vulkan-icd

            
            [PackageVulkanDev]: https://packages.ubuntu.com/focal/libvulkan-dev

            
            [PackageVulkanTools]: https://packages.ubuntu.com/focal/vulkan-tools

            
            [PackageVulkanValidation]: https://packages.ubuntu.com/eoan/vulkan-validationlayers

            
            [RenderDocSource]: https://github.com/baldurk/renderdoc
	# Generic Vulkan Development Environment Setup and Troubleshooting

	[Vulkan](https://www.khronos.org/vulkan/) is a new generation graphics and
	compute API that provides high-efficiency, cross-platform access to modern GPUs
	used in a wide variety of devices from PCs and consoles to mobile phones and
	embedded platforms.

	This page lists steps and tips for setting up and trouble shooting a Vulkan
	development environment. The information here is meant to be generic.

	## Vulkan architecture

	Vulkan adopts a layered architecture, which aims to better support extensiblity.
	There are four components involved in this architecture:

	* The Vulkan Application
	* [The Vulkan Loader][VulkanLoader]
	* [Vulkan Layers][VulkanLayer]
	* [Installable Client Drivers (ICDs)][VulkanICD]

	![High Level View of Loader][VulkanArchPicture]

	The Vulkan loader sits between the Vulkan application, which calls Vulkan APIs,
	and the ICDs, which implements these Vulkan APIs. Vulkan layers agument the
	Vulkan system to provide optional features like validation and debugging. The
	Vulkan loader composes a chain of requested layers, which processes the Vulkan
	application's API calls one by one, and finally redirects the API calls made by
	the Vulkan application to one or more ICDs.

	It's highly recommned to read the
	[Architecture of the Vulkan Loader Interfaces Overview][VulkanArchOverview] to
	get a general understanding of what these components are and how they interact
	with one another.

	## Vulkan development environment setup

	### Windows

	You need to install the [Vulkan SDK][VulkanSDK] from LunarG to get the Vulkan
	loader.

	Typically the Vulkan SDK will be installed at `C:\VulkanSDK\<version>\ ` and
	there will be an environment variable `VULKAN_SDK` pointing to it. You can run
	the `vulkancube` executable under the `Bin\` subdirectory of the Vulkan SDK to
	make sure everything works properly. If not, you probably need to check whether
	the graphics card is Vulkan capable or update the driver.

	### Debian/Ubuntu

	The following packages should be installed for a proper Vulkan runtime to test
	the runtime functions properly:

	* [libvulkan1][PackageLibVulkan1] for the Vulkan loader `libvulkan.so`.
	* [AMD] [mesa-vulkan-drivers][PackageMesaVulkan] for Mesa AMD Vulkan ICD.
	* [NVIDIA] [nvidia-vulkan-icd][PackageNvidiaVulkan] for NVIDIA Vulkan ICD.

	The above packages provide the Vulkan loader and ICDs. With them an Vulkan
	application should be able to run. You may additionally want to install

	* [vulkan-tools][PackageVulkanTools] for command-line tools like `vulkaninfo`
	(dumping available ICDs and their capabilities) and GUI application like
	`vulkancube` (rendering a rotating cube).

	In order to develop Vulkan applications, you additionally need the following
	packages:

	* [libvulkan-dev][PackageVulkanDev] for various Vulkan header files.
	* [vulkan-validationlayers][PackageVulkanValidation] for Vulkan validation
	layers like `VkLayer_standard_validation`.

	### Linux

	For other Linux distros, please consult the corresponding package management
	tools for the packages needed. (And please feel free to update this doc
	regarding them.)

	You can also download and install the [Vulkan SDK][VulkanSDK] from LunarG. It
	packages the loader with many useful layers and other shader tools. The source
	code of the SDK component projects are included, allowing you to recompile the
	artifacts if needed.

	You can also build the Vulkan SDK component projects like
	[Vulkan-Loader][VulkanLoaderSource] and
	[Vulkan-ValidationLayers][VulkanValidationLayersSource] from source. But note
	that building these components separately you need to make sure they are
	consistent with one another (e.g., using the same version of Vulkan headers) to
	function together.

	If you have multiple versions of Vulkan loaders exist, you may also need to set
	`LD_LIBRARY_PATH` and `LD_PRELOAD` to load the desired version of the loader.
	For example:

	```shell
	$ LD_LIBRARY_PATH={PATH_TO_VULKAN_SDK}/x86_64/lib/
	$ LD_PRELOAD=libvulkan.so.1
	```

	This can also be done by sourcing the proper `setup-env.sh` from one of the
	downloaded Vulkan SDKs.

	### Android

	Please make sure your Android device is Vulkan capable. Vulkan is supported on
	Android since 7, but we track newer Android versions (10+) closely and haven't
	set a clear min version yet.

	#### Troubleshooting Vulkan function `vkCreateInstance` not available

	Since Android 8 Oreo, Android re-architected the OS framework with
	[project Treble](https://source.android.com/devices/architecture#hidl).
	Framework libraries and
	[vendor libraries](https://source.android.com/devices/architecture/vndk) have a
	more strict and clear separation. Their dependencies are carefully scrutinized
	and only selected cases are allowed. This is enforced with
	[linker namespaces](https://source.android.com/devices/architecture/vndk/linker-namespace).

	`/data/local/tmp` is the preferred directory for automating native binary tests
	built using NDK toolchain. They should be allowed to access libraries like
	`libvulkan.so` for their functionality. However, there was an issue with fully
	treblized Android 10 where `/data/local/tmp` did not have access to the linker
	namespaces needed by `libvulkan.so`. This should be
	[fixed](https://android.googlesource.com/platform/system/linkerconfig/+/296da5b1eb88a3527ee76352c2d987f82f3252eb)
	now. But as typically in the Android system, it takes a long time to see the fix
	getting propagated, if ever.

	A known workaround is to symlink the vendor Vulkan implementation under
	`/vendor/lib[64]` as `libvulkan.so` under `/data/local/tmp` and use
	`LD_LIBRARY_PATH=/data/local/tmp` when invoking IREE executables.

	For Qualcomm Adreno GPUs, the vendor Vulkan implementation is at
	`/vendor/lib[64]/hw/vulkan.*.so`. So for example for Snapdragon 865:

	```shell
	$ adb shell ln -s /vendor/lib64/hw/vulkan.kona.so /data/local/tmp/libvulkan.so
	```

	For ARM Mali GPUs, there is only one monolithic driver
	(`/vendor/lib[64]/libGLES_mali.so`) for OpenGL and Vulkan and the Vulkan vendor
	driver (`/vendor/lib[64]/hw/vulkan.*.so`) is just a symlink to it. So for
	example:

	```shell
	$ adb shell ln -s /vendor/lib64/libGLES_mali.so /data/local/tmp/libvulkan.so
	```

	## Vulkan debugging and profiling

	### RenderDoc

	RenderDoc is an awesome tool that one can use to capture and introspect Vulkan
	applications. It can be downloaded from RenderDoc's
	[website](https://renderdoc.org/) or compiled from [source][RenderDocSource].

	## Vulkan development environment troubleshooting

	### Useful environment variables

	There are a few environment variables that can alter the default Vulkan loader
	behavior and print verbose information, notably:

	* `VK_LOADER_DEBUG`: enable loader debug messages. Setting it to `all` will
	enable the most verbose logging from the loader. This is especially useful
	when trying to see what layers/ICDs are searched and used.
	* `VK_ICD_FILENAMES`: force the loader to use a specific ICD. This is
	especially useful when you have multiple Vulkan capable devices and want to
	select which one to use manually.
	* `VK_INSTANCE_LAYERS`: force the loader to enable the given layers. For
	example, You can force enable `VK_LAYER_LUNARG_api_dump` to have a detailed
	dump of all Vulkan API calls made by the application. You can force enable
	`VK_LAYER_LUNARG_core_validation` to validate the API calls made by the
	application.
	* `VK_LAYER_PATH`: override the loader's standard layer library search folders.

	Please see the [Vulkan loader's documentation][VulkanLoaderEnvVars] for detailed
	explanation for these variables.

	### Setting environment variables for Bazel test

	Bazel runs tests in a sandbox and environment variables must be passed through
	to the test runner. Consider putting environment setup in a `user.bazelrc` to
	save typing. For example:

	```
	test --test_env="LD_LIBRARY_PATH=/absolute/path/to/vulkan/sdk/x86_64/lib/"
	test --test_env="LD_PRELOAD=libvulkan.so.1"
	test --test_env="VK_LAYER_PATH=/absolute/path/to/additional/layers/:$VK_LAYER_PATH"
	```

	### SSH on Linux and X forwarding

	Physical devices enumerated on NVIDIA drivers can be affected by the `DISPLAY`
	environment variable. If you are running under an SSH session to Linux or using
	chrome remote desktop and have problems with physical device enumeration, you
	probably want to check the `DISPLAY` environment and set it to point to a
	display at the server side, for example:

	```
	export DISPLAY=:0
	```

	[VulkanArchOverview]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#overview
	[VulkanArchPicture]: https://raw.githubusercontent.com/KhronosGroup/Vulkan-Loader/master/docs/images/high_level_loader.png
	[VulkanICD]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers
	[VulkanLayer]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#layers
	[VulkanLoader]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#the-loader
	[VulkanLoaderEnvVars]: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#table-of-debug-environment-variables
	[VulkanLoaderSource]: https://github.com/KhronosGroup/Vulkan-Loader
	[VulkanSDK]: https://www.lunarg.com/vulkan-sdk/
	[VulkanValidationLayersSource]: https://github.com/KhronosGroup/Vulkan-ValidationLayers
	[PackageLibVulkan1]: https://packages.ubuntu.com/focal/libvulkan1
	[PackageMesaVulkan]: https://packages.ubuntu.com/focal/mesa-vulkan-drivers
	[PackageNvidiaVulkan]: https://packages.debian.org/buster/nvidia-vulkan-icd
	[PackageVulkanDev]: https://packages.ubuntu.com/focal/libvulkan-dev
	[PackageVulkanTools]: https://packages.ubuntu.com/focal/vulkan-tools
	[PackageVulkanValidation]: https://packages.ubuntu.com/eoan/vulkan-validationlayers
	[RenderDocSource]: https://github.com/baldurk/renderdoc