| .. _module-pw_protobuf_compiler: | |
| -------------------- | |
| pw_protobuf_compiler | |
| -------------------- | |
| The Protobuf compiler module provides build system integration and wrapper | |
| scripts for generating source code for Protobuf definitions. | |
| Generator support | |
| ================= | |
| Protobuf code generation is currently supported for the following generators: | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Generator | Code | Notes | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | pw_protobuf | ``pwpb`` | Compiles using ``pw_protobuf``. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Nanopb | ``nanopb`` | Compiles using Nanopb. The build argument | | |
| | | | ``dir_pw_third_party_nanopb`` must be set to | | |
| | | | point to a local nanopb installation. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Nanopb RPC | ``nanopb_rpc`` | Compiles pw_rpc service and client code for | | |
| | | | nanopb. Requires a nanopb installation. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Raw RPC | ``raw_rpc`` | Compiles raw binary pw_rpc service code. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Go | ``go`` | Compiles using the standard Go protobuf | | |
| | | | plugin with gRPC service support. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| | Python | ``python`` | Compiles using the standard Python protobuf | | |
| | | | plugin, creating a ``pw_python_package``. | | |
| +-------------+----------------+-----------------------------------------------+ | |
| GN template | |
| =========== | |
| This module provides a ``pw_proto_library`` GN template that defines a | |
| collection of protobuf files that should be compiled together. The template | |
| creates a sub-target for each supported generator, named | |
| ``<target_name>.<generator>``. These sub-targets generate their respective | |
| protobuf code, and expose it to the build system appropriately (e.g. a | |
| ``pw_source_set`` for C/C++). | |
| For example, given the following target: | |
| .. code-block:: | |
| pw_proto_library("test_protos") { | |
| sources = [ "my_test_protos/test.proto" ] | |
| } | |
| ``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb`` | |
| compiles using Nanopb (if it's installed). | |
| Protobuf code is only generated when a generator sub-target is listed as a | |
| dependency of another GN target. | |
| GN permits using abbreviated labels when the target name matches the directory | |
| name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, the | |
| sub-targets for each generator are aliased to the directory when the target name | |
| is the same. For example, these two labels are equivalent: | |
| .. code-block:: | |
| //path/to/my_protos:my_protos.pwpb | |
| //path/to/my_protos:pwpb | |
| ``pw_python_package`` subtargets are also available on the ``python`` subtarget: | |
| .. code-block:: | |
| //path/to/my_protos:my_protos.python.lint | |
| //path/to/my_protos:python.lint | |
| **Supported Codegen** | |
| GN supports the following compiled proto libraries via the specified | |
| sub-targets generated by a ``pw_proto_library``. | |
| * ``${target_name}.pwpb`` - Generated C++ pw_protobuf code | |
| * ``${target_name}.nanopb`` - Generated C++ nanopb code (requires Nanopb) | |
| * ``${target_name}.nanopb_rpc`` - Generated C++ Nanopb pw_rpc code (requires | |
| Nanopb) | |
| * ``${target_name}.raw_rpc`` - Generated C++ raw pw_rpc code (no protobuf | |
| library) | |
| * ``${target_name}.go`` - Generated GO protobuf libraries | |
| * ``${target_name}.python`` - Generated Python protobuf libraries | |
| **Arguments** | |
| * ``sources``: List of input .proto files. | |
| * ``deps``: List of other pw_proto_library dependencies. | |
| * ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options`` | |
| files). | |
| * ``prefix``: A prefix to add to the source protos prior to compilation. For | |
| example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be | |
| compiled with protoc as ``"nested/foo.proto"``. | |
| * ``strip_prefix``: Remove this prefix from the source protos. All source and | |
| input files must be nested under this path. | |
| * ``python_package``: Label of Python package to which to add the proto modules. | |
| The .python subtarget will redirect to this package. | |
| **Example** | |
| .. code-block:: | |
| import("$dir_pw_protobuf_compiler/proto.gni") | |
| pw_proto_library("my_protos") { | |
| sources = [ | |
| "my_protos/foo.proto", | |
| "my_protos/bar.proto", | |
| ] | |
| } | |
| pw_proto_library("my_other_protos") { | |
| sources = [ "some/other/path/baz.proto" ] # imports foo.proto | |
| # This removes the "some/other/path" prefix from the proto files. | |
| strip_prefix = "some/other/path" | |
| # This adds the "my_other_protos/" prefix to the proto files. | |
| prefix = "my_other_protos" | |
| # Proto libraries depend on other proto libraries directly. | |
| deps = [ ":my_protos" ] | |
| } | |
| source_set("my_cc_code") { | |
| sources = [ | |
| "foo.cc", | |
| "bar.cc", | |
| "baz.cc", | |
| ] | |
| # When depending on protos in a source_set, specify the generator suffix. | |
| deps = [ ":my_other_protos.pwpb" ] | |
| } | |
| From C++, ``baz.proto`` included as follows: | |
| .. code-block:: cpp | |
| #include "my_other_protos/baz.pwpb.h" | |
| From Python, ``baz.proto`` is imported as follows: | |
| .. code-block:: python | |
| from my_other_protos import baz_pb2 | |
| Proto file structure | |
| -------------------- | |
| Protobuf source files must be nested under another directory when they are | |
| compiled. This ensures that they can be packaged properly in Python. The first | |
| directory is used as the Python package name, so must be unique across the | |
| build. The ``prefix`` option may be used to set this directory. | |
| Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to | |
| a completely different path. This can be useful when working with protos defined | |
| in external libraries. For example, consider this proto library: | |
| .. code-block:: | |
| pw_proto_library("external_protos") { | |
| sources = [ | |
| "//other/external/some_library/src/protos/alpha.proto", | |
| "//other/external/some_library/src/protos/beta.proto, | |
| "//other/external/some_library/src/protos/internal/gamma.proto", | |
| ] | |
| strip_prefix = "//other/external/some_library/src/protos" | |
| prefix = "some_library" | |
| } | |
| These protos will be compiled by protoc as if they were in this file structure: | |
| .. code-block:: | |
| some_library/ | |
| ├── alpha.proto | |
| ├── beta.proto | |
| └── internal | |
| └── gamma.proto | |
| .. _module-pw_protobuf_compiler-add-to-python-package: | |
| Adding Python proto modules to an existing package | |
| -------------------------------------------------- | |
| By default, generated Python proto modules are organized into their own Python | |
| package. These proto modules can instead be added to an existing Python package | |
| declared with ``pw_python_package``. This is done by setting the | |
| ``python_package`` argument on the ``pw_proto_library`` and the | |
| ``proto_library`` argument on the ``pw_python_package``. | |
| For example, the protos declared in ``my_protos`` will be nested in the Python | |
| package declared by ``my_package``. | |
| .. code-block:: | |
| pw_proto_library("my_protos") { | |
| sources = [ "hello.proto ] | |
| prefix = "foo" | |
| python_package = ":my_package" | |
| } | |
| pw_python_pacakge("my_package") { | |
| generate_setup = { | |
| name = "foo" | |
| version = "1.0" | |
| } | |
| sources = [ "foo/cool_module.py" ] | |
| proto_library = ":my_protos" | |
| } | |
| The ``hello_pb2.py`` proto module can be used alongside other files in the | |
| ``foo`` package. | |
| .. code-block:: python | |
| from foo import cool_module, hello_pb2 | |
| Working with externally defined protos | |
| -------------------------------------- | |
| ``pw_proto_library`` targets may be used to build ``.proto`` sources from | |
| existing projects. In these cases, it may be necessary to supply the | |
| ``strip_prefix`` argument, which specifies the protobuf include path to use for | |
| ``protoc``. If only a single external protobuf is being compiled, the | |
| ``python_module_as_package`` option can be used to override the requirement that | |
| the protobuf be nested under a directory. This option generates a Python package | |
| with the same name as the proto file, so that the generated proto can be | |
| imported as if it were a standalone Python module. | |
| For example, the ``pw_proto_library`` target for Nanopb sets | |
| ``python_module_as_package`` to ``nanopb_pb2``. | |
| .. code-block:: | |
| pw_proto_library("proto") { | |
| strip_prefix = "$dir_pw_third_party_nanopb/generator/proto" | |
| sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ] | |
| python_module_as_package = "nanopb_pb2" | |
| } | |
| In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via | |
| the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as | |
| ``#include "nanopb.pwpb.h"``. | |
| The ``python_module_as_package`` feature should only be used when absolutely | |
| necessary --- for example, to support proto files that include | |
| ``import "nanopb.proto"``. | |
| CMake | |
| ===== | |
| CMake provides a ``pw_proto_library`` function with similar features as the | |
| GN template. The CMake build only supports building firmware code, so | |
| ``pw_proto_library`` does not generate a Python package. | |
| **Arguments** | |
| * ``NAME``: the base name of the libraries to create | |
| * ``SOURCES``: .proto source files | |
| * ``DEPS``: dependencies on other ``pw_proto_library`` targets | |
| * ``PREFIX``: prefix add to the proto files | |
| * ``STRIP_PREFIX``: prefix to remove from the proto files | |
| * ``INPUTS``: files to include along with the .proto files (such as Nanopb | |
| .options files) | |
| **Example** | |
| .. code-block:: cmake | |
| include($ENV{PW_ROOT}/pw_build/pigweed.cmake) | |
| include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake) | |
| pw_proto_library(my_module.my_protos | |
| SOURCES | |
| my_protos/foo.proto | |
| my_protos/bar.proto | |
| ) | |
| pw_proto_library(my_module.my_protos | |
| SOURCES | |
| my_protos/foo.proto | |
| my_protos/bar.proto | |
| ) | |
| pw_proto_library(my_module.my_other_protos | |
| SOURCES | |
| some/other/path/baz.proto # imports foo.proto | |
| # This removes the "some/other/path" prefix from the proto files. | |
| STRIP_PREFIX | |
| some/other/path | |
| # This adds the "my_other_protos/" prefix to the proto files. | |
| PREFIX | |
| my_other_protos | |
| # Proto libraries depend on other proto libraries directly. | |
| DEPS | |
| my_module.my_protos | |
| ) | |
| add_library(my_module.my_cc_code | |
| foo.cc | |
| bar.cc | |
| baz.cc | |
| ) | |
| # When depending on protos in a source_set, specify the generator suffix. | |
| target_link_libraries(my_module.my_cc_code PUBLIC | |
| my_module.my_other_protos.pwpb | |
| ) | |
| These proto files are accessed in C++ the same as in the GN build: | |
| .. code-block:: cpp | |
| #include "my_other_protos/baz.pwpb.h" | |
| **Supported Codegen** | |
| CMake supports the following compiled proto libraries via the specified | |
| sub-targets generated by a ``pw_proto_library``. | |
| * ``${NAME}.pwpb`` - Generated C++ pw_protobuf code | |
| * ``${NAME}.nanopb`` - Generated C++ nanopb code (requires Nanopb) | |
| * ``${NAME}.nanopb_rpc`` - Generated C++ Nanopb pw_rpc code (requires Nanopb) | |
| * ``${NAME}.raw_rpc`` - Generated C++ raw pw_rpc code (no protobuf library) | |
| Bazel | |
| ===== | |
| Bazel provides a ``pw_proto_library`` rule with similar features as the | |
| GN template. The Bazel build only supports building firmware code, so | |
| ``pw_proto_library`` does not generate a Python package. The Bazel rules differ | |
| slightly compared to the GN build to be more in line with what would be | |
| considered idiomatic in Bazel. | |
| To use Pigweeds Protobuf rules you must first pull in the required dependencies | |
| into your Bazel WORKSPACE file. e.g. | |
| .. code-block:: python | |
| # WORKSPACE ... | |
| load("@pigweed//pw_protobuf_compiler:deps.bzl", "pw_protobuf_dependencies") | |
| pw_protobuf_dependencies() | |
| Bazel uses a different set of rules to manage proto files than it does to | |
| compile them. e.g. | |
| .. code-block:: python | |
| # BUILD ... | |
| load("@rules_proto//proto:defs.bzl", "proto_library") | |
| load("@pigweed//pw_protobuf_compiler:proto.bzl", "pw_proto_library") | |
| # Manages proto sources and dependencies. | |
| proto_library( | |
| name = "my_proto", | |
| srcs = [ | |
| "my_protos/foo.proto", | |
| "my_protos/bar.proto", | |
| ] | |
| ) | |
| # Compiles dependant protos to C++. | |
| pw_proto_library( | |
| name = "my_cc_proto", | |
| deps = [":my_proto"], | |
| ) | |
| # Library that depends on generated proto targets. | |
| pw_cc_library( | |
| name = "my_lib", | |
| srcs = ["my/lib.cc"], | |
| deps = [":my_cc_proto"], | |
| ) | |
| From ``my/lib.cc`` you can now include the generated headers. | |
| e.g. | |
| .. code:: cpp | |
| #include "my_protos/bar.pwpb.h" | |
| **Supported Codegen** | |
| Bazel supports the following compiled proto libraries via the specified | |
| sub-targets generated by a ``pw_proto_library``. | |
| * ``${NAME}.pwpb`` - Generated C++ pw_protobuf code | |
| * ``${NAME}.nanopb`` - Generated C++ nanopb code (requires Nanopb) | |
| * ``${NAME}.nanopb_rpc`` - Generated C++ Nanopb pw_rpc code (requires Nanopb) | |
| * ``${NAME}.raw_rpc`` - Generated C++ raw pw_rpc code (no protobuf library) |