GettingStarted.md - docs - Git at Google

 # What is this?

 This is Project Shodan, a project to research the fusion of novel hardware and
 software architectures to produce a low-power, ambient AI core. For more
 information, see our
 [internal site](https://sites.google.com/corp/google.com/cerebrahardware/projects/current-projects/shodan).

 ## Developing in this Codebase

 We've stored our code in Gerrit, and like the Android developers before us, we
 use `repo` to manage the projects in our Gerrit repositories.

 To get started, first make sure you have a Git login for all our projects
 by going to [googlesource.com/new-password](https://www.googlesource.com/new-password)
 and pasting the provided script into a terminal.

 Now you need to pull down a copy of the `repo`
 tool from our public facing sites and add it to your path:

 ```bash
 mkdir -p bin
 export PATH=$PATH:$HOME/bin
 curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
 chmod a+x ~/bin/repo
 ```

 Make sure you've initialized git with your name and email address, and have
 configured it properly for fetching the sources:

 ```bash
 git config --global user.name "Your Name"
 git config --global user.email "you@example.com"
 ```

 Once you've done this, you're actually ready to check out the sources. Make a
 new directory where you'd like it to live, and initialize `repo` with the
 current release branch.

 ```bash
 repo init -u https://spacebeaker.googlesource.com/shodan/manifest -g default,internal --no-use-superproject
 repo sync -j$(nproc)
 ```

 ## System Setup for Development

 Development for shodan requires that the necessary tools and prerequisites be
 installed.

 To setup the build system:

 ```bash
 source build/setup.sh
 ```

 For non-gLinux systems: Add the bazel apt repository to your machines sources.
 Instructions @ https://bazel.build/install/ubuntu#add-dis-uri

 Install the prerequisites:

 ```bash
 m prereqs
 ```

 Install the tools used for development:

 ```bash
 m tools -j$(nproc)
 ```

 ## Day-to-day Development Workflow

 In general, working with repo is relatively simple:

   1. Create a working topic branch to make changes on with `repo start
      ${TOPICNAME}`
   2. Make your changes to the files. Add them with `git add -A` and commit with
      `git commit`.
   3. Upload your changes with `repo upload --re ${REVIEWER} --cc ${CC_LIST}`
   4. Go to the URL repo spits out to read and reply to comments.
   5. Eventually your reviewer will give you a +2 LGTM on your change. To submit,
      click the "SUBMIT" button on the change in the web interface.
   6. Run `repo sync -j$(nproc)` to update.
   7. Run `repo prune` to remove your topic branch.

 For more information on how to use repo and git effectively, take a look at the
 [official documentation](https://source.android.com/setup/create/coding-tasks).

 ### A Note on Code Review Policies

 In the Shodan project, we follow a "sticky +2" policy.

 What this means is that you *must* get a +2 Code Review from a peer, or a TL.
 Gerrit will automatically pass any +2 Code Review scores you receive when you
 upload a patchset with minor changes. This policy is in place so that we can
 quickly get through code review and get our code submitted without requiring
 lots of round-trips with your reviewers.

 If you change significant parts of the CL post-+2, please ask for additional
 review from your peers in a reply. Note: two +1s does *not* count as a +2!

 You can't self-+2 your own CL, no matter the urgency or how simple the
 change is! This policy ensures the code change is peer reviewed, and prevent us
 from getting in trouble with security/compliance. If you don't know who to send
 a CL to, put one of the TLs on the review list, and they'll redirect it
 appropriately, or review your change.

 As a reviewer, please use good judgement to help the team keep the good
 momentum: If you provide a +1 score, please note in your comment who should
 provide the +2; if your comments are minor changes about style, format, etc.,
 and not related to the functionality of the code, please provide +2 so the code
 owner can address the comments and submit the code without further review.

 ### Who are the TLs?

 At the time of this writing, the TLs are:

 Cindy Liu <hcindyl@>
 June Tate-Gans <jtgans@>
 Bangfei Pan <pbf@>
 Steve Xu <stevexu@>

 Finally, if you can't get one of the above people, please put Kai Yick
 <kingkai@> on the review line.

 *Special note if you're listed above*: you still must get your +2 from someone
 else -- we need the TLs to lead by example and demonstrate the appropriate
 behavior.

 ### How to sync my local copy with latest?

 ```bash
 repo sync -j$(nproc)
 ```

 Then if you have any outstanding branches, a `repo rebase` will help.

 ### How to send code for review?

 To upload a branch to gerrit for review, do this:

 ```bash
 repo upload --re reviewer1,reviewer2 --cc email@host.com,email2@host2.com
 ```

 Reviewers can be specified as usernames or full email addresses, likewise for
 `--cc`.

 Repo will then output a URL for you to visit that allows you to make comments
 and abandon and merge the changes into the repository. To make changes during
 the review process, make your changes to the files, then:

 ```bash
 git add -A                                    # To add the files you've changed
 git commit --amend                            # To update the previous change
 repo upload -t --re ${REVIEWER} --cc ${CC_LIST} # To upload the change to Gerrit for review
 ```

 ### How to download CLs of a Gerrit topic?

 If you are reviewing a bunch of CLs from different projects, that are grouped
 under a single topic in Gerrit, you can download the latest CL of each project
 (replace `TOPIC` with the topic you are reviewing):

 ```bash
 download-gerrit-topic.py TOPIC
 ```

 By default, the new branches under which the CLs will be downloaded are named
 `TOPIC`. See `download-gerrit-topic.py --help` for more details.

 ### Help! I'm on a limited-bandwidth connection, and need to stop downloading tooling!

 There's an environment variable you can set called `PIN_TOOLCHAINS` to prevent
 download attempts for various toolchains in the system. The format is a
 space-delimited list of words that includes which toolchains you wish to prevent
 downloading automatically during a build.

 At the moment, this environment variable can be set to any of the following
 words:

   `renode` - prevents downloading of the latest Renode binary build.

   `iree`   - prevents downloading of the latest IREE compiler toolchain.

 Ie:

 ```bash
 export PIN_TOOLCHAINS="renode iree"
 ```

 *Note well*: if you set this variable, the output or behavior of any of the
 preceding toolchains may be bad or incorrect. Please do not file bugs against
 these components if you have this variable set!

 ### Help! IREE, Renode, or another toolchain is misbehaving!

 Did you set `PIN_TOOLCHAINS`? If so, please unset this environment variable and
 retry your build.

 ## Repository Layout

 Our layout is pretty simple:

 ### build/

 Contains build scripts for the whole tree. This is effectively just an
 orchestration layer to make building the whole shebang easier. Each subtree may
 have its own build systems and have their own ways of building.

 ### cache/

 The cached cross-compilation toolchain, including rust and RISC-V GCC/LLVM
 toolchain.

 ### cicd/

 Contains continuous integration scripts and tooling for Jenkins, our CI/CD tool.

 ### docs/

 Lots of extra documentation (we hope) about how the repo is laid out, how the
 build system works, code reviews, licensing, etc.

 ### hw/

 Contains all of the source code and RTL required to build the Shodan
 hardware, as well as simulate it in Verilator.

 #### opentitan

 Security core.

 #### ibex

 System management controller (SMC).

 #### springbok

 Vector core for ML acceleration.

 ### cantrip/

 Operating system software for the SMC; including seL4 kernel & CAmkES framework,
 and custom CAmkES components that support Shodan (or maybe CantripOS) applications.

 ### manifest/

 The repo manifest used to glue all the git repositories together.

 ### scripts/

 Contains utility scripts to help automate a few things.

 ### sim/

 Contains tools and src for simulators (Renode and Verilator) of the shodan system.

 ### sw/

 Contains the source code of applications running in all shodan cores.

 #### libtock-rs

 **TODO**: add more details

 #### multihart_boot_rom

 Bootstrap for System Management Controller, Security Core, and Vector Core. This
 is the first software to run after reset; it does low-level hardware setup and
 starts TockOs (SC) and seL4 (SMC).

 #### pigweed

 pigweed frameworks. Currently it is used for vector core functional tests.

 #### tock

 The operating system running on the Security Core.

 #### vec

 Springbok BSP, as well as the RVV instruction functional tests.

 #### vec_iree

 Springbok IREE application. It builds IREE runtime applications for ML models
 using IREE libraries and Spingbok BSP.

 ### toolchain/

 Contains the src to build the RISCV QEMU emulator, and
 [IREE](https://github/com/google/iree) toolchain for ML models.

 ## Build and Test ML Artifacts

 The ML executable is built with [IREE](https://github.com/google/iree) workflow,
 targeted to RISCV 32-bit bare-metal config.

 To build the IREE targets:

 ```bash
 m iree_no_wmmu
 ```

 The IREE compiler sits in `out/host/iree_compiler`, while the runtime library/example
 sits in `out/springbok_iree_no_wmmu`.
 To run the toy example (four-element vector element-wise multiplication) for
 testing:

 ```bash
 sim_springbok out/springbok_iree_no_wmmu/springbok_iree/samples/simple_vec_mul/simple_int_vec_mul_bytecode_static
 ```

 The output should be shown as:

 ```bash
 21:27:11.0241 [INFO] cpu2: simprint: "INFO |simple_vec_mul finished successfully", 0 (0x0)
 21:27:11.0248 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)
 ```

 (Enter `quit` or `q` to exit the Renode simulation)

 ## Running The Full Shodan System Simulation

 The simulator used for Shodan is [Renode](https://renode.io/). The configuration
 for the Shodan system is `sim/config/platforms/shodan.repl`, while and starting script
 is in `sim/config/shodan.resc`

 To run the full system simulation, build the default target:

 ```bash
 m
 ```

 After all the artifacts are bulit, the Renode simulation session automatically
 starts, and you should see the secured core and SMC booted

 ```bash
 10:11:43.1993 [INFO] uart5: [output] init_kernel()
 10:11:43.2012 [INFO] uart5: [output] Init local IRQ
 10:11:43.2018 [INFO] uart5: [output] Bootstrapping kernel
 10:11:43.2053 [INFO] uart5: [output] Initialing PLIC...
 10:11:43.2830 [INFO] uart5: [output] Booting all finished, dropped to user space
 10:11:43.2978 [INFO] uart5: [output] cantrip_os_rootserver::Bootinfo: (485, 131072) empty slots 1 nodes (15, 81) untyped 131072 cnode slots
 10:11:43.3008 [INFO] uart5: [output] cantrip_os_rootserver::Model: 1011 objects 32 irqs 0 untypeds 2 asids
 10:11:43.3123 [INFO] uart5: [output] cantrip_os_rootserver::capDL spec: 0.10 Mbytes
 10:11:43.3142 [INFO] uart5: [output] cantrip_os_rootserver::CAmkES components: 1.08 Mbytes
 10:11:43.3191 [INFO] uart5: [output] cantrip_os_rootserver::Rootserver executable: 0.39 Mbytes
 ```

 At this point, you should be able to connect to the SMC debug console.

 ### Setting Up Debug Console Communication

 Install socat to your machine

 ```bash
 sudo apt install socat
 ```

 In a new shell session, launch the debug console with

 ```bash
 scripts/kshell.sh
 ```

 You will see the CantripOS prompt showing up when you hit `enter`. Use `ctrl-c` to leave the console.

 ### Connecting to Renode Console

 Renode has its own console to control the emulation environment. You can connect to it by

 ```bash
 telnet localhost 1234
 ```

 For example, you can check the core status

 ```bash
 (matcha) cpu1 IsHalted
 False
 ```

 or stop the whole emulation with

 ```bash
 (matcha) quit
 ```

 ### Launch ML Job on the Vector Core

 You can launch the ML workload execution by communicating through the debug console

 Check existing builtin applications

 ```bash
 CANTRIP> builtins
 ```

 Install the embedded ML model app; for example,

 ```bash
 CANTRIP> install mobilenet_v1_emitc_static.model
 ```

 and you should see model installed.

 You can then launch the ML job with `test_mlexecute anything <model name>`

 ```bash
 CANTRIP> test_mlexecute anything mobilenet_v1_emitc_static
 ```

 with the Renode console log printed as

 ```bash
 15:17:59.9864 [INFO] uart5: [output] cantrip_ml_coordinator::Load successful.
 ...
 15:18:02.6868 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)

 ```

 ## More Information

 For more available Shodan build targets, please see [Build target lists](./BuildTargetsExplained.md),
 or use the in-project command of `hmm` and `hmm <target name>`.

 Also, [Information on how to use repo](https://go/repo)
	# What is this?

	This is Project Shodan, a project to research the fusion of novel hardware and
	software architectures to produce a low-power, ambient AI core. For more
	information, see our
	[internal site](https://sites.google.com/corp/google.com/cerebrahardware/projects/current-projects/shodan).

	## Developing in this Codebase

	We've stored our code in Gerrit, and like the Android developers before us, we
	use `repo` to manage the projects in our Gerrit repositories.

	To get started, first make sure you have a Git login for all our projects
	by going to [googlesource.com/new-password](https://www.googlesource.com/new-password)
	and pasting the provided script into a terminal.

	Now you need to pull down a copy of the `repo`
	tool from our public facing sites and add it to your path:

	```bash
	mkdir -p bin
	export PATH=$PATH:$HOME/bin
	curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
	chmod a+x ~/bin/repo
	```

	Make sure you've initialized git with your name and email address, and have
	configured it properly for fetching the sources:

	```bash
	git config --global user.name "Your Name"
	git config --global user.email "you@example.com"
	```

	Once you've done this, you're actually ready to check out the sources. Make a
	new directory where you'd like it to live, and initialize `repo` with the
	current release branch.

	```bash
	repo init -u https://spacebeaker.googlesource.com/shodan/manifest -g default,internal --no-use-superproject
	repo sync -j$(nproc)
	```

	## System Setup for Development

	Development for shodan requires that the necessary tools and prerequisites be
	installed.

	To setup the build system:

	```bash
	source build/setup.sh
	```

	For non-gLinux systems: Add the bazel apt repository to your machines sources.
	Instructions @ https://bazel.build/install/ubuntu#add-dis-uri

	Install the prerequisites:

	```bash
	m prereqs
	```

	Install the tools used for development:

	```bash
	m tools -j$(nproc)
	```

	## Day-to-day Development Workflow

	In general, working with repo is relatively simple:

	1. Create a working topic branch to make changes on with `repo start
	${TOPICNAME}`
	2. Make your changes to the files. Add them with `git add -A` and commit with
	`git commit`.
	3. Upload your changes with `repo upload --re ${REVIEWER} --cc ${CC_LIST}`
	4. Go to the URL repo spits out to read and reply to comments.
	5. Eventually your reviewer will give you a +2 LGTM on your change. To submit,
	click the "SUBMIT" button on the change in the web interface.
	6. Run `repo sync -j$(nproc)` to update.
	7. Run `repo prune` to remove your topic branch.

	For more information on how to use repo and git effectively, take a look at the
	[official documentation](https://source.android.com/setup/create/coding-tasks).

	### A Note on Code Review Policies

	In the Shodan project, we follow a "sticky +2" policy.

	What this means is that you must get a +2 Code Review from a peer, or a TL.
	Gerrit will automatically pass any +2 Code Review scores you receive when you
	upload a patchset with minor changes. This policy is in place so that we can
	quickly get through code review and get our code submitted without requiring
	lots of round-trips with your reviewers.

	If you change significant parts of the CL post-+2, please ask for additional
	review from your peers in a reply. Note: two +1s does not count as a +2!

	You can't self-+2 your own CL, no matter the urgency or how simple the
	change is! This policy ensures the code change is peer reviewed, and prevent us
	from getting in trouble with security/compliance. If you don't know who to send
	a CL to, put one of the TLs on the review list, and they'll redirect it
	appropriately, or review your change.

	As a reviewer, please use good judgement to help the team keep the good
	momentum: If you provide a +1 score, please note in your comment who should
	provide the +2; if your comments are minor changes about style, format, etc.,
	and not related to the functionality of the code, please provide +2 so the code
	owner can address the comments and submit the code without further review.

	### Who are the TLs?

	At the time of this writing, the TLs are:

	Cindy Liu <hcindyl@>
	June Tate-Gans <jtgans@>
	Bangfei Pan <pbf@>
	Steve Xu <stevexu@>

	Finally, if you can't get one of the above people, please put Kai Yick
	<kingkai@> on the review line.

	Special note if you're listed above: you still must get your +2 from someone
	else -- we need the TLs to lead by example and demonstrate the appropriate
	behavior.

	### How to sync my local copy with latest?

	```bash
	repo sync -j$(nproc)
	```

	Then if you have any outstanding branches, a `repo rebase` will help.

	### How to send code for review?

	To upload a branch to gerrit for review, do this:

	```bash
	repo upload --re reviewer1,reviewer2 --cc email@host.com,email2@host2.com
	```

	Reviewers can be specified as usernames or full email addresses, likewise for
	`--cc`.

	Repo will then output a URL for you to visit that allows you to make comments
	and abandon and merge the changes into the repository. To make changes during
	the review process, make your changes to the files, then:

	```bash
	git add -A # To add the files you've changed
	git commit --amend # To update the previous change
	repo upload -t --re ${REVIEWER} --cc ${CC_LIST} # To upload the change to Gerrit for review
	```

	### How to download CLs of a Gerrit topic?

	If you are reviewing a bunch of CLs from different projects, that are grouped
	under a single topic in Gerrit, you can download the latest CL of each project
	(replace `TOPIC` with the topic you are reviewing):

	```bash
	download-gerrit-topic.py TOPIC
	```

	By default, the new branches under which the CLs will be downloaded are named
	`TOPIC`. See `download-gerrit-topic.py --help` for more details.

	### Help! I'm on a limited-bandwidth connection, and need to stop downloading tooling!

	There's an environment variable you can set called `PIN_TOOLCHAINS` to prevent
	download attempts for various toolchains in the system. The format is a
	space-delimited list of words that includes which toolchains you wish to prevent
	downloading automatically during a build.

	At the moment, this environment variable can be set to any of the following
	words:

	`renode` - prevents downloading of the latest Renode binary build.

	`iree` - prevents downloading of the latest IREE compiler toolchain.

	Ie:

	```bash
	export PIN_TOOLCHAINS="renode iree"
	```

	Note well: if you set this variable, the output or behavior of any of the
	preceding toolchains may be bad or incorrect. Please do not file bugs against
	these components if you have this variable set!

	### Help! IREE, Renode, or another toolchain is misbehaving!

	Did you set `PIN_TOOLCHAINS`? If so, please unset this environment variable and
	retry your build.

	## Repository Layout

	Our layout is pretty simple:

	### build/

	Contains build scripts for the whole tree. This is effectively just an
	orchestration layer to make building the whole shebang easier. Each subtree may
	have its own build systems and have their own ways of building.

	### cache/

	The cached cross-compilation toolchain, including rust and RISC-V GCC/LLVM
	toolchain.

	### cicd/

	Contains continuous integration scripts and tooling for Jenkins, our CI/CD tool.

	### docs/

	Lots of extra documentation (we hope) about how the repo is laid out, how the
	build system works, code reviews, licensing, etc.

	### hw/

	Contains all of the source code and RTL required to build the Shodan
	hardware, as well as simulate it in Verilator.

	#### opentitan

	Security core.

	#### ibex

	System management controller (SMC).

	#### springbok

	Vector core for ML acceleration.

	### cantrip/

	Operating system software for the SMC; including seL4 kernel & CAmkES framework,
	and custom CAmkES components that support Shodan (or maybe CantripOS) applications.

	### manifest/

	The repo manifest used to glue all the git repositories together.

	### scripts/

	Contains utility scripts to help automate a few things.

	### sim/

	Contains tools and src for simulators (Renode and Verilator) of the shodan system.

	### sw/

	Contains the source code of applications running in all shodan cores.

	#### libtock-rs

	TODO: add more details

	#### multihart_boot_rom

	Bootstrap for System Management Controller, Security Core, and Vector Core. This
	is the first software to run after reset; it does low-level hardware setup and
	starts TockOs (SC) and seL4 (SMC).

	#### pigweed

	pigweed frameworks. Currently it is used for vector core functional tests.

	#### tock

	The operating system running on the Security Core.

	#### vec

	Springbok BSP, as well as the RVV instruction functional tests.

	#### vec_iree

	Springbok IREE application. It builds IREE runtime applications for ML models
	using IREE libraries and Spingbok BSP.

	### toolchain/

	Contains the src to build the RISCV QEMU emulator, and
	[IREE](https://github/com/google/iree) toolchain for ML models.

	## Build and Test ML Artifacts

	The ML executable is built with [IREE](https://github.com/google/iree) workflow,
	targeted to RISCV 32-bit bare-metal config.

	To build the IREE targets:

	```bash
	m iree_no_wmmu
	```

	The IREE compiler sits in `out/host/iree_compiler`, while the runtime library/example
	sits in `out/springbok_iree_no_wmmu`.
	To run the toy example (four-element vector element-wise multiplication) for
	testing:

	```bash
	sim_springbok out/springbok_iree_no_wmmu/springbok_iree/samples/simple_vec_mul/simple_int_vec_mul_bytecode_static
	```

	The output should be shown as:

	```bash
	21:27:11.0241 [INFO] cpu2: simprint: "INFO \|simple_vec_mul finished successfully", 0 (0x0)
	21:27:11.0248 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)
	```

	(Enter `quit` or `q` to exit the Renode simulation)

	## Running The Full Shodan System Simulation

	The simulator used for Shodan is [Renode](https://renode.io/). The configuration
	for the Shodan system is `sim/config/platforms/shodan.repl`, while and starting script
	is in `sim/config/shodan.resc`

	To run the full system simulation, build the default target:

	```bash
	m
	```

	After all the artifacts are bulit, the Renode simulation session automatically
	starts, and you should see the secured core and SMC booted

	```bash
	10:11:43.1993 [INFO] uart5: [output] init_kernel()
	10:11:43.2012 [INFO] uart5: [output] Init local IRQ
	10:11:43.2018 [INFO] uart5: [output] Bootstrapping kernel
	10:11:43.2053 [INFO] uart5: [output] Initialing PLIC...
	10:11:43.2830 [INFO] uart5: [output] Booting all finished, dropped to user space
	10:11:43.2978 [INFO] uart5: [output] cantrip_os_rootserver::Bootinfo: (485, 131072) empty slots 1 nodes (15, 81) untyped 131072 cnode slots
	10:11:43.3008 [INFO] uart5: [output] cantrip_os_rootserver::Model: 1011 objects 32 irqs 0 untypeds 2 asids
	10:11:43.3123 [INFO] uart5: [output] cantrip_os_rootserver::capDL spec: 0.10 Mbytes
	10:11:43.3142 [INFO] uart5: [output] cantrip_os_rootserver::CAmkES components: 1.08 Mbytes
	10:11:43.3191 [INFO] uart5: [output] cantrip_os_rootserver::Rootserver executable: 0.39 Mbytes
	```

	At this point, you should be able to connect to the SMC debug console.

	### Setting Up Debug Console Communication

	Install socat to your machine

	```bash
	sudo apt install socat
	```

	In a new shell session, launch the debug console with

	```bash
	scripts/kshell.sh
	```

	You will see the CantripOS prompt showing up when you hit `enter`. Use `ctrl-c` to leave the console.

	### Connecting to Renode Console

	Renode has its own console to control the emulation environment. You can connect to it by

	```bash
	telnet localhost 1234
	```

	For example, you can check the core status

	```bash
	(matcha) cpu1 IsHalted
	False
	```

	or stop the whole emulation with

	```bash
	(matcha) quit
	```

	### Launch ML Job on the Vector Core

	You can launch the ML workload execution by communicating through the debug console

	Check existing builtin applications

	```bash
	CANTRIP> builtins
	```

	Install the embedded ML model app; for example,

	```bash
	CANTRIP> install mobilenet_v1_emitc_static.model
	```

	and you should see model installed.

	You can then launch the ML job with `test_mlexecute anything <model name>`

	```bash
	CANTRIP> test_mlexecute anything mobilenet_v1_emitc_static
	```

	with the Renode console log printed as

	```bash
	15:17:59.9864 [INFO] uart5: [output] cantrip_ml_coordinator::Load successful.
	...
	15:18:02.6868 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)

	```

	## More Information

	For more available Shodan build targets, please see [Build target lists](./BuildTargetsExplained.md),
	or use the in-project command of `hmm` and `hmm <target name>`.

	Also, [Information on how to use repo](https://go/repo)