blob: 9825f1aeb6dec3bd72c9de5b0f1c3e46271142c7 [file] [log] [blame] [view]
June Tate-Gans4447d0e2020-06-25 14:31:51 -05001# What is this?
2
3This is Project Shodan, a project to research the fusion of novel hardware and
4software architectures to produce a low-power, ambient AI core. For more
Cindy Liuc1485e72021-08-25 19:39:36 -07005information, see our
6[internal site](https://sites.google.com/corp/google.com/cerebrahardware/projects/current-projects/shodan).
June Tate-Gans4447d0e2020-06-25 14:31:51 -05007
8## Developing in this Codebase
9
10We've stored our code in Gerrit, and like the Android developers before us, we
11use `repo` to manage the projects in our Gerrit repositories.
12
13To get started, first make sure you have a Git login for all our projects
14by going to [googlesource.com/new-password](https://www.googlesource.com/new-password)
15and pasting the provided script into a terminal.
16
17Now you need to pull down a copy of the `repo`
18tool from our public facing sites and add it to your path:
19
Cindy Liud9b324e2021-08-12 18:21:43 -070020```bash
June Tate-Gans4447d0e2020-06-25 14:31:51 -050021mkdir -p bin
22export PATH=$PATH:$HOME/bin
23curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
24chmod a+x ~/bin/repo
25```
26
27Make sure you've initialized git with your name and email address, and have
28configured it properly for fetching the sources:
29
Cindy Liud9b324e2021-08-12 18:21:43 -070030```bash
June Tate-Gans4447d0e2020-06-25 14:31:51 -050031git config --global user.name "Your Name"
32git config --global user.email "you@example.com"
33```
34
35Once you've done this, you're actually ready to check out the sources. Make a
36new directory where you'd like it to live, and initialize `repo` with the
37current release branch.
38
Cindy Liud9b324e2021-08-12 18:21:43 -070039```bash
Cindy Liu73984b02021-09-09 13:46:41 -070040repo init -u https://spacebeaker.googlesource.com/shodan/manifest -g default,internal --no-use-superproject
June Tate-Gans4447d0e2020-06-25 14:31:51 -050041repo sync -j$(nproc)
42```
43
Henry Herman0c341b42020-09-23 11:56:26 -070044## System Setup for Development
45
46Development for shodan requires that the necessary tools and prerequisites be
47installed.
48
49To setup the build system:
50
Cindy Liud9b324e2021-08-12 18:21:43 -070051```bash
Henry Herman0c341b42020-09-23 11:56:26 -070052source build/setup.sh
53```
54
Julian Mullings-Black1b833192022-07-19 00:47:15 +000055For non-gLinux systems: Add the bazel apt repository to your machines sources.
56Instructions @ https://bazel.build/install/ubuntu#add-dis-uri
57
Henry Herman0c341b42020-09-23 11:56:26 -070058Install the prerequisites:
59
Cindy Liud9b324e2021-08-12 18:21:43 -070060```bash
Henry Herman0c341b42020-09-23 11:56:26 -070061m prereqs
62```
63
Cindy Liuc1485e72021-08-25 19:39:36 -070064Install the tools used for development:
Henry Herman0c341b42020-09-23 11:56:26 -070065
Cindy Liud9b324e2021-08-12 18:21:43 -070066```bash
Jonas Larssonb490c4e2022-11-09 11:03:33 -080067m tools -j$(nproc)
Henry Herman0c341b42020-09-23 11:56:26 -070068```
69
June Tate-Gans63e07872020-06-25 16:23:28 -050070## Day-to-day Development Workflow
71
June Tate-Gans0ff47892020-06-25 17:11:03 -050072In general, working with repo is relatively simple:
June Tate-Gans63e07872020-06-25 16:23:28 -050073
74 1. Create a working topic branch to make changes on with `repo start
75 ${TOPICNAME}`
76 2. Make your changes to the files. Add them with `git add -A` and commit with
77 `git commit`.
June Tate-Gans0ff47892020-06-25 17:11:03 -050078 3. Upload your changes with `repo upload --re ${REVIEWER} --cc ${CC_LIST}`
June Tate-Gans63e07872020-06-25 16:23:28 -050079 4. Go to the URL repo spits out to read and reply to comments.
80 5. Eventually your reviewer will give you a +2 LGTM on your change. To submit,
81 click the "SUBMIT" button on the change in the web interface.
82 6. Run `repo sync -j$(nproc)` to update.
83 7. Run `repo prune` to remove your topic branch.
84
85For more information on how to use repo and git effectively, take a look at the
86[official documentation](https://source.android.com/setup/create/coding-tasks).
87
June Tate-Ganse8c4a092022-04-18 20:35:57 +000088### A Note on Code Review Policies
89
90In the Shodan project, we follow a "sticky +2" policy.
91
92What this means is that you *must* get a +2 Code Review from a peer, or a TL.
Cindy Liu9b2a2c12022-05-02 14:56:13 -070093Gerrit will automatically pass any +2 Code Review scores you receive when you
94upload a patchset with minor changes. This policy is in place so that we can
95quickly get through code review and get our code submitted without requiring
96lots of round-trips with your reviewers.
June Tate-Ganse8c4a092022-04-18 20:35:57 +000097
98If you change significant parts of the CL post-+2, please ask for additional
99review from your peers in a reply. Note: two +1s does *not* count as a +2!
100
Cindy Liu9b2a2c12022-05-02 14:56:13 -0700101You can't self-+2 your own CL, no matter the urgency or how simple the
102change is! This policy ensures the code change is peer reviewed, and prevent us
103from getting in trouble with security/compliance. If you don't know who to send
104a CL to, put one of the TLs on the review list, and they'll redirect it
105appropriately, or review your change.
June Tate-Ganse8c4a092022-04-18 20:35:57 +0000106
Cindy Liu9b2a2c12022-05-02 14:56:13 -0700107As a reviewer, please use good judgement to help the team keep the good
108momentum: If you provide a +1 score, please note in your comment who should
109provide the +2; if your comments are minor changes about style, format, etc.,
110and not related to the functionality of the code, please provide +2 so the code
111owner can address the comments and submit the code without further review.
June Tate-Ganse8c4a092022-04-18 20:35:57 +0000112
113### Who are the TLs?
114
115At the time of this writing, the TLs are:
116
117Cindy Liu <hcindyl@>
118June Tate-Gans <jtgans@>
119Bangfei Pan <pbf@>
120Steve Xu <stevexu@>
121
122Finally, if you can't get one of the above people, please put Kai Yick
123<kingkai@> on the review line.
124
125*Special note if you're listed above*: you still must get your +2 from someone
126else -- we need the TLs to lead by example and demonstrate the appropriate
127behavior.
128
June Tate-Gans63e07872020-06-25 16:23:28 -0500129### How to sync my local copy with latest?
130
Cindy Liud9b324e2021-08-12 18:21:43 -0700131```bash
June Tate-Gans63e07872020-06-25 16:23:28 -0500132repo sync -j$(nproc)
133```
134
135Then if you have any outstanding branches, a `repo rebase` will help.
136
137### How to send code for review?
138
139To upload a branch to gerrit for review, do this:
140
Cindy Liud9b324e2021-08-12 18:21:43 -0700141```bash
June Tate-Gans0ff47892020-06-25 17:11:03 -0500142repo upload --re reviewer1,reviewer2 --cc email@host.com,email2@host2.com
June Tate-Gans63e07872020-06-25 16:23:28 -0500143```
144
June Tate-Gans0ff47892020-06-25 17:11:03 -0500145Reviewers can be specified as usernames or full email addresses, likewise for
146`--cc`.
147
June Tate-Gans63e07872020-06-25 16:23:28 -0500148Repo will then output a URL for you to visit that allows you to make comments
149and abandon and merge the changes into the repository. To make changes during
150the review process, make your changes to the files, then:
151
Cindy Liud9b324e2021-08-12 18:21:43 -0700152```bash
June Tate-Gans63e07872020-06-25 16:23:28 -0500153git add -A # To add the files you've changed
154git commit --amend # To update the previous change
June Tate-Gans0ff47892020-06-25 17:11:03 -0500155repo upload -t --re ${REVIEWER} --cc ${CC_LIST} # To upload the change to Gerrit for review
June Tate-Gans63e07872020-06-25 16:23:28 -0500156```
157
June Tate-Gans31cd1c02022-08-23 15:33:50 -0500158### Help! I'm on a limited-bandwidth connection, and need to stop downloading tooling!
159
160There's an environment variable you can set called `PIN_TOOLCHAINS` to prevent
161download attempts for various toolchains in the system. The format is a
162space-delimited list of words that includes which toolchains you wish to prevent
163downloading automatically during a build.
164
165At the moment, this environment variable can be set to any of the following
166words:
167
168 `renode` - prevents downloading of the latest Renode binary build.
169
170 `iree` - prevents downloading of the latest IREE compiler toolchain.
171
172Ie:
173
174```bash
175export PIN_TOOLCHAINS="renode iree"
176```
177
178*Note well*: if you set this variable, the output or behavior of any of the
179preceding toolchains may be bad or incorrect. Please do not file bugs against
180these components if you have this variable set!
181
182### Help! IREE, Renode, or another toolchain is misbehaving!
183
184Did you set `PIN_TOOLCHAINS`? If so, please unset this environment variable and
185retry your build.
186
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500187## Repository Layout
188
189Our layout is pretty simple:
190
Cindy Liud9b324e2021-08-12 18:21:43 -0700191### build/
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500192
193Contains build scripts for the whole tree. This is effectively just an
194orchestration layer to make building the whole shebang easier. Each subtree may
195have its own build systems and have their own ways of building.
196
Cindy Liud9b324e2021-08-12 18:21:43 -0700197### cache/
198
199The cached cross-compilation toolchain, including rust and RISC-V GCC/LLVM
200toolchain.
201
202### cicd/
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500203
204Contains continuous integration scripts and tooling for Jenkins, our CI/CD tool.
205
Cindy Liud9b324e2021-08-12 18:21:43 -0700206### docs/
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500207
208Lots of extra documentation (we hope) about how the repo is laid out, how the
209build system works, code reviews, licensing, etc.
210
Cindy Liuc1485e72021-08-25 19:39:36 -0700211### hw/
212
213Contains all of the source code and RTL required to build the Shodan
214hardware, as well as simulate it in Verilator.
215
216#### opentitan
217
218Security core.
219
220#### ibex
221
222System management controller (SMC).
223
224#### springbok
225
226Vector core for ML acceleration.
227
June Tate-Gans280f4142022-11-21 12:54:01 -0600228### cantrip/
Cindy Liuc1485e72021-08-25 19:39:36 -0700229
Cindy Liu73984b02021-09-09 13:46:41 -0700230Operating system software for the SMC; including seL4 kernel & CAmkES framework,
June Tate-Gans280f4142022-11-21 12:54:01 -0600231and custom CAmkES components that support Shodan (or maybe CantripOS) applications.
Cindy Liuc1485e72021-08-25 19:39:36 -0700232
Cindy Liud9b324e2021-08-12 18:21:43 -0700233### manifest/
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500234
235The repo manifest used to glue all the git repositories together.
236
Cindy Liud9b324e2021-08-12 18:21:43 -0700237### scripts/
June Tate-Gans63e07872020-06-25 16:23:28 -0500238
239Contains utility scripts to help automate a few things.
240
Cindy Liud9b324e2021-08-12 18:21:43 -0700241### sim/
Henry Herman0c341b42020-09-23 11:56:26 -0700242
Cindy Liuc1485e72021-08-25 19:39:36 -0700243Contains tools and src for simulators (Renode and Verilator) of the shodan system.
244
245### sw/
246
247Contains the source code of applications running in all shodan cores.
248
249#### libtock-rs
250
251**TODO**: add more details
252
253#### multihart_boot_rom
254
Cindy Liu73984b02021-09-09 13:46:41 -0700255Bootstrap for System Management Controller, Security Core, and Vector Core. This
256is the first software to run after reset; it does low-level hardware setup and
257starts TockOs (SC) and seL4 (SMC).
Cindy Liuc1485e72021-08-25 19:39:36 -0700258
259#### pigweed
260
261pigweed frameworks. Currently it is used for vector core functional tests.
262
263#### tock
264
265The operating system running on the Security Core.
266
267#### vec
268
269Springbok BSP, as well as the RVV instruction functional tests.
270
271#### vec_iree
272
Cindy Liu73984b02021-09-09 13:46:41 -0700273Springbok IREE application. It builds IREE runtime applications for ML models
274using IREE libraries and Spingbok BSP.
Henry Herman0c341b42020-09-23 11:56:26 -0700275
Cindy Liud9b324e2021-08-12 18:21:43 -0700276### toolchain/
Cindy Liu1f238fa2021-03-17 10:59:05 -0700277
Cindy Liud9b324e2021-08-12 18:21:43 -0700278Contains the src to build the RISCV QEMU emulator, and
Cindy Liu1f238fa2021-03-17 10:59:05 -0700279[IREE](https://github/com/google/iree) toolchain for ML models.
280
Cindy Liuc1485e72021-08-25 19:39:36 -0700281## Build and Test ML Artifacts
Cindy Liu1f238fa2021-03-17 10:59:05 -0700282
Cindy Liud9b324e2021-08-12 18:21:43 -0700283The ML executable is built with [IREE](https://github.com/google/iree) workflow,
284targeted to RISCV 32-bit bare-metal config.
Cindy Liu1f238fa2021-03-17 10:59:05 -0700285
Cindy Liu1f238fa2021-03-17 10:59:05 -0700286To build the IREE targets:
Cindy Liud9b324e2021-08-12 18:21:43 -0700287
288```bash
Shaked Flur1b7191a2022-11-02 08:52:38 +0000289m iree_no_wmmu
Cindy Liu1f238fa2021-03-17 10:59:05 -0700290```
291
Cindy Liud9b324e2021-08-12 18:21:43 -0700292The IREE compiler sits in `out/host/iree_compiler`, while the runtime library/example
Shaked Flur1b7191a2022-11-02 08:52:38 +0000293sits in `out/springbok_iree_no_wmmu`.
Cindy Liud9b324e2021-08-12 18:21:43 -0700294To run the toy example (four-element vector element-wise multiplication) for
295testing:
296
297```bash
Shaked Flur1b7191a2022-11-02 08:52:38 +0000298sim_springbok out/springbok_iree_no_wmmu/springbok_iree/samples/simple_vec_mul/simple_int_vec_mul_bytecode_static
Cindy Liu1f238fa2021-03-17 10:59:05 -0700299```
300
301The output should be shown as:
Cindy Liu1f238fa2021-03-17 10:59:05 -0700302
Cindy Liud9b324e2021-08-12 18:21:43 -0700303```bash
Cindy Liu05320422021-09-29 21:31:10 -070030421:27:11.0241 [INFO] cpu2: simprint: "INFO |simple_vec_mul finished successfully", 0 (0x0)
30521:27:11.0248 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)
Cindy Liu1f238fa2021-03-17 10:59:05 -0700306```
Cindy Liud9b324e2021-08-12 18:21:43 -0700307
Cindy Liu05320422021-09-29 21:31:10 -0700308(Enter `quit` or `q` to exit the Renode simulation)
Cindy Liu1f238fa2021-03-17 10:59:05 -0700309
Cindy Liuc1485e72021-08-25 19:39:36 -0700310## Running The Full Shodan System Simulation
311
Cindy Liu73984b02021-09-09 13:46:41 -0700312The simulator used for Shodan is [Renode](https://renode.io/). The configuration
Henry Herman22889152022-01-06 22:03:58 +0000313for the Shodan system is `sim/config/platforms/shodan.repl`, while and starting script
314is in `sim/config/shodan.resc`
Cindy Liuc1485e72021-08-25 19:39:36 -0700315
316To run the full system simulation, build the default target:
317
318```bash
319m
320```
321
Cindy Liu73984b02021-09-09 13:46:41 -0700322After all the artifacts are bulit, the Renode simulation session automatically
323starts, and you should see the secured core and SMC booted
Cindy Liuc1485e72021-08-25 19:39:36 -0700324
325```bash
Cindy Liu1337cf72022-06-23 10:27:03 -070032610:11:43.1993 [INFO] uart5: init_kernel()
32710:11:43.2012 [INFO] uart5: Init local IRQ
32810:11:43.2018 [INFO] uart5: Bootstrapping kernel
32910:11:43.2053 [INFO] uart5: Initialing PLIC...
33010:11:43.2830 [INFO] uart5: Booting all finished, dropped to user space
33110:11:43.2978 [INFO] uart5: kata_os_rootserver::Bootinfo: 130550 empty slots 1 nodes 56 untyped 131072 cnode slots
33210:11:43.3008 [INFO] uart5: kata_os_rootserver::Model: 3750 objects 32 irqs 0 untypeds 2 asids
33310:11:43.3123 [INFO] uart5: kata_os_rootserver::capDL spec: 0.33 Mbytes
33410:11:43.3142 [INFO] uart5: kata_os_rootserver::CAmkES components: 1.02 Mbytes
33510:11:43.3191 [INFO] uart5: kata_os_rootserver::Rootserver executable: 0.40 Mbytes
Cindy Liuc1485e72021-08-25 19:39:36 -0700336```
337
338At this point, you should be able to connect to the SMC debug console.
339
340### Setting Up Debug Console Communication
341
342Install socat to your machine
343
344```bash
345sudo apt install socat
346```
347
348In a new shell session, launch the debug console with
349
350```bash
351scripts/kshell.sh
352```
353
June Tate-Gans280f4142022-11-21 12:54:01 -0600354You will see the CantripOS prompt showing up when you hit `enter`. Use `ctrl-c` to leave the console.
Cindy Liuc1485e72021-08-25 19:39:36 -0700355
356### Connecting to Renode Console
357
358Renode has its own console to control the emulation environment. You can connect to it by
359
360```bash
361telnet localhost 1234
362```
363
364For example, you can check the core status
365
366```bash
367(matcha) cpu1 IsHalted
368False
369```
370
371or stop the whole emulation with
372
373```bash
374(matcha) quit
375```
376
377### Launch ML Job on the Vector Core
378
379You can launch the ML workload execution by communicating through the debug console
380
Cindy Liu1337cf72022-06-23 10:27:03 -0700381Check existing builtin applications
382
Cindy Liuc1485e72021-08-25 19:39:36 -0700383```bash
Cindy Liu1337cf72022-06-23 10:27:03 -0700384KATA> builtins
Cindy Liuc1485e72021-08-25 19:39:36 -0700385```
386
Cindy Liu1337cf72022-06-23 10:27:03 -0700387Install the embedded ML model app; for example,
Cindy Liuc1485e72021-08-25 19:39:36 -0700388
389```bash
Cindy Liu1337cf72022-06-23 10:27:03 -0700390KATA> install mobilenet_v1_emitc_static.model
391```
392
393and you should see model installed with a bundle ID issued.
394
395```bash
396KATA> Bundle "fake.26" installed
397```
398
399You can then launch the ML job with `test_mlexecute <bundle ID> <model name>`
400
401```bash
402KATA> test_mlexecute fake.26 mobilenet_v1_emitc_static.model
403```
404
405with the Renode console log printed as
406
407```bash
40815:17:59.9864 [INFO] uart5: kata_ml_coordinator::Load successful.
Cindy Liu73984b02021-09-09 13:46:41 -0700409...
Cindy Liub650f672021-10-12 15:49:23 -070041015:18:02.6868 [INFO] cpu2: simprint: "main returned: ", 0 (0x0)
411
Cindy Liuc1485e72021-08-25 19:39:36 -0700412```
413
June Tate-Gans4447d0e2020-06-25 14:31:51 -0500414## More Information
415
Cindy Liu1337cf72022-06-23 10:27:03 -0700416For more available Shodan build targets, please see [Build target lists](./BuildTargetsExplained.md),
417or use the in-project command of `hmm` and `hmm <target name>`.
Cindy Liu9acf5522021-09-20 19:46:45 -0700418
419Also, [Information on how to use repo](https://go/repo)