Merge "Revert "Specify python3 version in apt install commands""
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..6272489 --- /dev/null +++ b/CONTRIBUTING.md
@@ -0,0 +1,28 @@ +# How to Contribute + +We'd love to accept your patches and contributions to this project. There are +just a few small guidelines you need to follow. + +## Contributor License Agreement + +Contributions to this project must be accompanied by a Contributor License +Agreement. You (or your employer) retain the copyright to your contribution; +this simply gives us permission to use and redistribute your contributions as +part of the project. Head over to <https://cla.developers.google.com/> to see +your current agreements on file or to sign a new one. + +You generally only need to submit a CLA once, so if you've already submitted one +(even if it was for a different project), you probably don't need to do it +again. + +## Code Reviews + +All submissions, including submissions by project members, require review. We +use GitHub pull requests for this purpose. Consult +[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more +information on using pull requests. + +## Community Guidelines + +This project follows [Google's Open Source Community +Guidelines](https://opensource.google/conduct/).
diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE
@@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License.
diff --git a/README.md b/README.md new file mode 100644 index 0000000..dce4529 --- /dev/null +++ b/README.md
@@ -0,0 +1,37 @@ +# Project Shodan Scripts + +Shodan is a project to build a low-power secure embeded platform +for Ambient ML applications. The target platform leverages +[RISC-V](https://riscv.org/) and [OpenTitan](https://opentitan.org/). + +This directory holds scripts used in developing Shodan: + +- *build-sparrow.sh*: + a script to demonstrate how to build and run seL4 CAmkES tests +- *kcargo.sh*: + a wrapper script that shows how Rust is run in the Shodan developement environment + +[more scripts will appear here as more of the project is released] + +## Source Code Headers + +Every file containing source code includes copyright and license +information. For dependent / non-Google code these are inherited from +the upstream repositories. If there are Google modifications you may find +the Google Apache license found below. + +Apache header: + + Copyright 2022 Google LLC + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License.
diff --git a/build-camkes.sh b/build-camkes.sh new file mode 100755 index 0000000..a88451f --- /dev/null +++ b/build-camkes.sh
@@ -0,0 +1,124 @@ +#! /bin/bash +# +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This script is derived from work by mwitkowski@antmicro.com + +# Script for building a CAmkES test setup using the minimal bits from +# KataOS. This is meant for testing kata-os-common portability and as +# a pathway to public use of the Rust code bits. +# +# This script assumes you have a gnu toolchain setup for the target +# platform and the associated "bin" dir in your shell's search path. +# The rust compiler likewise must be in your search path; the script +# will use rustup to request target support. +# +# Beware the KataOS Rust code currently uses a nightly build of Rust. +# Check projects/kata/apps/system/rust.cmake for cargo usage. +# +# The riscv* and arm/aarch* targets are build tested and in some cases +# tested under simulation. The x86* targets are untested and likely +# do not compile. There are many arm target flavors and only the basic +# stuff may work; in particular don't expect any hypervisor support to +# work without effort. + +# TODO(sleffler): add install-* scripts for necessary toolchains +# TODO(sleffler): import dependent simulators as needed: qemu, spike, renode + +TARGET_ARCH=${1:-aarch64} +EXTRA_INIT_ARGS= +MACHINE= + +case ${TARGET_ARCH} in +arm|aarch32) + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DSIMULATION=TRUE -DAARCH32=TRUE" + CROSS_COMPILER_PREFIX="arm-none-eabi-" + RUST_TARGET="arm-unknown-linux-gnueabi" + PLATFORM=omap3 + ;; +aarch64) + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DSIMULATION=TRUE -DAARCH64=TRUE" + CROSS_COMPILER_PREFIX="aarch64-none-linux-gnu-" + RUST_TARGET="${TARGET_ARCH}-unknown-none" + PLATFORM=rpi3 + MACHINE=raspi3b + ;; +riscv32) + # https://docs.sel4.systems/Hardware/spike.html + # assumes --enable-multilib toolchain + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DRISCV32=TRUE" + CROSS_COMPILER_PREFIX="riscv32-unknown-elf-" + RUST_TARGET="riscv32imac-unknown-none-elf" + PLATFORM=spike + ;; +riscv64) + # https://docs.sel4.systems/Hardware/spike.html + # assumes --enable-multilib toolchain + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DRISCV64=TRUE" + CROSS_COMPILER_PREFIX="riscv64-unknown-linux-gnu-" + RUST_TARGET="riscv64imac-unknown-none-elf" + PLATFORM=spike + ;; +esac + +mkdir camkes-${TARGET_ARCH}-rootserver +cd camkes-${TARGET_ARCH}-rootserver + +# NB: "no" disables colorization +echo 'no' | repo init -u https://spacebeaker.googlesource.com/shodan/manifest -m camkes-manifest.xml +repo sync -j$(nproc) + +# Export required variables +# TODO: requiring SEL4_DIR & SEL4_OUT_DIR in the environment is +# awkward; maybe add fallback/defaults in the build glue + +export ROOTDIR="$(pwd)" +export SEL4_DIR="${ROOTDIR}/kernel" +export SEL4_OUT_DIR="${ROOTDIR}/build/kernel" + +# NB: the gnu toolchain is expected to be in your shell search PATH; e.g. +# cd ~ +# wget https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz +# tar xf gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz +# PATH=~/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH + +# NB: use an existing toolchain but make sure the necessary target is installed +rustup target add --toolchain nightly-2021-11-05-x86_64-unknown-linux-gnu ${RUST_TARGET} + +# Run cmake to build the ninja files +test -d build || { + mkdir build + pushd build + ../init-build.sh \ + -DCROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX} \ + -DRUST_TARGET=${RUST_TARGET} \ + -DPLATFORM=${PLATFORM} \ + -DCAPDL_LOADER_APP=kata-os-rootserver \ + -DSIMULATION=TRUE \ + ${EXTRA_INIT_ARGS} + popd # build +} + +# Run ninja to do the actual build +pushd build +ninja -j$(nproc) + +# If there's a simulator you can test the rootserver +if test -x simulate; then + ./simulate -M ${MACHINE} +else + echo "No simulate script so don't know how to test machine ${MACHINE}" +fi +popd # build
diff --git a/build-sparrow.sh b/build-sparrow.sh new file mode 100755 index 0000000..449a07e --- /dev/null +++ b/build-sparrow.sh
@@ -0,0 +1,114 @@ +#! /bin/bash +# +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This script is derived from work by mwitkowski@antmicro.com + +# Script for building a CAmkES test setup using the minimal bits from +# KataOS. This is meant for testing kata-os-common portability and as +# a pathway to public use of the Rust code bits. +# +# This script assumes you have a gnu toolchain setup for the target +# platform and the associated "bin" dir in your shell's search path. +# The rust compiler likewise must be in your search path; the script +# suggests using rustup to request target support. +# +# Beware the KataOS Rust code currently uses a nightly build of Rust. +# Check projects/kata/apps/system/rust.cmake for cargo usage. +# +# The riscv* and arm/aarch* targets are build tested and in some cases +# tested under simulation. The x86* targets are untested and unsupported. +# There are many arm target flavors and only the basic stuff may work; +# in particular don't expect any hypervisor support to work without effort. + +# TODO(sleffler): maybe add install-* scripts for necessary toolchains +# TODO(sleffler): maybe import dependent simulators as needed: qemu, spike, renode + +TARGET_ARCH=${1:-aarch64} +EXTRA_INIT_ARGS= +MACHINE= + +case ${TARGET_ARCH} in +arm|aarch32) + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DSIMULATION=TRUE -DAARCH32=TRUE" + CROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX:-"arm-none-eabi-"} + RUST_TARGET=${RUST_TARGET:-"arm-unknown-linux-gnueabi"} + PLATFORM=${PLATFORM:-"omap3"} + ;; +aarch64) + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DSIMULATION=TRUE -DAARCH64=TRUE" + CROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX:-"aarch64-none-linux-gnu-"} + RUST_TARGET=${RUST_TARGET:-"${TARGET_ARCH}-unknown-none"} + PLATFORM=${PLATFORM:-"rpi3"} + MACHINE=${MACHINE:-"raspi3b"} + ;; +riscv32) + # https://docs.sel4.systems/Hardware/spike.html + # assumes --enable-multilib toolchain + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DRISCV32=TRUE" + CROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX:-"riscv32-unknown-elf-"} + RUST_TARGET=${RUST_TARGET:-"riscv32imac-unknown-none-elf"} + PLATFORM=${PLATFORM:-"spike"} + ;; +riscv64) + # https://docs.sel4.systems/Hardware/spike.html + # assumes --enable-multilib toolchain + EXTRA_INIT_ARGS="${EXTRA_INIT_ARGS} -DRISCV64=TRUE" + CROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX:-"riscv64-unknown-linux-gnu-"} + RUST_TARGET=${RUST_TARGET:-"riscv64imac-unknown-none-elf"} + PLATFORM=${PLATFORM:-"spike"} + ;; +esac + +BUILD_DIR="build-${TARGET_ARCH}" + +# Export required variables +# TODO: requiring SEL4_DIR & SEL4_OUT_DIR in the environment is +# awkward; maybe add fallback/defaults in the build glue + +export ROOTDIR="$(pwd)" +export SEL4_DIR="${ROOTDIR}/kernel" +export SEL4_OUT_DIR="${ROOTDIR}/${BUILD_DIR}/kernel" + +# NB: the gnu toolchain is expected to be in your shell search PATH; e.g. +# cd ~ +# wget https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz +# tar xf gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz +# PATH=~/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH + +# NB: use an existing toolchain but make sure the necessary target is installed +echo "If your rust toolchain is not setup use something like:" +echo "rustup target add --toolchain nightly-2021-11-05-x86_64-unknown-linux-gnu ${RUST_TARGET}" + +# Run cmake to build the ninja files +test -f ${BUILD_DIR}/build.ninja || { + mkdir -p ${BUILD_DIR} + pushd ${BUILD_DIR} + ../init-build.sh \ + -DCROSS_COMPILER_PREFIX=${CROSS_COMPILER_PREFIX} \ + -DRUST_TARGET=${RUST_TARGET} \ + -DPLATFORM=${PLATFORM} \ + -DCAPDL_LOADER_APP=kata-os-rootserver \ + -DSIMULATION=TRUE \ + ${EXTRA_INIT_ARGS} + popd # ${BUILD_DIR} +} + +# Run ninja to do the actual build +pushd ${BUILD_DIR} +ninja -j$(nproc) +popd # ${BUILD_DIR} + +echo "To run the simulator use: (cd ${BUILD_DIR} && ./simulate -M ${MACHINE})"
diff --git a/download-toolchain.sh b/download-toolchain.sh index ba7f769..2204c54 100755 --- a/download-toolchain.sh +++ b/download-toolchain.sh
@@ -22,12 +22,11 @@ exit 1 fi if [[ -z "$1" ]]; then - echo "Usage: download-toolchain.sh <gcc dir> [<TARGET> | GCC | LLVM] [<VARIANT> | master | RVV]" + echo "Usage: download-toolchain.sh <gcc dir> [<TARGET> | GCC | LLVM]" exit 1 fi TOOLCHAIN_TARGET=${2:-GCC} -TOOLCHAIN_VARIANT=${3:-master} TOOLCHAIN_GCC_SRC="$1" TOOLCHAIN_SRC="${OUT}/tmp/toolchain" @@ -53,22 +52,15 @@ fi mkdir -p "${TOOLCHAIN_GCC_SRC}" -# Download from the http://github.com/riscv/riscv-gnu-toolchain. For LLVM 32-bit RVV support or -# regular gcc, it requires a newer branch, whereas the native gcc rvv toolchain can use -# rvv-intrinsic branch. Use git init and git fetch to avoid creating extra layer of the -# source code. +# Download from the http://github.com/riscv/riscv-gnu-toolchain. For proper +# support of GDB symbol rendering, it requires a tag points to gcc 10.2. +# Use git init and git fetch to avoid creating an extra layer of the source code. pushd "${TOOLCHAIN_GCC_SRC}" > /dev/null git init git remote add origin https://github.com/riscv/riscv-gnu-toolchain -if [[ "${TOOLCHAIN_TARGET}" == "GCC" ]] && [[ "${TOOLCHAIN_VARIANT}" == "RVV" ]]; then - echo "Downloading the GNU toolchain source code for GCC RVV" - git fetch origin rvv-intrinsic - git reset --hard FETCH_HEAD -else - echo "Downloading the GNU toolchain source code from master" - git fetch origin --tags - git reset --hard ${TOOLCHAINLLVM_TAG} -fi +echo "Downloading the GNU toolchain source code from master" +git fetch origin --tags +git reset --hard ${TOOLCHAINLLVM_TAG} popd > /dev/null # Update the submodules. The riscv-binutils has to point to upstream binutil-gdb
diff --git a/download_iree_compiler.py b/download_iree_compiler.py index 7ecae06..43a6baf 100755 --- a/download_iree_compiler.py +++ b/download_iree_compiler.py
@@ -37,7 +37,7 @@ try: wget.download(download_url, out=out_file) break - except urllib.error.HTTPError as e: + except (urllib.error.HTTPError, ConnectionError) as e: if i == num_retries: raise print(f"{e}\nDownload failed. Retrying...")
diff --git a/fetch-rust-toolchain.sh b/fetch-rust-toolchain.sh index 7aed0e2..5771ce6 100755 --- a/fetch-rust-toolchain.sh +++ b/fetch-rust-toolchain.sh
@@ -77,7 +77,7 @@ try mkdir -p "${OUT}" - try wget -O "${OUT}/${tarball}" "${PUBLIC_ARTIFACTS_URL}/${tarball}" + try wget --progress=dot:giga -O "${OUT}/${tarball}" "${PUBLIC_ARTIFACTS_URL}/${tarball}" try wget -O "${OUT}/${checksum}" "${PUBLIC_ARTIFACTS_URL}/${checksum}" # Workaround the fact that we use the datestamped version of the filename
diff --git a/install-prereqs.sh b/install-prereqs.sh index d44c388..f91045c 100755 --- a/install-prereqs.sh +++ b/install-prereqs.sh
@@ -19,6 +19,8 @@ APT_PACKAGES=( asciidoctor + bazel + bazel-5.1.1 bison build-essential ccache @@ -46,8 +48,8 @@ llvm-11-tools gcc g++ - gcc-10 - g++-10 + gcc-11 + g++-11 libftdi1 libftdi1-dev libfl2
diff --git a/install-toolchain.sh b/install-toolchain.sh index 2347a46..d4fdc67 100755 --- a/install-toolchain.sh +++ b/install-toolchain.sh
@@ -75,7 +75,7 @@ DOWNLOAD_DIR="${OUT}/tmp" mkdir -p "${DOWNLOAD_DIR}" - wget -P "${DOWNLOAD_DIR}" "${DOWNLOAD_URL}" + wget --progress=dot:giga -P "${DOWNLOAD_DIR}" "${DOWNLOAD_URL}" wget -P "${DOWNLOAD_DIR}" "${DOWNLOAD_URL}.sha256sum" pushd "${DOWNLOAD_DIR}" > /dev/null try sha256sum -c "${TOOLCHAIN_TARBALL}.sha256sum"
diff --git a/kcargo.sh b/kcargo.sh new file mode 100755 index 0000000..8664a72 --- /dev/null +++ b/kcargo.sh
@@ -0,0 +1,48 @@ +#! /bin/bash +# +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Script for running Shodan cargo outside the build; useful +# for doing things like kcargo tree or kcargo clippy. + +if [[ -z "${ROOTDIR}" ]]; then + echo "Source build/setup.sh first" + exit 1 +fi + +# HACK: sel4-config needs a path to the kernel build which could be +# in debug or release +export SEL4_OUT_DIR="$ROOTDIR/out/kata/riscv32-unknown-elf/debug/kernel/" +if [[ ! -d "${SEL4_OUT_DIR}/gen_config" ]]; then + echo "No kernel build found at ${SEL4_OUT_DIR}; build a kernel first" + exit 2 +fi + +CARGO="${CARGO_HOME}/bin/cargo +${KATA_RUST_VERSION}" +CARGO_TARGET="--target riscv32imac-unknown-none-elf" +CARGO_OPTS='-Z unstable-options -Z avoid-dev-deps' + +export RUSTFLAGS='-Z tls-model=local-exec' + +cmd=${1:-build} +case "$1" in +fmt) + ${CARGO} $*;; +""|-*) + # TODO(sleffler): maybe set --target-dir to avoid polluting the src tree + ${CARGO} build ${CARGO_OPTS} ${CARGO_TARGET};; +*) + ${CARGO} $* ${CARGO_OPTS} ${CARGO_TARGET};; +esac
diff --git a/kgdb.sh b/kgdb.sh index 47af2c0..b3a745a 100755 --- a/kgdb.sh +++ b/kgdb.sh
@@ -12,27 +12,81 @@ TARGET=riscv32-unknown-elf GDB="${ROOTDIR}"/cache/toolchain/bin/${TARGET}-gdb -PROGRAM=out/shodan_boot_rom/build-out/multihart_boot_rom/multihart_boot_rom_sim_verilator.elf +PROGRAM=out/shodan_boot_rom/multihart_boot_rom/multihart_boot_rom.elf REMOTE=localhost:3333 KATA_OUT=out/kata/${TARGET}/debug MATCHA_OUT=out/matcha/riscv32imc-unknown-none-elf/debug +USE_SEL4_EXTENSIONS="true" +USE_SEL4_SYMBOL_AUTOSWITCHING="false" + export SOURCE_DIR=${ROOTDIR}/kata export BUILD_DIR=$KATA_OUT -# NB: -q suppresses the banner to workaround the banner msg triggering the pager -# NB: auto-start cpu0 & cpu1 but leave cpu2 (VC) halted -exec "${GDB}" -q -cd "${ROOTDIR}" \ - -ex "set pagination off" \ - -ex "directory sw/tock" \ - -ex "file ${PROGRAM}" \ - -ex "set confirm off" \ - -ex "add-symbol-file ${PROGRAM}" \ - -ex "add-symbol-file ${MATCHA_OUT}/matcha_platform" \ - -ex "add-symbol-file ${MATCHA_OUT}/matcha_app" \ - -ex "set pagination on" \ - -ex "target remote ${REMOTE}" \ - -ex "monitor cpu0 IsHalted false" \ - -ex "monitor cpu1 CreateSeL4 0xffffffef" \ - -ex "source sim/renode/tools/sel4_extensions/gdbscript.py" +function parseargv { + local usage="Usage: kgdb.sh [-h|--help] [-S|--no-sel4-extensions] [-a|--sel4-symbol-autoswitching]" + local args=$(getopt -o hSa --long no-sel4-extensions,symbol-autoswitching,help -n kgdb.sh -- "$@") + + set -- $args + + for i; do + case "$1" in + -S|--no-sel4-extensions) + echo "*** Disabling sel4 extensions" + USE_SEL4_EXTENSIONS="false" + shift + ;; + + -a|--symbol-autoswitching) + echo "*** Enabling sel4 symbol autoswitching" + echo "*** Warning: this can cause unexpected behaviors." + USE_SEL4_EXTENSIONS="true" + USE_SEL4_SYMBOL_AUTOSWITCHING="true" + shift + ;; + + --) + shift + break + ;; + + -h|--help|*) + echo "$usage" >/dev/stderr + exit 1 + ;; + esac + done +} + +function main { + local -a gdbargs=( + -ex "set pagination off" + -ex "directory sw/tock" + -ex "file ${PROGRAM}" + -ex "set confirm off" + -ex "add-symbol-file ${PROGRAM}" + -ex "add-symbol-file ${MATCHA_OUT}/matcha_platform" + -ex "add-symbol-file ${MATCHA_OUT}/matcha_app" + -ex "set pagination on" + -ex "target remote ${REMOTE}" + -ex "monitor cpu0 IsHalted false" + ) + + parseargv "$@" + + if [[ "${USE_SEL4_EXTENSIONS}" == "true" ]]; then + gdbargs+=( + -ex "monitor cpu1 CreateSeL4 0xffffffee" + -ex "source sim/renode/tools/sel4_extensions/gdbscript.py" + -ex "sel4 symbol-autoswitching ${USE_SEL4_SYMBOL_AUTOSWITCHING}" + ) + fi + + + # NB: -q suppresses the banner to workaround the banner msg triggering the pager + # NB: auto-start cpu0 & cpu1 but leave cpu2 (VC) halted + exec "${GDB}" -q -cd "${ROOTDIR}" "${gdbargs[@]}" +} + +main "$@"
diff --git a/prepare_bundle_image.sh b/prepare_bundle_image.sh new file mode 100755 index 0000000..2cbd4ac --- /dev/null +++ b/prepare_bundle_image.sh
@@ -0,0 +1,80 @@ +#! /bin/bash + +# Script for preparing a cpio archive of builtin applications and/or models. +# Usage: prepare_bundle_image [-n] -o target.cpio +# [-m input_model...] [-a input_app...] +# where +# -o target cpio archive filena,me +# -m identifies subsequent arguments as models +# -a identifies subsequent arguments as applications +# also +# -n do a dry-run where commands are just echo'd to the terminal + +# TODO(sleffler): redo with getopt + +if [[ -z "${ROOTDIR}" ]]; then + echo "No ROOTDIR, source build/setup.sh first" + exit 1 +fi +if [[ -z "${KATA_RUST_VERSION}" ]]; then + echo "No KATA_RUST_VERSION, source build/setup.sh first" + exit 1 +fi +if [[ -z "${CARGO_HOME}" ]]; then + echo "No CARGO_HOME, source build/setup.sh first" + exit 1 +fi + +TMP_DIR="${OUT}/tmp" +if [[ ! -d "${TMP_DIR}" ]]; then + echo "No tmp directory found at ${TMP_DIR}" + exit 2 +fi + +CARGO="${CARGO_HOME}/bin/cargo +${KATA_RUST_VERSION}" +CPIO_OPTS='-H newc -L --no-absolute-filenames --reproducible --owner=root:root' + +function prepare_bundle_image { + cd "${ROOTDIR}/kata/tools/seL4/misc/prepare_bundle_image" && \ + ${CARGO} run -q --target-dir "${OUT}/host/prepare_bundle_image" -- "$@" +} + +dry_run='false' +if [[ "$1" == '-n' ]]; then + dry_run='true' + shift +fi +if [[ "$1" != '-o' ]]; then + echo "Missing -o option to specify output cpio archive" + exit 3 +fi +OUTPUT_CPIO="$2" +shift 2 + +APPS='-a' +MODELS='-m' +file_type= +for arg; do + case "${arg}" in + -a) file_type='app';; + -m) file_type='model';; + *) case "${file_type}" in + app) + ln -sf "${arg}" "${TMP_DIR}" && \ + APPS="${APPS} ${TMP_DIR}/$(basename ${arg})" + ;; + model) + ln -sf "${arg}" "${TMP_DIR}" && \ + MODELS="${MODELS} ${TMP_DIR}/$(basename ${arg})" + ;; + *) echo 'Missing -m or -a option to identify file type'; exit -1;; + esac + esac +done + +if [[ "${dry_run}" == 'true' ]]; then + # TODO(sleffler): would be nice to print what prepare_bundle_image would do + echo "prepare_bundle_image ${MODELS} ${APPS} | cpio -o -D ${TMP_DIR} ${CPIO_OPTS} -O ${OUTPUT_CPIO}" +else + prepare_bundle_image ${MODELS} ${APPS} | cpio -o -D ${TMP_DIR} ${CPIO_OPTS} -O "${OUTPUT_CPIO}" +fi
diff --git a/preupload-hooks/GLOBAL-PREUPLOAD.cfg b/preupload-hooks/GLOBAL-PREUPLOAD.cfg new file mode 100644 index 0000000..b87fdec --- /dev/null +++ b/preupload-hooks/GLOBAL-PREUPLOAD.cfg
@@ -0,0 +1,13 @@ +# Global `repo upload` hook settings. +# https://android.googlesource.com/platform/tools/repohooks + +[Options] +ignore_merged_commits = true + +[Builtin Hooks] +pylint3 = true +cpplint = true +clang_format = true + +[Builtin Hooks Options] +clang_format = --commit ${PREUPLOAD_COMMIT} --style file --extensions c,h,cc,cpp
diff --git a/preupload-hooks/rustfmt.py b/preupload-hooks/rustfmt.py new file mode 100755 index 0000000..5e92401 --- /dev/null +++ b/preupload-hooks/rustfmt.py
@@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +# Copyright 2022 Google LLC +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Wrapper to run rustfmt for repo preupload.""" + +import argparse +import os +import subprocess +import sys + + +def get_parser(): + """Return a command line parser.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--rustfmt_path", + default="rustfmt", + help="The path to the rustfmt binary.") + parser.add_argument('files', + type=str, + nargs='*', + help='If specified, only consider rustfmt in ' + 'these files.') + return parser + + +def main(argv): + """The main entry.""" + parser = get_parser() + opts = parser.parse_args(argv) + + # Check and set rustfmt path in case `source build/setup.sh` is not run in + # the shell session. In repo preupload, the path should be set in + # PREUPLOAD.cfg. + if opts.rustfmt_path != "rustfmt": + # Add rustfmt path to system PATH and set up RUSTUP_HOME at one level up + # rustfmt has to have both variables set up to work properly. + path = os.path.realpath(opts.rustfmt_path + "/..") + os.environ["PATH"] = path + ":" + os.getenv("PATH") + os.environ["RUSTUP_HOME"] = path + "/.." + + # Only process .rs files + file_list = [f for f in opts.files if f.endswith("rs")] + if not file_list: + sys.exit(0) + + nightly_flag = os.getenv("KATA_RUST_VERSION") if os.getenv( + "KATA_RUST_VERSION") else "nightly-2021-11-05" + + cmd = [opts.rustfmt_path, f"+{nightly_flag}", "--check", "--color", "never"] + + for f in file_list: + cmd.append(f) + + # Run rustfmt on all the .rs files in the file list. `--check` flag + # prints out the formatting error and return with exit(1). + try: + subprocess.run(cmd, check=True) + except subprocess.CalledProcessError as e: + print(f"rustfmt check failed\ncmd: {cmd}\nexit code {e.returncode}") + sys.exit(e.returncode) + else: + sys.exit(0) + + +if __name__ == '__main__': + main(sys.argv[1:])
diff --git a/python-requirements.txt b/python-requirements.txt index 48a093d..b8e0d81 100644 --- a/python-requirements.txt +++ b/python-requirements.txt
@@ -1,3 +1,7 @@ +# pin to v4.0.0 since newer pushes are broken +# TODO(jtgans): Remove this pinning once upstream is unborked. +fixtures==4.0.0 + camkes-deps hjson lit @@ -10,6 +14,7 @@ pyyaml requests robotframework==4.0.1 +scipy sel4-deps setuptools tempita
diff --git a/run-chip-verilator-sim.sh b/run-chip-verilator-sim.sh index ff26460..67ec121 100755 --- a/run-chip-verilator-sim.sh +++ b/run-chip-verilator-sim.sh
@@ -1,8 +1,8 @@ #! /bin/bash # Run verilator testbench simulation. -if [[ $# -lt 4 || $1 == "--help" ]]; then - echo "Usage: run-chip-verilator-sim.sh <verilator testbech> <rom binary> <flash binary> <otp binary> [OPTIONS]" +if [[ $# -lt 5 || $1 == "--help" ]]; then + echo "Usage: run-chip-verilator-sim.sh <verilator testbench> <rom binary> <flash binary> <otp binary> <smc binary> [OPTIONS]" exit 0 fi @@ -10,8 +10,9 @@ ROM_BIN=$2 FLASH_BIN=$3 OTP_BIN=$4 +RAM_SMC_BIN=$5 -shift 4 +shift 5 if [[ ! -f $(realpath ${VCHIP_TB}) ]]; then echo "Verilator testbench not found. Please run \`m matcha_hw_verilator_sim\` or generate the testbench first." @@ -19,7 +20,7 @@ fi if [[ ! -f $(realpath ${ROM_BIN}) ]] || [[ ! -f $(realpath ${FLASH_BIN}) ]] || - [[ ! -f $(realpath ${OTP_BIN}) ]]; then + [[ ! -f $(realpath ${OTP_BIN}) ]] || [[ ! -f $(realpath ${RAM_SMC_BIN}) ]]; then echo "Software binaries not found. Please run \`m opentitan_sw_verilator_sim\` or generate the SW binaries first." exit 1 fi @@ -27,4 +28,5 @@ ${VCHIP_TB} \ "--meminit=rom,${ROM_BIN}" \ "--meminit=flash,${FLASH_BIN}" \ - "--meminit=otp,${OTP_BIN}" $@ + "--meminit=otp,${OTP_BIN}" \ + "--meminit=ram_smc,${RAM_SMC_BIN}" $@
diff --git a/run-spike-springbok.sh b/run-spike-springbok.sh index 50a8a75..32d0ef8 100755 --- a/run-spike-springbok.sh +++ b/run-spike-springbok.sh
@@ -17,9 +17,9 @@ fi # spike CLI options: -# -m<a:m,b:n>: specifies the memory layout. Springbok currently has 1MB IMEM at -# 0x3200_0000 and 16MB DMEM at 0x3400_0000 +# -m<a:m,b:n>: specifies the memory layout. Springbok currently has 16MB TCM +# at 0x3400_0000 # --varch: specifies the v-ext configuration w.r.t. vlen and elen. # --pc: ELF entry point. Set at the beginning of IMEM. -"${OUT}/host/spike/bin/spike" -m0x32000000:0x100000,0x34000000:0x1000000 \ - --varch=vlen:512,elen:32 --pc=0x32000000 $@ +"${OUT}/host/spike/bin/spike" -m0x34000000:0x1000000 \ + --varch=vlen:512,elen:32 --pc=0x34000000 $@