BuildRiscVToolchain.md

# Build RISC-V Toolchain

This doc lists the common config settings to build the RISC-V toolchain. It
generally involves two parts of the toolchain: GCC to build the headers and
libraries, and LLVM to build the compiler, linker, and utility tools.

## Prerequisites

Your host machine needs to have the following packages installed, which are
already part of the Shodan prerequisite pacakges

* CMake (>= 3.13.4)
* Ninja
* Clang

The source code of the toolchain is at

* [riscv-gnu-toolchain](https://github.com/riscv/riscv-gnu-toolchain): Checkout
the latest release tag, and checkout the submodule `riscv-binutils` at the master
branch at git://sourceware.org/git/binutils-gdb.git
* [llvm-project](https://github.com/llvm/llvm-project): Checkout the latest green
commit

## Build RISC-V Linux toolchain (64-bit)

### Build GCC

```bash
$ mkdir -p <GCC_BUILD_PATH>
$ cd <GCC_BUILD_PATH>
$ <GCC_SRC_PATH>/configure \
  --srcdir=<GCC_SRC_PATH> \
  --prefix=<TOOLCHAIN_OUT_DIR> \
  --with-arch=rv64gc \
  --with-abi=lp64d \
  --with-cmodel=medany
$ make -C <GCC_BUILD_PATH> linux
```

Notice Linux requires the full general CPU extension support, i.e., rv64imafdc,
and the ABI also needs to support hard double-float modules. For 32-bit Linux,
build the toolchain with the flags of `--with-arch=rv32gc --with-abi=ilp32d`.

### Build LLVM

```bash
$ cmake -B <LLVM_BUILD_PATH> \
    -DCMAKE_INSTALL_PREFIX=<TOOLCHAIN_OUT_DIR> \
    -DCMAKE_C_COMPILER=clang  -DCMAKE_CXX_COMPILER=clang++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_TARGETS_TO_BUILD="RISCV" \
    -DLLVM_ENABLE_PROJECTS="clang"  \
    -DLLVM_DEFAULT_TARGET_TRIPLE="riscv64-unknown-linux-gnu" \
    -DLLVM_INSTALL_TOOLCHAIN_ONLY=On \
    -DDEFAULT_SYSROOT=../sysroot \
    -G Ninja \
    <LLVM_SRC_PATH>/llvm
$ cmake --build  <LLVM_BUILD_PATH> --target install
```

For 32-bit, change the LLVM target triple to `riscv32-unknown-linux-gnu`.

## Build RISC-V bare-metal toolchain (32-bit)

### Build 32-bit GCC

```bash
$ mkdir -p <GCC_BUILD_PATH>
$ cd <GCC_BUILD_PATH>
$ <GCC_SRC_PATH>/configure \
  --srcdir=<GCC_SRC_PATH> \
  --prefix=<TOOLCHAIN_OUT_DIR> \
  --with-arch=rv32i2p0mf2p0 \
  --with-abi=ilp32 \
  --with-cmodel=medany
$ make -C <GCC_BUILD_PATH> newlib
```

Notice for bare-metal newlib there's no hard constraints on CPU feature and ABI
support. However, LLVM for bare-metal only supports soft-float modules, so the
GCC ABI setting needs to match that. Also, the ISA version needs to be specified,
since binuils ISA supports 20191213 spec and LLVM is at v2.2 spec

### Build 32-bit LLVM

```bash
$ cmake -B <LLVM_BUILD_PATH> \
    -DCMAKE_INSTALL_PREFIX=<TOOLCHAIN_OUT_DIR> \
    -DCMAKE_C_COMPILER=clang  -DCMAKE_CXX_COMPILER=clang++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_TARGETS_TO_BUILD="RISCV" \
    -DLLVM_ENABLE_PROJECTS="clang"  \
    -DLLVM_DEFAULT_TARGET_TRIPLE="riscv32-unknown-elf" \
    -DLLVM_INSTALL_TOOLCHAIN_ONLY=On \
    -DDEFAULT_SYSROOT=../riscv32-unknown-elf \
    -G Ninja \
    <LLVM_SRC_PATH>/llvm
$ cmake --build  <LLVM_BUILD_PATH> --target install
```

#### Build compiler-rt

This should not be necessary for the Shodan usage, but in case the compiler-rt
builtins is required in the project (instead of using libgcc), it can be built
with the additional commands:

```bash
$ export PATH=<TOOLCHAIN_OUT_DIR>/bin:${PATH}
$ cmake -B <LLVM_BUILD_PATH>/compiler-rt \
  -DCMAKE_INSTALL_PREFIX=$PREFIX \
  -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \
  -DCMAKE_AR=<TOOLCHAIN_OUT_DIR>/bin/llvm-ar \
  -DCMAKE_NM=<TOOLCHAIN_OUT_DIR>/bin/llvm-nm \
  -DCMAKE_RANLIB=<TOOLCHAIN_OUT_DIR>/bin/llvm-ranlib \
  -DCMAKE_C_FLAGS="-march=rv32i2p0mf2p0v1p0" \
  -DCMAKE_ASM_FLAGS="-march=rv32i2p0mf2p0v1p0" \
  -DCMAKE_C_COMPILER=<TOOLCHAIN_OUT_DIR>/bin/clang \
  -DCMAKE_C_COMPILER_TARGET=riscv32-unknown-elf \
  -DCMAKE_ASM_COMPILER_TARGET=riscv32-unknown-elf \
  -DCOMPILER_RT_OS_DIR="clang/14.0.0/lib" \
  -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
  -DCOMPILER_RT_BUILD_BUILTINS=ON \
  -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
  -DCOMPILER_RT_BUILD_XRAY=OFF \
  -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
  -DCOMPILER_RT_BUILD_MEMPROF=OFF \
  -DCOMPILER_RT_BUILD_PROFILE=OFF \
  -DCOMPILER_RT_BAREMETAL_BUILD=ON \
  -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
  -DLLVM_CONFIG_PATH=<LLVM_BUILD_PATH>/bin/llvm-config \
  -DCMAKE_C_FLAGS="-march=rv32i2p0mf2p0v1p0 -mno-relax" \
  -DCMAKE_ASM_FLAGS="-march=rv32i2p0mf2p0v1p0 -mno-relax" \
  -G "Ninja" <LLVM_SRC_PATH>/compiler-rt
$ cmake --build <LLVM_BUILD_PATH>/compiler-rt --target install
```

### Build newlib

Even with compiler_rt to replace libgcc, we still need to build libc, libm, and
libgloss as the toolchain prebuilts. They can also be built with clang
The source code is at <https://github.com/riscv/riscv-newlib>

***NOTE: The GCC utility tools needs to be built first.***

```bash
$ mkdir -p <NEWLIB_BUILD_PATH>
$ cd <NEWLIB_BUILD_PATH>
$ <NEWLIB_SRC_PATH>/configure \
  --target=riscv32-unknown-elf \
  --prefix=<TOOLCHAIN_OUT_DIR> \
  --enable-newlib-io-long-double \
  --enable-newlib-io-long-long \
  --enable-newlib-io-c99-formats \
  --enable-newlib-register-fini \
  CC_FOR_TARGET=clang \
  CXX_FOR_TARGET=clang++ \
  CFLAGS_FOR_TARGET="-march=rv32i2p0mf2p0v1p0 -O2 -D_POSIX_MODE -mno-relax" \
  CXXFLAGS_FOR_TARGET="-march=rv32i2p0mf2p0v1p0 -O2 -D_POSIX_MODE -mno-relax"
$ make -j32
$ make install
```

The newlib nano spec needs to be built separatedly and then merged with newlib.
GNU top-level Makefile lists the flags to build nano and the merging script.

## Test toolchain

Run

```bash
<TOOLCHAIN_OUT_DIR>/bin/<arch>-<os>-<abi>-gcc -v
```

to see the supported ABIs, architectures, library paths, etc.

Try to compile a simple c code (copied from CMake's package content, e.g.,
`/usr/share/cmake-3.18/Modules/CMakeTestCCompiler.cmake`)

```c
#ifdef __cplusplus
# error "The CMAKE_C_COMPILER is set to a C++ compiler"
#endif
#if defined(__CLASSIC_C__)
int main(argc, argv)
  int argc;
  char* argv[];
#else
int main(int argc, char* argv[])
#endif
{ (void)argv; return argc-1;}
```

To build (32-bit bare-metal example)

```bash
$<TOOLCHAIN_OUT_DIR>/bin/clang -c testCCompiler.c -O --target=riscv32
$<TOOLCHAIN_OUT_DIR>/bin/riscv32-unknown-elf-gcc testCCompiler.o -o testCCompiler -march=rv32gc -mabi=ilp32
```

You can also use `readelf` to inspect the object file before building the binary
to see if the architecture and ABI match.