About a week or two ago I started following some how-tos on the internet for building kernels and/or hypervisors in Rust. It’s been a fun adventure and I thought I would compile my notes together in case someone wants to follow along or see how an OS starts up. Most of the examples out (probably to keep things simpler…) are based on x86_64 hardware. To spice things up a bit I decided that I’d be targeting aarch64 (ARMv8) instead using QEMU’s virt board.
The intended audience here is someone with a background in programming, but not necessarily kernel programming or C. A familiarity with assembly syntax is helpful, but certainly not required. The same can be said for a working knowledge of Rust. This is mostly a vehicle to talk about all the exciting concepts in lower-level programming as opposed to a “Here’s how to write Rust.”
Some of this work was inspired by Philipp Oppermann’s excellent Writing an OS In Rust (Second Edition ) https://os.phil-opp.com/ . If you would rather see and write a kernel for x86_64 you should go read his instead.
Ash Wilding has an excellent tour of hypervisor concepts on his blog at https://ashw.io/blog/arm64-hypervisor-tutorial/1. He also includes examples of the assembly needed to setup some core functionality on an Arm system.
Preliminaries on OSX (10.14.5)
- homebrew
- crosstoolng (1.24.0) via homebrew
- Rustup if you’re following along with my code (not needed per se)
- rustup nightly (because we need compiler features hidden behind nightly)
Aside about cross compiling
What’s the deal with a cross compiler?! I already have a compiler.
That’s true, but…
Compilers target a “triple” on their output. The format is machine-vendor-
OS. In my case, the default “triple” is x86_64-apple-darwin. That is to say
it will use the x86_64 instruction set, on hardware provided by Apple, with
the Darwin/XNU
ABI (application
binary interface).
If I want to do operating system development I have to make a radically
different set of assumptions, especially if I’m targeting a different
instruction set like aarch64/ARMv8. What we really need is a compiler that can
target aarch64-unknown-elf. That is, it will use the aarch64 (ARMv8)
instruction set, without caring about hardware, and that the output need only
conform to the ELF spec.
Install crosstool-ng & Cross-compiling Toolchain
This has an easy part and a tricky part. The easy part is installing it via homebrew. The tricky part is that crosstool-ng is a toolchain that compiles other toolchains so you have to meet all the requirements of those toolchains too.
You’re probably going to need to create a new volume with Disk Utility for storing the cross compiler toolchain. Most of the compiler toolchains assume a case sensitive filesystem and crosstool-ng won’t even attempt to build (for good reasons) until that’s true.
You’re going to need an APFS volume w/ case sensitivity and probably about 10GB. The final required size will be considerably less, but we need to make sure to have enough space to store all the compilation intermediates.
I followed the instructions at <https://medium.com/coinmonks/setup- gcc-8-1-cross-compiler-toolchain-for-raspberry-pi-3-on-macos-high-sierra- cb3fc8b6443e>
Luckily for OSX 10.14.5 (as of this writing) I didn’t need most of the
instructions (for crosstool-ng 1.24.0). I started with creating the volume,
skipped to menuconfig, and then went right to build and it all worked.
Once you have the mount point /Volumes/xtool-build-env/ __ you can skip to
configuring crosstool-ng .
The key things that need to be set via ct-ng menuconfig:
- Paths and misc options -> Working directory ->
/Volumes/xtool-build-env/.build - Paths and misc options -> Prefix directory ->
/Volumes/xtool-build-env/${CT_TARGET} - Target options -> Target architecture -> ARM
- Target options -> Endianness -> Little endian
- Target options -> Bitness -> 64
- Operating System -> Target OS -> Bare-metal
- Binary utilities -> binary format -> ELF
- Debug facilities -> gdb (Not strictly required, but you will regret not having it if you decide to experiment)
Here’s a gist of my local config just in case you decide to dig around the config file and want a reference point.
You should be able to make the toolchain now with ct-ng build. Don’t be like
me and do this step at home, or at least with a power cable. It took my
relatively recent laptop about 20 minutes of work and 25% of my battery to
finish this step.
When it’s done you should have a directory /Volumes/xtool-build-
env/aarch64-unknown-elf/bin with a bunch of tools prefixed aarch64-unknown-
elf.
Installing Rust
I did this project in Rust, so if you want to build along, you’ll need it. That being said, if you just want the details of how the pieces fit together, feel free to ignore it. Get the Rust toolchain installer and follow the instructions.
The only additional instructions you’ll need are to enable + set nightly, and
add the aarch64-unknown-linux-gnu target.
rustup default nightly
rustup target add aarch64-unknown-linux-gnu
Additional Rust tools
We need a few additional Cargo command line tools that aren’t strictly needed to compile our OS crate, but do need to be on the command line
rustup component add llvm-tools-preview
cargo install xargo
cargo install cargo-binutils
Installing QEMU
Super simple: brew install qemu.
Sanity Checks Before Continuing
If you’re trying to get everything working then you should now have (on $PATH):
- aarch64-unknown-elf-gdb
- qemu-system-aarch64
- cargo-size (from cargo-binutils)
