You Too Can Build A Kernel/Hypervisor Part 1 — Preliminaries

About a week or two ago I started following some how-tos on the internet for building kernels and/or hypervisors in Rust. It’s been a fun adventure and I thought I would compile my notes together in case someone wants to follow along or see how an OS starts up. Most of the examples out (probably to keep things simpler…) are based on x86_64 hardware. To spice things up a bit I decided that I’d be targeting aarch64 (ARMv8) instead using QEMU’s virt board.

The intended audience here is someone with a background in programming, but not necessarily kernel programming or C. A familiarity with assembly syntax is helpful, but certainly not required. The same can be said for a working knowledge of Rust. This is mostly a vehicle to talk about all the exciting concepts in lower-level programming as opposed to a “Here’s how to write Rust.”

Some of this work was inspired by Philipp Oppermann’s excellent Writing an OS In Rust (Second Edition ) . If you would rather see and write a kernel for x86_64 you should go read his instead.

Ash Wilding has an excellent tour of hypervisor concepts on his blog at He also includes examples of the assembly needed to setup some core functionality on an Arm system.

Preliminaries on OSX (10.14.5)

  • homebrew
  • crosstoolng (1.24.0) via homebrew
  • Rustup if you’re following along with my code (not needed per se)
  • rustup nightly (because we need compiler features hidden behind nightly)

Aside about cross compiling

What’s the deal with a cross compiler?! I already have a compiler.

That’s true, but…

Compilers target a “triple” on their output. The format is machine-vendor- OS. In my case, the default “triple” is x86_64-apple-darwin. That is to say it will use the x86_64 instruction set, on hardware provided by Apple, with the Darwin/XNU ABI (application binary interface).

If I want to do operating system development I have to make a radically different set of assumptions, especially if I’m targeting a different instruction set like aarch64/ARMv8. What we really need is a compiler that can target aarch64-unknown-elf. That is, it will use the aarch64 (ARMv8) instruction set, without caring about hardware, and that the output need only conform to the ELF spec.

Install crosstool-ng & Cross-compiling Toolchain

This has an easy part and a tricky part. The easy part is installing it via homebrew. The tricky part is that crosstool-ng is a toolchain that compiles other toolchains so you have to meet all the requirements of those toolchains too.

You’re probably going to need to create a new volume with Disk Utility for storing the cross compiler toolchain. Most of the compiler toolchains assume a case sensitive filesystem and crosstool-ng won’t even attempt to build (for good reasons) until that’s true.

You’re going to need an APFS volume w/ case sensitivity and probably about 10GB. The final required size will be considerably less, but we need to make sure to have enough space to store all the compilation intermediates.

I followed the instructions at < gcc-8-1-cross-compiler-toolchain-for-raspberry-pi-3-on-macos-high-sierra- cb3fc8b6443e>

Luckily for OSX 10.14.5 (as of this writing) I didn’t need most of the instructions (for crosstool-ng 1.24.0). I started with creating the volume, skipped to menuconfig, and then went right to build and it all worked.

Once you have the mount point /Volumes/xtool-build-env/ __ you can skip to configuring crosstool-ng .

The key things that need to be set via ct-ng menuconfig:

  • Paths and misc options -> Working directory -> /Volumes/xtool-build-env/.build
  • Paths and misc options -> Prefix directory -> /Volumes/xtool-build-env/${CT_TARGET}
  • Target options -> Target architecture -> ARM
  • Target options -> Endianness -> Little endian
  • Target options -> Bitness -> 64
  • Operating System -> Target OS -> Bare-metal
  • Binary utilities -> binary format -> ELF
  • Debug facilities -> gdb (Not strictly required, but you will regret not having it if you decide to experiment)

Here’s a gist of my local config just in case you decide to dig around the config file and want a reference point.

You should be able to make the toolchain now with ct-ng build. Don’t be like me and do this step at home, or at least with a power cable. It took my relatively recent laptop about 20 minutes of work and 25% of my battery to finish this step.

When it’s done you should have a directory /Volumes/xtool-build- env/aarch64-unknown-elf/bin with a bunch of tools prefixed aarch64-unknown- elf.

Installing Rust

I did this project in Rust, so if you want to build along, you’ll need it. That being said, if you just want the details of how the pieces fit together, feel free to ignore it. Get the Rust toolchain installer and follow the instructions.

The only additional instructions you’ll need are to enable + set nightly, and add the aarch64-unknown-linux-gnu target.

rustup default nightly  
rustup target add aarch64-unknown-linux-gnu

Additional Rust tools

We need a few additional Cargo command line tools that aren’t strictly needed to compile our OS crate, but do need to be on the command line

rustup component add llvm-tools-preview  
cargo install xargo  
cargo install cargo-binutils

Installing QEMU

Super simple: brew install qemu.

Sanity Checks Before Continuing

If you’re trying to get everything working then you should now have (on $PATH):

  • aarch64-unknown-elf-gdb
  • qemu-system-aarch64
  • cargo-size (from cargo-binutils)