Quickly Boot a Minimal Linux Kernel + Initrd in Cloud Hypervisor or QEMU
For various reasons, such as OS development, it is convenient to quickly start up an instance of QEMU or Cloud Hypervisor with a custom kernel and initrd. However, often the major struggle is to get some kernel and some initrd, i.e., the boot items, quickly and easily.
Scope
Foremost, this blog post is a guideline for direct (Linux) kernel boot, possibly along with a matching initrd. If you can boot an ISO installer image (-cdrom parameter in QEMU), you are good to go. If you have a disk image following the bootloader specification (such as Ubuntu cloud-init image), you can boot that file relatively easily in Cloud Hypervisor with the hypervisor firmware. But sometimes, you need or want direct kernel boot. This is where this blog post presents a solution.
My approach targets people who have Nix (nix-build, nix eval, …) installed on their system. For people who don’t have Nix, this post is less helpful, unfortunately.
Quickly Get Bootitems
Well, how do we get some bootitems, e.g., Linux kernel and matching initrd, quickly? As part of my public NixOS configurations, I also maintain various Nix helper libraries. One of them is my collection of bootitems. My Nix library contains tooling to build a minimal Linux kernel based on a minimal kernel configuration (no audio or video drivers, for example) for different versions of the Linux source tree. In the end, we get the following attribute structure from the library:
{
linux = {
initrds = {
default = <derivation initrd>;
minimal = <derivation initrd>;
};
kernels = {
latest = <derivation linux-6.13.6>;
linux_6_12 = <derivation linux-6.12.18>;
linux_6_12_18 = <derivation linux-6.12.18>;
linux_6_13 = <derivation linux-6.13.6>;
linux_6_13_6 = <derivation linux-6.13.6>;
lts = <derivation linux-6.12.18>;
stable = <derivation linux-6.12.18>;
};
};
}Okay, cool, but how do we actually can use the files now? Well, good question! Here is a bash script that you can use as template:
# Get Nix store path to bootitems library.
export LIB=$(nix eval github:phip1611/nixos-configs#lib.bootitems)
# Lets start by getting and building kernel and initrd.
# The build might take a few minutes.
# Select kernel version
export KERNEL="stable"
export KERNEL=$(nix-build -E "
let
pkgs = import <nixpkgs> {};
lib = import $LIB { inherit pkgs; };
in
lib.linux.kernels.$KERNEL
")
export KERNEL="$KERNEL/bzImage"
# Select initrd
export INITRD="default"
export INITRD=$(nix-build -E "
let
pkgs = import <nixpkgs> {};
lib = import $LIB { inherit pkgs; };
in
lib.linux.initrds.$INITRD
")
export INITRD="$INITRD/initrd"
echo "Booting kernel=$KERNEL, initrd=$INITRD"
cloud-hypervisor \
--kernel "$KERNEL" \
--cmdline "console=ttyS0" \
--initramfs "$INITRD" \
--serial "tty" \
--console "off"This will show you something like this:
And that’s all we need! In case you prefer QEMU, a similar QEMU invocation is the following:
qemu-system-x86_64 \ -kernel $KERNEL \ -append console=ttyS0 \ -initrd $INITRD \ -serial stdio \ -machine q35,accel=kvm \ -m 1G
Convenience of NixOS
This is quite cool, isn’t it? But wouldn’t it be much more convenient if we don’t need this magic script all the time, but instead have the bootitems available anytime in an easy to find and accessible location? This is where my bootmodules NixOS module comes into the game! When you use it as part of your NixOS configuration, you will find the following files always available:
$ tree /etc/bootitems /etc/bootitems ├── ... ├── linux │ ├── initrd_default -> /etc/static/bootitems/linux/initrd_default │ ├── initrd_minimal -> /etc/static/bootitems/linux/initrd_minimal │ └── kernel_minimal │ ├── latest.bzImage -> /etc/static/bootitems/linux/kernel_minimal/latest.bzImage │ ├── latest.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/latest.vmlinux │ ├── linux_6_13_6.bzImage -> /etc/static/bootitems/linux/kernel_minimal/linux_6_13_6.bzImage │ ├── linux_6_13_6.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/linux_6_13_6.vmlinux │ ├── linux_6_13.bzImage -> /etc/static/bootitems/linux/kernel_minimal/linux_6_13.bzImage │ ├── linux_6_13.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/linux_6_13.vmlinux │ ├── linux_6_6_82.bzImage -> /etc/static/bootitems/linux/kernel_minimal/linux_6_6_82.bzImage │ ├── linux_6_6_82.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/linux_6_6_82.vmlinux │ ├── linux_6_6.bzImage -> /etc/static/bootitems/linux/kernel_minimal/linux_6_6.bzImage │ ├── linux_6_6.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/linux_6_6.vmlinux │ ├── lts.bzImage -> /etc/static/bootitems/linux/kernel_minimal/lts.bzImage │ ├── lts.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/lts.vmlinux │ ├── stable.bzImage -> /etc/static/bootitems/linux/kernel_minimal/stable.bzImage │ └── stable.vmlinux -> /etc/static/bootitems/linux/kernel_minimal/stable.vmlinux ├── ...
I personally think this is extremely useful and helpful, I use it at least a few times a month. I hope this helps or improves your workflow as well, or at least is an inspiration to you.
Is it really quick? What about the build time?
It is fairly quick; in the sense that you quickly know what to do; just a few commands. The build of the (very small!) kernel however may take one minutes (reference: AMD Ryzen 7 7840U with 16 cores). Don’t forget that once you built that, it is part of your Nix store and cached for a while.
0 Comments