skip to content
Hacktivate
Table of Contents

It is the tenth of August, it is thirty-eight degrees in my room, and the fan in this laptop has been making a noise I can only describe as pleading. So naturally I decided this was the ideal week to spend my evenings with a piece of software called Firecracker.

In my defence, half the country is already at the beach, nobody is looking at the cluster, and there is no better moment to break something than the week before Ferragosto. If it catches fire, so to speak, there’s nobody around to notice.

Why microVMs at all

Containers are processes. Fast, cheap processes that share one kernel with every other container on the node, and with the node itself. For your own services that’s fine: you wrote them, you trust them, and the blast radius is a problem you already own.

The moment you start running other people’s code, that shared kernel becomes the whole security story. CI runners building arbitrary pull requests, customer-supplied functions, a multi-tenant PaaS, notebooks where “please don’t” is the only thing between a user and /proc. One kernel bug and the isolation you thought you had turns out to have been a namespace and some good manners.

The old answer was a full VM per tenant, which works and costs you thirty seconds of boot time and a few hundred megabytes of QEMU. The microVM answer is: keep the hardware boundary, throw away everything else.

What Firecracker actually is

Firecracker is a VMM that AWS open-sourced in 2018, and it quietly powers a great deal of Lambda and Fargate. It’s written in Rust, built on KVM and the rust-vmm crates, and its defining characteristic is what it refuses to do.

There’s no BIOS. No PCI bus. No emulated graphics, no USB, no CD-ROM, no legacy anything. The device model is virtio-net, virtio-block, virtio-vsock, a serial console, and a partial keyboard controller that exists purely so the guest has a way to ask for a reset. That’s the list. The whole VMM is a few tens of thousands of lines of Rust, and the attack surface it presents to a hostile guest is small enough that you can actually reason about it.

What you get in return is a virtual machine that boots in around 125 ms with less than 5 MiB of memory overhead, on a host that can launch roughly 150 of them per second. It is the only thing in this apartment currently capable of starting up quickly.

The gap Kubernetes has to cross

Here’s the problem. Kubernetes has no idea what a microVM is, and shouldn’t have to. It talks to a CRI runtime, the CRI runtime talks to an OCI runtime, and the OCI runtime is expected to produce something that looks and behaves like a container.

So somebody has to sit in that slot and quietly boot a virtual machine instead. Two projects do it:

  • Kata Containers: a containerd shim that starts a microVM per pod and runs an agent inside it. It supports several hypervisor backends, Firecracker among them. This is the path that actually works with stock Kubernetes today.
  • firecracker-containerd: AWS’s own containerd integration. Excellent as a building block if you’re constructing your own control plane; considerably more assembly required if what you want is pods.

We’ll do Kata.

Path A: Kata Containers with the Firecracker backend

1. Check the node can do it

Terminal window
ls -l /dev/kvm
lscpu | grep -iE 'vmx|svm|hypervisor'

If /dev/kvm isn’t there, nothing below this line matters.

2. Install Kata with kata-deploy

Kata ships a DaemonSet that drops the binaries, kernel and rootfs image into /opt/kata on every node and wires up containerd for you:

Terminal window
kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml

Label the nodes you want it on, and watch it land:

Terminal window
kubectl label node <node> katacontainers.io/kata-runtime=true
kubectl -n kube-system get pods -l name=kata-deploy -w

kata-deploy also creates the RuntimeClass objects, so after this you should see kata-fc among them:

Terminal window
kubectl get runtimeclass

3. The devmapper detour

This is the step that eats your afternoon, and the why matters more than the commands.

Kata normally shares the container’s root filesystem into the guest with virtio-fs, a filesystem passthrough that lets the VM read the image straight off the host. It’s elegant and it’s how kata-qemu works.

Firecracker doesn’t have virtio-fs. Remember the device list: network, block, vsock, serial. No filesystem passthrough anywhere on it. So the container image has to reach the guest as a block device, which means containerd has to hand Kata a block snapshot rather than an overlayfs directory. Which means the devmapper snapshotter.

For a test cluster, a loopback-backed thin pool will do:

Terminal window
DATA=/var/lib/containerd/devmapper/data
META=/var/lib/containerd/devmapper/meta
mkdir -p /var/lib/containerd/devmapper
truncate -s 100G "$DATA"
truncate -s 2G   "$META"


DATA_DEV=$(losetup --find --show "$DATA")
META_DEV=$(losetup --find --show "$META")


SECTORS=$(blockdev --getsize "$DATA_DEV")
dmsetup create contd-thin-pool \
  --table "0 $SECTORS thin-pool $META_DEV $DATA_DEV 128 32768 1 skip_block_zeroing"

Then tell containerd about it, in /etc/containerd/config.toml:

/etc/containerd/config.toml
[plugins."io.containerd.snapshotter.v1.devmapper"]
  pool_name = "contd-thin-pool"
  root_path = "/var/lib/containerd/devmapper"
  base_image_size = "10GB"
  discard_blocks = true


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc]
  runtime_type = "io.containerd.kata-fc.v2"
  snapshotter = "devmapper"

That second block is the nice part. Since containerd 1.7 you can set the snapshotter per runtime, so only your kata-fc pods pay the devmapper tax; everything else on the node carries on with overlayfs as before. Before 1.7 it was all-or-nothing, and moving a whole node onto devmapper for the sake of a handful of pods was a miserable trade.

Terminal window
systemctl restart containerd

4. Run a pod in a microVM

microvm-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: hello-microvm
spec:
  runtimeClassName: kata-fc
  containers:
    - name: app
      image: alpine:3.20
      command: ["sleep", "3600"]
      resources:
        limits:
          memory: "256Mi"
          cpu: "500m"

One line. runtimeClassName: kata-fc is the entire Kubernetes-facing surface of everything above.

5. Convince yourself it worked

The satisfying check is the kernel version. A normal container reports the node’s kernel, because it is running on the node’s kernel. A Kata pod reports the guest kernel that kata-deploy installed:

Terminal window
$ uname -r                                                    # on the node
6.9.7-arch1-1


$ kubectl exec hello-microvm -- uname -r                      # inside the microVM
6.1.62


$ kubectl exec hello-microvm -- cat /proc/1/comm
sh

Different kernel, PID 1 of its own. That’s a real machine boundary, not a namespace wearing a convincing hat.

For extra confidence, kubectl exec hello-microvm -- dmesg | head -20 will show a guest boot log that mentions virtio-mmio devices and no PCI bus whatsoever. That’s the Firecracker signature.

Path B: firecracker-containerd

AWS’s own integration is a snapshotter, a runtime shim and a firecracker-control service that manages VM lifecycle directly. It gives you finer control over the microVMs, including running several containers inside one VM, and it’s the closest thing to how AWS actually does this internally.

What it doesn’t give you is a Kubernetes story. There’s no RuntimeClass integration to speak of, and you’d be writing the glue yourself. It’s the right choice if you’re building a platform like Lambda; it’s the wrong choice if you’re trying to get pods onto microVMs before the weekend.

What you’re giving up

Now for the honest list, because this is the part the marketing pages tend to skim over:

  • No virtio-fs, as covered. Beyond the devmapper setup, this makes hostPath volumes awkward and gives ConfigMaps and Secrets a more roundabout journey into the guest than you might expect.
  • No device passthrough. No PCI bus means no GPUs. If your workload wants a GPU, Firecracker is simply the wrong VMM and you want kata-qemu or Cloud Hypervisor.
  • Resources are fixed at boot. Kata sizes the microVM from the pod’s limits when it starts. Vertical scaling of a running pod isn’t a thing here in the way you may be used to.
  • No privileged pods, no host namespaces. Which is, to be fair, rather the point. But it does mean your DaemonSets full of node agents aren’t moving to kata-fc.
  • Networking goes through a tap device per microVM, bridged to your CNI. It works, and tc-redirect-tap handles the awkward parts, but it is one more layer between a packet and where it’s going.
  • Startup isn’t 125 ms. Firecracker boots in 125 ms. Your pod also has to pull an image, convert it into a thin snapshot, boot the guest and start an agent. You’ll land somewhere in the hundreds of milliseconds to low seconds. Still excellent for a VM, still not a container.

So is it worth it

If you’re running your own trusted services, no. You’d be buying an elaborate solution to a threat model you don’t have, and paying for it in operational surface every single day. Namespaces, seccomp and a decent RuntimeDefault profile are the right answer, and they don’t require you to understand thin pools.

If you’re running code that arrives from outside your organisation (CI for public pull requests, customer functions, tenant workloads, anything where “what if this one is hostile” is a Tuesday rather than a hypothetical), then a hardware boundary per pod for a few hundred milliseconds and a few tens of megabytes is one of the better deals in infrastructure.

And if you find yourself wanting the microVM isolation but keep tripping over the missing virtio-fs, look at Kata’s Cloud Hypervisor backend (kata-clh). Same rust-vmm foundations, same minimal spirit, but it does have virtio-fs and device hotplug, so the whole devmapper detour disappears. You trade a little boot time for a lot less yak-shaving. Kata 3.x also ships Dragonball, a VMM built directly into the runtime, which removes a process hop entirely.

Anyway. The cluster survived, the microVMs boot in a fraction of a second, and the only thing in this building running hot tonight is me. Firecracker is the coolest thing with “fire” in the name that I’ve touched all month. On the tenth of August in Italy, that’s a lower bar than it sounds.

Go outside. The pods will still be there.