Rebuilding My WebDAV Server: A Bigger /boot and NixOS's Declarative Payoff

A while back I wrote about building a personal cloud storage server with WebDAV, S3, and NixOS on EC2. It ran quietly for months — game saves syncing, music streaming, backups landing in S3. Then one morning it stopped upgrading itself, and I didn’t notice until I tried to deploy a config change and the rebuild failed outright.

The culprit was a full /boot partition. The fix turned into a small lesson about how NixOS’s design makes boot partition sizing a first-class concern, and a reminder of why the declarative approach is worth the trouble: getting the whole thing rebuilt and back online took minutes, not an afternoon.

The Server That Stopped Upgrading Link to heading

The server auto-upgrades on a schedule. One of those upgrades had been failing every single night, silently, for weeks. The error only surfaced when I ran a manual rebuild:

cannot copy .../linux-6.18.49/Image to /boot/kernels/...-Image.tmp: No space left on device
Failed to install bootloader

A quick look confirmed it:

$ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  249M  249M     0 100% /boot

The /boot partition was completely full. Worse, it had been full long enough that each failed nightly upgrade left behind a zero-byte .tmp file — the stub of a kernel it never had room to write. The server was still serving WebDAV traffic fine, but it had quietly stopped applying any updates, including security patches. On a box exposed to the internet, that’s the kind of silent failure you want to understand and prevent, not just clear.

So why does a NixOS /boot fill up in the first place?

Why NixOS Makes /boot a First-Class Concern Link to heading

NixOS builds every system change as a generation — a complete, bootable snapshot of your system. That’s the feature that gives you atomic upgrades and instant rollback. But each generation that changes the kernel keeps its own kernel image and initial ramdisk (initrd) in /boot.

On the arm64 instance this server runs, each generation’s kernel + initrd is roughly 90MB. The stock NixOS Amazon Machine Image (AMI) ships a 256MiB EFI System Partition (ESP) — the FAT partition mounted at /boot. Do the math and the partition only ever had room for two or three generations before it was full.

There’s a second, sneakier factor. switch-to-configuration copies the new kernel into /boot before it prunes the old ones (nixpkgs#23926). So even if your steady state fits, the brief moment during an upgrade needs room for one extra kernel. On a 256MiB partition, that transient is exactly what tipped it over.

My original config tried to cap the number of retained generations, but the setting was on the wrong bootloader:

# WRONG: the amazon-image profile boots with GRUB, not systemd-boot,
# so this option was silently ignored and nothing capped /boot.
boot.loader.systemd-boot.configurationLimit = 2;

The NixOS Amazon image boots with GRUB. Setting a limit on systemd-boot is a no-op there — the option applies to a bootloader that isn’t in use, so no pruning ever happened. The correct option is:

# RIGHT: cap retained generations on the bootloader actually in use.
boot.loader.grub.configurationLimit = 3;

That stops the unbounded accumulation. But capping generations only helps if the partition is big enough to hold them in the first place — and 256MiB isn’t. The real fix had to make /boot bigger.

Why You Can’t Just Resize /boot Link to heading

The obvious move is “make the partition bigger.” On this setup, that’s harder than it sounds.

• growPartition only grows root. The NixOS Amazon image expands the root filesystem on boot, but it never touches the ESP. Giving the instance a bigger EBS volume gets you a bigger root partition and the same 256MiB /boot.
• The ESP sits before root on disk. Growing /boot in place would mean shrinking and moving the root partition that follows it — risky surgery on a live server’s root filesystem.
• Relaunching doesn’t help. The 256MiB size is baked into the stock AMI’s disk image. A fresh instance from the same AMI has the same small /boot.

The partition layout is decided when the disk image is built. So the durable fix is to build our own image — one that lays down a bigger ESP from the start.

Baking a Bigger Boot Partition into a Custom AMI Link to heading

NixOS builds its Amazon images with a function called make-disk-image.nix, and that function takes a bootSize parameter. The stock amazon image never overrides it, so it falls back to the 256MiB default. By calling the image builder ourselves and passing a larger bootSize, we get an AMI identical to stock except for a roomier /boot.

The image expression imports the standard amazon-image profile, then overrides the resulting build to request a 1GiB ESP. It also pins nixpkgs to nixos-unstable, which matters later for the upgrade channel.

# ami/image.nix — a stock NixOS amazon image, but with a 1GiB /boot
system.build.amazonImage = lib.mkForce (
  import "${pkgsSrc}/nixos/lib/make-disk-image.nix" {
    inherit lib config pkgs;
    inherit (config.virtualisation) diskSize;

    fsType = "ext4";
    partitionTableType = "efi";   # GPT with an EFI System Partition
    bootSize = "1024M";           # the whole point: a 1GiB ESP, not 256MiB
    format = "vpc";               # VHD, what AWS import-snapshot expects
  }
);

Building an arm64 image is fastest on an arm64 host, so the build runs on a throwaway Graviton instance that gets terminated afterward. The resulting disk image becomes an AMI in three AWS steps: upload the VHD to S3, import it as an EBS snapshot, then register that snapshot as an AMI. The commands, in order:

# 1. Upload the built VHD to an S3 bucket in the target region
aws s3 cp result/nixos.vhd s3://my-import-bucket/nixos-bigboot.vhd

# 2. Import the VHD as an EBS snapshot (requires the 'vmimport' IAM role)
aws ec2 import-snapshot \
  --disk-container "Format=VHD,UserBucket={S3Bucket=my-import-bucket,S3Key=nixos-bigboot.vhd}"

# 3. Register an AMI from the imported snapshot (arm64, UEFI boot)
aws ec2 register-image \
  --name nixos-bigboot \
  --architecture arm64 --boot-mode uefi \
  --root-device-name /dev/xvda \
  --block-device-mappings "DeviceName=/dev/xvda,Ebs={SnapshotId=snap-XXXX,VolumeType=gp3}"

The whole build-and-register flow is scripted in the repository under ami/, including the one-time vmimport role setup that import-snapshot requires. The output is a new AMI ID. Now the interesting part: putting it to use without a painful migration.

The Declarative Payoff: Back Online in Minutes Link to heading

Here’s where the original design earned its keep. The server from the first post is stateless — all data lives in S3, and a static Elastic IP points at whatever instance is current. The EC2 box is pure cattle. That means “replace the instance” is a safe, boring operation instead of a nerve-wracking one.

The cutover is a single CloudFormation parameter change: point the stack’s AMI ID at the new image. CloudFormation launches a fresh instance from the bigger-/boot AMI, terminates the old one, and re-associates the Elastic IP automatically.

The new instance comes up as a bare NixOS system. Bringing back the entire service — Caddy with its TLS certs and auth, the rclone WebDAV bridge, fail2ban — was two actions: copy the same configuration.nix the first post described, and run one command.

# On the fresh instance: apply the exact same declarative config
nixos-rebuild switch

That single rebuild reconstructed every service, firewall rule, and hardening tweak identically. Within minutes the endpoint was answering again:

$ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 1015M  129M   887M  13% /boot   # 1GiB now, 13% used

$ curl -o /dev/null -w "%{http_code}\n" https://saves.example.com/
401   # Caddy up, TLS valid, auth enforced

This is the difference between declarative and hand-managed infrastructure. There was no runbook to follow from memory, no half-remembered apt incantations, no “which services did I enable again?” The config is the server, so rebuilding the server is just applying the config.

One key detail is worth calling out from the cutover: stateVersion stays pinned. It records the NixOS version at initial deployment (26.05 here) and must never change, even though the new AMI runs a newer release. It’s a compatibility marker, not a version selector.

Sizing /boot Before It Bites Link to heading

If you run NixOS on a cloud image with auto-upgrades, treat /boot as something to size deliberately on day one. The lessons from this incident distill to three:

• Set the generation limit on the bootloader you actually use. Confirm whether your image boots GRUB or systemd-boot, and cap configurationLimit there. A limit on the wrong bootloader is silent — it looks configured but does nothing.
• Match the partition size to your kernel footprint. At roughly 90MB per arm64 generation plus the write-then-prune transient during upgrades, a 256MiB ESP can’t hold even three generations. A 1GiB /boot leaves comfortable headroom.
• Bake the layout into the image, not the running system. Partition geometry is set at image-build time. Building a custom AMI with the right bootSize fixes the problem at its source, so no instance you launch from it ever hits the wall.

The full source — CloudFormation template, NixOS configuration, and the ami/ build tooling that produces the bigger-/boot image — is in the repository on GitHub. If you’re running a self-upgrading NixOS server on a stock cloud image, check your /boot size before it checks you.