Firecracker Developer Setup¶

This is the in-depth Firecracker guide: the full manual setup, building from source, and custom kernels/images. Firecracker is Linux-only and requires KVM. The remote-Linux-host workflow is the common case: you run shed on your laptop and shed-server on a separate Linux box you SSH into.

Just want a server running?

Use the Linux Quickstart — the shed-server deb plus sudo shed-server setup. This page is for the manual/from-source setup, custom kernels, and operations detail.

Prerequisites¶

Linux host with KVM support (/dev/kvm)
Root access (for network and capability setup)
Docker CE (for pulling and converting images). Install from Docker's official repository, not the docker.io package in Ubuntu's default repositories.
For source builds: Go 1.24+ on the host where you run scripts/build-firecracker-rootfs.sh (the script unconditionally builds shed-agent and shed-firstboot with Go). If you don't want Go on the remote Linux host, cross-compile the binaries on another machine and copy them in, or use the deb package.

1. Install shed¶

deb Package (Recommended)¶

Download and install the .deb from the latest release:

# Find the latest version at https://github.com/charliek/shed/releases
# or, with the gh CLI:
VERSION=$(gh release view --repo charliek/shed --json tagName -q .tagName | sed 's/^v//')

wget https://github.com/charliek/shed/releases/download/v${VERSION}/shed-server_${VERSION}_amd64.deb
sudo dpkg -i shed-server_${VERSION}_amd64.deb

This installs:

shed CLI and shed-server binaries to /usr/local/bin/
Default Firecracker server config at /etc/shed/server.yaml
Systemd unit file (enabled but not started)

Build from Source (Alternative)¶

On the remote Linux host (or wherever you'll run shed-server):

ssh charliek@your.linux.host.example.com
git clone https://github.com/charliek/shed.git
cd shed
make build

When building from source you must also do the steps that shed-server setup automates for the deb install:

Set up the bridge network
Check and join the KVM group
Set CAP_NET_ADMIN on the binaries (or run as root)
Build or pull VM images — see Set Up Images

Use shed-server install to create the systemd service, or run ./bin/shed-server serve --config <path> directly.

2. Run Automated Setup¶

The setup command handles all Firecracker infrastructure in a single step:

sudo shed-server setup

This performs:

Step	What it does
KVM check	Verifies `/dev/kvm` is available
Docker check	Verifies Docker CE is installed
Firecracker	Downloads Firecracker v1.14.1 and jailer to `/usr/local/bin/`
Kernel	Downloads a fallback CI kernel to `/var/lib/shed/firecracker/images/vmlinux`
Directories	Creates `/var/lib/shed/firecracker/{instances,images}` and `/var/run/shed/firecracker/`
Bridge	Creates `shed-br0` at `172.30.0.1/24`
NAT	Enables IP forwarding and iptables masquerade rules
Capabilities	Sets `CAP_NET_ADMIN` on `shed-server` and `firecracker`

The command is idempotent — safe to re-run after upgrades or if something needs to be reconfigured.

3. Pre-cache Images (Optional)¶

Pull and convert VM images before creating your first shed:

sudo shed-server pull-images

This pulls the default_image and every image_aliases ref registry-direct and lands the OCI blobs under images_dir/blobs/sha256/, including the pre-built rootfs erofs blob — mounted as-is on boot, with no host-side mkfs.erofs (since v0.5.2).

The OCI image store lives under /var/lib/shed/firecracker/images/. See Storage Model for the full layout and the upgrade cookbook for how to bump image versions and free disk space.

4. Configure¶

Edit /etc/shed/server.yaml to configure mounts and extensions:

name: my-server

http_port: 8080
ssh_port: 2222

default_backend: firecracker
log_level: info

# Host directories to mount into VMs via 9P (map of name -> {source, target, readonly}).
# The older `credentials:` key is still honored as a fallback when `mounts:`
# is absent, but `mounts:` is preferred.
mounts:
  claude:
    source: /root/.shed/mounts/claude
    target: /home/shed/.claude
    readonly: false

# Environment variables passed to git clone and provisioning hooks
env_file: /root/.shed/env

# Extensions provide credential brokering from host to VM.
# Requires shed-host-agent to be installed and running.
# extensions:
#   enabled:
#     - ssh-agent
#     - aws-credentials
#     - docker-credentials

firecracker:
  # default_image / image_aliases omitted -> resolved from the server
  # version: ghcr.io/charliek/shed-fc-{base,extensions,full}:vX.Y.Z. A
  # released (deb) shed-server fills these in automatically; pin them
  # explicitly only on an ahead-of-tag/dirty source build or to use a
  # custom registry (see "Set Up Images" below).
  pull_policy: missing
  instance_dir: /var/lib/shed/firecracker/instances
  socket_dir: /var/run/shed/firecracker
  # uppers_dir is optional. Defaults to <images_dir>/uppers. If you set
  # a non-default instance_dir (e.g. for testing under /tmp), set
  # uppers_dir to a matching location so per-shed writable layers live
  # alongside their sheds.
  # uppers_dir: /var/lib/shed/firecracker/uppers
  default_cpus: 2
  default_memory_mb: 4096
  default_disk_gb: 20
  vsock_base_cid: 100
  console_port: 1024
  notify_port: 1026
  start_timeout: 120s
  stop_timeout: 10s
  bridge_name: shed-br0
  bridge_cidr: 172.30.0.1/24
  tap_prefix: shed-tap

You usually don't need to name the images: a released shed-server resolves default_image and image_aliases from its own version, so the config carries no tag to bump on upgrade. Pin an explicit ref only on an ahead-of-tag/dirty source build (which has no matching published image), or to use a custom registry. To pin a mirror while still tracking the server version, use the ${shed.version} token:

firecracker:
  default_image: myregistry.example.com/shed-fc-full:${shed.version}

Check https://github.com/charliek/shed/pkgs/container/shed-fc-full for tags. Pre-v0.5.0 published images use the legacy flattened layout and will not work with v0.5.0+ shed-server.

The deb package generates this config with a matching version automatically.

Private Repo Access¶

Private Git authentication is handled via the SSH agent forwarding extension. For Git configuration inside the VM (e.g., .gitconfig), use shed sync to push it as a dotfile rather than mounting single files as credentials.

5. Start the Server¶

sudo systemctl start shed-server

Check status:

sudo systemctl status shed-server

View logs:

sudo journalctl -u shed-server -f

The service starts automatically on reboot since the deb package enables it during installation.

6. Create a Firecracker Shed¶

# Create a shed with the Firecracker backend (default `full` variant)
shed create myproject --backend=firecracker

# Or with custom resources
shed create myproject --backend=firecracker --cpus=4 --memory=8192

# Or with a specific variant
shed create myproject --backend=firecracker --image extensions

Upgrading from v0.4.x¶

The image store schema changed from v2 to v3 with the OCI image rollout. Pre-v3 sheds, snapshots, and cached blobs are not migrated automatically — the in-guest initramfs rejects them.

To upgrade:

sudo systemctl stop shed-server

# Wipe the legacy store. Backup first if anything in there is precious.
sudo rm -rf /var/lib/shed/firecracker/images/blobs
sudo rm -rf /var/lib/shed/firecracker/images/instances
sudo rm -rf /var/lib/shed/firecracker/images/snapshots
sudo rm -rf /var/lib/shed/firecracker/images/uppers
sudo rm -rf /var/lib/shed/firecracker/instances

# Install the new release. Pick the latest v0.5.x deb from the releases page
# (https://github.com/charliek/shed/releases), or with the gh CLI:
VERSION=$(gh release view --repo charliek/shed --json tagName -q .tagName | sed 's/^v//')
wget https://github.com/charliek/shed/releases/download/v${VERSION}/shed-server_${VERSION}_amd64.deb
sudo dpkg -i shed-server_${VERSION}_amd64.deb

sudo systemctl start shed-server
sudo shed-server pull-images

See the v0.4 → v0.5 Upgrade Guide for what's new in v0.5.0, what carries over, and recovery scenarios.

Workspace data under --local-dir mounts is unaffected. Workspace data that lived only inside the deleted upper layers is lost, by design.

Manual Setup Reference¶

The sections below document the individual steps that shed-server setup automates. If you built from source (no deb), you need to walk through these manually — the deb install path is the only one where sudo shed-server setup does it all for you.

Source-build users should at minimum do:

Check KVM Support (add user to kvm group)
Set Up Bridge Network (bridge + NAT + IP forwarding)
Set Capabilities so shed-server can create TAP devices without sudo
Create Required Directories

Then continue with Set Up Images and the main flow.

Check KVM Support¶

# Check if KVM is available
ls -la /dev/kvm

# If not accessible, add your user to the kvm group
sudo usermod -aG kvm $USER
# Log out and back in for changes to take effect

Download Firecracker¶

Run the download script to get the Firecracker binary:

./scripts/download-firecracker.sh

This installs /usr/local/bin/firecracker (v1.14.1). When using published images, the kernel is included in the image and extracted automatically.

Set Up Images¶

Option A: Use published images (recommended)¶

Shed pulls published OCI references registry-direct (no Docker daemon needed) and stacks the layers as overlayfs lowers on first boot. Published images include a custom Firecracker kernel with Docker, 9P, and BPF support — no separate kernel build needed.

A released shed-server resolves the images from its own version, so the common config names none of them:

firecracker:
  pull_policy: missing
  images_dir: /var/lib/shed/firecracker/images
  # default_image / image_aliases omitted -> resolved from the server version

Pin an explicit ref only on an ahead-of-tag/dirty source build, or to use a different release/registry:

firecracker:
  default_image: ghcr.io/charliek/shed-fc-full:v0.6.2
  image_aliases:
    base: ghcr.io/charliek/shed-fc-base:v0.6.2
    extensions: ghcr.io/charliek/shed-fc-extensions:v0.6.2
    full: ghcr.io/charliek/shed-fc-full:v0.6.2
  pull_policy: missing
  images_dir: /var/lib/shed/firecracker/images

Check https://github.com/charliek/shed/pkgs/container/shed-fc-full for tags. See Images for available images and configuration details.

Option B: Build from source¶

Build rootfs images locally. Requires Go 1.24+ on the host where the script runs (the script compiles shed-agent and shed-firstboot).

# Build the default variant (full)
./scripts/build-firecracker-rootfs.sh

# Build a specific variant
./scripts/build-firecracker-rootfs.sh --variant base
./scripts/build-firecracker-rootfs.sh --variant extensions

# Build all variants (base, extensions, full)
./scripts/build-firecracker-rootfs.sh --all

The script writes directly into the local blob store at /var/lib/shed/firecracker/images/blobs/ and advances the base, extensions, and full tags. Confirm with shed image ls.

Configure server for local builds¶

When you build images locally, label them with shed image build/pull -t <label> and reference the label. Either set default_image to a label (used when no --image is passed) or pass --image <label> per create:

firecracker:
  instance_dir: /var/lib/shed/firecracker/instances
  socket_dir: /var/run/shed/firecracker
  images_dir: /var/lib/shed/firecracker/images
  default_image: full   # a local label, or a ghcr.io/...:vX ref

Then create (omit --image to use default_image, or override per shed):

shed create test            # uses default_image
shed create dev --image base

default_image may be a Docker ref (ghcr.io/...:vX), an image_aliases name, a local label set with -t, or an absolute path to an ext4/erofs image. Docker refs are pulled on first use per pull_policy.

Set Up Bridge Network¶

Firecracker VMs need a bridge network for connectivity. This is a one-time setup.

Create the Bridge¶

sudo ip link add shed-br0 type bridge
sudo ip addr add 172.30.0.1/24 dev shed-br0
sudo ip link set shed-br0 up

Enable IP Forwarding¶

# Enable IP forwarding (temporary)
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

# Make permanent
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-ip-forward.conf
sudo sysctl -p /etc/sysctl.d/99-ip-forward.conf

Configure NAT for Internet Access¶

sudo iptables -t nat -A POSTROUTING -s 172.30.0.0/24 -j MASQUERADE
sudo iptables -A FORWARD -i shed-br0 -j ACCEPT
sudo iptables -A FORWARD -o shed-br0 -j ACCEPT

Make Network Persistent (Optional)¶

To persist the bridge across reboots, create a systemd-networkd configuration:

# /etc/systemd/network/shed-br0.netdev
cat << 'EOF' | sudo tee /etc/systemd/network/shed-br0.netdev
[NetDev]
Name=shed-br0
Kind=bridge
EOF

# /etc/systemd/network/shed-br0.network
cat << 'EOF' | sudo tee /etc/systemd/network/shed-br0.network
[Match]
Name=shed-br0

[Network]
Address=172.30.0.1/24
ConfigureWithoutCarrier=yes
EOF

# Enable systemd-networkd
sudo systemctl enable systemd-networkd
sudo systemctl restart systemd-networkd

For iptables persistence, install iptables-persistent:

sudo apt install iptables-persistent
sudo netfilter-persistent save

Create Required Directories¶

sudo mkdir -p /var/lib/shed/firecracker/instances
sudo mkdir -p /var/run/shed/firecracker
sudo chown -R $USER:$USER /var/lib/shed/firecracker
sudo chown -R $USER:$USER /var/run/shed/firecracker

Set Capabilities (Alternative to Running as Root)¶

To run shed-server without sudo, grant capabilities to both binaries:

sudo setcap cap_net_admin+ep /usr/local/bin/shed-server
sudo setcap cap_net_admin+ep /usr/local/bin/firecracker

# Verify
getcap /usr/local/bin/shed-server /usr/local/bin/firecracker

Firecracker is spawned as a child process, and Linux capabilities don't inherit to child processes. That's why both binaries need the capability set directly.

9P Kernel Configuration¶

The --local-dir and --add-dir flags and mounts: directory mounts use the 9P filesystem protocol over the TAP bridge network (each --local-dir/--add-dir host directory is mounted at /home/shed/<basename> in the guest). This requires the following kernel configuration options to be built into the guest kernel:

CONFIG_NET_9P=y
CONFIG_NET_9P_FD=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y

The custom kernel built by build-firecracker-kernel.sh already includes these options. Published images also include a compatible kernel. If you are building your own kernel, add these options to your kernel config fragment.

To verify 9P support inside a running VM:

shed exec myproject -- grep 9p /proc/filesystems

If the output includes nodev 9p, the kernel has 9P support.

Known Limitations¶

Server restart does not recover 9P mounts¶

If shed-server restarts while VMs with --local-dir/--add-dir or mounts: directory mounts are running, the 9P servers are not automatically restarted. Running VMs will have stale mounts that return I/O errors. Recovery requires stopping and starting the affected sheds:

shed stop myproject
shed start myproject

UID mapping¶

9P maps UIDs directly (1:1). Host UID 1000 corresponds to guest UID 1000. If the host user running shed-server has a different UID than the shed user inside the VM (UID 1000), file ownership may appear incorrect. This matches the behavior of VirtioFS on Linux. Apple VirtioFS has transparent UID mapping, but that is not available on Linux.

Firewall and iptables¶

The 9P TCP servers bind to the bridge IP on dynamically assigned ports. If the host has restrictive iptables INPUT rules, these ports may be blocked. Ensure that traffic on the bridge network (default 172.30.0.0/24) is allowed:

# Allow all traffic on the shed bridge interface
sudo iptables -A INPUT -i shed-br0 -j ACCEPT

This rule is typically not needed if the default INPUT policy is ACCEPT, which is common on desktop Linux systems.

Troubleshooting¶

KVM Permission Denied¶

failed to create firecracker machine: permission denied

Solution: Add your user to the kvm group:

sudo usermod -aG kvm $USER
# Log out and back in

Bridge Not Found¶

failed to find bridge shed-br0

Solution: Run sudo shed-server setup or create the bridge network manually (see Manual Setup Reference above).

TAP Device Creation Failed¶

failed to create TAP device: operation not permitted

or

Could not create the network device: Open tap device failed: ... Resource busy

Solution: Run shed-server as root or with CAP_NET_ADMIN capability on both binaries:

sudo shed-server serve

# Or with capabilities (must set on BOTH binaries)
sudo setcap cap_net_admin+ep $(which shed-server)
sudo setcap cap_net_admin+ep $(which firecracker)

If you see "Resource busy", clean up stale TAP devices:

sudo ip link delete shed-tap-0  # or whichever TAP is stale

Vsock Address In Use¶

Cannot create backend for vsock device: UnixBind(Os { code: 98, kind: AddrInUse, message: "Address in use" })

Solution: Remove stale vsock socket files:

sudo rm -f /var/run/shed/firecracker/*.vsock

VM Timeout During Start¶

agent health check failed: context deadline exceeded

Possible causes: 1. Rootfs image is corrupted - rebuild with build-firecracker-rootfs.sh 2. Kernel is incompatible - try a different kernel version 3. shed-agent is not starting - check VM console output 4. Stale socket files - remove /var/run/shed/firecracker/*.sock and *.vsock

No Network Connectivity in VM¶

How network is configured: The VM's IP address is passed via kernel command line arguments using the kernel IP autoconfig format: ip=<client>::<gateway>:<netmask>::<device>:off. For example: ip=172.30.0.2::172.30.0.1:255.255.255.0::eth0:off. The kernel configures the network interface automatically during boot.

Verify: 1. Bridge is up: ip link show shed-br0 2. IP forwarding is enabled: cat /proc/sys/net/ipv4/ip_forward 3. NAT rules are in place: sudo iptables -t nat -L -n 4. TAP device is attached to bridge: ip link show 5. Check kernel args inside VM: shed exec myproject -- cat /proc/cmdline 6. Check network-setup ran: shed exec myproject -- systemctl status network-setup 7. Check interface is up: shed exec myproject -- ip addr show eth0

Docker Fails to Start in VM¶

Docker is configured with the vfs storage driver and cgroupfs driver for compatibility with the Firecracker kernel. If Docker fails to start:

# Check Docker daemon status
shed exec myproject -- systemctl status docker

# View Docker daemon logs
shed exec myproject -- journalctl -u docker

# Verify daemon.json exists
shed exec myproject -- cat /etc/docker/daemon.json

# Check cgroup support (kernel args should include cgroup_enable=memory)
shed exec myproject -- cat /proc/cmdline | grep cgroup

If you see cgroup errors, ensure the kernel args include cgroup_enable=memory cgroup_memory=1. This is set automatically by shed-server.

Docker Containers Fail to Start (BPF Error)¶

If you see errors like bpf_prog_query(BPF_CGROUP_DEVICE) failed:

runc create failed: unable to start container process: error during container init:
error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE)
failed: invalid argument

This means the kernel lacks BPF cgroup support. The custom 6.1 kernel built by build-firecracker-kernel.sh has full BPF support. If you're using the CI fallback kernel, build a custom kernel which includes: - CONFIG_BPF=y - CONFIG_BPF_SYSCALL=y - CONFIG_CGROUP_BPF=y

To build and use the Docker-capable kernel:

./scripts/build-firecracker-kernel.sh

Then set kernel_path in your server.yaml to point to the built kernel:

firecracker:
  kernel_path: /var/lib/shed/firecracker/vmlinux

Network Architecture¶

                    ┌─────────────┐
                    │   Host      │
                    │  eth0/wlan  │
                    └──────┬──────┘
                           │ NAT (iptables MASQUERADE)
                    ┌──────┴──────┐
                    │  shed-br0   │  172.30.0.1/24
                    │   (bridge)  │
                    └──────┬──────┘
              ┌────────────┼────────────┐
              │            │            │
        ┌─────┴─────┐┌─────┴─────┐┌─────┴─────┐
        │shed-tap-0 ││shed-tap-1 ││shed-tap-2 │
        └─────┬─────┘└─────┬─────┘└─────┬─────┘
              │            │            │
        ┌─────┴─────┐┌─────┴─────┐┌─────┴─────┐
        │   VM 0    ││   VM 1    ││   VM 2    │
        │172.30.0.2 ││172.30.0.3 ││172.30.0.4 │
        └───────────┘└───────────┘└───────────┘

Each VM gets: - A dedicated TAP device attached to the bridge - A static IP in the 172.30.0.0/24 network - Internet access via NAT - vsock for communication with the host