Upgrade Guide: v0.5.7 to v0.5.8¶
v0.5.8 is a maintenance release that closes two operational bugs discovered while rolling v0.5.7 out to mini2 / mini3 / the mac, plus a documented playbook for the routine "I upgraded shed, clean up the old images and reclaim disk space" workflow. No manifest format changes, no image cache wipe required — this is not the v0.5.1 → v0.5.2 kind of upgrade.
What changed¶
shed image prune no longer deletes tagged manifests (#147)¶
Pre-v0.5.8 shed image prune followed Docker's "tags are
informational" model: only sheds, snapshots, and in-flight create
markers were protective. That made the very natural sequence
a footgun on a fresh host — prune would delete the manifest just
pulled (no shed was yet pinning it), leaving either a tag pointing
at a missing blob (mac VZ behavior — image ls then says "No
images available") or a tag silently reverted to an older
locally-cached manifest (mini2 saw base flip from the v0.5.7
manifest back to a v0.5.3 manifest with the missing zip).
v0.5.8 makes tags protective: the prune walker now treats every
tag's manifest digest as live, including the manifest's transitive
blobs (config, layers, kernel, initrd, rootfs erofs). The
documented cleanup workflow is now shed image rm <tag> first,
then shed image prune — same shape as Docker's docker rmi
followed by docker image prune. See the
Image cache cleanup playbook
below.
Local image builds pin the Ubuntu kernel package (#148)¶
Pre-v0.5.8 initramfs/Dockerfile and vz/Dockerfile each
installed the linux-image-virtual apt metapackage independently.
The two installs run in separate docker buildx build invocations
with their own BuildKit cache; when those caches diverge (common
in iterative local rebuilds via ./scripts/build-vz-rootfs.sh),
the initramfs's staged erofs.ko + libcrc32c.ko target a
different kernel ABI than the booted vmlinuz. The VZ guest then
panics in shed-initramfs:
shed-initramfs: insmod 10-libcrc32c.ko.zst returned non-zero
shed-initramfs: insmod 20-erofs.ko.zst returned non-zero
mount: mounting /dev/vdb on /lower failed: No such device
shed-initramfs PANIC [SHED-INIT-03]: failed to mount /dev/vdb at /lower (erofs)
VZ kernels lack erofs as built-in, so the .ko load is load-bearing. FC's custom kernel has erofs built-in, so the FC side merely logs a confusing "insmod failed" warning that the kernel transparently recovers from — no panic, but still a mismatched-kernel state.
GitHub Actions Publish Images and Release doesn't hit this
(every runner has a fresh BuildKit cache, so both stages see the
same apt snapshot), so the published images on ghcr.io are
fine. The bug only bit operators iterating on the image scripts
locally.
v0.5.8 pins both Dockerfiles to
ARG LINUX_IMAGE_VERSION=6.8.0-124 and installs
linux-image-${LINUX_IMAGE_VERSION}-generic directly. A new
make check-kernel-pin target (wired into make check) fails the
build if the two values drift apart. See
Kernel version pinning
in the images reference.
Operator upgrade steps¶
Linux (.deb)¶
curl -fsSL -o /tmp/shed-server.deb \
https://github.com/charliek/shed/releases/download/v0.5.8/shed-server_0.5.8_amd64.deb
sudo dpkg -i /tmp/shed-server.deb
sudo systemctl restart shed-server
The .deb does NOT restart the service automatically
dpkg -i installs the new binary but leaves the old
shed-server process running. Without the explicit
systemctl restart shed-server, the prune fix and any other
behavioral changes won't take effect. A follow-up patch to the
.deb postinst is tracked separately; for v0.5.8 the workaround
is the manual restart above.
After restarting, verify the running version:
macOS (Homebrew)¶
Then bump the image refs in /opt/homebrew/etc/shed/server.yaml
(or /usr/local/etc/shed/server.yaml on Intel Macs) to the new
release. Homebrew does NOT manage this file across upgrades — it
keeps your existing config in place — so the refs stay at
whatever they were when you first installed:
vz:
base_rootfs: ghcr.io/charliek/shed-vz-full:v0.5.8
images:
base: ghcr.io/charliek/shed-vz-base:v0.5.8
extensions: ghcr.io/charliek/shed-vz-extensions:v0.5.8
full: ghcr.io/charliek/shed-vz-full:v0.5.8
Restart again after editing:
Image cache cleanup playbook¶
This is the routine "I bumped shed to a new release, now remove the old images and reclaim disk space" workflow. With the v0.5.8 prune fix the sequence is safe to run any time — pre-v0.5.8 step 3 would delete the tags you just pulled in step 1.
Step 1. Re-pull the configured images at the new release version. This advances the on-disk tag to the new manifest digest; the old manifest blob is now unreferenced by a tag (only by any shed that's still booted off it).
Step 2 (optional). Remove any stale tags you added by hand.
The base/extensions/full tags are advanced in place by
step 1, so this is only needed for ad-hoc tags (e.g. shed image
tag <digest> experimental).
Step 3. GC anything no longer reachable from a tag, a running shed, or a snapshot pin. With the v0.5.8 fix, this is safe to run any time; the configured tags survive.
Step 4 (optional). Docker layer cache cleanup on the host
running ./scripts/build-vz-rootfs.sh or similar. Only relevant
if you do local image builds. Frees BuildKit-cached apt layers +
the cached ubuntu:24.04 base.
docker buildx prune --all --force
# And/or, for the broader docker layer cache:
docker system prune -af
Verify after cleanup¶
Every tag in server.yaml's images: map should appear with a
current digest. shed image inspect <tag> confirms the
io.shed.rootfs.erofs.digest annotation and the configured
source ref.
Troubleshooting¶
"image ls says No images available after prune"¶
You're on a pre-v0.5.8 server hitting the prune bug — upgrade per
the steps above. On v0.5.8+ this symptom usually means a registry
pull was interrupted; re-run the shed image pull from step 1
of the cleanup playbook.
"Locally-built rootfs panics with SHED-INIT-03"¶
You're rebuilding images locally on pre-v0.5.8 source. Upgrade
the source tree (git pull then make build) and the next local
build will pin the kernel version. The safe escape hatch on any
version is
which forces both apt-get install invocations to share the same
apt snapshot.
"After dpkg -i, shed --server <name> version still reports the old version"¶
The .deb postinst does not restart shed-server. Run
sudo systemctl restart shed-server explicitly. The version in
shed --version (the CLI) is independent from the server's
reported version; check the server's via
shed --server <name> version.
Rollback¶
The v0.5.8 changes are independent and reversible:
# Linux
sudo apt install --reinstall shed-server=0.5.7
sudo systemctl restart shed-server
# macOS
brew install shed@0.5.7 # or pin via your local Brewfile
brew services restart shed
Rolling back re-introduces the v0.5.7 prune footgun (avoid shed
image prune after a fresh pull) and the v0.5.7 local-build
panic (run docker buildx prune --all --force before rebuilding
images locally). No on-disk state changes are required.