Remote Backend Development¶

This page sets up a development shed-server that runs default_backend: remote on your machine, beside any installed shed-server. Each shed it creates is a virtual machine in your own Google Cloud project. For what the backend does and how clients see it, see Remote Backend (Cloud).

Ports and paths¶

The remote dev server never touches an installed server's ports (8080 and 2222) or its state.

One-time cloud setup¶

Use a Google Cloud project that runs nothing but sheds. scripts/remote/gce-setup.sh prints the gcloud commands for one environment and runs none of them:

scripts/remote/gce-setup.sh --print \
  --project <project-id> \
  --env <env-name> \
  --region <region> \
  --source-range <your-public-address>/32 \
  --key-out ~/.config/shed/remote-dev/gce-key.b64

Use a documentation address such as 203.0.113.7/32 when you only want to read the output.

Read each printed block, then run it yourself. It enables the Compute Engine API and creates a custom-mode network and subnet, one firewall rule (tcp:22 from your address to the environment's tag), the custom role, a service account holding only that role, and its key. The printed text ends with the environment variables to set.

The server never creates or deletes the firewall rule, but it reads it. Unless the env file says otherwise, the dev config is rendered with firewall.mode: static and source_ranges_env: SHED_REMOTE_SOURCE_RANGES: the rule's source ranges must equal that variable, and while they do not, shed create and shed start answer 503 FIREWALL_CHECK_FAILED. When your public address changes, update both:

gcloud compute firewall-rules update shed-<env-name>-ssh --project <project-id> \
  --source-ranges=<new-address>/32
# then set SHED_REMOTE_SOURCE_RANGES=<new-address>/32 in the env file and
make dev-server-restart-remote

If the address changes often, let the server follow it: run the block gce-setup.sh --print --firewall-auto adds (pass the --role-id your project's base role was made with, and check the printed service account is the one your key belongs to), then set two variables in the env file and restart:

SHED_REMOTE_FIREWALL_MODE=auto
SHED_REMOTE_ECHO_URLS=https://<first-echo-service>/,https://<second-echo-service>/

The config is then rendered with mode: auto and those echo_urls, and SHED_REMOTE_SOURCE_RANGES is no longer read. The start-up log line firewall check passed (mode auto, ...) shows the address the services agreed on. See The firewall check for every clause, the two extra roles and what each failure means.

Create the SSH key the server uses to log in to every VM:

mkdir -p ~/.config/shed/remote-dev && chmod 700 ~/.config/shed/remote-dev
ssh-keygen -t ed25519 -N '' -C shed-remote-link -f ~/.config/shed/remote-dev/link_ed25519

The private env file¶

The cloud settings live in a private env file, never in the repository or the config file. The Make targets source ~/.config/shed/remote-dev/env (override with SHED_REMOTE_ENV_FILE). It holds VAR=value lines:

SHED_REMOTE_ENV=<env-name>
SHED_GCE_PROJECT=<project-id>
SHED_GCE_ZONE=<zone>
SHED_GCE_NETWORK=shed-<env-name>
SHED_GCE_SUBNET=shed-<env-name>
SHED_GCE_TAG=shed-<env-name>
SHED_GCE_CREDENTIALS_FILE=<path to the key file>
SHED_REMOTE_SSH_KEY_FILE=<path to link_ed25519>
SHED_REMOTE_SOURCE_RANGES=<your-public-address>/32

Keep the file mode 0600. The server refuses to start, naming the variable, when one it needs is unset.

Baking an image¶

A remote shed boots from an image baked by scripts/build-gce-image.sh. The bake boots a builder VM from stock Ubuntu 24.04, provisions and verifies it, saves its disk as an image in the family shed-extensions-amd64, and deletes the builder.

set -a; . ~/.config/shed/remote-dev/env; set +a
make gce-image GCE_IMAGE_ARGS=--print   # show every step, run nothing
make gce-image                          # bake (the builder VM is billed while it runs)
make test-gce-image                     # offline checks of the image tooling

The dev config boots the newest image of the family (default_image: family/shed-extensions-amd64, alias cloud). Rebake after any change to cmd/shed-agent or gce/: the server does not copy an agent into a VM (agent_install is not built), so a VM whose agent is incompatible fails its create with reason layout.

Running the dev server¶

make dev-server-up-remote        # build, render the config, start
make dev-server-status-remote    # running? reachable through the CLI?
make dev-server-logs-remote      # follow the log
make build && make dev-server-restart-remote   # pick up a rebuild
make dev-server-down-remote      # stop the server; its sheds keep running

dev-server-up-remote refuses to start when the env file is missing or a dev server is already running. Like the other dev servers it runs under nohup with a PID file: no unit, no restart on crash, nothing that survives a reboot. Stopping the server does not stop or delete any VM; at its next start it lists the cloud and adopts what it finds (see Server restarts).

SHED_REMOTE_GITHUB_USER (a Make variable) names the GitHub account whose SSH keys may enrol a CLI. The rendered config also defines a nocturn service client; issue its bundle with:

bin/shed-server client issue nocturn --config ~/.shed/dev-remote/server.yaml --ttl 72h --out <file>

Registering the CLI entry¶

Once, after the first start:

shed server add localhost --port 18090 --ssh-port 12290 --name remote-dev

First contact is over SSH, so name the SSH port. The CLI enrols with the server's client CA and records an entry with the HTTPS API URL (port 18490), the pinned certificate fingerprint and a client certificate under ~/.shed/creds/remote-dev/, which it renews when it nears expiry. Then:

shed -s remote-dev create t1
shed -s remote-dev exec t1 -- uname -a
ssh t1@localhost -p 12290
shed -s remote-dev stop t1
shed -s remote-dev start t1
shed -s remote-dev delete t1 --force

Checking nothing is left¶

Before you stop for the day:

shed -s remote-dev list
gcloud compute instances list --project <project-id> --filter='labels.shed_env=<env-name>'
gcloud compute disks list --project <project-id> --filter='labels.shed_env=<env-name>'

Both gcloud lists must be empty once every shed is deleted. shed delete returns only when the VM is gone (about 30 s); a JSON DELETE through the API answers in about 3 s and the VM goes in the background, so it can show for a short while. A destroy that fails is retried by the server's periodic pass, which runs once a minute. A VM that should not exist can be deleted with gcloud compute instances delete; the server forgets its record at the next pass.

Every VM also carries a maximum run duration (the larger of remote.max_run_duration, 24 h in the dev config, and the owning client's max_age), after which the cloud stops it and the shed is listed stopped (run_limit). A stopped VM still bills for its disk. To see it happen, render the config with max_run_duration: 10m, the shortest the server accepts.

Integration tests¶

The integration suite's [remote] cases run only when SHED_REMOTE_SERVER names this entry:

make test-integration-remote                                          # every [remote] case
make test-integration-remote PYTEST_ARGS="test_remote_refusals.py"    # no VMs

The target is manual only and never starts the server for you. It sets every vz and fc server variable empty, so no other backend is touched. A full run creates about 25 VMs one after another, never more than remote.limits.max_running (2) at once, and takes about 42 minutes. Tests the backend cannot run (mounts, image store, guest MTU, vz/fc timing and image bakes) skip with a reason. See tests/integration/README.md for the list.

Measured timings¶

From three live runs on the dev project (e2-standard-2, 20 GB image, one zone). The cloud's own share varies from run to run: the provider phase of a create took between 12 and 35 s.