Runbook: Roll a New cm-runner Image onto the Locally Built BYOVM Agents

Owner: Platform Team Last validated: 2026-11-04 (PR #4063 client, commit a0a76e4d1) Required access: SSH to $DEPLOY_USER@$PLATFORM_VPS_IP (platform host, read gateway logs) and $DEPLOY_USER@$RUNNERS_VPS_IP (runners host).

CD recreates the two agents on the platform host (curateme-runner-agent, curateme-runner-agent-blog). The two agents on the runners host are built by hand, so they keep running an old client until someone rolls them. Do this after any change to packages/cm-runner.

Never print config files, the env, or docker inspect without a --format. The agent token (cmagent_...) must not change: a changed token means a re-registration and a new agent id.

1. Find the stale agents

The gateway logs agent_ws_token_in_query_deprecated (with agent_id, never the token) once per connection from an agent that still puts the token in the URL.

ssh $DEPLOY_USER@$PLATFORM_VPS_IP 'docker logs --since 60m curateme-backend-gateway 2>&1 \
  | grep agent_ws_token_in_query_deprecated | grep -o "byovm_[a-z0-9]*" | sort | uniq -c'

Map an id to a container with docker exec <c> sh -c 'grep -o "agent_id[^,]*" <config path>' (id only).

2. Build the image on the runners host

The checkout in ~/platform there is old, dirty, and its git remote credential is stale. Do not pull or reset it. Ship a clean archive of the commit from your laptop instead.

# laptop
git archive <sha> packages/cm-runner deploy/runner-images/cm-runner > rollout.tar
scp rollout.tar $DEPLOY_USER@$RUNNERS_VPS_IP:/tmp/
 
# runners host: keep the old images under a rollback tag, then build
mkdir ~/agent-rollout && tar -x -C ~/agent-rollout -f /tmp/rollout.tar && cd ~/agent-rollout
docker tag platform-runner-agent:latest platform-runner-agent:pre-4063-c934bf4b
docker tag cm-runner:gwauth3 cm-runner:pre-4063-gwauth3
docker build -q -t platform-runner-agent:<sha> --label ai.curate-me.git-sha=<sha> \
  -f deploy/runner-images/cm-runner/Dockerfile .
docker tag platform-runner-agent:<sha> cm-runner:<sha>

3. Cipher (compose, volume holds the credentials)

Cipher’s volume held only the old agent-credentials.json. The new client reads config.json and would otherwise re-register. Write config.json from the old file, with the same id and token, using the new image as a helper (nothing is printed):

docker run --rm --entrypoint python -v platform_runner_agent_credentials:/etc/curateme \
  platform-runner-agent:<sha> -c '<read agent-credentials.json, build cm_runner.config.AgentConfig
  (agent_id, auth_token=agent_auth_token, gateway_url=control_plane_url, org_id,
  workspace_base_path=/data/workspaces, org_drive_base_path=/data/org-drives),
  write /etc/curateme/config.json mode 0600>'

Compose names the image platform-runner-agent:latest, so retag first, then recreate without building (the file on this host still points at the removed services/runner-agent):

docker tag platform-runner-agent:<sha> platform-runner-agent:latest
cd ~/platform && docker compose -f docker-compose.runners.yml --env-file .env.production \
  up -d --no-build --no-deps --force-recreate runner-agent

Check docker inspect curateme-runner-agent --format '{{.Image}}' is the new image. If it is not, you recreated on the old latest; retag and run it again.

4. Drift (plain docker run, config lives in the container layer)

Drift has no volume for /root/.cm-runner. A plain recreate loses the registration. Copy config.json out, move it to the /etc/curateme volume, and keep the old container stopped under another name for rollback:

umask 077; D=$(mktemp -d)
docker inspect cm-runner --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -E '^CM_' > $D/env
docker cp cm-runner:/root/.cm-runner/config.json $D/config.json
docker stop cm-runner && docker rename cm-runner cm-runner-pre4063
docker create --name cm-runner --restart unless-stopped --env-file $D/env \
  -v cm-runner-orgdrive:/data/org-drives:z -v cm-runner-workspaces:/data/workspaces:z \
  -v cm-runner-data:/etc/curateme -v /var/run/docker.sock:/var/run/docker.sock \
  --label com.curateme.component=cm-runner cm-runner:<sha>
docker cp $D/config.json cm-runner:/etc/curateme/config.json
docker start cm-runner && rm -rf $D

5. Verify (within 2 minutes, per agent)

ssh $DEPLOY_USER@$PLATFORM_VPS_IP 'docker logs --since 3m curateme-backend-gateway 2>&1 \
  | grep -E "agent_ws_token_in_query_deprecated|agent_ws_connected|Agent WebSocket registered" | grep <agent_id>'

Pass: agent_ws_connected and Agent WebSocket registered for the agent id, and no agent_ws_token_in_query_deprecated for it after the restart. The agent’s own log should end with WS bridge connected.

Rollback

• Cipher: docker tag platform-runner-agent:pre-4063-c934bf4b platform-runner-agent:latest and rerun the compose up command. The old agent-credentials.json is still in the volume.
• Drift: docker rm -f cm-runner && docker rename cm-runner-pre4063 cm-runner && docker start cm-runner.

Delete cm-runner-pre4063 and the pre-4063-* tags once both agents have been healthy for a few days.