Runbook — verify the layer-3 rollout (end to end)¶
Verifies the work from Phases 0–6 (see architecture.md §7)
live on the brain-pascal instance (4090) driving the rap-1 R1Pro sim: the
grasp-service (replacing the prompt loop), the declarative robots.yaml attach, the
admin Skills panel, and — optionally — the cloud SAM3 perception server. Do the
stages in order; each says what to run, what to expect, and what to do if it fails.
Warning
Deploy touches the live 4090 brain (briefly recreated) and the rap-1 sim (a real grasp mission moves the robot). It does not rebuild the sim. Rollback is at the end.
Prerequisites¶
- rap-1 sim up:
ssh ros@rap-1 'ss -ltn | grep -E ":(9206|9207)"'→ both listening. ssh ros@rap-1andssh 4090work from the Mac.- 4090
brain.envhasANTHROPIC_API_KEY(+GROQ_API_KEY); rap-1~/binabik/robot-mcp-perception/.envhasGEMINI_API_KEY. - Repos checked out on the Mac under
/Users/sage/Documents/code.
Stage 0 — Deploy the latest code¶
0a. (Mac) Build the frontend + stage the brain UI (the 4090 has no npm):
cd /Users/sage/Documents/code/binabik-orchestrator/robot-voice-chat/frontend
npm run build
rsync -a --delete dist/ /Users/sage/Documents/code/binabik-brain-host/brain-ui/
cd /Users/sage/Documents/code/binabik-brain-host
git add brain-ui && git commit -m "chore: rebuild brain-ui (Skills panel)" && git push
0b. (rap-1) Ensure grasp-service is up (it's a detached process — restart if the VM rebooted):
ssh ros@rap-1 'ss -ltn | grep -q :9210 && echo "grasp-service up" || \
(cd ~/binabik/grasp-service && setsid env MCP_TRANSPORT=sse GRASP_PORT=9210 \
.venv/bin/grasp-service >grasp.log 2>&1 </dev/null & sleep 5; ss -ltn | grep :9210)'
:9210 listening.
0c. (4090) Pull + rebuild + re-attach from the manifest. git pull brain-host
first (its new brainctl + gen_mcp_config.py + robots/pascal.yaml), then build
+ up (auto-loads robots/pascal.yaml, so grasp attaches via the manifest, not a flag):
ssh 4090 'set -e
cd ~/sage/orchestrator/brain-host && git pull --ff-only
git -C ~/sage/orchestrator/robot-voice-chat pull --ff-only
./brainctl build
./brainctl up pascal --port 8001
./brainctl ls'
built binabik-brain:latest; up: brain-pascal … (manifest: …/robots/pascal.yaml);
ls shows brain-pascal 8001 rap-1 Up.
If build can't find robot-voice-chat/backend: check BRAIN_SRC / that ~/sage/orchestrator/robot-voice-chat exists.
Stage 1 — Phase 2: manifest → mcp.yaml¶
Expect: three sse servers — rap-1:9207, rap-1:9206, and rap-1:9210 with
name: grasp; blocking_tools includes grasp. This is gen_mcp_config.py
expanding robots/pascal.yaml.
If grasp is missing: the manifest wasn't picked up — confirm the brain.manifest
label (docker inspect brain-pascal --format '{{index .Config.Labels "brain.manifest"}}')
points at robots/pascal.yaml.
Stage 2 — Phase 3: admin Skills panel¶
Open https://phyai4090.<tailnet>.ts.net/admin/skills → admin login → Skills.
Expect: one card per MCP server (rap-1 capability, rap-1 perception, grasp),
each showing a connected badge, its tools (with a blocking chip on grasp,
pick, …), and grasp's advertised prompt://system fragment ("To pick an object up,
call grasp(object)…").
Headless sanity (no token needed):
ssh 4090 'curl -s localhost:8001/api/config/system | python3 -c "import sys,json;print(\"grasp\" in [t[\"name\"] for t in json.load(sys.stdin)[\"data\"][\"tools\"]])"'
True.
Stage 3 — Phase 1 flagship: the grasp mission (kills the prompt loop)¶
Tail the grasp-service, then drive a mission from the chat UI:
In the brain UI chat (or Visual view), send: “pick up the coke can” (or “put the coke can on the left table” for the full pick-and-place).Expect:
- The plan (Debug/observer panel) shows a single
graspstep — not agrasp_pose+pickloop. grasp.logshows the loop running:goto_named → where_to_stand → go_to_stance → grasp_pose → pick, thenheld.- The robot lifts the can; the mission finishes succeeded. With a destination, it
then
navigate_to_named/locate_surface → go_to_stance(carrying) → place. - On a first miss, the deterministic policy re-approaches / switches arm and retries (a second attempt in the log) — the refinement the prompt loop couldn't do.
If it fails: read grasp.log's reason (not_found / unreachable /
grasp_missed); the brain will replan or finish "incomplete". This is the empirical
A/B vs. the old loop — capture the result. To go back to the loop, see Rollback.
Stage 4 (optional) — Phase 5: fleet-scoped memory¶
Mostly covered by unit tests (robot-mcp-memory, 48 passing incl. a scoping test).
Live smoke, if the memory server is deployed on rap-1: save a shared lesson + a
rap-1 lesson, then confirm a rap-1-scoped find_processes sees both and another
robot's scope does not. (Auto-binding a connection's token→robot is the documented
follow-up.)
Stage 5 (optional) — 6b: SAM3 cloud segmenter as MCP¶
Not needed for the R1 (perception runs on-robot); this proves off-robot cloud perception as MCP.
# 1. Deploy the Modal endpoint (from binabik-modal-bottle):
cd /Users/sage/Documents/code/binabik-modal-bottle && uv run modal deploy -m sam3_modal.app
# 2. Run the MCP wrapper (anywhere on the tailnet), pointed at the endpoint URL:
SAM3_URL=https://<workspace>--sam3-segmenter-segmenter-web.modal.run \
MCP_TRANSPORT=sse SAM3_MCP_PORT=9212 uv run --extra mcp sam3-mcp
# 3. Add a manifest entry (robots/pascal.yaml → servers: - name: sam3, url: …:9212/sse)
# and `brainctl up pascal --port 8001`; confirm `segment` appears in the Skills panel.
segment(image_b64, prompt) tool attached; segmenting a frame returns
detections from the cloud.
Stage 6 (deferred) — Phase 4: pinned vendor stack¶
Only verifiable on a fresh VM (it clones + pins galaxea + the ROS plugins). Verify at the next rap-1 re-provision:
Expect: logsgalaxea pinned @ 309130d…, …pinned @ <ver> for STVL + TRAC-IK, then
r1pro already built / a clean colcon build. (galaxea is private → the VM needs a
GitHub SSH key.)
Rollback (restore the pre-rollout prompt loop)¶
The RAP prompt swap is on the grasp-service-integration branch; main still has the
loop. To revert the robot to the loop and detach grasp:
# 1. (Mac) redeploy main's (loop) planner prompt to rap-1 — the capability server
# reads it per-request, so the brain picks it up on reconnect:
git -C /Users/sage/Documents/code/rap-integration show main:brain/planner_prompt.txt \
| ssh ros@rap-1 'cat > ~/binabik/rap/brain/planner_prompt.txt'
# 2. (4090) re-attach WITHOUT the manifest (temporarily move it aside), robot-only:
ssh 4090 'cd ~/sage/orchestrator/brain-host && mv robots/pascal.yaml /tmp/ && \
./brainctl up pascal --port 8001 --robot rap-1'
# 3. (optional) stop grasp-service on rap-1:
ssh ros@rap-1 'pkill -f "grasp_service.server" || pkill -f grasp-service'
grasp.