AuthorPascal DateJuly 14, 2026 Rev1.0

Integrating a robot with the brain — the full contract

The brain runs centrally on a shared host (e.g. PhyAI4090) and is an LLM orchestrator with no robot knowledge of its own. You run your robot — real or simulated — on your own machine, expose it to the brain as a couple of MCP servers over Tailscale, and drive it from a web page. You get your own named instance, isolated from everyone else.

This is the single doc for connecting a robot: from the minimum needed to run at all, to everything that unlocks every brain feature. You write robot code + wrap it as MCP; the brain does the planning, watching, recovering, and learning.

Canonical reference implementations to copy from: - motion/capability → rap-integration/rap_capability_server.py - perception/locate → robot-mcp-perception (locate_server.py) - processes/skills memory → robot-mcp-memory - async perception events → mcp-perception-buffer

---

Quick connect to the shared brain

   your machine                          shared host
 ┌──────────────────────┐   Tailscale   ┌──────────────────┐
 │ your robot / sim      │◀─────────────▶│ brain-<you>      │──▶ http://<host>:<port>
 │ + MCP servers (SSE)   │               │ (Claude planner) │
 │   motion   :9206      │               └──────────────────┘
 │   perception :9207    │
 └──────────────────────┘

The brain is just an orchestrator (no ROS, no GPU). It plans and then calls tools on your robot; your machine does the actual sensing and acting.

Two MCP servers reachable on the tailnet, bound to 0.0.0.0 (so the remote brain can connect), implementing the tools the planner drives:

Server	Default port	Provides
capability / motion	9206	navigation + manipulation: navigate_to, navigate_to_named, pick, place, go_to_stance, …
perception / locate	9207	3D grounding: locate_3d, grasp_pose, place_pose, …

Ports are configurable per instance (CAP_PORT / LOCATE_PORT). The exact tool names the brain expects are the authoritative list in the running instance's generated config/tools/mcp.yaml (blocking_tools). Any robot that speaks this tool contract over MCP/SSE works — the brain doesn't care whether it's Gazebo, Isaac, or hardware.

Reference implementation (ZHAW RAP / R1Pro): the rap-integration repo ships these two servers — start them with rap.sh capability and rap.sh locate (they already bind 0.0.0.0). If you're bringing a different robot, run your own MCP servers that expose the same tools.

Steps:

1. Join the tailnet (once):

tailscale up
tailscale status | head -1   # note YOUR machine's tailnet name

2. Start your robot + its two MCP servers, then sanity-check they listen:

ss -ltn | grep -E ':(9206|9207)'   # both should show 0.0.0.0:...

3. Ask the admin for an instance — give them your tailnet hostname:

# admin runs, on the shared host:
brainctl up <yourname> --robot <your-tailnet-host>

They reply with your URL, e.g. http://<host>:8002.

4. Use it — open the URL, type a command, open the Debug panel to watch the brain reason and call your robot's tools. Improvements the admin makes to the brain reach everyone after a brainctl update. Everything stays private on the tailnet.

Notes:

• Your machine needs enough CPU/GPU for your robot/sim — that's the heavy part. The brain is light and lives on the shared host.
• Nothing is exposed publicly; lock the tailnet down with Tailscale ACLs.

The rest of this doc is the full contract: every tool + return field to implement, how to publish over the tailnet, and what each optional capability requires to unlock all brain functionality — with a minimum-vs-full checklist.

---

0. The shape of it

The brain is an LLM orchestrator with no robot knowledge of its own. It calls tools you expose over MCP; your code senses and acts. Tools must be:

• coarse, blocking, self-verifying — one call = one whole primitive (pick, navigate_to_named) that returns only when finished and reports success/failure;
• locally safe — keep your own e-stop/reflexes; a dropped connection mid-call must fail safe on the robot (the brain auto-reconnects);
• clearly described — the planner picks tools from their names + docstrings;
• structured — return JSON ({"ok": true, …}); the plan machine branches on it;
• bare-named — expose them with no server prefix (the brain calls pick, not motion.pick); dotted names intermittently break tool routing.

response_text is auto-injected into every tool's schema by the brain (spoken text alongside a call) — ignore it in your handler.

---

0b. Where your code goes — the layer model

Everything you build sits in one of a few layers, each owning a different kind of knowledge. Putting a capability in the right layer is what keeps one brain image serving every robot and customer — and lets you make the robot better at a task without ever changing the brain.

Layer	Owns	Per-what	You build it as
Core brain	conversation, planning, the generic plan runtime (data-flow, expect gates, loops, survey/retry), oversight, memory	universal (one image)	— (you don't touch it)
Configuration	prompt fragments, plan templates, named places, model choice, which servers to attach	customer / robot / task	prompt + config (data, not code)
Skill / task services	a closed-loop, self-verifying competence behind ONE high-level tool	object / task	an MCP server
Capability primitives	atomic motion/IO — move, navigate, pick, place, get_pose	robot / gripper	an MCP server (§2)
Perception / world model	ground, segment, localize, grasp geometry, scene capture	sensor / model (swappable)	an MCP server (§2)

The load-bearing idea: competence lives in the lowest layer that can own it end-to-end and verify itself, and a refinement loop belongs where the feedback signal is — never in a prompt or in the brain.

• Primitives (§2) are atomic, blocking, self-verifying, and dumb: pick attempts one grasp and reports {ok, held}. It does not re-perceive or retry.
• A skill/task service wraps a hard, object-specific competence as a single high-level MCP tool — e.g. grasp("coke can") — and owns the closed loop internally: perceive → grasp → check the grip → refine (nudge the base, adjust standoff, try the other arm) → repeat until it verifies success or fails honestly. It calls your pick primitive + grasp-pose perception beneath it, and may bundle its own AI models (a task-specific segmenter, a grasp net). To the brain it's just another tool, so the compiled plan shrinks to one step.

So a new object, gripper, robot, or customer is a new MCP server or a config change — never a brain fork. Add generic capability to the brain (everyone benefits); add specific competence here (isolated). Everything from §1 on is how you build these MCP layers.

---

1. Publish + connect (every robot)

1. Tailscale on the robot's machine: tailscale up.
2. Run your MCP servers as SSE, bound to 0.0.0.0 so the remote brain reaches them over the tailnet — conventionally two: motion on :9206, perception on :9207 (ports are configurable per instance). With FastMCP:

   mcp.settings.host = "0.0.0.0"; mcp.settings.port = 9206
   # Binding non-localhost trips the MCP SDK's DNS-rebinding guard (HTTP 421
   # "Invalid Host header"). Relax it for a tailnet-only server:
   from mcp.server.transport_security import TransportSecuritySettings
   mcp.settings.transport_security = TransportSecuritySettings(
       enable_dns_rebinding_protection=False
   )
   mcp.run(transport="sse")
   

3. Ask the admin for an instance (send your tailnet hostname): brainctl up <you> --robot <your-host> → your brain at http://<host>:<port>.

That's enough for the brain to connect. What you implement next decides how much of the brain you can actually use.

A server needn't run on the robot. The brain connects to each MCP server by URL, so a capability too heavy for the robot's onboard compute — e.g. a VLA wrapped as an MCP server, or a heavy segmenter — can run in the cloud (or any host) and be attached alongside the robot-local ones. The admin adds it with brainctl up … --server <url> (auth via a header in brain.env); to your robot code nothing changes. Bind cloud servers with TLS + a bearer token; the brain sends the configured Authorization header and connects to each server independently.

But a camera-consuming service off-robot needs the frames. Attaching its tools is easy (MCP is just a URL); getting it the robot's camera stream is the real work, because the camera lives in ROS on the robot and the brain is not in that path. Two patterns (full detail in architecture.md §3a): (A) keep a thin MCP server on the robot holding the camera subscription and have it call a stateless cloud model per request (this is how robot-mcp-perception uses cloud Gemini-ER today) — best for request/response; or (B) run the whole server off-robot and stream the camera to it (WebRTC / RTSP / gRPC over Tailscale — robot-mcp-perception's frame_source: rtsp does exactly this) — best for a continuous, high-rate consumer such as a closed-loop VLA. A tight VLA loop over the public internet is latency-bound; prefer a GPU box on the robot's own LAN. Either way the control plane is unchanged: tools + prompt://system over MCP.

---

2. Core tool contract (to drive at all)

Implement these (names as the planner expects them). Args/returns below are the essentials — the reference servers show the exact shapes.

Navigation | Tool | Args | Returns | |---|---|---| | navigate_to_named | name | {ok, name, x, y, yaw_deg}; unknown name → {ok:false, known:[…]} — drive to a saved place | | navigate_to | x, y, yaw_deg | {ok, x, y, yaw_deg} — drive to a raw map-frame goal | | list_waypoints | – | {ok, waypoints:{name:{name,x,y,yaw_deg}}} — the known places, injected into planning | | get_pose / get_state | – | current base/arm state |

Don't confuse navigate_to_named (drive the base to a saved place) with a goto_named(group,state)-style tool (move an arm/gripper to a named posture, e.g. open the gripper). Both say "named" but they're different subsystems.

Perception (usually the :9207 server) | Tool | Args | Returns | |---|---|---| | locate_3d | query (e.g. "the blue box") | {found, label, x, y, z, frame} | | grasp_pose / model_object | target | {reachable, grasp, …} | | place_pose / locate_surface | surface | {ok, pose} |

Manipulation | Tool | Args | Returns | |---|---|---| | pick | target/grasp, side | {ok, held, side} — held+side are tracked as "carrying" across commands | | place | surface, side | {ok} — clears "carrying" | | go_to_stance, reach, fine_approach, turn_by, clear_octomap, allow_gripper_touch | – | {ok, …} |

Return-field conventions the brain relies on — keep these consistent, because the compiled plans reference them: a pick result saved as pk is checked with expect: pk.held; a grasp saved as gp with expect: gp.reachable; a locate_3d saved as box with expect: box.found. So every tool returns ok plus the task-specific booleans (held, found, reachable, …) and values (side, coordinates) that later steps consume. Match the reference servers' field names and the stock plan templates work unchanged.

---

3. Fully utilizing the brain — optional capabilities

Each is independent; add the ones you want.

a) Named navigation + teaching places ("move to the kitchen table")

This is how the brain reaches somewhere not currently in view — the core "navigate to the named place first, then perceive + act" rule depends on it. Implement four tools that persist base poses (map frame) to disk:

Tool	Does
save_waypoint(name)	record the robot's current base pose under name (teach a place by standing there) → {ok, name, x, y, yaw_deg}
save_waypoint_from_goal(name)	save the last map goal you placed (e.g. an RViz "2D Nav Goal" arrow) — map the space without driving → {ok, name, x, y, yaw_deg, age_s}
list_waypoints()	the places the robot knows → {ok, waypoints:{…}}
navigate_to_named(name)	drive the base (your nav stack) to a saved place → {ok, name, …}; unknown → {ok:false, known:[…]}

Flow: seed places once (save_waypoint*), and the brain injects their names into planning, so "take the coke to the other table" becomes navigate_to_named("other table") → perceive → pick → navigate_to_named(dest) → place — instead of guessing coordinates or assuming a place is already visible. Operators can teach new places by name mid-session. Persist waypoints across restarts (a JSON file is enough); keep names lowercase-insensitive for matching.

b) The surveillance watchdog (live oversight + auto-replan)

Expose a VLM observer tool observe_task (configurable via agent.surveillance.vlm_tool) and, optionally, a robot-property monitor robot_state (agent.surveillance.state_tool):

observe_task(context: str) -> {ok: bool, issue: str, reasoning: str}
robot_state()             -> {ok: bool, issue: str, reasoning: str}

While a plan runs, the brain polls these every interval_s and streams findings to the Debug panel. vlm_strikes consecutive ok:false VLM frames (default 3), or any ok:false from robot_state, aborts the plan and makes the brain replan. Not connected → that source is simply inactive.

c) Ship your robot's prompt (this is how robot guidance reaches the brain)

The brain image is robot-agnostic — it bakes no robot prompt. Your robot's identity, skill/tool descriptions, plan templates, and manners reach the planner by being advertised from an MCP resource prompt://system:

@mcp.resource("prompt://system")
def system_prompt() -> str:
    return "Robot: <arms/grippers, reach, constraints>. <How to use your tools>. <Plan templates>."

The brain fetches it on connect and composes it after its universal base (you refine the shared rules, you don't replace them). So the prompt travels with the tools: change it → restart your server → only your robot picks it up, no brain rebuild. Put as much or as little as you need here (a one-liner, or your full plan templates). Reference: the RAP capability server serves its whole planner prompt this way (rap-integration/brain/planner_prompt.txt).

d) Learned processes / skills (repeatable how-tos)

Run a process-store MCP server named memory (agent.memory_server; reference: robot-mcp-memory) exposing find_processes(query) / save_process(…). The brain retrieves matching processes into the planner at mission start and can save a taught procedure. Leave it out (agent.memory_server="") and the brain still works, just without process memory.

e) Recovery lessons (learning from failures)

Nothing to implement — the brain distills lessons from failed→fixed→succeeded runs (and from your mid-mission corrections) and injects them into future planning via the shared store. You only need consistent tool names/returns so a lesson learned once applies again.

f) Async perception events ("what do you see", advanced)

For the robot to proactively tell the brain something mid-mission (not just when polled), POST watch-events to the brain's ingress POST http://<brain-host>:<port>/api/agent/events with the shared PERCEPTION_EVENT_SECRET. Reference: mcp-perception-buffer (subscribes your perception topics and pushes events). Optional; the watchdog in (b) covers most oversight needs.

g) Clarification (ask-back) — free

When a request is ambiguous the brain asks the operator with options; nothing to implement — it uses the tools + prompt you already provide to decide when to ask.

h) Visual Command GUI (direct manipulation)

Powers the operator's segmented-camera view (click an object to pick, tap a spot to place, draw an arrow to arrange, jog the base). Expose:

Tool	Does
get_frame()	the live head-camera frame, image only, cheap enough to poll → {ok, image_jpeg_b64, width, height, stamp}
capture_scene()	detect + segment the scene → {ok, image_jpeg_b64, width, height, segmenter, objects:[{id, label, score, bbox, polygon:[[u,v]…], centroid_uv, point_3d:{x,y,z,frame}}]}
pixel_to_3d(u, v)	deproject a clicked pixel to a 3D point → {ok, x, y, z, frame}
grasp_from_pixel(u, v, side?)	grasp geometry for the object under a clicked pixel (same shape as grasp_pose)

The brain streams get_frame continuously for a live view and only calls capture_scene on a key step or an explicit Re-segment — so segmentation can be as slow/expensive as it needs to be. segmenter reports "sam3" (per-pixel masks) or "box" (bounding-box fallback) so the UI knows the overlay fidelity. Reference: robot-mcp-perception/locate_server.py.

Manual jog / teleop — the robot provides the base-motion primitives; the brain only relays button presses and enables only the buttons for the tools you expose:

Tool	Button
turn_by(degrees)	rotate in place (⟲ left = +, ⟳ right = −)
drive_by(distance_m)	drive straight (↑ forward = +, ↓ backward = −)
strafe_by(distance_m) (optional; holonomic bases only)	strafe (← left = +, → right = −)

Keep each jog bounded (small step, speed cap, timeout) with a local watchdog — the brain calls these directly (not through the planner) for immediate manual control, and blocks them while a mission is running.

You get the observer/livestream view for free. Once get_frame + capture_scene work, the same Visual view has an observer mode: top-bar toggles hide the operator controls and overlay the robot's live reasoning + plan in translucent panels over the camera stream (the marketing/broadcast shot). That overlay is entirely brain-side — it reads the mission/reasoning streams — so there is nothing extra to implement for it.

---

4. Checklist

Minimum (drive it): - [ ] Tailscale up; MCP SSE servers on 0.0.0.0 (DNS-rebinding guard relaxed) - [ ] navigation (navigate_to_named, list_waypoints) + perception (locate_3d) + manipulation (pick, place) with structured {ok, held, side, found, …} returns - [ ] asked the admin for an instance

Full utilization (add as needed): - [ ] save_waypoint* — teach named places - [ ] observe_task (+ robot_state) — the live watchdog + auto-replan - [ ] prompt://system resource — your robot's prompt - [ ] a memory process-store server — learned procedures - [ ] perception-event POST — proactive mid-mission events - [ ] Visual Command: get_frame + capture_scene (+ pixel_to_3d, grasp_from_pixel) - [ ] manual jog: turn_by + drive_by (+ strafe_by for holonomic bases)

The brain's controller/watcher, ask-back, recovery-learning, and multi-instance config are all brain-side — you get them for free once connected.

---

Admin quick reference (on the shared host)

cd ~/sage/orchestrator/brain-host
./brainctl up <name> --robot <host>   # start an instance
./brainctl ls                          # names, ports, robots
./brainctl logs <name>                 # follow logs
./brainctl down <name>                 # stop + remove
./brainctl update                      # git pull source + rebuild + recreate all

For using and operating the brain (mental model, driving it, the Visual Command / observer view, run/update/configure), see using-the-brain.md.