AuthorPascal DateJuly 14, 2026 Rev1.0

Using and operating the brain¶

The brain is an LLM orchestrator: you hand it your robot as a set of MCP tools, it turns a plain-language command into a plan, calls your tools, watches the run, asks when unsure, recovers from failures, and learns. You never touch the LLM — you just expose good tools and drive it from a web page.

This doc covers using the brain (the mental model, the layer model, driving it, the Visual Command / observer view, and how it learns) and operating it (running, updating, and configuring instances on the shared host, how prompts are composed, and what's inside). To connect a robot — every tool + return field, how to publish, and the optional capabilities — see integrating-a-robot.md.

Using the brain¶

You know how to write robot code and wrap it as MCP. That's all you need.

Mental model¶

"bring the blue box to the workbench"
        │
        ▼   (brain: plan → call tools → observe → react)
   your MCP tools:  locate_3d → get_grasp → pick → navigate_to → place
        │
        ▼
   your robot code / sim does the actual sensing + acting

The brain is high-level only — it issues intent and awaits results. It never runs a tight control loop; your controllers do.

Where competence lives (the layer model)¶

The brain is one layer of a stack, and each layer owns a different kind of knowledge. Putting a capability in the right layer is what keeps the system maintainable: one brain image serves every robot and customer, and making the robot better at a task never means changing the brain.

  ── CORE BRAIN ────────────────────────   one shared image; robot/
     conversation · planning · a generic     task/customer-agnostic.
     plan runtime (data-flow · success       Parameterized only by
     gates · loops · survey/retry) ·         CONFIG (prompt fragments ·
     oversight · memory                      plan templates · named
                                             places · model · servers).
            │  discovers + calls, over MCP
            ▼
  ── SKILL / TASK SERVICES ─────────────   per object/task — the
     one high-level tool per competence,     "competence layer".
     e.g. grasp("coke can"):                 The closed loop lives
     perceive → grasp → check → refine       INSIDE one MCP tool.
     → retry, until verified or honest fail
            │  built on
            ▼
  ── CAPABILITY PRIMITIVES ──┬── PERCEPTION / WORLD MODEL ──
     move · navigate · pick  │   ground · segment · localize ·
     · place · get_pose ·    │   grasp geometry · scene capture
     get_frame — atomic,     │   per sensor/model, swappable;
     self-verifying          │   may host its own AI models
     per robot / gripper     │

Rules of thumb:

The brain orchestrates intent; it never pilots a control loop. Anything real-time or robot-specific lives below it.
Competence lives in the lowest layer that can own it end-to-end and verify itself. "Reliably pick up a coke can with this gripper" is neither a brain feature nor a bare pick primitive — it's a skill service grasp("coke can") that perceives, grasps, checks the grip, and refines/retries (nudge the base, try the other arm) until it verifies success or gives up honestly. It calls the pick primitive + grasp-pose perception underneath, and may bundle its own models. The brain just calls the one tool; the plan stays a single step.
A refinement loop belongs in the layer that holds the feedback signal — the skill service or the primitive — never in a prompt.
Everything robot/task/customer-specific is a registered MCP server or config, never a fork of the brain. Add generic capability to the brain (everyone benefits); add specific competence as a service or config (isolated — changing task A on robot B for customer C touches only that service/config).
A skill service is just another MCP server the brain discovers and calls like any tool — so you add the competence layer exactly the way you expose primitives.

Connect your robot¶

Connecting a robot means exposing it as MCP (SSE) servers on the tailnet, making your tools coarse/blocking/self-verifying with clear descriptions and structured returns, and shipping your robot's prompt via a prompt://system resource. The full connect-a-robot contract — every tool + return field, how to publish over the tailnet, the layer model, and each optional capability — lives in integrating-a-robot.md. In short: MCP SSE servers on the tailnet bound 0.0.0.0; coarse/blocking/self-verifying tools; local safety independent of the brain; the expected tool set (or adapted prompt/plan templates); then ask the admin for an instance (brainctl up <you> --robot <your-host>).

The planner LLM (e.g. Claude Opus vs a Groq model) is chosen per robot by the admin at launch — brainctl up … --planner anthropic --planner-model claude-opus-4-8. You provide the tools + prompt; the admin picks the brain's model.

Drive it¶

Open your instance URL (http://<host>:<port>), type a command, and open the Debug panel to watch it: streamed reasoning, the compiled plan, each tool call, and the vision watchdog.

It asks when unsure ("which shelf?") — answer in chat; it then plans.
It's honest — if a request needs a capability you didn't expose as a tool, it says so instead of flailing.

Visual Command + observer view (the camera GUI)¶

The header Visual toggle switches the main area to a live head-camera view with the scene segmented + labelled (needs your get_frame/capture_scene tools — see integrating-a-robot.md §3h). Two things in one:

Direct manipulation — click an object to Pick, tap a spot to Place, pick a named place to Navigate, drag an arrow to Arrange, and jog the base. Each click becomes a grounded mission through the planner.
Observer / livestream mode — a top bar with a ● LIVE badge, a status pill (Reasoning / Executing / Monitoring / …), and two toggles: Controls and Reasoning. Turn Controls off for a clean broadcast shot — the tool buttons and jog pad disappear and the robot's chain-of-thought + the live plan render in translucent glass panels over the camera stream (plus per-arm hold status and a speech caption). This is the marketing/stream view: camera + segmentation + what the robot is thinking, with no operator chrome.

The reasoning/plan overlay is brain-side (it reads the mission + reasoning streams), so it needs no extra robot work beyond the camera tools. Preview it with no robot at http://<host>:<port>/?mockVision=1.

Correct it on the fly — and it learns¶

If the robot gets stuck, tell it what it was missing in chat. If your hint leads to success, the brain distills a generalized lesson (not the one-off state) and reuses it next time — e.g. you say "the object is in the left arm" and it learns "track which arm holds the object and place with that arm," not "object is in the left arm." Genuine failure→fix→success runs can be Remembered and are shared with compatible robots.

Install it as a desktop app (PWA)¶

The UI is a Progressive Web App — it can run as a standalone desktop window (own dock/taskbar icon, no browser chrome). When you open it in a normal browser tab, a small "Install Binabik" toast appears for a few seconds (bottom-right); click Install, or use the browser's install control (the ⊕ in the address bar / "Install app" in the menu). Installed users don't see the toast.

Requires a secure context. Browsers only offer install over HTTPS (or localhost) — over plain http://<host>:<port> the prompt never appears. To enable install on the shared host, serve the brain over HTTPS, e.g. Tailscale tailscale serve for an https://<host>.<tailnet>.ts.net URL, or a TLS reverse proxy. The app still works fine over plain HTTP; it just can't be installed.

Operating the brain¶

Admin/maintainer guide for the host that runs the brain instances (e.g. the 4090).

Run¶

Prereqs: Docker, Tailscale up, brain.env with ANTHROPIC_API_KEY + GROQ_API_KEY, and the brain source as git clones under $BRAIN_SRC (~/sage/orchestrator/src: robot-voice-chat, robot-mcp-kit). The prebuilt UI and default prompts live in this repo (brain-host/{brain-ui,prompts}). No robot integration repo is needed — the image is robot-agnostic; each robot advertises its own identity, skills, and plan templates to the brain at runtime over MCP (prompt://system).

cd ~/sage/orchestrator/brain-host
./brainctl build                       # build the image (first time)
./brainctl up  <name> --robot <host>   # start an instance → http://<this-host>:<port>
./brainctl ls                          # names, ports, robots, status
./brainctl logs <name>                 # follow logs
./brainctl down <name>                 # stop + remove

Each instance is one Docker container (brain-<name>), one robot, its own state.

Update (propagate your changes to everyone)¶

cd ~/sage/orchestrator/brain-host
./brainctl update      # git pull the source clones → rebuild image → recreate ALL instances

That's the "change the brain once, everyone's on it" button. Under the hood it git pulls this repo (brain-host: brainctl + brain-ui + prompts + Dockerfile) and the source clones (robot-voice-chat, robot-mcp-kit), rebuilds binabik-brain:latest, and recreates every running instance on it (same names, ports, robots, and extra servers). No robot repo is pulled.

Configuring an instance (per robot) — incl. a different planner LLM¶

Every instance runs the same image; config differs only by brainctl up flags:

brainctl up alice --robot alice-pc \
  --planner anthropic --planner-model claude-opus-4-8 \
  --profile r1pro \
  --set agent.max_llm_steps=30

Flag	Effect
`--planner <provider>`	planner LLM group: `anthropic` or `groq` (the chat LLM is always Groq)
`--planner-model <id>`	planner model, e.g. `claude-opus-4-8`, `claude-opus-4-6`, `claude-sonnet-4-6`, or a Groq model
`--set key=value`	any valid Hydra override (repeatable), e.g. `llm.model=<groq-chat-model>`, `agent.max_llm_steps=30`. Keys are validated against the config schema at startup — an unknown key fails the instance loudly.
`--profile <name>`	robot capability/model tag for shared-lesson compatibility
`--server <url>`	add an MCP server at ANY URL (repeatable) — beyond the robot-local `--robot` pair. Use for a server the robot can't host locally (e.g. a cloud-hosted VLA wrapped as MCP). Auth via `EXTRA_MCP_HEADER` in `brain.env` (e.g. `Authorization: Bearer …`). Omit `--robot` entirely for an all-remote instance.

Verified working: --planner, --planner-model, and --set llm.model=… / --set agent.max_llm_steps=…. A --set on an unknown key (or a nested key not exposed at that path) is rejected by Hydra on startup — check config/schema.py.

So each robot can run a different planner LLM with no rebuild. The flags are merged (winning) into that instance's Hydra overrides and preserved across brainctl update. Requirements: the provider's key must be in brain.env (ANTHROPIC_API_KEY / GROQ_API_KEY); model ids follow the Claude API list (Opus 4.8 = claude-opus-4-8).

How prompts are composed (per instance, per robot)¶

At mission start the brain assembles the system prompt in layers:

Universal base — the brain-wide planning rules (every robot), in the brain's code (general_system_prompt). Baked, generic.
Robot/scenario fragments — the robot's identity, skill/tool descriptions, plan templates, and manners. Advertised by the robot over MCP, not baked: each connected server may expose a prompt://system resource the brain fetches and composes in, so a robot ships its own prompt with its tools. (The RAP capability server serves its full planner prompt this way, from rap-integration/brain/planner_prompt.txt.) The image bakes only a robot- agnostic default (prompts/agent_prompt.default.yaml, effectively empty; the chat scenario prompts/system_prompts.default.yaml).
Runtime context — operator facts, known waypoints, learned lessons, current situation.

To change a prompt: - universal base → edit robot-voice-chat → brainctl update (everyone). - a robot's own prompt/plan → edit that server's prompt://system (for RAP, edit agent_prompt.rap.yaml + run scripts/gen_planner_prompt.py) + restart that server (only that robot; no brain redeploy). - per-instance tweak → --set (e.g. a stricter surveillance threshold).

What the brain contains (rough overview)¶

The brain is robot-voice-chat — a FastAPI + Socket.IO backend serving a React SPA. The image bakes the prebuilt UI + robot-agnostic default prompts (both from this repo); robot-specific prompts arrive at runtime over MCP.

Mission executor — the planner loop (Anthropic Opus by default, Groq fallback). Turns a command into a plan, calls tools, observes results, reacts.
FSM plan runner — a compiled step machine (loops, if, optional) that runs a whole plan over your MCP tools without the LLM in the tight loop.
MCP tool backend — connects to the instance's MCP servers: the robot-local capability (:9206) + locate (:9207) pair, plus any --server URLs (e.g. a cloud-hosted VLA). Each connects independently with auto-reconnect; a down server is skipped, not fatal. Tools + prompt://system are discovered per server.
Surveillance — an independent VLM watchdog that watches the run and can pause on hazards/mismatches.
Memory + recoveries — operator context, learned processes, and distilled recovery lessons, injected into the planner prompt at mission start.
Web UI — chat + a Debug panel (reasoning, compiled plan, tool calls, watchdog), a Context panel (operator facts), a Config panel, and an admin area. Plus a Visual Command / observer view (header "Visual" toggle): the segmented live camera with click-to-Pick/Place/Navigate/Arrange + jog, and top-bar toggles to hide the controls and overlay the reasoning + plan in glass panels over the stream — the marketing/livestream shot. See the Using part above.
Config — Hydra, with layered system prompts (general base + robot + task + manners fragments) and .env for keys.

For depth, read the robot-voice-chat repo (backend/src/robot_voice_chat/).

Config & keys layout (on the host)¶

What	Where
API keys + `EXTRA_MCP_HEADER` (shared by all instances)	`brain-host/brain.env` (gitignored)
Brain source (git clones)	`$BRAIN_SRC` = `~/sage/orchestrator/src` (`robot-voice-chat`, `robot-mcp-kit`)
Prebuilt UI + default prompts	`brain-host/{brain-ui,prompts}` (this repo)
Per-instance runtime config (incl. MCP server list)	generated in each container at start (`entrypoint.sh`)

The prebuilt SPA is built on a Mac and committed here (brain-ui/); refresh it with rap-integration/build-ui.sh, which writes both this copy (for brainctl) and the single-box VM copy.

The robot side (not here)¶

The robot + its MCP servers run on each robot's own machine (reachable on the tailnet), and a server too heavy for the robot (e.g. a VLA) can run in the cloud and be attached with --server <url>. Either way it's just an MCP endpoint to the brain — the brain host runs only the lightweight orchestrators. Bring-up for the ZHAW RAP stack lives in rap-integration (rap.sh capability / rap.sh locate); those servers advertise their own tools + prompts to the brain.