Skip to content
AuthorPascal DateJuly 14, 2026 Rev1.0

Binabik — architecture & onboarding

Read this first if you're new. It explains how the whole system fits together — the ideas before the code — then names the actual repositories, tells you where everything runs, is honest about what is and isn't built yet, and shows how to add a new robot or a new skill.

1. What binabik is

Binabik turns a plain-language request ("put the coke can on the left table") into real robot behaviour. A central brain (a large language model with tools) does the thinking — understanding, planning, watching, recovering — and a robot does the acting. The brain never touches the robot's hardware directly; it only calls tools. This split is the single most important idea in the codebase, so before any repository is named, understand the layers.

Two guiding rules:

  1. The brain orchestrates intent; it never runs a tight control loop. Anything real-time, hardware-specific, or physically detailed lives below the brain.
  2. Competence lives in the lowest layer that can own a job end-to-end and verify its own result. The brain says what; lower layers own how and prove it worked.

2. First principles: the layer model

Everything in binabik belongs to exactly one of five layers. Learn these five boxes before you learn any service name — every repo, every future change, and every design argument maps back to them.

  ┌──────────────────────────────────────────────────────────────┐
  │ 1. CORE BRAIN            universal · one shared build          │
  │    conversation · planning · the generic plan runtime          │
  │    (data-flow between steps, success gates, loops,             │
  │     survey/retry) · oversight · memory                         │
  │    ── parameterized by ──                                      │
  │ 2. CONFIGURATION         per customer / robot / task · DATA     │
  │    prompt fragments · plan templates · named places ·          │
  │    model choice · WHICH MCP servers to attach                  │
  └───────────────────────────────┬──────────────────────────────┘
              discovers + calls, over MCP (the control plane)
                                   ▼
  ┌──────────────────────────────────────────────────────────────┐
  │ 3. SKILL / TASK SERVICES     per object / task                 │
  │    a closed-loop, self-verifying COMPETENCE behind ONE         │
  │    high-level tool, e.g. grasp("coke can"):                    │
  │    perceive → grasp → check grip → refine → retry              │
  └───────────────────────────────┬──────────────────────────────┘
                        built on ↓
        ┌──────────────────────────┴───────────────────────────┐
        ▼                                                        ▼
  ┌─ 4. CAPABILITY PRIMITIVES ─┐              ┌─ 5. PERCEPTION / WORLD MODEL ─┐
  │  atomic, self-verifying    │              │  ground · segment · localize ·│
  │  motion/IO: move,          │              │  grasp geometry · scene       │
  │  navigate_to, pick, place, │              │  capture. May host its own    │
  │  gripper, get_pose         │              │  AI models.                   │
  │  per robot / gripper       │              │  per sensor / model           │
  └────────────────────────────┘              └───────────────────────────────┘

Layer 1 — Core brain. The universal orchestrator. It plans, runs plans, watches execution, learns, and talks. It contains no robot-, gripper-, object-, or task-specific knowledge. A change here benefits every deployment.

Layer 2 — Configuration. Data, not code. Which model, which prompts, which plan templates, which named places, and — critically — which MCP servers this instance attaches to. Re-targeting to a new task/robot/customer is a config change; it never forks the brain.

Layer 3 — Skill / task services. The competence layer. A hard, object-specific ability wrapped as one high-level tool — e.g. "reliably grasp a coke can" — that owns its own closed loop internally (perceive → act → check → refine → retry) and can bundle its own AI models. This is not the brain and not a bare primitive; it sits between them. A VLA (vision-language-action model — pixels + a goal → motions) is the canonical layer-3 citizen: it owns a closed-loop competence and brings its own perception, and to the brain it is just one tool. Because it is GPU-hungry it often runs off-robot — see §3a for how it still gets the camera stream.

Layer 4 — Capability primitives. The robot's atomic, blocking, self-verifying motions and I/O — navigate_to, pick, place, open/close gripper, get_pose. Each does one thing and reports ground truth ({ok, held, reachable, …}). It does not retry or re-perceive; that's layer 3's job.

Layer 5 — Perception / world model. Grounding, segmentation, 3-D localization, grasp geometry, scene capture. Swappable per sensor/model; may run heavy AI. It is the heaviest consumer of the data plane (the camera + depth), so if it runs off-robot it must be fed those frames — the central data-plane wrinkle, covered in §3a.

The load-bearing consequence: anything robot/task/customer-specific is a registered service or config — never a fork of the brain. One brain build serves every robot.

3. The two planes, and where things run

Binabik has a strict control/data split — internalize it, because it decides where every new piece of code is allowed to live and how it is allowed to talk:

  • Control plane = MCP (Model Context Protocol, over SSE, over Tailscale). The brain speaks only this. Every tool call, every advertised prompt, every result travels here. It is request/response, low-bandwidth, and location-independent: an MCP server is just a URL, so the brain neither knows nor cares whether it answers from the robot, the central host, or a cloud GPU.
  • Data plane = ROS 2 (topics/services). Cameras, depth, MoveIt 2, Nav 2, point clouds, /cmd_vel, TF. It is high-bandwidth, real-time, and — by default — robot-local: it never leaves the robot, and the brain never sees a ROS message. When the brain wants to know what do you see, it calls a tool; the pixels themselves stay down on the data plane.

Physically there are up to three places code runs:

  CENTRAL HOST (no ROS, no GPU)     ROBOT HOST (ROS 2 + GPU)      CLOUD / off-robot GPU
  ─────────────────────────────     ────────────────────────      ─────────────────────
  the BRAIN (+ web UI)              capability + perception        a model too heavy for
  fleet-shared memory (optional)    MCP servers, the ROS 2         the robot (a VLA, a big
                                    stack, cameras, hardware       segmenter), wrapped as MCP
        │                                  │      ▲                          │  ▲
        └───────────── MCP (SSE) over Tailscale, by URL ──────────────────────┘  │
                                           └──── camera / data stream ────────────┘
                                                 (only if an off-robot service needs it)

The brain connects to a list of MCP server URLs. Most are on the robot; a skill or model too heavy for the robot can run in the cloud. On the control plane they all look identical — tools + an advertised prompt. What differs is the data plane: a service that needs the robot's sensors has to be fed them, and that is a separate path the brain is not part of (next section).

3a. Off-robot compute and the camera stream

Not everything can run on the robot. A VLA (a vision-language-action model that maps camera frames — and often proprioception — directly to motions), or a heavy segmenter / grasp-net, can need far more GPU than a mobile base carries. Such a service can run in the cloud, or on any off-robot GPU box, and still be attached to the brain — on the control plane it is just another MCP URL (--server, with auth; see §5 and integrating-a-robot.md §1).

The catch is the data plane. A VLA or a perception model is useless without the camera stream (and sometimes depth / joint states), and those live in ROS on the robot. So "run it in the cloud" is really two problems: attach its tools (easy — MCP by URL) and get it the robot's frames (the real work). There are two patterns, and choosing between them is a latency-vs-simplicity call:

Pattern A — thin robot-side MCP shim + cloud inference endpoint. The MCP server (and its ROS camera subscription) stays on the robot and is thin; for each call it grabs the latest frame, sends it to a stateless cloud model endpoint (HTTP/gRPC), and returns the answer as a normal tool result. The data plane never leaves the robot except as per-call frames. This is what perception already does today: robot-mcp-perception runs on the robot, holds the ROS camera + point-cloud subscriptions, and calls Gemini Robotics-ER in the cloud per request for grounding, while SAM runs on the robot's local GPU. Best when the model is request/response ("which object is the mug?", "grasp pose for this box?") and a per-call round-trip is acceptable.

Pattern B — full cloud MCP server + a robot→cloud frame stream. The whole MCP server runs off-robot, and a thin robot-side egress bridge streams the camera to it continuously — WebRTC / RTSP / gRPC over Tailscale. robot-mcp-perception already supports this shape via its frame_source: rtsp adapter: point it at a stream instead of a local ROS topic and the same node runs anywhere. Best when the off-robot service needs a continuous, high-rate stream rather than per-call frames — e.g. a VLA closing a control loop at video rate.

Trade-offs to weigh:

  • Latency. Pattern A pays one WAN round-trip per tool call — fine at planning rate (a call every few seconds), painful for a tight loop. Pattern B pays continuous bandwidth but keeps the loop running next to the model. A closed-loop VLA over the public internet is usually a bad idea — put it on a GPU box on the same LAN as the robot (still "off-robot", still MCP by URL, but metres not continents away). Reserve the true cloud for occasional big-model calls.
  • Who owns the ROS subscription. Pattern A keeps ROS strictly robot-local (only frames egress, per call). Pattern B moves the consumer off-robot but still needs a robot-side process to publish the stream — ROS itself still doesn't leave.
  • Statefulness & bandwidth. A stateless "score this frame" model → A. A model that needs temporal context or runs continuously → B.
  • The control plane is identical either way. However the frames reach it, the service still advertises tools + prompt://system, and the brain still speaks only MCP. Frame transport is out-of-band from the brain — a deployment decision, not a brain feature.

The same reasoning applies to robot-mcp-perception itself: today it is a robot-side node using a cloud grounder (Pattern A); its frame_source adapter means the whole node could instead run off-robot against an RTSP stream (Pattern B) if the robot's onboard GPU is too small for SAM. The layer-5 boundary never moves — only where the box runs and how the frames get there.

4. How the layers map to the real repositories

Now the concrete components. Keep the layer numbers from §2 in mind.

Repo Layer What it is Runs where
binabik-orchestrator — Umbrella repo: product-general specs, binabik.sh, and it holds the component repos below. This file lives here. — (dev)
robot-voice-chat 1 + 2 (+ UI) The brain. FastAPI + Socket.IO + Hydra: chat, the planner LLM, the generic plan runtime, the surveillance watchdog, recovery learning, and the React web UI (chat + the Visual Command / observer view). Central host
binabik-brain-host deployment How the brain is run. brainctl (launch/update one lightweight brain container per robot), the Docker image, the prebuilt UI, robot-agnostic default prompts, and Tailscale-Serve HTTPS. Central host
robot-mcp-kit framework Shared library for MCP servers — the Async Task Contract (long-running work → task_id → signed task_done event). Imported by the servers; vendored into the brain image. Not a running service. Both (a dependency)
robot-mcp-perception 5 The robot's vision module: a Gemini-ER grounder (what/where — a cloud API call) + a SAM 3 segmenter (pixels — the robot's local GPU). Exposes locate_3d, grasp_pose, capture_scene, pixel_to_3d, grasp_from_pixel. Robot host (can run off-robot — §3a)
mcp-perception-buffer 5 (async) The "what do you see" server: subscribes the /vision/** topics, keeps a rolling latest snapshot, and proactively pushes events to the brain. Optional. Robot host
ros2_vision_interfaces interfaces Pure ROS 2 message definitions for the vision data plane. Built first — the perception node compiles against it. Robot host (build dep)
robot-mcp-memory memory Process/skill memory (:9103): markdown "how-to" documents distilled from teaching-mode missions, retrieved into the planner when a similar instruction arrives. Fleet-shared. Central/shared (see §6)
ros2-memory memory Episodic memory: records configured ROS topics into time-segmented bags and answers temporal queries ("where was I 10 s ago?"). Robot host
rap-integration 2 + 4 The reference robot integration for the Galaxea R1Pro: the capability MCP server (pick/place/navigate_to/…), perception launch, the robot's advertised planner prompt, and deploy tooling. It is the counterpart to binabik-brain-host, not an instance of it. Robot host (the sim VM)
(pulled by rap-integration) robot vendor The robot's own stack: galaxea_isaac_moveit (the R1Pro description + MoveIt 2 config + Nav 2 params + Gazebo world), plus MoveIt 2 and Nav 2 themselves. Not ours — our capability server wraps their client APIs. Robot host

The brain is layer 1+2; a robot is layers 4+5 (+3 when built); config (layer 2) says which of a robot's servers to attach. That's the whole map.

5. The MCP contract — how a robot or skill plugs in

A server joins the system by exposing, over MCP/SSE:

  1. Tools — bare-named (pick, not motion.pick), each with a clear docstring (what it does, args, and a structured return like {ok, held, side}). The docstring is what the planner reads to decide whether and how to call it.
  2. prompt://system — a text fragment the brain fetches on connect and composes into the planner's system prompt after the universal base. This is how a robot ships its identity/skills/plans, and how a skill says when it should be used. Change it, restart that server, and only that robot/skill is affected — no brain rebuild.
  3. Blocking behaviour — tools listed as "blocking" make the brain await the result (and gate the compiled plan on it); everything else is fire-and-forget.
  4. The Async Task Contract (from robot-mcp-kit) for work longer than a request.

The brain discovers all of this at connect time and recomposes its prompt. So adding a capability is: run an MCP server + tell the instance to attach it. There is no brain code change to add a robot or a skill.

5b. A mission, end to end (the loop, concretely)

To make the layers concrete, here is what actually happens when an operator types "put the coke can on the left table" into a connected instance:

  1. Compose the prompt. At mission start the brain takes its universal base prompt (layer 1) and appends every connected server's prompt://system fragment (layer 2 data): the robot's identity + tools + plan templates, plus any skill service's "when to use me" text. The planner LLM now knows this specific robot.
  2. Plan. The planner reads the instruction + the composed prompt + the tool docstrings and emits a plan — a small program in the brain's plan runtime, e.g. navigate to "left table" → locate the can → grasp it → place it. Named places and return-field names (pk.held, box.found) come from the advertised prompt.
  3. Execute, step by step, over MCP. The runtime calls tools one at a time: navigate_to_named("left table") (layer 4) → locate_3d("coke can") (layer 5) → the grasp (today a prompt loop over grasp_pose + pick; should be one grasp("coke can") layer-3 call — §6a) → place(…). Each returns {ok, …}; the runtime gates the next step on it (expect: pk.held) and the brain speaks a line of narration alongside each call.
  4. The data plane does the real work — invisibly to the brain. Every one of those tool calls turns into ROS activity on the robot: MoveIt plans an arm trajectory, Nav 2 drives the base, the camera + point cloud feed perception. The brain never sees a pixel or a joint angle — only the {ok, held, side} summary. (If perception ran off-robot, this is exactly where the camera-stream path of §3a would carry the frames out to it.)
  5. Watch + recover. While the plan runs, the surveillance watchdog polls observe_task (a VLM) every few seconds; a run of bad frames aborts and replans. Failures that later get fixed become recovery lessons for next time (layer 1).

Every arrow between the brain and the robot in that list is one MCP tool call; everything the robot does inside a call is the data plane. That division is the whole architecture in one mission.

6. The current situation — honest

Built and deployed today: layers 1, 2, 4, 5. The brain is robot-agnostic; robots advertise their prompts over MCP; instances attach arbitrary MCP servers (including cloud ones, with auth) via brainctl … --server; per-API + per-MCP + robot-GPU stats are in the admin panel; the UI is an installable PWA served over Tailscale HTTPS; multiple users can share a view (one controls, others watch).

Not built yet: layer 3 (skill / task services). This is the biggest gap and the reason for the next section. Today there is no skill-service anywhere.

6a. The "prompt loop", where it is, and why it is bad

Because layer 3 doesn't exist yet, the hardest, most object-specific competence — grasping — is currently faked in two places:

  • the actual motion is a single pick primitive (layer 4) in rap-integration/rap_capability_server.py, and
  • the reliability logic is a loop written in a prompt: rap-integration/brain/agent_prompt.rap.yaml → plan_templates, generated into planner_prompt.txt and advertised to the brain. The relevant shape is roughly:
{loop: {steps: [grasp_pose, pick], until: "pk.held", max_iterations: 3}}

i.e. "call grasp_pose, then pick; if the gripper isn't holding, do it again, up to three times."

Why this is bad — read this carefully, it motivates the whole roadmap:

  1. It's prose, not a program. The single most safety- and success-critical behaviour on the robot has no code, no unit tests, and no metrics — it's a few lines of English in a robot's prompt file.
  2. It can't actually refine. "Retry the same two steps" is not a closed loop. A real grasp loop nudges the base, adjusts the standoff/approach, re-reads with fresh depth, or switches arms based on feedback. A prompt loop just repeats.
  3. It leaks robot detail into the plan. The brain's plan template now knows a specific gripper's quirks. Improving grasping means editing a robot's prompt rather than shipping a better, independently-versioned skill.
  4. It doesn't scale to variants. Top-grasp for small flat items, front-grasp for cans, two-handed side-grasp for boxes, special moves for bobbins — as prose, each becomes another branch the planner must reason about mid-plan.
  5. Wrong layer. The generic loop/expect runtime (layer 1) is fine for orchestration ("did the step succeed?"). Grasp competence is a different thing and belongs in layer 3 as real, testable code.

The fix is layer 3: move this into a skill-service MCP server — one grasp(object) tool whose body runs perceive → grasp → verify → refine as code, calling the primitives + perception underneath. The prompt loop then collapses to a single plan step.

6b. The robot's upstream stack is not cleanly pinned

The Galaxea R1Pro's own software — galaxea_isaac_moveit (its URDF + MoveIt 2 config + Nav 2 params + Gazebo world), plus the MoveIt 2 and Nav 2 packages it needs — is currently assembled by hand on the sim VM:

  • galaxea_isaac_moveit sits on the VM's persistent data volume, pulled from GitHub at no fixed commit, then symlinked into ~/colcon_ws/src and colcon build-ed by rap-integration/setup.sh;
  • the MoveIt/Nav2 pieces are apt install-ed ad-hoc (ros-jazzy-trac-ik-kinematics-plugin, ros-jazzy-spatio-temporal-voxel-layer, …) with no version pin.

Why this is bad: the source of truth is "whatever snapshot happens to be on that volume." A fresh VM, a second robot, or a colleague can't reproduce the exact stack, and upgrades are unrepeatable. It works, but it isn't clean.

Target: pin it and clone it reproducibly (see §7).

6c. Other honest gaps

  • Attach config is imperative. Which servers an instance attaches is passed as brainctl … --server flags. There's no declarative per-robot manifest, so the full server set for a robot lives only in the admin's command history.
  • Fleet memory placement. robot-mcp-memory is conceptually fleet-shared but in the RAP deployment it has run robot-side. It should settle as a central shared service.
  • Off-robot perception / VLA is design, not deployment. §3a describes two patterns for running camera-consuming compute off-robot. Today only Pattern A exists in practice — perception is a robot-side node calling a cloud grounder per request. No service runs the whole node off-robot over a streamed camera (Pattern B), and there is no VLA in the stack yet. Both are supported shapes the architecture allows, not running code — the frame-egress bridge for Pattern B would be the new piece to build.

7. Where each repo must go — the change roadmap

7a. Build layer 3 (the four brain-side items)

# Change Where Why
1 Skill-service template repo + a convention (one high-level tool + prompt://system "when to use" + {ok,…} returns). new repo, e.g. skill-service-template Makes new skills copy-paste, consistent, and testable.
2 First real skill service — a grasp-service wrapping perceive→grasp→verify→refine as code, replacing the prompt loop (§6a). new repo, e.g. skill-grasp Kills the prompt loop; proves the pattern.
3 Declarative attach config — a per-robot manifest (e.g. robots.yaml) listing each robot's MCP servers (robot-local + skill services + cloud), read by brainctl. binabik-brain-host Turns §6c's --server flags into reproducible config.
4 Skills panel (read-only) in the admin UI listing attached skill servers + their advertised descriptions. robot-voice-chat Visibility into what a deployment can do.

Note the brain mechanism is already done — it discovers tools + composes advertised prompts + attaches arbitrary servers. Items 1–4 are new repos + config + UI, not core-brain surgery.

7b. Clean up the robot's upstream stack (moveit / nav2 / vendor sim)

The robot vendor stack must become reproducible from version control, not a volume snapshot:

  • Pin the vendor repo. Record galaxea_isaac_moveit's clone URL and a commit or tag in rap-integration (a deps.lock, or a git submodule). setup.sh should git clone --branch <pinned-ref> (or git submodule update) instead of relying on whatever is on the data volume.
  • Pin the ROS packages. Record exact versions of the MoveIt 2 / Nav 2 plugins (and the ROS 2 distro) in a rosdep/apt manifest so a fresh VM installs the same stack every time.
  • Keep it separate from our code. We don't vendor MoveIt/Nav2 — they're standard ROS 2 packages — but we document and pin them so a from-scratch bring-up is deterministic.
  • Result: a new robot box goes from bare image → identical, working stack by running one pinned script, with no hand-symlinking.

7c. Per-repo summary

Repo Change needed
robot-voice-chat Add the read-only skills panel (7a#4). Otherwise stable — it already discovers tools + composes advertised prompts.
binabik-brain-host Add the declarative robots.yaml attach config (7a#3).
rap-integration Pin + clone the vendor stack reproducibly (7b). Move the grasp competence out of the prompt loop into the new grasp-service (7a#2); shrink the plan template to a single grasp step.
robot-mcp-perception / -buffer / ros2_vision_interfaces / ros2-memory Stable. Keep pinned + reproducible clones like everything else.
robot-mcp-memory Settle as a central shared fleet service (§6c).
robot-mcp-kit Stable — the shared contract.
(new) skill-service-template, skill-grasp, … The layer-3 repos (7a#1–2).

8. How to add a new robot

A robot is layers 4 + 5 behind MCP. To onboard one:

  1. Create a robot-integration repo (model it on rap-integration) implementing the contract in integrating-a-robot.md:
  2. a capability MCP server — navigate_to, navigate_to_named, pick, place, get_pose, … each returning {ok, …};
  3. a perception server — locate_3d, grasp_pose, capture_scene, …
  4. Bind them 0.0.0.0 over SSE; use bare tool names.
  5. Advertise the robot via a prompt://system resource (identity, how to use its tools, plan templates).
  6. Pin the robot's own stack (drivers / MoveIt / Nav2) reproducibly (§7b).
  7. Run the servers on the robot; join the tailnet.
  8. Launch a brain instance: brainctl up <name> --robot <its-tailnet-host>.

No brain change, no rebuild. The instance discovers the robot's tools + prompt.

9. How to add a new skill (a grasp variant, an inspection routine, …)

A skill is layer 3 — a self-contained competence behind one tool:

  1. Clone the skill-service template into a new repo, e.g. skill-grasp-box.
  2. Implement one high-level MCP tool — e.g. grasp_box_twohand(object) — whose body runs the closed loop internally (perceive → grasp → verify → refine), calling the robot's primitives + perception underneath. Bundle its own model if needed.
  3. Advertise prompt://system: when to use this skill versus others (this is how the planner chooses — "for boxes >20 cm use grasp_box_twohand; for cans use grasp_can_front; for flat items use grasp_top"), plus constraints and return fields.
  4. Run it — on the robot, or in the cloud if it's heavy — on the tailnet.
  5. Attach it to the deployment: brainctl up … --server https://…/sse (or add it to the robot's robots.yaml entry once 7a#3 lands).
  6. The brain discovers the tool + composes the prompt → the planner uses it automatically, choosing among skills from their advertised "when to use" text.

No brain change, no rebuild. That is the entire point of the layering.

10. Glossary

  • Brain — the central LLM orchestrator (robot-voice-chat). Plans; never actuates.
  • MCP — Model Context Protocol; the control-plane RPC the brain speaks to every server.
  • prompt://system — an MCP resource a server exposes so its prompt travels with its tools; the brain composes it in at mission start.
  • Primitive (layer 4) — one atomic, self-verifying robot action.
  • Skill / task service (layer 3) — a closed-loop competence behind one high-level tool; its own repo + MCP server.
  • Capability server — a robot's layer-4 MCP server.
  • Instance — one brainctl brain container pointed at one robot.
  • Control/data split — brain speaks MCP only; ROS stays on the robot.