AuthorPascal DateAugust 15, 2026 Rev3.0

Binabik — architecture¶

The whole system on one page: the layers, the seam between robot-agnostic and robot-specific code, and which repository is which. Everything operational — bring-up, brainctl, r1ctl, the tool-by-tool contract — lives on the pages in §0, and only there.

0. Where to read what¶

Each page owns one job, so nothing is documented twice:

Page	Owns
this page	the layer model, the abstraction seam, where things run, the repo map
Setting up your own stack	from-zero bring-up on your own machine
Running the simulator	the R1 in Gazebo — the Linux/VM path and the macOS (RoboStack) path
Running a brain	all `brainctl` operation — commands, flags, `brain.env`, central memory, HTTPS
Using the brain	driving it — chat, the Debug panel, Visual Command
Integrating a robot	the robot-developer MCP contract — the canonical tool set, conventions, checklists
R1 robot stack	the R1 robot-side reference — `r1ctl`, ports, tmux windows, smoke tests
Design specs	per-service design specs (grasp, the world state, memory, the kit, …)
Coding guidelines	the conventions every binabik codebase follows — read before your first edit

Credentials are the one subject with two owners, because there are two config files and a service on either side of the tailnet reads the same secret. brain.env and the brain's identity at the store are Running a brain → per-instance secrets; r1.env, the robot's identity and its per-service cache are R1 robot stack → credentials from a store. The resolver both sides use is one library — see §4 — so the pages differ in deployment, not in mechanism.

Cost has three owners, not two — a robot's spend is a contract, not a config file. What a call is charged at is host state — the price table, its mount and its refresh, on Running a brain → the price table. What a call cost, and where the money went, is a surface you drive: Using the brain → what it is costing you. Both of those are brain state. The third owner is outside the brain entirely: a robot-side spender is a robot developer's obligation, spec'd on Integrating a robot → reporting spend — send tokens plus identity, never a price, because the ledger and the table it prices against are both brain state and a robot has neither. All three read from and write to one module in the brain, so the pages differ in what they let you do, not in where the number comes from.

Live issue tracking is Linear (the Brain Dev project), not a file in this repo. There is no archive: a page that stops being true is deleted, and git history keeps what it said.

1. What binabik is¶

Binabik turns a plain-language request — "put the coke can on the left table" — into robot behaviour. A central brain does the thinking: understanding, planning, watching, recovering. The robot does the acting. The only thing that crosses between them is an MCP tool call; the brain never touches ROS.

Two rules decide where any piece of code belongs:

The brain orchestrates intent; it never runs a tight control loop. Anything real-time, hardware-specific or physically detailed lives below it.
Competence lives in the lowest layer that can own a job end-to-end and verify its own result. The brain says what; lower layers own how, and prove it worked.

2. First principles: the layer model¶

  ┌──────────────────────────────────────────────────────────┐
  │ BRAIN — core (L1) + config (L2)                          │
  │ a VLM: plans, narrates, watches, replans — never a loop  │    ┌────────────────────────────┐
  │ · planner: the snapshot + one frame, injected at every   │    │ WORLD STATE     :9240      │
  │   decision point                                         │───▶│ body · cameras · scene     │
  │ · watchdog: the same snapshot + a frame, judged here     │    │ ONE snapshot, three        │
  └───────┬───────────────────────────────────┬──────────────┘    │ readers. Free unless a     │
grasp()   │                                   │                   │ scene refill is asked ($)  │
          ▼                                   │                   └───┴─────────────────────┬──┘
  ┌────────────────────────────────────────┐  │  navigate_to · stop   │                     │
  │ COARSE SKILLS — L3:  grasp · place     │  │  pick · place         │                     │
  │ one tool each, owning perceive →       │  │  locate_3d · capture  │                     │
  │ act → verify → retry. Each may run     │◀─┼───────────────────────┘                     │
  │ its own finer VLM tool loop; its       │  │                                             │
  │ recovery reads the snapshot too        │  │                                             │
  └───────┬────────────────────────────────┘  │                                             │
          │                                   │                                             │
          │                                   │                                             │
══════════▼═══════════════════════════════════▼═════════════════════════════════════════════▼═══
      THE BINABIK ROBOT ABSTRACTION LAYER — the canonical MCP contract
════════════════════════════════════════════════════════════════════════════════════════════════
  ┌────────────────────────────────────────────────────────────────────────┐
  │ ROBOT ADAPTER — primitives (L4) + perception (L5)                      │
  │ pick · place · move_arm · navigate_to · stop · locate_3d ·             │
  │ capture_scene · verify_held …   the ONLY robot-specific code           │
  └───────────────────────────────────┬────────────────────────────────────┘
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │ ROS 2 — MoveIt 2 · Nav 2 · drivers · cameras · TF                      │
  │ the Gazebo sim, or the real robot                                      │
  └────────────────────────────────────────────────────────────────────────┘

The brain (L1 + L2). A vision-language model with tools. It reads the world snapshot — the robot's posture, joints, end-effector and base poses, grippers, plus the scene inventory and camera frames — turns the operator's instruction into a plan, runs that plan step by step over MCP, narrates it, and replans when something goes wrong. It holds no robot-, gripper-, object- or task-specific knowledge; what it knows about a particular deployment is layer 2: configuration — prompt fragments, plan templates, named places, the model choice, and which MCP servers to attach. That is data, not code, which is why re-targeting to a new robot or customer never forks the brain.

The world state. One service composes the single "what is the world like" answer, and it has exactly three readers: the brain's planner, the brain's surveillance watchdog, and an L3 skill's recovery controller. It sits in front of the abstraction (it is a client of the contract, and strictly ROS-free), so its caching, staleness handling, refresh budgets and per-consumer projections are written once and inherited by every robot. See the world-state spec.

How the brain reads it is worth being exact about, because it is not a tool call. The planner is given the snapshot — MissionExecutor composes a "what you can see right now" block at each decision point, plus exactly one current camera frame — rather than being allowed to call for it. The service is deliberately not in the brain's attach manifest: a refresh runs the grounder and bills, and a paid tool on the planner's surface would let the model spend on its own initiative. The watchdog reads the same client (deterministic body checks, then a brain-side VLM over frame + body), and the operator's World panel reads that same client too — so what the panel shows is what the planner was given. On the robot, grasp-service's recovery controller reads the service directly (WORLD_URL, default :9240) to look before each of its steps. Every one of those reads is free; only an explicit refresh fills the scene.

Coarse skills (L3). A hard, object-specific competence wrapped as one high-level tool — grasp("coke can"), and place — which owns its loop internally: perceive → act → check → refine → retry, with its own recovery and learning. A skill is free to be a small planner in its own right: grasp-service runs a deterministic state machine first and then, on exhaustion, a scoped LLM tool-use loop over its own tools. A VLA (vision-language-action model) is the canonical L3 citizen for the same reason — it owns a closed-loop competence, brings its own perception, and to the brain is just one tool. Because a skill only speaks the contract, it is robot-agnostic. grasp is a skill service today; place is still a bare primitive at the seam and is the next competence to move up here.

The abstraction layer (L4 + L5). The canonical MCP tool contract, and the highest layer that may contain anything robot-specific. L4 is atomic, self-verifying motion and I/O (move_arm, navigate_to, pick, place, set_gripper, get_pose, stop); L5 is perception (locate_3d, grasp_pose, capture_scene, verify_held). The brain reaches it directly for anything that needs no closed loop — drive forward, turn, go to a named place, stop, look — and through a skill for anything that does. Each tool does one thing and reports ground truth ({ok, held, reachable, …}); it does not retry or re-perceive, because that is L3's job.

ROS 2 and below. MoveIt 2, Nav 2, the drivers, the cameras, TF — the robot's own software, reached only through the adapter above it. Nothing above the seam imports rclpy or subscribes to a topic.

The load-bearing consequence: anything robot-, task- or customer-specific is a registered MCP server or config — never a fork of the brain. One brain build serves every robot.

2a. The abstraction layer — where agnostic meets robot-specific¶

   BRAIN + CONFIG · WORLD STATE · SKILLS   ┐  AGNOSTIC — written once,
   the L4/L5 TOOL SURFACE (names, schemas) ┘  serves every robot
  ═══ BINABIK ROBOT ABSTRACTION LAYER ═══  the standard MCP contract  ← the seam
   the L4/L5 IMPLEMENTATION                   ROBOT-SPECIFIC — the only place
   + ROS 2, drivers, the machine              specificity is allowed to live

Note where the line falls: the tool surface — the exact names, argument/return schemas, frames, units and semantics — is agnostic and written once. Only its implementation is robot-specific. Swap the robot, or swap sim for real, and only the servers below the line change; two implementations of the same contract make sim→real transfer free, because nothing above the line moves.

What is, and isn't, in the contract. It is deliberately small and generic — the operations every mobile manipulator has:

In it: atomic motion, gripper, IK/FK, planning-scene, base navigation, perception primitives, and an object-centric pick (the caller says "grasp this object"; the robot chooses the mechanics). Every tool returns at least ok, plus a short reason on failure; frames and units are explicit.
Above it: the generic grasp skill loop — one L3 service, identical across robots, that only calls the contract.
Below it: the robot's grasp mechanics — the choreography and every empirical constant in it — which implement pick.

A service can sit above the line and still deploy on the robot: the world state does exactly that, so L3 reads it locally and the brain gets a whole snapshot in one round-trip. Deployment and architecture are independent choices; what puts something below the line is robot-specific code, not the host it runs on.

Robot-dependent capability is handled in one place, in one shape: every part of the world snapshot is a block of {label, available, reason}, so a capability the robot cannot answer arrives as available: false with a plain-words reason and no data at all — never as an empty list that reads like "nothing wrong". Consumers iterate; nobody branches per robot.

The tool-by-tool contract (names, args, returns, frames, units, the integration checklist) is Integrating a robot — this page only names the seam and says which side of it a thing belongs on.

2b. Fleet, groups, and memory scoping¶

Deployments span many robots, factories and customers, so memory is multi-tenant, and the split turns on one question — is this knowledge transferable, or robot-specific?

Robot-private (what this robot can see right now) is scoped robot:<id>, never shared, and not persisted: the robot answers "what do you see" live, on demand.
Transferable learning (grasp lessons, recovery corrections, taught how-tos) is scoped to a group — the explicit sharing boundary. Robots in a group pool it; an ungrouped robot is isolated.

The group registry is central and propagating: robots and their group membership live once in robot-mcp-memory, and every brain reads and writes it over MCP, so grouping a robot from any brain is seen by all. An operator manages this from the brain's Fleet admin view. Details: the memory spec and Running a brain.

3. The three planes, and where things run¶

Binabik has a strict control/data split, and it decides how every new piece of code is allowed to talk:

Control plane = MCP (over SSE, over Tailscale). The brain speaks only this. Every tool call and result travels here. It is request/response and location-independent — an MCP server is just a URL, so the brain neither knows nor cares whether it answers from the robot, the central host, or a cloud GPU.
Data plane = ROS 2 (topics/services). Cameras, depth, MoveIt 2, Nav 2, point clouds, /cmd_vel, TF. High-bandwidth, real-time and robot-local: it never leaves the robot.
Media plane = WebRTC (H.264 over SRTP, over Tailscale), and it is the newest of the three (BIN-262). The operator's browser and the robot are the two peers; the brain is neither. It brokers two SDP strings per viewing session over the control plane and carries no media at all — no encoder, no decoder, no aiortc, nothing on its event loop. host↔host UDP over the tailnet, with no TURN and no STUN — the tailnet already routes both peers, so there is no address for STUN to discover. What the tailnet does not promise is that the path is direct: Tailscale falls back to a DERP relay when it cannot NAT-traverse, silently and without changing anything an application can see (BIN-576). A relayed path carries TCP perfectly well and unprotected fixed-rate video badly, so the stream's own loss resilience is what has to carry it — see §3a.

The third plane is what the note below used to say the stack did not have. It exists to serve a human looking at a screen, not the brain: the mission planner still reads the camera the control-plane way, through get_frame (which is also the fallback the Visual view keeps running underneath the stream, at ~1.4 fps, for a robot that cannot send one). Nothing in the brain's reasoning path depends on it, which is exactly why it can be a browser↔robot connection rather than something the brain has to mediate.

3a. The media plane does not get a direct path for free¶

Being on the tailnet gets both peers routable, not adjacent. Tailscale attempts NAT traversal and, when it fails, relays through a DERP node instead — tailscale ping is the only thing that will tell you which happened (direct connection not established versus a direct <addr>:<port> line). Measured 2026-09-10: a robot and an operator's laptop, both on the tailnet, both reporting a healthy stream at the sender, relayed via DERP in Helsinki at ~50 ms with the operator seeing constant freezing.

This matters because the failure is invisible from the sender's side. The same rig measured from a host that was direct showed a flat 30.0 fps and zero packet loss, which is why a LAN measurement is a regression check on the source and never an acceptance test for the stream. r1-abstraction on a --tun=userspace-networking host is the same lesson one step further along: same tailnet, no usable path at all.

Physically there are up to three places code runs — the central host (the brain and the shared memory; no ROS, no GPU), the robot host (the skills, the world state, the adapter, the robot's own servers, ROS), and optionally an off-robot GPU for a model too heavy for the robot, wrapped as MCP.

An off-robot model still needs the frames

Attaching a cloud service's tools is trivial — MCP is a URL. Getting it the robot's camera stream is the real work, because the camera lives in ROS on the robot. The pattern in use for every model today keeps a thin MCP server on the robot holding the camera subscription and has it call a stateless cloud model per request.

A robot-side camera-egress bridge does now exist — binabik-r1-vision's WebRTC sender, the media plane above (BIN-262) — but note what it is and is not. It is pointed at a browser: the receiver is a human's video element, and each viewer negotiates its own peer connection. Feeding a continuous, high-rate model consumer such as a closed-loop VLA is the same transport aimed at a different receiver, and the shared-encode tee on the sender is the piece that makes adding one cheap — but no such consumer exists yet, and nothing in the stack today reads frames off this plane.

The R1, concretely. The same layers, instantiated for the one robot we run:

  brain (central, one container per colleague)
     │  MCP over Tailscale
     ├── grasp(object) ─────────────────▶  grasp-service        :9210
     ├── world_state · get_frame ───────▶  binabik-world-state  :9240
     └── pick · place · navigate_to_named · stop · locate_3d · capture_scene ·
         stream_start · stream_stop · …
                                      │
                                      ▼
                        r1-abstraction  :9220   ← THE SEAM
                                      │
                    ┌──────────────────┴───────────────────┐
                    ▼                                      ▼
     galaxea_agent — the R1's own                          binabik-r1-vision  :9230
     MCP servers (r1_manipulation ·                        capture_scene · get_frame ·
     perception · nav2 · ros)                              pixel_to_3d · stream_start ·
                    │                                      stream_stop    (with vision)
                    ▼   ROS 2 — the data plane                     │
     MoveIt 2 / Nav 2  →  the Gazebo sim, or the real R1           │ H.264 / SRTP
                                                                   ▼   the media plane
                                              the operator's browser, over the
                                              tailnet — never through the brain

grasp-service and binabik-world-state are themselves clients of :9220 — every path to the robot goes through the seam. The same wiring serves sim and real — only the ROS build under galaxea_agent changes, which is exactly what the abstraction buys. Ports, tmux windows, env vars and the bring-up order are the R1 robot stack; the brain side is Running a brain.

4. The repositories¶

Repo	Layer	One-liner	Runs where
robot-voice-chat	1 + 2 (+ UI)	The brain — planner, plan runtime, watchdog, recovery, MCP client, web UI + admin.	Central host
binabik-brain-host	deployment	How the brain is run — `brainctl`, the Docker image, `robots/<name>.yaml` attach manifests, HTTPS.	Central host
binabik-orchestrator	—	Umbrella dev checkout holding the brain-side repos + product-general specs.	— (dev)
robot-mcp-kit	framework	Shared MCP library: server helpers + the Async Task Contract, `ToolClient`, `ParamStore`, `RecoveryAgent`, and the credential registry (`keys`, `ServiceCredentials`) every service on both planes resolves its secrets through.	Dependency
binabik-world-state	agnostic, in front of the seam	The one world snapshot (`:9240`) — body, cameras, scene — read by the planner, the skills and the watchdog. ROS-free. One per robot, not per brain: brains derive its URL from `ROBOT_HOST`, so two brains on the same robot share one process, its scene cache and its fill budget. A plain read is free (~2 s) and never captures; the paid calls are `refresh=true` and an annotated `get_frame`, which share one budget (BIN-536).	Robot host
grasp-service	3	The grasp skill (`:9210`) — `grasp(object)` as one tool: strategy selection, the FSMs, verification, recovery, learning.	Robot host
skill-service-template	3 (scaffold)	Copy-me scaffold for a new skill service.	— (dev)
robot-mcp-memory	memory + fleet	Central fleet memory (`:9103`) — robot/group registry, process how-tos, the group-scoped lesson store. Deployed with every brain.	Central/shared
r1-abstraction	⅘ (adapter)	The R1's implementation of the contract (`:9220`) — presents the canonical tools over `galaxea_agent`'s.	Robot host
binabik-r1-vision	5	The R1's on-demand scene service (`:9230`) — `capture_scene` / `get_frame` / `pixel_to_3d`, plus `locate_marker` / `detect_markers` and a `PoseStamped` marker feed at camera rate (KOE-33, the L5 half of the AprilTag docking module). Opt-in (`VISION_ENABLE=1`), because grounding bills per capture — markers themselves cost nothing and need no depth.	Robot host
binabik-scene-perception	5 (core)	Robot-agnostic scene core (no ROS): grounding → segmentation → the canonical object list.	Dependency
binabik-docking	agnostic (core) + 5	Two packages: the robot-agnostic docking core (no ROS, asserted by a test): the holonomic control law behind a `ControlStrategy` Protocol, the slowdown/saturation/acceleration envelope, the staging pose and goal test, the abort predicates, and the tag-relative dock database — a `DEFINITION` table, so unlike a waypoint it survives a restart (KOE-34) — and the servo service on `:9250` beside it (KOE-35): the command/marker/odometry ports, the FSM, `undock`, a 0.4 s dead-man, and `dock_to_marker` / `undock` / `save_dock` / `list_docks` / `dock_status`. R1 and G2 backends; the G2's has never run on hardware and says so. Launched by `r1ctl` behind `DOCKING_ENABLE=1` and re-exported by `r1-abstraction` on `:9220` since KOE-36, so nothing above the seam learns this address.	Robot host
segmentation-modal	5 (cloud, opt.)	SAM 3 segmenter on Modal + an MCP wrapper — off-robot perception, paid, never started implicitly.	Cloud (Modal)
galaxea_agent	⅘	The R1's own MCP servers (`r1_manipulation` · `perception` · `nav2` · `ros`) over its ROS stack.	Robot host
galaxea_isaac_moveit	ROS substrate	The R1's ROS stack — description, MoveIt 2, Nav 2, the sim. One unified branch, `main`, for both sim and robot since BIN-276 (`jazzy` was merged into it and deleted — BIN-478).	Robot host
binabik-r1-host	deployment	`r1ctl` — one command to install, reset and relaunch the whole robot-side stack.	Robot host
binabik-docs	docs	This site. Code repos keep only a `README.md`.	— (dev)

Gone, so that old references resolve: rap-integration, robot-mcp-perception and ros2_vision_interfaces were archived (BIN-95) and mcp-skill-control deleted; mcp-perception-buffer (:9202) and ros2-memory (:9203) were retired (BIN-306) — the repos exist, nothing launches them, and the world state answers "what do you see" instead. Their specs are gone from this site; git history has them.

5. What a server owes the brain¶

A server joins the system by exposing, over MCP:

Tools — bare-named (pick, not motion.pick), each with a clear docstring and a structured {ok, …} return. The docstring is what the planner reads to decide whether and how to call it.
prompt://system — a text fragment the brain fetches on connect and composes into the planner's system prompt. This is how a robot ships its identity and named places, and how a skill says when it should be used. Change it, restart that server, and only that robot or skill is affected — no brain rebuild.
Blocking behaviour — tools declared blocking make the brain await the result and gate the plan on it.
The Async Task Contract (from robot-mcp-kit) for work longer than a request.

The brain discovers all of this at connect time. So adding a capability is: run an MCP server, and tell the instance to attach it. There is no brain code change to add a robot or a skill.

6. A mission, end to end¶

What actually happens when an operator types "put the coke can on the left table":

Compose the prompt. The brain appends every connected server's prompt://system fragment to its universal base prompt. The planner now knows this robot.
Read the world. It reads the world snapshot — posture, grippers, base pose, the scene inventory, a camera frame — so it plans against what is actually there.
Plan. The planner emits a small program for the brain's plan runtime: navigate to "left table" → grasp the can → place it, with data flowing between steps and each step gated on the previous one's result.
Execute over MCP. navigate_to_named("left table") (the adapter, directly) → grasp("coke can") (the skill, which internally perceives → picks → verifies → refines) → place(…). Each returns {ok, …}, and the brain narrates alongside each call.
Watch and recover. A surveillance watchdog runs for the whole mission — both when the brain compiles a plan and when it drives the tools itself, since the planner picks between those on its own and the robot is no less worth watching either way (BIN-416). Its free half reads the body every tick; its paid half judges camera frames when the observer is armed, and replans on a run of bad ones. Failure is absorbed at the lowest competent layer: the primitive self-verifies, then the skill retries and switches approach, then the skill's recovery loop, and only then does the brain replan.

Every arrow between the brain and the robot is one MCP tool call; everything the robot does inside a call is the data plane. That division is the whole architecture in one mission.

7. Adding a robot, or a skill¶

A new robot is layers 4 + 5 behind MCP, and nothing above the line changes:

Write a robot-integration server (model it on r1-abstraction) that implements the contract — the canonical tool set, args/returns, frames and units, per Integrating a robot. Bind 0.0.0.0; use bare tool names.
Advertise the robot via prompt://system — identity, named places, plan templates.
Make the robot's own stack reproducible (drivers / MoveIt / Nav 2), at minimum by pinning the branch a fresh host builds from.
Run the servers on the robot, join the tailnet, and launch a brain instance for it (Running a brain).

A new skill is layer 3 — a self-contained competence behind one tool:

Clone the skill-service-template into a new repo.
Implement one high-level MCP tool whose body runs the closed loop internally, calling the contract underneath. Bundle its own model if it needs one.
Advertise prompt://system: when to use this skill versus others, its constraints, and its return fields — this is how the planner chooses.
Run it (on the robot, or off-robot if it is heavy) and attach it to the deployment.

7a. The skill-service contract¶

Every skill honours the same behavioural contract; grasp-service is the reference implementation and the template demonstrates each clause:

One high-level tool owning its loop internally, with structured {ok, …} returns and a specific reason on failure. ok is authoritative — never claim a success you did not achieve.
Patient and multi-approach. Try a different approach on each miss until it works or you genuinely cannot. A miss is not a failure.
Static first, then a recovery loop. Run the fast deterministic path; when it exhausts, convert into a scoped LLM tool-use loop over this skill's own tools (robot_mcp_kit.RecoveryAgent) before handing {ok:false} back to the brain. A recovery path may never fail the thing it recovers — an unusable backend degrades to "no recovery", loudly, and returns the original failure.
One skill, many approaches. Variation within a competence (top / front / two-arm) is a strategy the skill selects, not another tool and not another skill. The skill is an agnostic engine plus a robot strategy pack; the mechanics stay below the seam.
Default → override → learn. Run the default for the object's signature, let recovery override it, store the winning override keyed by that signature, and promote it once corrections converge — so the skill needs the LLM less over time.
A deadline is a safety net, not a schedule. Bound the whole skill with a generous wall-clock deadline checked between attempts. The real "that's too long" control is the operator's abort, which cancels the in-flight call and the live motion goal.
Fail fast on a wedge only. A hung downstream call is terminal and never retried; report it as {ok:false, reason:"timeout"}.
Stay promptly abortable and legible. Never catch CancelledError; emit a progress event per phase so a slow run is readable rather than a black box.

8. Glossary¶

Brain — the central LLM/VLM orchestrator (robot-voice-chat). Plans; never actuates.
MCP — Model Context Protocol; the control-plane RPC the brain speaks to every server.
prompt://system — an MCP resource a server exposes so its prompt travels with its tools; the brain composes it in at mission start.
World state — the one snapshot of body + cameras + scene, read by the brain, the skills and the watchdog.
Skill (L3) — a closed-loop competence behind one high-level tool; its own repo and MCP server.
Primitive (L4) — one atomic, self-verifying robot action.
The abstraction layer — the canonical MCP contract, and the highest layer that may hold robot-specific code.
Instance — one brainctl brain container pointed at one robot.