AuthorPascal DateJuly 29, 2026 Rev1.1

Running a brain (`brainctl`)¶

The single reference for operating brain instances: every command, every flag, brain.env, central memory, HTTPS. Every other page links here rather than explaining a brainctl line — the two exceptions are deliberate and one-way, both on Set up your own robot + sim: the single onboarding command that page exists to walk you through, and the same-host --robot forms, which this page delegates to it. Anything else that explains, defaults or qualifies a brainctl flag outside this page is a bug, not a second opinion.

The brain host already exists — it is the 4090

You almost never stand up a new brain host. ssh 4090 (user bot), the image is built, and instances are running: brain-pascal on :8001 is the primary and owns the installable PWA root at https://phyai4090.tailc7b34f.ts.net/. A shared brain-memory (:9103) is up alongside them.

Onboarding a colleague or a new robot is one command on that host — see Launch an instance. Building a brain host from scratch is an exception; the prerequisites below are for that case and for understanding what the 4090 already has.

And keep the running instances running. Recreating a container kills that colleague's in-flight mission and drops their control lease, so name the instance you mean (brainctl update pascal) and use a scratch instance to test changes.

One brain instance = one Docker container (brain-<name>) = one robot, with its own port, own state, own planner config. Every instance runs the same image; all that differs is the flags it was started with. Adding a robot never means rebuilding or forking the brain — the robot advertises its own tools and prompt over MCP at runtime.

Standing up a brain (and a robot) from nothing, on your own machine or VM → Set up your own robot + sim.
Driving the brain once it runs (chat, Debug panel, Visual Command) → Using the brain.
The robot side (r1ctl, the R1's MCP servers) → the R1 robot stack.

Prerequisites¶

On the host that will run the containers:

Need	Why	Notes
Docker (daemon + CLI, ≥ 20.10)	one container per instance	`--add-host …:host-gateway` needs 20.10+. `docker compose` is not used.
`rsync`, `git`, bash ≥ 4	`build` stages a context; `update` pulls the sources	`brainctl` uses `declare -A` and expands possibly-empty arrays under `set -u` — macOS's `/bin/bash` 3.2 aborts with `a[@]: unbound variable`. `getent` is glibc-only. Treat it as Linux-only.
`brain.env`, or a key store	somewhere for the provider keys to come from	Since BIN-146 the chat provider is per instance, so no single key is unconditionally required. A provider whose key is absent is offered but not selectable, and says so in the UI — the brain boots either way. See keys.
The brain source, as git clones under `$BRAIN_SRC`	the image is built from them	`robot-voice-chat`, `robot-mcp-kit`, `robot-mcp-memory`. `robot-mcp-kit` is private — the host's git needs SSH access to the `binabik-ai` org.
Outbound network for the build	npm, PyPI, Debian, `ghcr.io` base images	the SPA is built inside the image (Node stage)
Tailscale (optional but recommended)	resolves the robot's name; publishes HTTPS	Without it you lose the HTTPS/PWA URL and must give `--robot` something this host can already resolve.

No GPU and no ROS — the brain container has neither. It is a lightweight orchestrator; any host with Docker will do.

$BRAIN_SRC does not default to where you cloned things

BRAIN_SRC defaults to $HOME/sage/orchestrator/src, which matches the 4090 and nothing else. Point it at the directory that actually holds the three source clones:

export BRAIN_SRC=/path/to/dir-containing-robot-voice-chat-robot-mcp-kit-robot-mcp-memory

The commands¶

brainctl build                    # build binabik-brain:latest (first time, and after source changes)
brainctl up <name> [flags]        # create/recreate one instance
brainctl down <name>              # stop + remove one instance
brainctl ls                       # every instance: name, port, robot, status + the URL to open
brainctl logs <name>              # follow that instance's logs
brainctl memory {up|down|logs|build}   # the one shared central-memory container
brainctl update [<name> …] [--dry-run] [-y]   # pull + rebuild + recreate selected instances

That is the whole surface. Two things people reach for that do not exist: there is no brainctl status (it is ls, aliased ps), and no restart (up on an existing name recreates it).

up is idempotent-by-recreate: it stops and removes any container of that name first, so re-running it is the normal way to change an instance's flags. Every one of these that stops a container gives it a grace period to shut down rather than killing it.

Re-running up without --port moves the instance to a new port

The auto-assigned port skips every port already claimed by a brain container — including the one you are about to replace. So up pascal --robot rap-1 on an existing brain-pascal lands it on 8002, not 8001, and whoever had the old URL bookmarked is now pointed at nothing. Always pass --port explicitly when recreating by hand. (update replays the recorded port, so it is stable there.)

What the build actually bakes in¶

The git pin in robot-voice-chat/pyproject.toml is not what ships. build rsyncs the build host's own robot-mcp-kit checkout into the context and installs it editable from there, so whatever is checked out beside you on the 4090 is what ends up in the image. build therefore opens by naming both commits going in, and fetches each checkout before measuring it:

==> source going into the image (this, not the git pin in pyproject.toml, is what ships)
   robot-voice-chat: 8a1c4f2 (up to date with origin/main)
   robot-mcp-kit: 5e59db1 — 12 commit(s) BEHIND origin/main
   WARNING: a checkout above is behind its remote, and it is baked in VERBATIM.

A behind checkout warns and continues. Building against an unpushed local kit is a legitimate thing to do, and a build that refused would only get worked around; dirty is reported the same way, without a warning, for the same reason.

A stale kit crash-loops the brain, and blames the app

The symptom is ModuleNotFoundError: No module named 'robot_mcp_kit.keys' on start. Two things make it vicious. It names the application's import, not the deploy environment, so the obvious investigation — check the pin, find it correct, check the lock, find it agrees — ends nowhere. And it is invisible until someone rebuilds: every already-running brain keeps working, having been built before the divergence, so the bomb goes off for whoever builds next. That is what BIN-412 was. git -C $BRAIN_SRC/robot-mcp-kit pull is the fix; the warning names the error so the two connect.

The fetch is not decoration. git rev-list HEAD..origin/main against a tracking ref nothing has fetched compares HEAD with itself and reports 0 for an arbitrarily old checkout — the same trap that let the sim tree read as current while sitting 124 commits back on a branch that had been deleted upstream (BIN-478). A behind-count that can only ever say "up to date" is worse than none, so a fetch that fails is printed as the fetch FAILED so that count may be stale rather than presented as a measurement.

Both commits are stamped onto the image as well, so the question survives the scrollback:

docker image inspect binabik-brain:latest --format '{{json .Config.Labels}}' | jq

gives binabik.src.robot-voice-chat and binabik.src.robot-mcp-kit. To ask a running instance instead, inspect its container's image — which is the only way to answer "which kit is in this brain?" for an instance that has been up since before the last rebuild.

update rebuilds too, so it prints the same report; read it there before assuming a recreate shipped what you pushed.

Launch an instance¶

On the 4090 the image is already built, so onboarding a colleague or a new robot is a single command:

ssh 4090
cd ~/sage/orchestrator/brain-host
./brainctl up <name> --robot <their-tailnet-host> --port <free port ≥ 8001>
./brainctl ls                                  # prints the HTTPS URL to hand them

That is genuinely all of it for the converged R1: robots/r1.yaml is the built-in default manifest, so you do not have to name it. (./brainctl build is only needed on a host that has never built the image, or after changing the brain source.)

Give each instance its own --port and leave the existing ones alone — brain-pascal on :8001 is the primary. To try something out, use a scratch instance (brainctl up scratch --robot rap-1 --port 8003) rather than recreating someone else's.

What the manifest does. It lists the MCP servers this brain attaches, each URL parametrized by the robot host — r1-abstraction (:9220), grasp (:9210) and the central memory server. One file serves every robot; --robot fills in the host. Resolution order:

an explicit --robots-file <path>;
else robots/<name>.yaml, if that instance has its own;
else the default robots/r1.yaml.

A relative --robots-file is fine — brainctl resolves it to an absolute path before handing it to Docker.

The world service (:9240) is not on the manifest — the planner reads it another way

The brain reaches it over its own connection, configured by world_state: in the brain's config.yaml, not as one of the planner's MCP servers. brainctl already injects ROBOT_HOST, so attaching a robot stays one flag with no manifest edit, and the World panel, the planner's injected body block and the watchdog all reach it without an entry.

The planner does get to read it on its own initiative (BIN-392) — through a brain-local read_world_state tool, naming the cameras it wants. Deliberately not through a manifest entry: that would put world_state(refresh=true) on the model's tool surface, and the paid capture must stay unreachable from a model's own initiative. The local tool has no refresh in its schema at all, which is the difference between a parameter a model cannot pass and one it might be talked into. The per-layer rule this belongs to is on one page — the world-state spec.

For a non-R1 robot, write a manifest

The default manifest names the R1's servers. A different robot needs its own robots/<name>.yaml listing its servers (same shape: name, url, blocking, optional header/event_secret/prefixed). Without one you would attach the R1's ports to a robot that isn't serving them.

Your manifest is now the only source of blocking tools — gen_mcp_config.py adds none of its own, where until BIN-543 it merged 26 R1 tool names under every manifest. What that means for a robot developer is on the page that owns it: the contract.

--robot is what makes the manifest resolve

Every URL in r1.yaml is http://${ROBOT_HOST}:…, and ROBOT_HOST comes from --robot. up refuses to start with neither --robot nor --server. But if you pass only --server while the R1 manifest is still in play, ${ROBOT_HOST} expands to nothing and the brain attaches a meaningless http://:9220/sse. Pass --robot whenever the manifest has robot-hosted servers in it.

--robot-id is a different fact from --robot, and a tailnet robot needs both (BIN-494)

--robot is a host to reach; --robot-id is the label the fleet knows that robot by, and it has to equal the robot's own ROBOT_ID (the one in its r1.env — see the robot stack runbook). It defaults to --robot's first DNS label, so rap-1.tailc7b34f.ts.net gives rap-1 — which is a guess at a fact only the robot holds, so up prints the label it chose. Pass --robot-id when that guess is wrong.

They cannot always match. The robot spends its ROBOT_ID on robot_mcp_kit.keys.KeyScope, which expands it into a Secret Manager id (robot__<id>__ANTHROPIC_API_KEY) and therefore validates it as ^[A-Za-z0-9_-]+$. A tailnet FQDN has dots, so a robot handed its own FQDN as ROBOT_ID raises inside credential resolution before it can bill anything — the robot must keep a short label, and the brain must be told that label.

Both costs of getting this wrong are silent. The ledger drops every usage row that robot reports (reported_usage_rejected, now at ERROR with the fix in the line), so its spend is invisible; and the planner asks central memory for skill lessons under a name the robot never files them under, so it plans as though the robot had learned nothing (BIN-279, with no log line at all). Per-robot memory tokens are keyed the same way — the MEMORY_ROBOT_TOKENS map ("rap-1:tokA") is looked up by the fleet id, not the host.

# robot's r1.env:  ROBOT_ID=rap-1
./brainctl up pascal --robot rap-1.tailc7b34f.ts.net --robot-id rap-1 --port 8100

Until BIN-587 the default was the whole host, and a dotted one was fatal rather than silent. KeyScope.from_env() raises at import time in the brain's main, so the container never served anything: docker ps showed Restarting (1), forever, over an error naming neither --robot nor brainctl. Two production instances still carried a dotted brain.robot label, and brainctl update is fleet-wide by construction — so the next bare update would have recreated both onto the current image and taken both down. brainctl now refuses an unusable --robot-id before creating the container, the way it already checks --port and the memory auth combinations.

Running the brain on the same machine as the robot?

Then --robot needs care: the container is on a bridge network, so localhost means the container, and a brain started with --robot localhost comes up reporting ready with zero robot tools and no warning. The forms that work are in Set up your own robot + sim.

Verify it came up attached, not merely healthy:

curl -s http://localhost:8001/api/readyz | jq .

Expect status: ready, tools_connected: true, and each server in mcp_servers true. Note the route is /api/readyz — /api/admin/readyz returns the SPA's HTML with a 200 and reads as a false pass.

readyz is not a robot-liveness probe

The per-server booleans say the brain holds a session object, not that the robot still answers. When a robot vanishes without a clean TCP close — relay drop, host powered off — the session lingers and the probe keeps reporting ready for a robot that is completely gone. To test the robot, probe the robot:

curl -s -o /dev/null -w '%{http_code}\n' --max-time 6 http://<robot>:9220/sse   # 200 = serving
tailscale ping -c 2 <robot>                                                     # pong = path OK

Every `up` flag¶

brainctl up <name> — <name> is required and becomes the container brain-<name>.

Flag	Arg	Default	What it does
`--robot`	host	from the manifest's `robot:` key, else none	The robot host — where to reach it. Fills `${ROBOT_HOST}` in the manifest and is resolved to an IP for the container's `/etc/hosts`. Required unless every server is a `--server` URL.
`--robot-id`	label	`--robot`'s first DNS label	The robot's fleet identity — the scope its spend is booked against and its lessons are pooled under. Must equal that robot's own `ROBOT_ID`. Validated as `^[A-Za-z0-9_-]+$` before the container is created (BIN-587). See the warning below: for a tailnet-addressed robot this cannot be the host.
`--port`	N	first free ≥ `BRAIN_PORT_BASE` (8001)	Host port → container's 8000.
`--robots-file`	path	`robots/<name>.yaml`, else `robots/r1.yaml`	The attach manifest. Relative paths are resolved for you.
`--secrets-file`	path	`secrets/<name>.env` if it exists, else none	This instance's identity at the key store, layered after `brain.env` so a per-instance value wins. There is deliberately no shared fallback file.
`--profile`	tag	(empty)	Robot capability/model tag (`ROBOT_PROFILE`). Stamped on every recovery lesson this instance stores and used as the retrieval filter (BIN-545): a tagged lesson only comes back to a brain carrying the same tag, an untagged one comes back to any. Empty — the default — is the generic profile: it stores generic lessons and receives generic lessons only. Changing it therefore retires the lessons tagged for the old robot rather than handing them to the new one.
`--planner`	`anthropic`\|`groq`	`anthropic` — from `config/config.yaml`, not from which keys are present	Planner LLM provider group. Before BIN-239 the entrypoint sniffed `ANTHROPIC_API_KEY` and silently switched to Groq without it; it no longer does. A planner with no key refuses missions and names the variable.
`--planner-model`	model id	provider default	e.g. `claude-opus-4-8`.
`--set`	`key=value`	—	Any Hydra override, repeatable, e.g. `agent.max_llm_steps=30`. Validated against the config schema at startup — an unknown key fails the instance loudly.
`--no-models`	—	every reasoner on	Switch all four reasoners off — the model-free instance. Nothing on this brain can reach an LLM provider; the Programming view is the whole UI. See Running a brain with the models switched off.
`--no-chat-llm`	—	on	No conversational model: the Chat view is not rendered at all.
`--no-planner`	—	on	No mission planner: removes both Chat and Visual Command. Stored plans still run. Not the negation of `--planner`, which picks the planner's provider.
`--no-watchdog-observer`	—	on	No paid mid-mission VLM eye. The free body checks keep running.
`--no-watchdog-body`	—	on	No free deterministic body checks. A real safety reduction — ask for it deliberately.
`--server`	URL	—	Attach an MCP server at any URL, repeatable — beyond the manifest. For something the robot can't host (a cloud VLA). Auth via `EXTRA_MCP_HEADER` in `brain.env`. Omit `--robot` entirely for an all-remote instance.
`--no-memory`	—	memory on	Don't attach the shared central memory.
`--memory-scoped`	—	admin	Authenticate to memory with this robot's token → sees only its group. Needs `MEMORY_ROBOT_TOKENS`.
`--memory-admin`	—	(this is the default)	Authenticate with the fleet-wide `MEMORY_TOKEN`. Required for the Fleet admin.
`--primary`	—	off	Own the Tailscale-Serve `:443` root — the installable PWA URL.
`--https-port`	N	443	Serve this instance on a distinct HTTPS port.
`--no-https`	—	off	Keep this instance off Serve entirely (plain HTTP only).
`--overrides`, `--servers-list`	string	—	Internal. `update` uses them to replay an instance's full override/server string.

--primary is mutually exclusive with --no-https, and with any --https-port other than 443; brainctl refuses those combinations rather than letting them half-work.

Every flag is stored as a container label, and update reads them back — so per-instance config survives a rebuild without you re-typing it.

Configuring an instance¶

Same image, different flags:

brainctl up alice --robot alice-pc --port 8002 \
  --planner anthropic --planner-model claude-opus-4-8 \
  --profile r1pro \
  --set agent.max_llm_steps=30

So each robot can run a different planner LLM with no rebuild. Requirements: the provider's key must be in brain.env, and model ids follow the Claude API list (Opus 4.8 = claude-opus-4-8).

Verified working: --planner, --planner-model, --set llm.model=…, --set agent.max_llm_steps=…. A --set on an unknown key is rejected by Hydra at startup — check config/schema.py in robot-voice-chat.

How prompts are composed¶

At mission start the system prompt is assembled in layers:

Universal base — brain-wide planning rules, baked into the image, generic.
Robot fragments — identity, tool descriptions, plan templates, manners. Advertised by the robot over MCP: each server may expose a prompt://system resource the brain fetches and composes in, so a robot ships its own prompt with its tools. (The R1's r1-abstraction serves its planner prompt this way.) The image bakes only a robot-agnostic default.
Runtime context — operator facts, known waypoints, learned lessons.

To change one: the universal base → edit robot-voice-chat, then update (everyone). A robot's own prompt → edit that server's prompt://system and restart that server — no brain redeploy, and it only affects that robot. A per-instance tweak → --set.

The chat profiles, and which file defines them¶

The conversational layer is separate from the planner prompt above, and comes from one file in the image: prompts/system_prompts.default.yaml in binabik-brain-host. entrypoint.sh force-sets scenarios_file to it on every instance, which is the part that catches people — a profile added to robot-voice-chat's own backend/config/system_prompts.yaml never loads in a deployed brain. (That file's four Unitree G1-era personas are dead for exactly this reason, and are marked retired.)

Since BIN-419/BIN-485 a profile carries capability as well as text. Per profile:

Key	Default	Meaning
`extends`	none	compose from another profile — its prompt is prepended, so shared text is written once. An unknown or circular name fails the container at startup rather than yielding a short prompt
`abstract`	`false`	a base that exists only to be extended: never selectable, never in the picker
`unlocked`	`[]`	the allow-list of features this profile has. Anything not named is locked. `[""]` unlocks everything including features that do not exist yet*, so only the developer profile should carry it
`starters`	`[]`	the use cases this profile offers — shown as clickable options when a chat opens
`skill_library`	`null`	which skills the Programming view's palette offers under this profile (KOE-26) — tool names as the robot advertises them. Absent means "no opinion", so `SKILL_LIBRARY_VISIBLE` and then the shipped curated set answer. Unlike `unlocked` this is visibility, not capability: a hidden skill still validates and still runs inside a stored plan, and a name no attached server advertises is simply never shown. Because this file is host-mounted, a customer profile and the developer profile can offer different palettes on one brain with no redeploy. What an operator sees: using the brain

unlocked defaults to empty, so a profile that forgets it gets the customer treatment. That direction is deliberate, and so is the choice of an allow-list over a deny-list: a demo that cannot do enough is noticed in a minute, whereas a customer instance quietly writing recovery episodes is noticed by nobody — and a feature added next month is locked everywhere until somebody says otherwise, rather than opening itself to every customer by default.

The features, and what locking one looks like¶

The canonical list lives in one file — robot-voice-chat's backend/src/robot_voice_chat/models/features.py — which is also where a new feature is added. An unknown key in a profile fails the container at startup, because a typo must not present as a feature that is deliberately locked.

Key	Locked looks like	Covers
`robot_control`	—	making the robot act at all: `start_mission`, and the raw robot tools on a brain with no planner
`recoveries`	Coming soon	the "Remember this run" control, the paid distillation, Admin → Recoveries
`fleet`	Coming soon	Admin → Fleet, group lessons, and announcing this robot at startup
`cost`	hidden	the spend chip in the top bar and Admin → Cost
`config`	hidden	Admin → Config and the Config tab in the main UI's right-hand rail. One feature, both surfaces — they answer the same question about the instance. `GET /api/config` itself stays ungated; the app boots from it
`waypoints`	hidden	the Waypoints tab in that rail, and the `/api/waypoints` + `/api/robot_pose` routes behind it
`demo`	hidden	the Demo events card on Admin → Overview, which injects tool-call cards the robot never produced
`credentials`	hidden	Admin → Credentials
`api`	hidden	Admin → API. Unlocked for the customer profiles: Swagger UI and ReDoc are a fair demonstration that the thing has an interface, and `/openapi.json` is filtered to the active profile's unlocks — so a locked feature's endpoints are not published in it, and `/docs` + `/redoc` follow because both fetch it. A profile that unlocks everything gets the schema untouched, byte for byte, since that is what client generators read
`logs`	hidden	Admin → Logs
`stats`	hidden	Admin → Statistics (the KPIs)
`missions`	hidden	Admin → Missions and per-mission traces
`skills`	hidden	Admin → Skills

The two customer profiles ship with unlocked: ["missions", "stats", "api"] — the mission history, the KPIs and the API reference are worth showing a demo audience — and everything else locked.

Branding a profile¶

A profile may also carry a brand, so a customer-facing profile looks like theirs. Every field is optional and absence degrades cleanly: no accent means our red, no logo means our logo alone.

brand:
  accent: "#0060ff"                  # must be #rrggbb; anything else FAILS AT STARTUP
  logo: "/images/logos/koerber.svg"  # a file in robot-voice-chat's frontend/public/
  logo_alt: "Körber"
  font_family: "Noto Sans"           # a face the browser can already get

The accent replaces --primary-rgb. The darker shade used for text on a primary tint is derived from it — the accent at 70% over black, the relationship index.css documents — rather than configured, because two values could drift out of that relationship and the symptom is unreadable text on a tint. An invalid colour is refused at startup rather than passed to CSS: a --primary-rgb the stylesheet cannot parse does not fall back to red, it renders every primary-coloured element transparent.
The logo appears as a Binabik ✕ <customer> lockup, not a replacement — it is our brain running their use case. A missing file leaves our logo standing alone rather than a broken image.
font_family cannot ship a customer's corporate typeface. That is a licensing question, not a code one: "Font Koerber" and "Audi Type" are licensed faces, and naming one here would render as nothing on almost every machine. It names a family the browser already has — Körber's body face, Noto Sans, is open-licensed and is requested alongside the app's own — prepended to the stack so an unavailable family degrades to ours.
A brand is not inherited through extends. The shared base carries shared text; a customer's colour is the least shared thing there is.

Two treatments, and the difference is the point. Coming soon keeps the capability visible and unreachable — a customer should see that fleet learning exists. Hidden removes it outright, for machinery and money: a blurred Cost page still tells a customer we are counting dollars, and a blurred Credentials page still names our providers.

The lock is enforced on the routes, not only in the UI. A locked feature's endpoints answer 409, so a removed button is not the only thing standing between a customer and a fleet write. GET /api/cost/summary is gated too — it is the one cost route with no admin gate, on the grounds that an operator must see spend from anywhere, and that reasoning was never about showing it to a customer.

Which profile an instance boots with is the first non-abstract entry in the file, and that one matters beyond the picker: fleet registration happens once at startup and reads it. A real customer instance should therefore have their profile first, either in a file of its own or via up <name> --set scenarios_file=/opt/prompts/<yours>.yaml. What each profile does once the brain is running, and what the operator sees, is using the brain.

Running a brain with the models switched off¶

Four models think in a brain, and since BIN-527 each can be absent rather than merely unused:

Reasoner	Flag	`brain.env`	What goes with it
Chat LLM	`--no-chat-llm`	`REASONER_CHAT_LLM=false`	The whole Chat view — no feed, no composer, no voice path in
Mission planner	`--no-planner`	`REASONER_PLANNER=false`	Chat and Visual Command; stored plans keep running
Watchdog eye (paid VLM)	`--no-watchdog-observer`	`REASONER_WATCHDOG_OBSERVER=false`	Its arm/disarm control, observer window, cost banner and findings
Watchdog body checks (free)	`--no-watchdog-body`	`REASONER_WATCHDOG_BODY=false`	The deterministic checks: e-stop, a declared fault, stale joints, frozen-while-moving, `at_limit`

--no-models is all four at once — the configuration a customer instance ships as:

brainctl up koerber --robot rap-1 --port 8010 --no-models

Why four keys and not one. The watchdog is two halves on different terms: the eye costs money per look and is armed by hand, the body checks are free contract reads that run every tick. One shared flag is how the default deployment ended up watching nothing (BIN-377), so switching the eye off deliberately leaves the body checks running. "No models" must never silently mean "no supervision".

Every switch is a veto, and nothing re-enables. A reasoner is available only if this instance booted it, the active chat profile names it, and its dependencies hold. A profile carrying unlocked: ["*"] — the Binabik developer profile — does not get the planner back on a brain started with --no-planner. There is no "which one wins" rule to remember, because none of them wins.

planner is the root of the chain, so --no-planner alone is model-free: the chat model goes with it (a chat view that cannot command the robot is not a view we render), and so do both watchdog halves, because a watchdog exists to pull the brain back in to replan.

What a planner-free brain gives up

With no planner there is no watchdog either, so a hard body fault — e-stop, self-collision, a declared driver fault — no longer aborts a running stored plan. For a supervised but model-free instance, leave the planner on and switch off only the chat model and the eye:

brainctl up koerber --robot rap-1 --no-chat-llm --no-watchdog-observer

The planner is then built and never asked to plan, and the free body checks keep watching.

A switched-off reasoner has no client at all. The brain installs a stand-in instead of constructing the provider, so no key is resolved and no SDK handle exists — the instance cannot reach that provider even by a code path nobody thought of. It also means the credential is not required: a brain started --no-planner needs no ANTHROPIC_API_KEY.

The flags survive update. They are recorded in the same override string as --planner and --set, so a model-free instance stays model-free across a deploy. Repeating an off switch is fine; contradicting one (--set reasoners.planner=true --no-planner) is refused up front.

What the operator sees on such an instance — which views disappear, and the read-only Reasoners card that says whether a model is off at boot or locked for the profile — is using-the-brain.md.

`brain.env` — keys and shared settings¶

One file, <brain-host>/brain.env (gitignored), shared by all instances on the host. up refuses to start without it. Override the location with BRAIN_ENV.

Every key below can instead live in a key store, fetched at runtime per instance — see Per-instance secrets. brain.env remains the fallback, per instance, so a host can migrate one brain at a time.

Var	Required?	What
`OPERATOR_KEY`	Yes — the instance will not start without it	The credential every client of this brain must present, the browser's own Socket.IO connection included (BIN-534). At least 16 characters; generate with `python -c 'import secrets; print(secrets.token_urlsafe(32))'`. There is no unauthenticated mode: an unset or too-short key makes the container exit at startup with a message naming this variable. One key per deployment, shared by its operators. Full detail, and why it fails closed: Who can drive a brain.
`GROQ_API_KEY`	For the default chat provider	Groq is the default chat provider, so without a key the chat role has none. The brain still boots: the role answers with "this provider has no credential", Groq is greyed out in the config panel, and any provider whose key is present stays selectable.
`ANTHROPIC_API_KEY`	For the default planner	Anthropic is the default planner. Without a key the brain boots and the UI works, but a mission is refused up front, naming the variable — it does not silently fall back to Groq any more (BIN-239). Either set the key, or pick a planner that has one from the config panel.
`MEMORY_TOKEN`	Optional	Bearer token for the central memory. Must match the robot side (`r1.env`).
`MEMORY_ROBOT_TOKENS`	Optional	`"rap-1:tokA,rap-2:tokB"` — per-robot tokens for multi-tenant scoping. Setting it turns auth on for every connection, so `MEMORY_TOKEN` must be set too.
`RVC_EVENT_SECRET`	Only for a server that pushes events	HMAC secret for `POST /api/agent/events`. A pushing server signs with it and the manifest entry carries the same value as `event_secret`; without a match the brain 401s every push. No attached server pushes today — the one that did, the retired perception buffer, used `PERCEPTION_EVENT_SECRET` (BIN-306).
`RVC_USAGE_SECRET`	Only for a robot-side spender that pushes its spend	HMAC secret for `POST /api/agent/usage` (BIN-240 phase 2) — a robot's grasp recovery tier, vision, or segmenter reporting a paid call the brain didn't ask about. Fleet-wide, not per-server: unlike `RVC_EVENT_SECRET`, no manifest `event_secret:` entry is involved, and the same value must match the robot side's own `RVC_USAGE_SECRET`. Without a match every push 401s — the return-field channel (a tool result carrying its own `usage` key) needs no secret and still works.
`EXTRA_MCP_HEADER`	Only with `--server`	`Authorization: Bearer …`, sent to flat `--server` URLs.
`REASONER_CHAT_LLM` · `REASONER_PLANNER` · `REASONER_WATCHDOG_OBSERVER` · `REASONER_WATCHDOG_BODY`	No	`false` switches that reasoner off for every instance on this host — the shared-file version of the `--no-*` flags. Prefer the flags: this file reaches every brain, and a model-free instance is normally one instance's decision. Where each key belongs, and the planner ⇒ everything dependency: above.
`SKILL_LIBRARY_VISIBLE`	No	The Programming view's skill palette for every instance on this host — a comma-separated list of tool names, e.g. `"navigate_to_named,drive_by,grasp,place"`. Unset means the shipped curated seven. It is only a default: a per-profile `skill_library` list overrides it, and an operator's ticks in Config → Skills override both, per profile, stored on the brain. Visibility only — it can neither refuse a stored plan nor offer a tool no server advertises (using the brain).
`ADMIN_PASSWORD_HASH`	Only for the admin UI	Argon2 hash, not a plaintext password (plaintext is rejected and logged as an error). Generate with `python -m robot_voice_chat.scripts.hash_password`. Without it the admin area stays disabled. Not in `brain.env.example` — a documentation gap in that file.
`ADMIN_SECRET_KEY`	Optional	Unset → admin tokens are ephemeral and die on restart.

Two things brain.env cannot control, because brainctl passes them as explicit -e flags that win over the env file: the per-instance values it derives itself (MEMORY_URL, ROBOT_ID, ROBOT_HOST, BRAIN_NAME, PERCEPTION_URL, EPISODIC_URL, EXTRA_MCP_SERVERS, ROBOT_PROFILE, …), and ROBOT_CUSTOMER/ROBOT_FACTORY, which are read from the operator's own shell environment — put them in brain.env and they are silently overwritten with empty. Export them in your shell instead.

ROBOT_ID no longer depends on central memory (BIN-240)

It used to be injected only when memory was attached, because register_robot was its only reader. It has three others now — the "which robot am I controlling" chip, the mission payload, and the cost ledger's robot scope — and none of them involve memory. So a brain started --no-memory used to record spend with a null robot, which is a legitimate state for a spender that has no robot and therefore said nothing was wrong. Since a per-robot budget is set against exactly that field, the cap would have had nothing to enforce against.

ROBOT_CUSTOMER and ROBOT_FACTORY are still memory-only, deliberately: register_robot remains their only reader and it is skipped without a memory server.

BRAIN_NAME is brain-<name> — the same string as the container and as the first column of brainctl ls. It is how an instance identifies itself to a service whose budget is shared between instances. The robot's world service meters scene captures per robot, so every brain attached to that robot draws on one ceiling, and it records who spent what in budget.by_caller. Before BIN-415 the brain sent no name and an operator's own refresh landed as unattributed — the one value that answers nothing when four instances share a 40-capture budget.

Never put a per-robot URL in brain.env

brain.env is shared by every instance on this host, so a URL that belongs to one robot puts all of them on that robot. Anything robot-specific is derived from --robot — via the manifest's ${ROBOT_HOST}, or a flag that builds the URL for you. (The retired --perception / --episodic flags were the instance of this rule that got written down; the rule outlives them.)

Who can drive a brain — the operator key¶

A brain can move a robot and spend money. Until BIN-534 the answer to "who is allowed to" was anyone who could open a socket to it: the Socket.IO handshake took an auth argument and never read it, POST /api/plans/{id}/run launched a stored plan on real hardware with no credential, and one anonymous POST /api/scenarios/active switched the active profile — which is what every feature gate resolves against, so it unlocked the whole allow-list on a customer instance.

Now every client presents a credential.

Setting it up¶

One line, once, in brain.env — it is shared by every instance on the host, so this covers all of them:

echo "OPERATOR_KEY=$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" \
  >> ~/sage/orchestrator/brain-host/brain.env

To give one instance its own key, put OPERATOR_KEY in its --secrets-file instead. brainctl layers that file after brain.env and docker applies --env-file in order, so a per-instance value wins over the shared one — the same layering the key store uses for provider keys.

Then, per person, per device: open the brain, paste the key into the prompt once. It is kept in that browser's localStorage, so an installed PWA does not ask again. Somebody who has the admin password can sign in under Admin instead and never needs the key — an admin session is issued against a password and is accepted everywhere the operator key is.

What it does and does not buy¶


HTTP header	`X-Operator-Key`. A header rather than a cookie, so it cannot be attached by a cross-site form post, an `<img>` tag or a prefetch
Socket handshake	`io({ auth: { key, bearer } })` — either the operator key or an admin bearer authenticates
What it gates	Everything that moves the robot, edits what the planner is told, or spends money. Plus the control lease: its members are authenticated clients, so "cooperative takeover" is now between operators rather than open to a stranger
*What needs admin* instead**	Switching the active profile (`POST /api/scenarios/active` and the `change_scenario` socket event). The operator key is shared by every client of a deployment, and whoever sets the profile unlocks every hidden feature on a customer instance. Set `ADMIN_PASSWORD_HASH` to be able to switch profiles
What stays open	The app shell — the prompt that asks for the key lives inside it — and the free world reads (`GET /api/world_state`, `GET /api/world_frame`). They are free ROS reads that the World panel polls; a credential there would gate the panel without saving a cent
What it is not	Per-user identity. One key per deployment says "you are an operator here", never which operator, and it cannot tell two colleagues apart. That is the Binabik Console's job (BIN-364)

The two paid world reads are POSTs for the same reason the credential is a header: POST /api/world_state/capture and POST /api/world_frame/annotated. The old refresh=true and annotated=true query flags are gone — a GET that runs the grounder is reachable from a link, a prefetch or a crawler, which made the cost guard a convention about who calls the endpoint rather than a control.

It fails closed, and that is deliberate

An instance with no OPERATOR_KEY exits at startup instead of serving an open socket behind a warning. Under --restart unless-stopped it will crash-loop; docker logs brain-<name> names the variable and prints the command to generate a value.

So forgetting the line takes every brain on the host down, loudly. That is the intended direction of the trade. The alternative — warn and serve anyway — is this codebase's recurring defect written on purpose: BIN-347's paid default, BIN-359 and BIN-377 were all guards that were present but inert, and a credential check that switches itself off when unconfigured is exactly that shape. A brain that is down gets fixed in a minute; a brain that is quietly open does not get noticed.

The threat model, stated out loud¶

Worth writing down because the code used to claim one thing and the deployment did another — main.py's own docstring says "in production, bind to 127.0.0.1 and front the app with a TLS reverse proxy", and no deployed instance does.

What the credential fixes: the audience for "drive this robot" is no longer everything that can route to the 4090. brainctl publishes with -p "$port:8000", which binds 0.0.0.0 — so the plain-HTTP port is reachable from anything on the host's network, not only through tailscale serve. Before BIN-534 that was an unauthenticated remote control with a spend button on it; now it is a locked door on a port that is wider than it should be.

What is still true, and is a separate issue: the port is still published on 0.0.0.0, and server.environment is dev with allowed_origins: ["*"] on every instance. Binding the container to loopback and letting Serve be the only ingress belongs to brainctl (BIN-560 — it is not a robot-voice-chat change), and the deployment-preset half is BIN-539. The honest summary of today's posture is:

Anyone on the tailnet, or on the 4090's network, can reach a brain — and now needs this deployment's operator key to do anything with it. Six brains share one host and one robot, and one key per deployment cannot tell them apart, so the key is a perimeter, not an audit trail. Attribution of spend is separate and does work: each brain sends BRAIN_NAME as the caller, so the robot's capture budget records who spent what.

Per-instance secrets — the key store¶

brain.env is shared by every instance on the host, which means every instance runs on one Anthropic key: spend cannot be attributed, and one customer's key cannot be isolated from another's. --secrets-file gives each instance its own identity at a secret store instead, so two brains on the same host can read different customers' keys and neither can read the other's.

cp secrets/example.env secrets/pascal.env
chmod 600 secrets/pascal.env          # it holds a credential; brainctl warns if you skip this
$EDITOR secrets/pascal.env            # backend + bootstrap identity, NOT the API keys
./brainctl up pascal --robot rap-1    # picked up by name, like robots/<name>.yaml

What goes in the file is the bootstrap credential and the choice of store — never the API keys themselves. That split is the whole point. Docker env is create-time (BIN-78), so anything injected at docker run can only change by re-creating the container; a key the registry holds is refreshed in place, which is what makes "rotate without a rebuild" reachable at all. Put an ANTHROPIC_API_KEY in this file and you have simply moved the old problem to a new path.

Backends are chosen by BINABIK_SECRETS_BACKEND: env (today's behaviour, keys straight from brain.env), google (Secret Manager), or infisical. Unset falls back to the environment; an unrecognised value is fatal — a deployment that configured a store and then typo'd its name must not quietly resume reading brain.env and look healthy doing it.

Details worth knowing:

No shared default. Discovery is --secrets-file, then secrets/<name>.env, then nothing. A secrets/default.env fallback would quietly hand every instance one identity — the host-level identity this replaces. An instance with no file keeps reading keys from brain.env, which is the supported mid-migration state, per instance.
brain.env first, the identity file second. Docker applies --env-file in order, so a per-instance value wins while everything an instance has not moved to the store yet still comes from the shared file.
The label is the path, never the contents. docker inspect prints labels; the credential stays in the file. brainctl update replays the path, so a recreate keeps the identity.
Google ADC names a file. If the identity sets GOOGLE_APPLICATION_CREDENTIALS, brainctl mounts that file read-only at the same path inside the container, because injecting the variable alone would point at nothing — and the brain would then start with every credential silently falling back to brain.env. A missing path is fatal here rather than a puzzling ADC error from inside the container. On GCP itself, omit it: the workload's service account needs no file.
No service-account key? Impersonate one. Where an org policy blocks key creation, gcloud auth application-default login --impersonate-service-account=<sa> gives a file with no long-lived key in it while keeping the instance's identity, so per-customer isolation still holds. Logging in as yourself works too and is a dev-box answer only: the brain then acts as you, with everything you can read, and the credential expires. secrets/example.env in binabik-brain-host has all three routes side by side.

Provisioning it: `provision-identity` (BIN-503)¶

Until BIN-503 nothing created this identity — secrets/example.env delegated the account, the grant and the key to a gcloud recipe run by hand, and it had been run once, for a single shared project-wide admin every instance then read the same secrets with (BIN-482). provision-identity is robot-mcp-kit's CLI front end over its provisioning library; r1ctl's first run is the robot-side twin over the same code, and the brain's Credentials tab (BIN-429) is the third — it provisions on first write, the same call below, over the operator's own forwarded browser token rather than a pasted one.

provision-identity provision --project-id binabik-dev --robot-id pascal --kind brain

From the brain host itself, brainctl provision <name> is the front door — it runs this exact call from the robot-mcp-kit checkout brainctl already knows about ($BRAIN_SRC, via uv run) and, for --kind brain (its default — provision-identity's own default is robot, because r1ctl's first run is its more common caller), writes a ready-to-use secrets/<name>.env on success: BINABIK_SECRETS_BACKEND=google, GOOGLE_SECRETS_PROJECT, and GOOGLE_APPLICATION_CREDENTIALS pointing at the new key, mode 600. A plain ./brainctl up <name> … then picks it up with no by-hand editing:

./brainctl provision pascal --project-id binabik-dev
./brainctl up pascal --robot rap-1

It defaults --robot-id to the instance's own brain.robot_id label when the instance is already running (else the instance name — BIN-494's distinction applies here too), and remembers --project-id in secrets/<name>.env so a re-run to rotate the key needs no flags at all: ./brainctl provision pascal. Where key creation is blocked by org policy it prints the exact gcloud commands to finish the impersonation grant by hand (constraint 2 below) instead of guessing, and writes nothing until a real key exists. See binabik-brain-host's README ("Per-instance secrets at the store") for the full flag list.

What it produces, all idempotent (safe to re-run after a half-finished attempt):

a service account for this instance alone (brain-<name>@<project>.iam.gserviceaccount.com);
one conditional secretAccessor binding covering only this instance's own prefixes — robot__<name>__*, site__<name>__* with --site-id — plus the unprefixed company defaults, and nothing else (BIN-502 decision 3);
the identity itself: a key file, or an iam.serviceAccountTokenCreator grant where constraints/iam.disableServiceAccountKeyCreation blocks key creation — both are first-class outcomes, not a happy path and a fallback (BIN-482 constraint 2).

--kind brain, not the default, is the part worth noticing. One deployment gets two service accounts, not one shared key: a brain and its robot read the same secrets through the one binding above, but are separate principals, so either can be revoked, rotated and audited without touching the other. Passing no --kind provisions the robot's account instead — the flag defaults to robot because r1ctl's first run is the more common caller, not because it is the right default here.

Authority is the operator's, and transient — decision 4 (BIN-502): provision-identity holds no credential of its own. It shells out to gcloud auth print-access-token unless $BINABIK_OPERATOR_TOKEN or --token is given, and this host must never hold standing admin in order to obtain its own non-admin credential (BIN-482) — the same reason this is a separate, operator-run command and never a step brainctl up takes on its own.

Verified live against binabik-dev (BIN-503 step 5): a provisioned identity cannot create, overwrite or list secrets, and cannot read another instance's. The condition is keyed on the project number, not its id — an id-keyed condition matched nothing, and through the double-negated defaults clause that swung the whole expression universally true, granting every secret to every identity while reading as least privilege.

Reading the boot line¶

The brain reports its credential position once, at startup, before anything uses a key — the point being that an under-scoped identity says so while someone is still watching the boot, rather than answering 401 an hour later on the one call path that needed the missing key.

brain-pascal: 5/7 credentials (3 from store, 2 from environment) | absent (optional) OPENAI_API_KEY, LLM_API_KEY

docker logs brain-pascal 2>&1 | grep credentials — 2>&1 matters, the brain logs to stderr. Read it as: the store is reachable, three keys came from it, two are still coming from brain.env and have not been migrated yet. Two phrases are worth reacting to:

In the line	Means
`IDENTITY REFUSED — this will not fix itself`	The store answered and rejected this identity (401/403) — a missing IAM binding or the wrong service account, not an outage. It will never recover on its own.
`STORE MISCONFIGURED — this will not fix itself: …`	The brain could not ask: the backend's SDK is not in the image, or no project id / URL is set. Nothing reached the store at all, so no rotation can arrive. Logged at ERROR, unlike the two either side of it.
`store unavailable: …`	The store could not be reached — a genuine outage. The brain degraded onto `brain.env` and will pick the store up on the next refresh.
`n from environment` where you expected `from store`	That key has not actually been migrated. A single count would have hidden this, which is why the sources are reported separately.

STORE MISCONFIGURED is new in robot-mcp-kit 0.11.0, and it used to be indistinguishable from the row below it. That mattered: an outage's wording licenses waiting, and on rap-1 a permanent packaging fault wearing it cost a day of running on cached credentials (BIN-572). If you see it on a brain, the image is missing robot-mcp-kit[google] — a store backend's SDK is an extra, so depending on the kit does not install it, and uv sync removes anything undeclared. Declare it; do not uv pip install it.

Central memory (shared, automatic)¶

up also starts one shared container — brain-memory, the robot-mcp-memory server on :9103 — and attaches every brain to it over the host gateway (http://host.docker.internal:9103/sse). That is what lights up teaching mode, process recall, the Fleet admin (/admin/fleet), and group-scoped grasp learning. Nothing to hand-wire; data lives in $BRAIN_SHARED/memory.

Opt one instance out: --no-memory.
Manage the container: brainctl memory {up|down|logs|build}.
Rotating MEMORY_TOKEN takes effect on the next up/update — brainctl notices the drift and recreates brain-memory, printing memory auth changed in brain.env — recreating brain-memory. Confirm with docker exec brain-memory printenv MEMORY_TOKEN.
update leaves it alone unless the memory image actually changed. Recreating it drops the memory SSE stream of every brain, so a deploy of your own instance must not take everyone else's memory down with it (BIN-359).

It only started being true on 2026-09-04 — and your first update will still recreate once

That guard was written in BIN-359 and never once held (BIN-493). With buildx as the default builder, docker build tags a manifest list whose digest folds in freshly generated provenance, so the image id moved on every build even with all five steps CACHED. The comparison behind the guard was a constant, not a signal, and every targeted update destroyed the shared container.

Builds now pass --provenance=false --sbom=false, which puts the tag back on a plain image manifest and makes the id content-addressed. The first update after picking this up recreates brain-memory one more time — the new build genuinely is a different image from the manifest-list one it replaces. Every update after that leaves it alone.

Check it rather than trusting it, which is the whole lesson of this one:

docker inspect -f '{{.Created}}' brain-memory   # before
./brainctl update <you>
docker inspect -f '{{.Created}}' brain-memory   # unchanged, from the second run onward

Scoped vs admin¶

With MEMORY_ROBOT_TOKENS configured, which token an instance presents is a per-instance deploy choice:

brainctl up alice  --robot alice-pc --memory-scoped   # alice's robot token → its group only
brainctl up pascal --robot rap-1    --memory-admin    # fleet-wide + /admin/fleet
brainctl up bob    --robot bob-pc                     # default: admin (brainctl prints a NOTE)

A scoped brain sees only its robot's group plus the fleet-wide shared docs, and is denied fleet administration. Its Fleet page degrades to an explained read-only state rather than erroring. Keep at least one admin brain or nobody can regroup a robot.

brainctl refuses combinations that could only fail at runtime — a wrong token fails silently (the brain starts, looks healthy, 401s every memory call): --memory-scoped with no token for that robot, a malformed token map, --memory-scoped with --no-memory, or MEMORY_ROBOT_TOKENS set while MEMORY_TOKEN is empty. Full semantics: memory spec.

Turning memory auth on in production (BIN-113)¶

The central memory ships unauthenticated (tailnet-only). Switching it on touches the brain host and every robot, and a mismatch silently kills teaching/recall on the side that's wrong — so roll both together, in this order:

# 1) brain host (4090) — generate a token and put it in brain.env
ssh 4090
openssl rand -hex 32                                    # the admin token
cd ~/sage/orchestrator/brain-host
$EDITOR brain.env                                       # MEMORY_TOKEN=<tok>
./brainctl update pascal                                # name the instance — see "Update" below
#   expect: "(memory auth changed in brain.env — recreating brain-memory)" + "(memory auth: on)"
docker exec brain-memory printenv MEMORY_TOKEN          # confirm the container really has it

2) Every robot gets the same token, before or immediately after step 1 — the robot half is R1 robot stack → fleet grasp-learning (MEMORY_TOKEN in r1.env, then a relaunch). Do not skip a robot — "A mismatch is silent", below, is why.

# 3) verify
curl -s -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer wrong' \
     http://localhost:9103/sse                          # → 401
./brainctl logs pascal | grep mcp_server_connected      # → server=memory
#   then in the UI: teach a fact, reload, ask for it back; run a grasp and check the lesson lands

A mismatch is silent

If the two sides disagree, every memory call 401s: the brain still starts and reports healthy, it just loses teaching and recall. Roll both sides together, keep brain-pascal up by naming it (brainctl update pascal, not a manual down/up), and check brainctl logs for mcp_server_connected server=memory afterwards.

Per-robot tokens (MEMORY_ROBOT_TOKENS + --memory-scoped) are the optional next step once the shared token is on — see above. Rotating either value takes effect on the next brainctl up/update: brainctl recreates brain-memory when the token in brain.env drifts from the running container's.

HTTPS, and who owns the PWA URL¶

Tailscale Serve publishes each instance over HTTPS with the tailnet's certificate. The HTTPS URL is the one to open: it is a secure context, so the full UI works and the app is installable as a PWA. Plain http://<host>:<port> works too but browsers won't offer installation over it.

brainctl up pascal   --robot rap-1   --primary          # owns https://<host>/
brainctl up giovanni --robot giov-pc --https-port 8443  # https://<host>:8443
brainctl up scratch  --robot rap-1   --no-https         # off Serve; plain HTTP only

Declare the root owner with --primary. Without it the alphabetically first instance takes :443 — which is how brain-giovanni once took the PWA URL from brain-pascal. brainctl ls prints the URL to open per instance and marks the primary.

Requires Serve enabled on the tailnet and, once per machine, sudo tailscale set --operator=$USER. If tailscale isn't installed, Serve is skipped silently and everything stays on plain HTTP.

tailscale serve reset is node-global

There is no per-port "off" in the CLI, so brainctl resets the node's whole Serve config and rebuilds it from the running instances' labels on every up/down/update. Anything else you serve on that tailnet node gets wiped.

Update — propagate a change¶

./brainctl update pascal            # just brain-pascal
./brainctl update pascal giovanni   # these two
./brainctl update --dry-run         # print the recreate plan, change nothing
./brainctl update                   # everyone — asks first at a terminal (-y to skip)

It pulls the brain-host repo and the source clones, rebuilds binabik-brain:latest — printing what it is baking in as it does — and recreates the selected instances from their labels — replaying port, robot, manifest, extra servers, memory scope and HTTPS settings. No robot repo is pulled. update prints what it will recreate and what it leaves alone before it pulls or builds.

Recreating a container interrupts that colleague

Their in-flight mission ends and their control lease drops. A change that matters to one instance should name it. In particular, don't take down the primary (brain-pascal) as collateral for someone else's deploy. The mission is now finalised on the way out rather than simply vanishing — see the grace period — but it is still interrupted.

update touches the shared memory container only when its image changed

brain-memory is shared, so recreating it drops the :9103 SSE stream of every brain — each logs mcp_server_unhealthy and reconnects (~60 ms, all 18 tools back), including the instances you deliberately left alone.

update used to recreate it unconditionally, on the reasoning that the image had just been rebuilt. But a rebuild with unchanged source is a cache hit producing the byte-identical image, so almost every one of those interruptions bought nothing: on the 4090 the container was replaced several times in one afternoon for an image untouched for three days. It now compares the image id across the rebuild and prints memory image changed — recreating the shared brain-memory only when it really does (BIN-359). Memory is still started if it was down, still recreated on token drift (see Central memory — docker env is create-time), and still removed once no instance wants it.

This is worth knowing when reading someone else's logs: a memory reconnect in an otherwise-idle brain is the signature of another engineer deploying, not of a fault in that instance. Diagnosing one, compare instances over the same window — a brain created after the last event will show zero drops no matter what, which is how BIN-359 came to be filed as a pascal-specific fault when the busiest victim was in fact a third instance nobody had looked at.

A rebuild is global; the recreate is what you select

All instances share the one image, so update always rebuilds it. Instances you left out keep running on the image they started with — untouched, not version-pinned. Roll them forward whenever it suits them.

Stopping an instance, and the grace period¶

Every path that takes a container down — down, up over an existing name, update (which recreates through up), and memory down — sends SIGTERM, waits up to BRAIN_STOP_TIMEOUT seconds, and only then removes it. Nothing has to be passed; this is simply what stopping means now.

It has not always been. Until BIN-535 every one of those paths used docker rm -f, which sends SIGKILL and does not send SIGTERM at all — so uvicorn's shutdown never ran on a real deploy and the brain's whole graceful path was unreachable code:

the in-flight pipeline turn was never drained (observability.shutdown_drain_timeout_s);
the mission was never finalised — nothing marked its running step failed, recorded its finished trace event, or stopped the skills the robot was executing, so a deploy during a grasp left the arm mid-motion with no brain behind it, and the run kept a blank outcome in the Missions view forever;
and brain.db's queued writes died in its writer thread with the WAL never checkpointed.

That last one is why this is worth a section rather than a footnote. Since BIN-531 everything a brain remembers is one WAL'd SQLite file behind a queue and a dedicated writer thread, so a SIGKILL is data loss on every deploy — in the store that exists to stop losing data on every deploy. The file was left as database-plus-journal, and anything still queued was gone.

The timeout is a ceiling, not a delay. docker stop returns the moment the container exits, so an idle brain — the normal case for a deploy — costs a fraction of a second, and brainctl update across six instances is no slower than before. The 30 s is spent only by a brain that genuinely will not go.

Knob	Default	What it sets
`BRAIN_STOP_TIMEOUT`	`30`	seconds between SIGTERM and the daemon's SIGKILL, for every brainctl stop and as the container's own `--stop-timeout`

30 s is chosen to cover the app's 10 s pipeline drain plus a real mission teardown (registry cancels, skill stops, the finished event, the WAL checkpoint). It is deliberately not long enough to outlast a hung MCP call, which runs at the deployed call_timeout_s of 180 s: six instances waiting out one wedged skill is not a deploy anyone would sit through. Raise it for a brain whose teardown genuinely needs longer — BRAIN_STOP_TIMEOUT=90 ./brainctl update pascal — and note that the value is also written onto each container as --stop-timeout, so a host reboot or a bare docker stop gets the same grace without going through brainctl.

A brain that refuses to drain is named, and it did lose data

docker stop exits 0 whether the container drained or was killed at the end of the grace period, so brainctl reads the container's exit code in between and reports the difference:

!! brain-giovanni ignored SIGTERM for 30s and was killed — its mission was not
   finalised and its store was left un-checkpointed (raise BRAIN_STOP_TIMEOUT?)

This is per instance on purpose. During a fleet update you need to know which brain refused, because that is the one whose mission is un-finalised and whose brain.db was left beside a journal. Silence means every instance drained.

What the brain does with those 30 seconds¶

The grace period is only half the bargain; the app has to finish inside it. Since BIN-552 the whole teardown runs under one deadline, and these are the knobs that set it — all in observability, all overridable per instance through RVC_OVERRIDES:

Key	Default	What it bounds
`observability.shutdown_budget_s`	`20` s	the hard total for every awaitable step together — no combination of wedged steps exceeds it
`observability.shutdown_drain_timeout_s`	`10` s	the in-flight pipeline turn
`observability.shutdown_teardown_timeout_s`	`10` s	`executor.aclose()` — the mission teardown that records the `finished` trace event
`observability.shutdown_call_timeout_s`	`5` s	each robot/memory MCP call made while shutting down (the halt, `cancel_task`, `stop_skill`, `record_outcome`)

Three things follow from those numbers and are worth knowing before changing any of them:

20 s + a ~3 s synchronous store close fits inside 30 s with room to spare, which is the whole point. Raise shutdown_budget_s and you must raise BRAIN_STOP_TIMEOUT with it; the brain checks the arithmetic at boot and says so if it no longer works, because boot is the only moment anyone can act on the answer:
```
shutdown_budget_exceeds_grace worst_case_s=43.0 grace_s=30.0
```
The mission teardown's share is reserved from the start. The halt and the drain are only ever offered what is left after setting the teardown's cap aside, so a pipeline turn that will not finish cannot cost the mission its terminal trace event.
shutdown_call_timeout_s is deliberately far below the deployed call_timeout_s of 180 s. A shutdown that waits out a hung skill is the bug, not the fix — and this is the number that makes 30 s a grace period the app can actually live inside.

SIGTERM halts the robot, before anything else in the teardown, and only when this brain has a mission running. A brainctl update during a grasp or a navigate_to used to leave the arm or the base executing that goal with the brain gone, and on the R1 there is no second line of defence. The gate matters as much as the halt: stop raises the robot's cooperative flag, so an idle brain firing it on every deploy could halt a robot a different instance is commanding. The flag is not cleared afterwards — the next plan to run clears it, under a brain that means to move.

Missions orphaned by a kill are closed out at the next boot. A brain that was SIGKILLed — every deploy before BIN-535, and any teardown that still overruns — leaves a mission with no finished event, which the Missions view rendered as running for a run that ended weeks ago. Startup now writes a terminal event marked interrupted for each, and reports the count on the boot line, so interrupted=0 is how you read "the last stop was graceful":

brain_store_ready schema=2->2 loaded={...} invalidated=0 interrupted=0 wal_bytes=0

Where things live on the host¶

What	Where	Default
Keys + shared settings	`brain.env` (gitignored)	`<brain-host>/brain.env`, override `BRAIN_ENV`
Brain source clones	`$BRAIN_SRC`	`$HOME/sage/orchestrator/src`
Shared state (recoveries, memory data)	`$BRAIN_SHARED`	`$HOME/sage/orchestrator/shared`
Per-instance store — everything a brain remembers	`$BRAIN_SHARED/brain/<name>` → `/app/data`	one `brain.db` per instance
LLM price table (config — read-only to the brain)	`$BRAIN_SHARED/pricing` → `/app/config/pricing`	one table for the host, seeded from the image
Attach manifests	`<brain-host>/robots/<name>.yaml`	—
Image name	`$BRAIN_IMAGE`	`binabik-brain:latest`
First host port tried	`$BRAIN_PORT_BASE`	`8001`
Shutdown grace before SIGKILL	`$BRAIN_STOP_TIMEOUT`	`30` s — see Stopping an instance
Central memory port / data	`$BRAIN_MEMORY_PORT`, `$BRAIN_MEMORY_DATA`	`9103`, `$BRAIN_SHARED/memory`
Per-instance runtime config	generated in the container at start	`entrypoint.sh` → `gen_mcp_config.py`

The SPA is built inside the image from robot-voice-chat/frontend in a Node stage — there is no prebuilt-UI directory to rsync into and no manual UI step. update always ships the current frontend.

One store per instance, and mounting it is the point. Everything a brain remembers is one SQLite file, /app/data/brain.db, which brainctl up backs with $BRAIN_SHARED/brain/<name> (BIN-531). Missions and their events, recovery lessons, stored plans, the curated skill palette, the event log, the daily cost rollups, operator spend caps, the operator's Context facts, the admin-session revocation watermark, the fetched rate catalogue and persisted warnings — all of it. The state model — which kind each table holds, why no belief is in there, and the standing rule that everything written goes in a database — is the brain-app service spec's (§8a); what this page owns is the mount and the knobs.

It is deliberately not shared the way recoveries/ is: one SQLite file has one writer, spend recorded against two robots in one file cannot be separated again afterwards, and a mission id (m-<hex>) is unique per brain rather than across the fleet.

Anything under /app/data that is not on the mount lives in the container's writable layer, and update recreates containers. That failure has been found three times — the cost ledger (BIN-240), mission traces (BIN-461, measured on 2026-09-09 as three instances holding zero traces the morning they were recreated while one untouched since the 4^th still had its eight) and data/context.json plus data/admin_session.json — the operator's Context panel and the "log everyone out" watermark, neither of which anything had ever mounted. All four are tables in brain.db now (BIN-547), and one mount at /app/data, on a host directory called brain, is what stops there being a fifth. brainctl down leaves it in place: spend already recorded, a plan an operator authored and the record of what a mission did are not the container's to delete.

Mounting it is necessary and not sufficient: the file also has to be closed on the way out, or what is queued is lost and the WAL is never folded back in. That is why stopping a container sends SIGTERM and waits instead of killing it — for two weeks it did not, and a mounted store was still left as database-plus-journal on every deploy (BIN-535).

Retention runs once at startup, per kind. The rule is that records are pruned and definitions never are:

Knob	Default	What it prunes
`observability.retention_days`	`30`	raw `events` rows — folded into a never-pruned daily-totals table first, so month-to-date survives the 31^st while per-call detail does not (BIN-240)
`agent.trace_retention_days`	`30`	whole missions and their events. A full day of eight missions measured 316 KB, so 30 days is single-digit MB per instance
`agent.recovery_retention_days`	`0`	recovery lessons — `0` keeps every one, because a lesson is cheap to keep and expensive to re-learn
`observability.log_retention_rows`	`5000`	persisted log records (those at `observability.log_persist_level`, default `WARNING`, and above). Bounded by count, not age — the value of a persisted warning is the last thing that went wrong, which a quiet fortnight would age out of existence. Since BIN-592 this table is what the admin Logs page reads, so it is also how deep that page goes; the in-memory ring it used to read is gone, and lowering `log_persist_level` is the deliberate way to get INFO lines into the UI.

Stored plans, the skill palette, operator spend caps, the operator's Context facts and the admin-session watermark are definitions and no knob prunes them. 0 disables a knob; it never means "prune everything". model_prices — the fetched rate catalogue — is a record and is also deliberately unprunable: it is bounded by (providers × models) rather than by traffic, and deleting a rate by age would quietly move a priced call to unpriced.

One line reports the whole startup — grep the container log for brain_store_ready:

brain_store_ready path=/app/data/brain.db schema=0->2 loaded={"missions":8,"mission_events":214,
  "recoveries":0,"events":9143,"events_daily":74,"logs":0,"model_prices":383,"plans":2,
  "skill_visibility":1,"context":4} invalidated=0 wal_bytes=0 migrated={"events":9143,"plans":2,
  "skill_visibility":1,"missions":8,"mission_events":214}

context and model_prices are on that line on purpose: an instance an operator has curated reading context: 0, or weeks of uptime reading model_prices: 0, is the sentence a reader needs said out loud — the Context facts disappearing silently on every deploy is exactly what BIN-547 fixed. migrated appears only on the first boot after the upgrade and pruned only when something aged out, so a steady-state restart is a short line. loaded and invalidated are always present — "nothing was loaded" and "the field is missing" must not look the same in a log.

Upgrading an existing instance needs nothing by hand. up (and therefore update) moves the two old host directories — $BRAIN_SHARED/metrics/<name> and $BRAIN_SHARED/missions/<name> — onto the new mount, and the brain then folds their contents into brain.db and parks the sources in a migrated/ subdirectory. Nothing is deleted. If both an old and a new copy somehow exist, up says so on stderr and touches neither, because merging two homes would let a stale ledger overwrite a live one. $BRAIN_SHARED/recoveries stays mounted at /shared/recoveries as a migration source; lessons move into brain.db, and fleet-wide sharing is the central memory container's job.

A store written by a newer build is refused, not written through. Rolling an image back past a schema bump makes the brain refuse to start and say the version it found. That is the cheap outcome: the file is the only copy of the fleet's history, and writing through a schema this build does not understand is how a column silently becomes garbage for the build that does.

The price table is host state, and it maintains itself. What a call costs is data — per-model rates in micro-USD per million tokens — and brainctl up mounts $BRAIN_SHARED/pricing at /app/config/pricing so correcting one is an edit on the host rather than an image build and a fleet recreate. One table for every instance, like recoveries/: a rate is a property of the provider, not of a colleague's brain. A directory is mounted rather than the file itself, because a bind-mounted file keeps its inode and an editor that writes a replacement would leave the container serving the old rates silently.

The first container to start against an empty mount seeds it from the image's copy and gives the file to whoever owns the directory (the container runs as root; a root-owned table would mean correcting a rate needs sudo). From then on the host's file wins and a running brain re-reads it within a minute of a change.

The file is yours; the daily refresh is a table (BIN-547). Nothing in a running container writes pricing.yaml any more — the brain fetches the catalogue daily into brain.db's model_prices and the mounted file is read-only to it, under the standing rule that a file a human is meant to edit is config while anything the application produces is a row. Until then the refresher rewrote this very file, so one document had two authors and only a convention kept them apart.

What that means at the keyboard:

Hand corrections go in overrides: (or endpoints:), which beat the fetched rows and the shipped providers: seed alike. Nothing overwrites them, and the layers merge per token class, so one line leaves the other three tracking the daily refresh.
providers: is the seed, not the live table: the copy baked into the image, regenerated deliberately with uv run python scripts/refresh_prices.py in the repo and committed. It is the bottom layer, and it is what prices a brain that has never fetched — an air-gapped host, pricing_refresh.enabled: false, or the first minute after a start.
A failed fetch changes nothing: pricing a turn never touches the network, the stored rates stay exactly as they are, and a table stale for more than two refresh intervals says so in the log. A refresh that resolves nothing is treated as a failure rather than as a successful empty one, so it retries on the backoff instead of claiming the rates are current.
Every brain on the host now fetches for itself, once a day, where the shared file used to mean the first to wake refreshed for all of them. That is a deliberate trade for the split: the staleness check exists to prevent six requests a minute, not six a day.
Admin → Cost shows all four layers, labelled, so "what will this call be priced from" is answered by the page rather than by re-deriving the merge from the file.

Troubleshooting¶

The container exits immediately, or restarts in a loop, and brainctl logs <name> says OPERATOR_KEY is not set. It is doing what it is supposed to: there is no unauthenticated mode, so a brain with no operator key stops instead of serving an open socket (the operator key). Add the line to brain.env and re-run up. A key shorter than 16 characters is refused the same way, with the length in the message.

The UI says "This brain is locked" and asks for an operator key. Expected on a first visit from a new browser or device. Paste the key from brain.env, or sign in under Admin. If you believe you already entered it, the browser may not be keeping it — private mode and restricted storage both prevent that, and the prompt says so underneath the field.

The socket never connects and brainctl logs <name> shows ws_connect_refused. The client presented no valid credential. offered_auth says whether it sent one at all — false means a browser with nothing stored (it should be showing the prompt), true means the key it holds is wrong for this instance, which happens when a brain has a per-instance key from its --secrets-file and the operator is using the shared one. The credential itself is never logged, only whether one was offered.

The brain is "healthy" but has no robot tools. The single most common failure. The container has no MagicDNS, so brainctl resolves the robot's name on the host (tailscale ip -4, then getent hosts) and injects an /etc/hosts entry. Check the up output for the (rap-1 → 100.x.y.z) line; a WARNING: could not resolve … means every robot MCP URL will fail. In order:

tailscale status on the brain host — is the robot reachable, does tailscale ip -4 <robot> return an address?
On the robot, are the servers bound to 0.0.0.0 rather than 127.0.0.1? ss -ltn | grep -E ':(9220|9210)'. Bound to localhost → a remote brain can't connect.
brainctl logs <name> — a down server is skipped, not fatal, so the brain looks healthy with zero tools. Bring it up and the brain reconnects on its own (30 s).
The /etc/hosts entry is written at container-create time, so if the robot's tailnet IP changed, re-run up/update.

gen_mcp_config: WARNING — no MCP servers resolved. Nothing attached at all: the manifest's ${VAR}s didn't resolve. Usually a missing --robot.

"I answered the robot's question and it says it got no reply." brainctl logs <name> and look for these, all at INFO since BIN-199 — before that a turn's arrival was log.debug and therefore invisible, which is why this class of failure could not be diagnosed at all:

Log line	Means
`ws_send_message`	the turn reached the brain. If it is absent, the browser never delivered it — a dropped socket, or a click the UI swallowed
`ws_action_refused` (`action=send_message`)	it arrived and was rejected because that client was a watcher, not the controller
`turn_routed to=mission` \| `late_answer` \| `chat`	which branch took it. `chat` for something that was meant as a mission answer means the mission had already ended
`answer_never_arrived` (mission trace)	the mission finished still blocked, with the measured wait
`ws_connect` / `ws_disconnect` with `took_control_from` / `handed_control_to` / `reason` / `user_agent`	the control lease churning, and why the socket dropped (`ping timeout` = the client stopped answering, e.g. a sleeping laptop; `transport close`/`transport error` = the connection died beneath engine.io)

The socket reconnecting every minute or two is expected when the operator's machine sleeps — engine.io's deadline is ping_interval + ping_timeout (25 s + 20 s), so a laptop doing macOS Power Nap dark-wakes produces sessions that are multiples of that. Sustained churn on an awake client is not expected; that is BIN-200.

The planner keeps asking questions instead of acting. By design it asks when unsure. Usual causes: no named places (teach waypoints — their names are injected into planning), a missing tool (it's honest about capabilities you never exposed), or no robot prompt (prompt://system), leaving it only the universal base.

A mission is slow — was it the model, or was it us? brainctl logs <name> and look for planner_call on each planner turn (BIN-202). If its ttft_ms accounts for nearly all of its duration_ms, we were waiting on Anthropic and there is nothing local to fix. A small ttft_ms inside a long duration_ms means the brain's own event loop was late draining the stream — cross-check the event_loop_lag lines, and restart the instance with RVC_ASYNCIO_DEBUG=1 in brain.env to have asyncio name the blocking callback (it costs real overhead, so take it back out afterwards). A duration_ms far above the configured timeout is an SDK retry inside the awaited call. Every field → robot-voice-chat's README. Before restarting an instance whose CPU is pegged, grab a py-spy dump — the evidence dies with the container.

Memory isn't attaching. brainctl ls should show a brain-memory row; if not, brainctl memory up and check brainctl memory logs. Was the instance started --no-memory? If MEMORY_TOKEN is set it must match on both sides — a mismatch 401s every memory call while the brain still reports healthy. brainctl logs <name> showing no value for '${MEMORY_AUTH_TOKEN}' … attached WITHOUT its Authorization header means the brain is talking to memory unauthenticated — fine while memory is open, a misconfiguration if you meant auth to be on.

The Fleet admin is read-only or sees one robot. That's --memory-scoped working as intended. Confirm with docker inspect -f '{{index .Config.Labels "brain.memory_scope"}}' brain-<name>.

Port already in use. --port selection only checks other brain containers' labels, not whether something else on the host owns 8001. Pass --port explicitly if you have other services around.

Running a brain (brainctl)¶