AuthorPascal DateJuly 29, 2026 Rev1.8

robot-mcp-memory — Implementation Spec¶

robot-mcp-memory is the per-robot process memory MCP server for the robot fleet. It stores process documents — markdown distilled from successfully executed (or instructively failed) missions in "teaching mode" — and retrieves the most relevant ones when a new instruction arrives, so the planner LLM can reuse what the robot has already learned.

Repo: robot-mcp-memory

Audience

Fleet operators (teaching + recall) and brain developers wiring the planner. The server is optional — the brain plans fine without it; attaching one makes past missions reusable.

Everything this service writes is in one SQLite file — MEMORY_DATA_DIR/index.db — under the stack-wide rule that written state goes in a database and never a JSON or markdown file (BIN-546; implemented here by BIN-548). The process documents, the robot registry, the groups and the self_improve gate are all tables. Markdown is the wire format, rendered from the row on the way out, not the storage format; fleet.json and MEMORY_DATA_DIR/processes/ no longer exist.

The lesson store went with it, in BIN-573. params/<scope>.json was the kit's JsonFileBackend, not this service's writer, so BIN-546 part 3 moved it: BIN-549 removed that backend and the kit offers SqliteBackend, where scope is a column rather than a filename — one table with a (scope, key) primary key serves every scope with one connection, one retention policy and one backup. ScopedParams keeps its job (resolving a robot to its scope through FleetRegistry and gating on self_improve_for) and changed only what it hands ParamStore; the lessons live in MEMORY_DATA_DIR/lessons.db, opened lazily so a service where nobody switched self-improvement on still leaves nothing behind. Nothing is migrated, because nothing was ever written: measured 2026-09-09, no params/ directory existed on the 4090.

Quick start / usage¶

Operators — press Teach in the UI before a mission. On success (or an instructive failure) the chat app distills the run into a markdown process document and calls save_process — armed teaching auto-saves, nothing else to do.
Developers — memory is optional. With no server attached the brain still plans; attach one and every mission start calls find_processes(instruction, k=3), injecting the matched processes into the planner's context as advice (the planner deviates when reality disagrees).
Deployment — it is deployed by default by brainctl as one shared central brain-memory container (SSE on :9103). Commands, flags and token handling: Running a brain.

Planned

Learned-good skills (vision configs) may later be persisted here alongside processes. That would be an additive use and does not change the interface described here.

Fleet registry + group-scoped learning (v0.2)¶

Beyond process memory, this server is also the central fleet registry and the group-scoped store of transferable learning — run one central instance for the fleet.

FleetRegistry (fleet.py, one JSON file) — robots + groups (with customer / factory tags) + a per-group / global self-improve toggle. It's the single, propagating source of truth: every brain reads/writes it over MCP, so a grouping edit from any brain is seen by all. Tools: register_robot, list_robots, list_groups, set_group, upsert_group, delete_group, set_self_improve.
ScopedParams (params.py) — the grasp-lesson / LLM-correction store (a robot_mcp_kit.ParamStore per scope), scoped by the robot's group. Tools: recall_params, record_params, list_lessons.

Scoping — the tenancy model. Every lesson is keyed to a scope resolved from the registry:

Scope	Holds	Shared?
`group:<G>`	transferable learning for robots grouped in G	within the group
`solo:<robot>`	transferable learning for an ungrouped robot	no (isolated)
`robot:<id>`	robot-private / episodic ("what did you see")	never

So connected robots share; ungrouped robots are isolated; and robot-specific observations never enter a shared scope. The env-aware signature (place) still isolates environments within a group (a lesson from one table doesn't leak to another until earned). Grasp-service reaches this via its RemoteLessonStore (MEMORY_URL). The brain Fleet admin surfaces robots, groups, and the per-group toggle. See architecture.md §2b.

Scopes above are the data model — which store a lesson lands in. Who may reach which scope is enforced per connection from the caller's token (§6a): with MEMORY_ROBOT_TOKENS set, a brain or grasp-service authenticating as rap-1 cannot read, write, or regroup outside rap-1's group.

1. System context¶

Each colleague gets a brain instance (robot-voice-chat, FastAPI — port 8000 inside the container, published on the host from 8001 upward) whose agentic MissionExecutor plans missions with an LLM and calls tools on capability MCP servers (HTTP/SSE). This server is the fleet/process memory — deployed centrally by brainctl (port 9103, shared by every brain). The robot-side capability servers it sits alongside are r1-abstraction (9220) and grasp-service (9210), plus the opt-in perception buffer (9202) and episodic ros2-memory (9203).

The flow this server enables:

Operator presses Teach in the UI → the chat app records the next mission's full trace.
Mission ends → the chat app distills the trace into a markdown process document (the chat app does the LLM distillation, not this server) and calls memory.save_process(...).
On every later mission start, the chat app's executor calls memory.find_processes(instruction, k=3) and injects the matches into the planner's context as advice.
After a mission that used a process, the executor calls memory.record_outcome(...) so stale or flaky processes lose weight over time.

Unlike navigation/perception, everything here is fast — no Async Task Contract and no robot-mcp-kit task manager. All tools are short request/response MCP tools, and this server posts no events.

It does, however, depend on robot-mcp-kit (>=0.4, the release that introduced ParamStore): the lesson store in §2 is a robot_mcp_kit.ParamStore per scope. The kit is used as a library, not as the async-task/event framework — which is the distinction this paragraph used to collapse into a flat "does not depend on robot-mcp-kit", contradicting §2 two pages above and the pyproject.toml floor. Practically it means this repo needs the private-kit deploy-key wiring in CI like any other consumer.

2. Repo layout¶

robot-mcp-memory/
├── pyproject.toml                  # package: robot_mcp_memory
│                                   # deps: mcp (floored + CEILED, see below), pydantic,
│                                   #   pydantic-settings, pyyaml, structlog,
│                                   #   robot-mcp-kit>=0.12.0 (ParamStore + the store and its
│                                   #   kind vocabulary — this is a KIT CONSUMER, so CI needs
│                                   #   the RMK_DEPLOY_KEY wiring)
│                                   # (sqlite3 is stdlib; FTS5 ships with CPython's sqlite)
├── README.md
├── src/robot_mcp_memory/
│   ├── __init__.py
│   ├── server.py                   # FastMCP("memory") + all 18 tools (§5) + entrypoint
│   ├── store.py                    # ProcessStore: the documents, as rows (§4)
│   ├── docmodel.py                 # ProcessDoc parsing/serialization (§3)
│   ├── fleet.py                    # FleetRegistry: robots + groups + self-improve toggle
│   ├── params.py                   # ScopedParams: a robot_mcp_kit.ParamStore per scope
│   ├── tenancy.py                  # per-connection tenant scoping for SSE (MEMORY_ROBOT_TOKENS)
│   └── db.py                       # index.db: the DDL, each table's declared kind, and the
│                                   #   two things only this service can say — the pre-BIN-548
│                                   #   shape fix and StoreUnreadable. The store machinery and
│                                   #   the kind vocabulary are the kit's (BIN-573/BIN-574);
│                                   #   the `db/` package that held copies of both is gone
├── deploy/robot-mcp-memory.service
└── tests/                          # per-module; count and coverage live in the repo, not here

mcp carries a ceiling, not just a floor

mcp>=1.28.1,<2 — the floor clears PYSEC-2026-3483, and the ceiling exists because mcp 2.0.0 deleted mcp.server.fastmcp, which this server imports. Without it a routine uv lock resolves a version the server cannot import. pydantic-settings>=2.14.2 is a transitive floored directly, because mcp itself still accepts a version with GHSA-4xgf-cpjx-pc3j. Both are decisions with an exit — see Versioning & packaging metadata.

Tooling: Python ≥ 3.11, uv, ruff, mypy --strict, pytest, GitHub Actions CI. Entrypoint/transport convention as siblings: console script robot-mcp-memory, MCP_TRANSPORT (stdio|sse), MCP_PORT default 9103. FastMCP name must be "memory".

3. The process document format¶

A process is one row in processes; this is the markdown it is transmitted as. save_process(markdown) parses a document into the row and get_process / find_processes render the row back into the identical markdown. YAML frontmatter + free markdown body:

---
id: fetch-package-mailroom            # kebab-case slug, unique; server enforces
title: Fetch a package from the mailroom
task_pattern: "fetch|get|bring ... package|parcel ... mailroom"   # free-text hint, also indexed
tags: [delivery, mailroom]
robot: g1-lab-01                      # provenance (env ROBOT_ID at save time)
created: 2026-06-12T14:00:00Z
updated: 2026-06-12T14:00:00Z
success_count: 3
failure_count: 1
last_used: 2026-06-12T16:20:00Z
source_mission_id: m-9af31c           # the teaching mission that produced it
---

## Steps
1. navigation.navigate_to(mailroom) with trigger eta_below:10
2. on trigger → perception.prewarm("mailroom, parcel shelf")
3. perception.find_object("package labeled <addressee>")
...

## Pitfalls & fixes
- After 18:00 the door is often closed → DOOR_CLOSED is recoverable: ask reception via ...

## Operator notes
- Visitor packages are on the lower shelf.

docmodel.py provides ProcessDoc (pydantic) with parse(text) -> ProcessDoc / render() -> str (round-trip stable), validating frontmatter and slug format. The body is opaque advice text — the server never interprets steps; only the planner LLM reads them.

The row is the source of truth (BIN-548). Until then the .md file was, with the table as its index — two sources of truth for one document, written without a transaction between them, which is why a directory-reconciliation pass had to exist. There is no directory now and no reindex() after editing a file: to change a stored process, call save_process with the corrected markdown.

That trade is deliberate and it has a cost worth naming: documents are no longer hand-editable or diffable on disk. The property being removed is "a filesystem anyone can edit under a running service". If reviewability is wanted back it belongs as an explicit dump/load operator command, not as the live format.

4. ProcessStore (SQLite FTS5)¶

MEMORY_DATA_DIR/index.db:

CREATE TABLE processes (          -- the document itself: `body` is what BIN-548 moved in,
  id TEXT PRIMARY KEY,            -- replacing the `path` + `mtime` columns that pointed at a file
  title TEXT, task_pattern TEXT, tags TEXT,
  robot TEXT, created TEXT, updated TEXT,
  success_count INTEGER, failure_count INTEGER, last_used TEXT,
  source_mission_id TEXT, body TEXT
);
CREATE VIRTUAL TABLE processes_fts USING fts5(
  id UNINDEXED, title, task_pattern, tags, body, tokenize='porter unicode61'
);

CREATE TABLE fleet_robots   (robot_id TEXT PRIMARY KEY, group_id TEXT, name TEXT, ...);
CREATE TABLE fleet_groups   (group_id TEXT PRIMARY KEY, name TEXT, self_improve INTEGER, ...);
CREATE TABLE fleet_settings (key TEXT PRIMARY KEY, value TEXT);   -- global_self_improve
CREATE TABLE store_meta     (key TEXT PRIMARY KEY, value TEXT);   -- schema_version

Kinds and retention¶

Every table declares its kind (db.py) in BIN-546's stack-wide vocabulary — record (prunes by age or count), definition (never removed by elapsed time), belief (needs the epoch it was observed in), derived (a projection of another table, rebuilt rather than pruned). A startup sweep runs on every boot through one chokepoint that refuses any table that is not a declared record. Since BIN-573 the vocabulary and the chokepoint are the kit's; what this service declares is which of its tables is which.

Table	Kind	Retention
`processes`	definition	none — a human, or a distillation they armed, authored it. `success_count` / `failure_count` / `last_used` are records about it and sit on its row
`fleet_robots`, `fleet_groups`, `fleet_settings`	definition	none — an operator registered the robot, arranged the groups and flipped `self_improve`
`store_meta`	definition	none — the file's schema version: its identity, not its history
`processes_fts`	derived	never pruned on its own; `reindex()` rebuilds it from `processes`

So this service holds only definitions plus one search index, and nothing ages out, which is the answer rather than an omission — a learned process vanishing on day 31 would be data loss wearing the word "retention". What matters is that it is declared: a table that arrives without a kind fails a test, and Schema.pruned cannot name a non-record. The day someone adds a records table the decision is forced instead of forgotten. Boot logs one store_ready line with the schema transition and the row counts. (It was memory_store_ready with a prunable field until BIN-573; prunable reported len(SCHEMA.pruned), a static fact about the schema, and is now enforced when the schema is built — the kit refuses a non-record in pruned — rather than reported once per boot.)

processes_fts is Kind.DERIVED, and this service is the reason that kind exists (BIN-574). The other three all answer "what does elapsed time do to this content?"; a search index has no content of its own, so pruning it independently of its source is a bug in either direction — by age it silently unindexes documents that still exist; left alone while the source prunes it indexes documents that do not.

Until BIN-574 this was a local overlay: SCHEMA declared the conservative definition and a DERIVED_TABLES set recorded what it really was, with a bridge test keeping them honest. All of it is deleted. definition did get the operational half right by accident — both are unprunable — but it could not say the part that matters for recovery: a projection is rebuildable, so losing this table costs a reindex() rather than a document, and a backup may skip it.

The kind also earns the exemption. Its five fts5 shadow tables (processes_fts_data, _idx, _content, _docsize, _config) are sqlite's own, and Schema.not_ours() derives their names from the DERIVED declaration — so this service names none of them, where BIN-573 had it holding DERIVED_TABLES, a copy of sqlite's five suffixes and a shadow_tables() to combine them. Declaring a Kind for a shadow table instead would claim retention could reason about it, and omitting them entirely put a WARNING naming all five in the log on every boot of a container six brains share, which is how a real warning goes unread.

Worth recording, because it is the strongest evidence the taxonomy was short rather than one author over-modelling: r1-abstraction independently found the same vocabulary too narrow in a different direction (kind-per-column, for a waypoint row that is three kinds at once). Two agents, no contact, same conclusion.

Writing: one connection, one writer thread¶

The store is the kit's SqliteStore (floor >=0.12.0, the version that can say derived). It was a near-copy of the brain's BrainStore here — 826 lines across db/kinds.py, db/schema.py, db/memory_store.py and db/lifecycle.py, written in the same hour BIN-549 was lifting the same machinery into the kit — and BIN-573 deleted it. Hot record writes — the outcome counter every mission step bumps — are enqueued and run on a dedicated thread, bounded and countably droppable, so a record never blocks the one asyncio loop serving every brain in the fleet (BIN-203). Cold definition writes — a document an LLM just distilled, operator CRUD, the retention sweep — run synchronously and are never dropped. Which path a write takes follows its kind.

Opening is versioned: older migrates forward, a newer file raises SchemaTooNew rather than being written through, and a file that is not a database raises StoreUnreadable naming the path and the remedy. That last one matters more than it looks: brainctl runs this container with --restart unless-stopped, and the unguarded json.loads in the old FleetRegistry.__init__ meant 1 KB of truncated JSON crash-looped teaching, recall, group learning and the Fleet admin for all six brains at once.

Search (find(query, k)):

FTS5 MATCH over title+task_pattern+tags+body with bm25 ranking. Sanitize the query (strip FTS operators, OR the remaining terms) so arbitrary natural-language instructions never produce syntax errors.
Re-rank: score = bm25 * reliability, where reliability = (success_count + 1) / (success_count + failure_count + 2) (Laplace smoothing).
Return top k with score and a reliability field; the chat app decides how to present them.

Planned

Embedding search is a planned upgrade inside this repo (swap or augment find). The tool surface does not change for it.

Store API: save(doc) -> ProcessDoc (insert or update by id; bumps updated), get(id), get_markdown(id), find(query, k), record_outcome(id, success, note=""), delete(id), list(limit, offset), count(), reindex().

A document's row and its FTS entry are written in one transaction, so they cannot disagree and there is no crash window between them. scan() is gone with the directory it reconciled. reindex() stays but is no longer a repair tool — it returns {"indexed": n} and exists for a tokenizer or column-weight change.

5. MCP tools¶

All tools return JSON strings. Their docstrings tell the LLM that processes are advice from past executions, that it should deviate when reality disagrees, and that find_processes is normally called automatically by the system (though the LLM may still call it mid-mission for sub-tasks).

Tool	Behavior
`find_processes(query: str, k: int = 3)`	§4 search → `{"processes": [{id, title, score, reliability, success_count, failure_count, updated, markdown}]}`. `markdown` is the full document — the chat app injects it verbatim.
`get_process(id: str)`	Full document or `{"error": "not_found"}`.
`save_process(markdown: str)`	Parse + validate (`ProcessDoc.parse`); slug collision ⇒ update that process (preserve counters, bump `updated`) unless frontmatter differs in `source_mission_id`, in which case suffix the id (`-2`). Returns the stored doc's frontmatter.
`record_outcome(id: str, success: bool, note: str = "")`	Update counters/`last_used`; if `note` non-empty, append a line under an `## Outcome log` section (create if missing) with date + success flag + note.
`delete_process(id: str)`	Remove the row and its index entry, in one transaction.
`list_processes(limit: int = 50, offset: int = 0)`	Frontmatter summaries (no bodies).
`reindex()`	Rebuild the FTS index from the `processes` rows → `{"indexed": n}`. Admin-only. Its old `errors`/`removed` counts are gone with the directory: reporting a reassuring zero for a failure that can no longer happen is worse than dropping the key.

6. Configuration + tenancy¶

Env var	Meaning	Default
`MCP_TRANSPORT` / `MCP_PORT`	transport / SSE port	`stdio` / `9103`
`MEMORY_DATA_DIR`	`index.db` (everything this service writes) + `params/` (the kit's per-scope lessons, still files — BIN-549)	`data/`
`ROBOT_ID`	provenance written into saved docs	`unknown`
`MEMORY_TOKEN`	optional shared bearer token gating the SSE transport (401 without it); unset ⇒ no auth (tailnet-only). With `MEMORY_ROBOT_TOKENS` set it is the unscoped admin token	unset
`MEMORY_ROBOT_TOKENS`	optional per-robot tokens (`"rap-1:tokA,rap-2:tokB"`) — each scopes its connection to that robot's group (§6a)	unset

Deployed by default via brainctl

In production this server is not installed per-robot by hand — one shared central brain-memory container (SSE on 0.0.0.0:9103) serves every brain. How it is built, started, opted out of, and how a token rotation is propagated: Running a brain. What matters here is only the server-side contract: with MEMORY_TOKEN set the SSE transport is gated by a constant-time bearer check, so every client — each brain and grasp-service's RemoteLessonStore — must send Authorization: Bearer <token>.

6a. Multi-tenant scoping (per-connection)¶

Auth has two levels. MEMORY_TOKEN alone authenticates a connection as unscoped — full fleet access, the original behavior, and what the brain's Fleet admin needs. MEMORY_ROBOT_TOKENS ("rap-1:tokA,rap-2:tokB") additionally issues one token per robot: the SSE middleware resolves the token to a tenant and binds it for the connection's lifetime, so every tool call is scoped to that robot's group.

Tool group	Scoped tenant (robot token)	Unscoped (`MEMORY_TOKEN`)
`find_processes`, `get_process`, `list_processes`	own group + fleet-wide `shared` docs only; the `robot` argument cannot escape the scope; out-of-scope ids read as `not_found`	everything
`save_process`	pinned to the tenant (`robot: unknown` → its own id); a foreign robot or a fleet-wide `shared` doc → `forbidden`	any robot
`record_outcome`, `delete_process`	in-scope only; deleting a fleet-wide doc is `forbidden` (it serves every tenant)	any doc
`register_robot`, `set_robot_meta`	only for itself	any robot
`list_robots`, `list_groups`	only its own group's robots / its own group	whole fleet
`set_group`, `upsert_group`, `delete_group`, `set_self_improve`, `reindex`	`forbidden` — fleet administration	allowed
`recall_params`, `record_params`, `list_lessons`	always act as the tenant's own robot, whatever `robot` says	as given

So a compromised robot host can neither read another customer's processes nor regroup itself into someone else's learning scope. Each robot's grasp-service sends its own token (r1.env MEMORY_TOKEN).

Which token a brain presents (BIN-111)¶

Scoping is per connection, so it's the token the instance sends that decides — a per-instance deploy choice, not a property of the manifest. Which flag selects which token, and how it reaches the container: Running a brain — scoped vs admin.

The admin path stays available by designating an admin brain, rather than by having one brain hold both tokens. A brain that switched tokens per call would need two memory server entries (and therefore two tool prefixes) in the attach manifest; keeping one token per connection matches how the middleware binds a tenant, and the Fleet admin (/admin/fleet → list_robots, set_group, set_self_improve) simply lives on the admin instance.

A scoped brain's Fleet page now degrades gracefully (BIN-112) rather than showing the refusal. The refusal wire shape is unchanged — this server keeps answering {"error": "forbidden", "detail": …} — but the brain no longer passes that through as data. Because it is well-formed JSON, the brain's /api/admin/fleet* proxies used to return it as a successful result, so a denied write reported success and silently did nothing; they now translate it into {"data": null, "error": <detail>, "code": "forbidden"}, kept distinct from a transport fault (code: "unavailable") so it is never retried as one. The Fleet page reads that code and switches to a read-only state: the group select, the self-improve toggles and the add-robot form are disabled with an explanatory tooltip, under a banner naming the robot the brain is scoped to.

This server exposes no "what scope am I?" tool, so the brain cannot ask up front — it learns the scope from the first refusal and remembers it for the session (a later successful write clears it). That's deliberate: the reads in the table above are filtered, not refused, so a scoped brain still renders its own group and only discovers the boundary when it tries to cross it. Adding such a tool would let the UI disable the controls before the first attempt; it isn't needed for correct behaviour and no client asks for it today.

A wrong token fails silently — the brain starts, reports healthy, and 401s every teaching/recall call — so the launcher refuses the impossible token combinations up front rather than letting them reach this server; the refusal list lives with the flags in Running a brain. Backwards compatible: with only MEMORY_TOKEN set nothing is scoped, and with neither var set the transport is open (the tailnet-only default). The stdio transport is always unscoped.

Implementation (tenancy.py): fail-fast token-map parsing (tenancy.parse_robot_tokens defines the token-map grammar), constant-time TokenAuth, and a pure-ASGI TenantAuthMiddleware. Pure ASGI is required — Starlette's BaseHTTPMiddleware runs the downstream app in a separate task, so a tenant contextvar set there would not reach the MCP session loop under GET /sse.

Fleet sync: there is no directory to rsync since BIN-548. Sharing between robots is what the fleet registry is for — put them in a group and they pool their learning through the one central instance, which is how this is deployed. Moving documents between two separate deployments is list_processes + get_process out, save_process in; the markdown is unchanged, so it is a copy rather than a conversion. Backup is one file copy of index.db.

7. Tests¶

docmodel: parse/render round-trip; invalid frontmatter rejected; slug validation.
store: save→find by phrasing variants ("get me a parcel from the mail room" finds fetch-package-mailroom); reliability re-ranking (a 0/5-failure process ranks below a 5/0 one at equal bm25); FTS query sanitization (queries containing ", *, (, NEAR don't crash); record_outcome updates the row and survives a reindex; the body is in the column and no markdown reaches the disk. 2b. the database (BIN-548): every table declares its kind and the prune chokepoint refuses a definition without deleting first; a hot counter write executes off the caller's thread, asserted on sqlite's own statement trace, while a document save stays on it; the versioned open refuses a newer file and names an unreadable one; _MIGRATIONS is driven with injected steps. The BIN-547 CI guard is lifted here and walks src/ for write-shaped calls.
tools (FastMCP in-memory client): full surface; save collision behavior; not-found paths.

8. Acceptance checklist¶

All gates pass (pytest, ruff check, ruff format --check, mypy --strict).
MCP_TRANSPORT=sse robot-mcp-memory serves on 9103; chat app logs mcp_server_connected server=memory.
Demo: save_process a hand-written mailroom doc, then find_processes("bring me the parcel from the mail room") returns it ranked first with the full markdown.
Kill -9 between file write and index write, restart → reindex heals.

Appendix A — Repository & tooling conventions¶

Scope. This is an MCP server. Every convention applies, including the MCP-server ones. Config shape: env vars only — not Hydra (no hydra-core dependency, no config/ tree); see Configuration.

The full list — repo structure, uv/ruff/mypy/pytest tooling, the pre-commit hook set, ports-and-adapters, the two config shapes (Hydra where implementations are selectable, a frozen env dataclass where they are not), secrets, structlog, the CI gate, and what an MCP server owes its consumer (transport choice, blocking tools, the ~30 s call timeout, reconnect tolerance, the hardened systemd unit) — is Coding guidelines. It lives there and only there: this appendix was one of five near-identical copies, and they drifted from each other and from the code.