AuthorPascal DateJuly 14, 2026 Rev1.0

robot-voice-chat — Agent Orchestration Spec¶

robot-voice-chat is the per-robot voice/chat app that hosts the mission brain: an agentic MissionExecutor layered on top of the single-turn voice/chat pipeline. It handles multi-step planning, parallel long-running tasks, triggers and replanning, the HTTP event ingress for capability servers, a Claude-backed planner LLM, teaching mode with process-memory integration, mid-mission barge-in (safety stop → pause → confirm), and a live mission pipeline view in the UI (every planned tool call colored by state).

Repo: robot-voice-chat

It also contains the fleet deployment runbook (§12): how to clone this repo plus the companion repos onto a robot and bring the whole system up.

This document is self-contained for an engineer who knows this codebase (ports-and-adapters, Hydra DI via composition.py, FastAPI + Socket.IO, McpToolBackend). External contracts it depends on are restated inline.

0. Capability model — control plane / data plane split¶

The brain configures a ROS stack and observes it, rather than calling MCP "async task" capabilities that stream events and hand back results (pixels included). This is the control plane / data plane split: robot modules talk to each other over ROS at sensor rate (segmentation → manipulation) with the brain out of that loop.

The brain is a pure MCP client (no rclpy). Two ROS↔MCP bridge servers front the per-capability ROS nodes:
mcp-skill-control (specs/mcp-skill-control.md) — discover + instantiate skills.
mcp-perception-buffer (specs/mcp-perception-buffer.md) — "what do you see" read buffer.
A "capability use" is configure-then-observe. find_object is not a call that returns a mask; it is skill_control.instantiate_skill("find_object", {query}) → a running ROS vision task publishing to /vision/find_object/detections, which the brain reads via perception.get_latest("find_object") (label/box/score; mask by reference, never over MCP) and a planned manipulation ROS node consumes directly.
The barge-in / safety-stop machinery (§9) stops motion ROS nodes via the motion bridge and stop_skills active vision skills.
The Async Task Contract (robot-mcp-kit) is push-only: it carries watch_events from the perception buffer (a watcher skill firing) and long-running ROS completions. Most observation is the pull buffer, not pushed events.

Note

The eta_below / zone_entered / navigate_to examples in §1–§14 are illustrative of an async-task capability; read them as skill instantiation against the current stack.

Companion repos:

Repo	Role	Plane	Port	Server name
`robot-mcp-kit`	shared lib: Async Task Contract models + HMAC signing (push events only)	both	—	—
`robot-mcp-perception`	configurable perception ROS node (Gemini ER understanding + SAM 3 segmentation), publishes `/vision/**`	data (ROS)	—	`vision_node`
`mcp-skill-control`	discover + instantiate skills (MCP → ROS services)	control	9201	`skill_control`
`mcp-perception-buffer`	"what do you see" rolling buffer (MCP ← ROS topics)	control	9202	`perception`
`robot-mcp-memory`	process/skill store: `find_processes`, `save_process`, `record_outcome`	control	9103	`memory`
`ros2-memory`	episodic ROS-bag memory: temporal queries over recorded state + `/vision/**` ("what did you see N s ago")	control	9203	`episodic`
`ros2-manipulation` (planned)	grasps; subscribes `/vision/` directly**	data (ROS)	—	—

How the capability model maps onto the brain:

Area	Behavior
Planner tool surface (§5.3)	MCP capability tools + local mission tools. The MCP tools come from `skill_control` (`list_skill_templates`, `instantiate_skill`, `stop_skill`) + `perception` (`what_do_you_see`, `get_latest`). Planner protocol: discover skills → instantiate → observe the buffer → stop.
Async-task registry / events (§5.4, §3)	The `/api/agent/events` ingress + registry handle `watch_event` pushes and ROS completions. `instantiate_skill` returns a topic, not a `task_id`/`running` TaskInfo — so skill use does not register an async task; only watchers do.
Safety stop (§9)	`stop_skill` all active vision skills + motion-node stop; `mcp.yaml` `motion: true` flags motion bridge servers.
`mcp.yaml` (§10)	`skill_control` + `perception` + `memory` entries; `perception` keeps `event_secret` (watcher push).
Teaching distillation (§8)	A process reads as skills instantiated + topics observed (e.g. "instantiate `find_object(scissors)`, watch `/vision/find_object/detections`").
Deployment runbook (§12)	Clone `robot-mcp-perception` (ROS node) + `mcp-skill-control` + `mcp-perception-buffer` + `robot-mcp-memory`; bring up ROS first, then the bridge servers, then the brain.

1. The Async Task Contract¶

This backend takes one dependency on robot-mcp-kit (for the shared pydantic TaskEvent / TaskFailure models and signing.verify). Everything else is in-repo.

1.1 Consumer view¶

Capability servers POST events to this app:

POST /api/agent/events
X-RVC-Server: navigation
X-RVC-Timestamp: <unix-seconds>            # ±300 s tolerance
X-RVC-Signature: sha256=<hex>              # HMAC-SHA256(secret, f"{ts}.{raw_body}")
Body: TaskEvent (robot_mcp_kit.models) —
  {schema_version, event_id, task_id, server, tool, type, seq, ts, payload}
  type ∈ progress | trigger_fired | task_done | task_failed | watch_event
  task_failed payload = TaskFailure {error_code, message, recoverable, suggestions}

Delivery is at-least-once and per-task ordered; dedupe on event_id. Reconciliation: every task-capable server exposes get_task(task_id) / cancel_task(task_id) / list_tasks(states=["running"]) MCP tools — poll after restart or event silence. Async-task tools return a TaskInfo JSON ({task_id, state: "running", est_duration_s, ...}) immediately; the executor recognizes this shape (§5.4).

2. Architecture overview (what goes where)¶

backend/src/robot_voice_chat/
├── agent/                          # NEW package — the mission brain
│   ├── executor.py                 # MissionExecutor: the agentic loop (§5)
│   ├── mission.py                  # Mission, PlanStep, MissionEvent dataclasses + states (§4)
│   ├── plan_tools.py               # set_plan / update_plan / finish_mission local tools (§5.3)
│   ├── reactions.py                # ReactionTable: event-pattern → tool call, zero LLM hop (§6)
│   ├── task_registry.py            # live TaskHandle map + reconciliation (§5.4)
│   ├── trace.py                    # TraceRecorder — full mission trace (always on) (§8)
│   └── teaching.py                 # teaching flag + distillation prompt + save to memory (§8)
├── services/llm/anthropic.py       # NEW AnthropicLLM adapter (§7)
├── api/routes/agent.py             # NEW /api/agent/* routes incl. events ingress (§3)
├── pipeline/…                      # MODIFIED: start_mission routing (§5.1), barge-in (§9)
├── services/tools/mcp.py           # MODIFIED: per-server event_secret + motion flag exposure
└── config/…                        # NEW groups + agent section (§10)
frontend/src/…                      # Mission pipeline view + Teach button (§11)

Design rules: the planner lives here, MCP servers are capabilities not coordinators; the fast path (the base single-turn pipeline) stays untouched for small talk; one mission at a time.

3. Event ingress — `POST /api/agent/events`¶

Route module api/routes/agent.py:

Read the raw body (signature is over raw bytes), look up the server's event_secret from the MCP config by the X-RVC-Server header, robot_mcp_kit.signing.verify(secret, ts, body, sig). Failures → 401 (log agent_event_rejected). Unknown server name → 401. Server entries without an event_secret configured cannot post (401).
Parse into TaskEvent; unknown schema_version > supported → 400.
Dedupe on event_id (in-memory LRU set, size 4096). Duplicates → 200 {"status": "duplicate"}.
Hand to MissionExecutor.on_task_event(event) (and to the ReactionTable first, §6). Events for unknown task_ids are logged agent_event_orphan and dropped with 200 (the task may belong to a previous process life).
Rate-limit: 120/min per server via the existing slowapi limiter.

Other routes (admin-token not required; these power the main UI):

Route	Behavior
`GET /api/agent/missions`	last N missions (id, instruction, state, started/ended, step summary) from the in-memory store + trace dir
`GET /api/agent/missions/{id}`	full mission snapshot (plan, steps, events) — same shape as the `mission_plan` socket event
`POST /api/agent/teaching`	`{"enabled": true}` → arm teaching for the next mission (§8); GET returns current state

4. Domain model (`agent/mission.py`)¶

MissionState = Literal["planning", "running", "paused", "succeeded", "failed", "cancelled"]
StepStatus   = Literal["pending", "running", "succeeded", "failed", "skipped"]

@dataclass
class PlanStep:
    id: str                    # "s1", "s2", … assigned by set_plan
    title: str                 # human-readable, shown in the UI ("Drive to the mailroom")
    tool: str | None           # dotted tool name if known ("navigation.navigate_to")
    status: StepStatus = "pending"
    detail: str = ""           # live detail line: progress %, ETA, result snippet, error
    task_id: str | None = None # linked async task, if any
    started_at: datetime | None = None
    ended_at: datetime | None = None

@dataclass
class Mission:
    id: str                    # "m-" + uuid4.hex[:6]
    instruction: str
    state: MissionState
    steps: list[PlanStep]
    created_at: datetime
    summary: str = ""          # set by finish_mission
    teaching: bool = False
    used_process_ids: list[str] = field(default_factory=list)

A MissionStore (in-memory dict + the trace files of §8 for history) holds the current mission and recent finished ones. Single-mission invariant: starting a mission while one is planning/running/paused is rejected — the fast path tells the user the robot is busy and what it's doing (it has the mission summary available).

5. MissionExecutor (`agent/executor.py`)¶

One asyncio task per mission, owned by a singleton MissionExecutor built in composition.py with: planner LLMProvider (§7), ToolBackend (existing MCP backend), EventEmitter, ScenarioRegistry (for the system prompt base), AppState, MemoryClient (thin wrapper that calls memory.find_processes / record_outcome through the ToolBackend), config (§10).

5.1 Entry: routing from the fast path¶

The fast-path LLM has one additional local tool (not an MCP tool — intercepted by the dispatcher exactly like use_predefined_answer):

start_mission(instruction: str, ack: str)
  description: "Call this when the user asks for a multi-step physical task (fetch, deliver,
  patrol, find-and-bring …) rather than a question or a single gesture. `instruction` is the
  task restated with all context resolved; `ack` is one short sentence to say immediately."

On call: speak/emit ack through the normal output path, then executor.start(instruction). The fast turn ends; the mission runs in the background. While a mission is active, every subsequent user turn is routed into the mission (executor.on_user_turn(turn)) instead of the fast path — status questions, corrections, and "stop" all land in the mission conversation. Small talk unrelated to the mission stays in the mission conversation too; the planner decides and can answer briefly via say.

5.2 The loop¶

async def _run(mission):
    processes = memory.find_processes(mission.instruction, k=3)     # executor-driven, not LLM
    messages  = [system prompt (§5.5) + processes] + [user: instruction]
    while mission.state in ("planning", "running", "paused"):
        step_llm:  response = await planner.chat(messages, tools=self._tools())
        execute tool calls (possibly several in parallel via asyncio.gather)
        append assistant msg + tool results to messages
        if no pending work and no tool calls → nudge once, then fail with NO_PROGRESS
        if awaiting events/user (LLM said so via wait_for_events) →
            suspend on self._wakeup (asyncio.Event), set by:
              · on_task_event(...)   — task progress/trigger/done/failed
              · on_user_turn(...)    — user spoke/typed mid-mission
              · timeout (config agent.wakeup_timeout_s, default 60) → inject a synthetic
                "still waiting; tasks: <registry snapshot>" tool result
        each wakeup: drain queued events → append as user-role messages (formatted §5.4) → next LLM step
    finalize (§5.6)

Budgets (config, hard caps): max_llm_steps (default 40), max_wall_clock_s (default 1800), max_cost_usd optional (computed from usage if the provider reports it). Exceeding one → mission failed, summary explains the budget, all live tasks cancelled.

5.3 The planner's tool surface (`agent/plan_tools.py` + MCP)¶

The planner sees, in one flat list:

All MCP tools from the existing ToolBackend (dotted names, unchanged schemas).
Local mission tools (executed in-process, never reach MCP):

Tool	Effect
`set_plan(steps: [{id, title, tool?}])`	Replace the mission plan. Must be the planner's first call. Emits `mission_plan` (§11.1).
`update_plan(steps)`	Same shape; used on replanning. Steps already terminal keep their status (match by id); new ids start `pending`; removed pending steps become `skipped`.
`mark_step(step_id, status: running\|succeeded\|failed\|skipped, detail?)`	Manual status control for steps without a 1:1 tool call.
`say(text)`	Speak/emit to the user mid-mission via the existing `SpeechOutput` + emitter (chunked TTS path reused).
`register_reaction(event: {server, task_id?, type, match?}, call: {tool, arguments})`	Add a prepared reaction (§6).
`wait_for_events(reason: str)`	Explicitly suspend until the next wakeup. The loop also auto-suspends when a step ends with running tasks outstanding and no tool calls.
`ask_user(question)`	`say` + suspend until `on_user_turn`.
`finish_mission(outcome: succeeded\|failed, summary: str)`	Terminal.

Step linkage: every MCP tool call may carry step_id (the executor injects an optional step_id property into each MCP tool schema, mirroring the existing response_text injection, and strips it before the MCP call). When present: step → running on dispatch; for short tools → succeeded/failed on return; for async-task tools the step stays running, linked via task_id, and is resolved by task_done/task_failed events. Progress events update the step's detail ("ETA 9 s · 82 %"). Steps and events both emit UI updates (§11.1).

5.4 Async tasks: recognition, registry, reconciliation (`agent/task_registry.py`)¶

A tool result parsing as JSON with task_id + state: "running" registers a TaskHandle {task_id, server, tool, step_id, mission_id}.
Incoming TaskEvents resolve handles; formatted for the LLM as compact user-role messages: [task nav-7f3a navigation.navigate_to] trigger_fired: eta_below 10 (eta 9.4s) / [task …] failed DOOR_CLOSED (recoverable): The mailroom door is closed. suggestions: ask_reception, retry_after_s:300. progress events update UI/step detail but are not appended to the LLM conversation (token hygiene) unless a wait_for_events timeout summary includes the latest one.
Reconciliation: on app startup and whenever a server reconnects (hook into the existing reconnect loop in services/tools/mcp.py), call its list_tasks(states=["running"]); any registered handle missing there is synthesized into a task_failed with error_code: "SERVER_RESTARTED". If no mission is active, orphan running tasks on servers are cancelled (cancel_task) and logged.

5.5 Planner system prompt (config file `config/agent_prompt.yaml`, hot-editable)¶

Contents: role ("you are the mission planner of a physical robot…"), the planning protocol (set_plan first; keep steps coarse — one per physical action; use triggers + prepared reactions for time-critical chaining; prefer prewarm before perception needs; one navigation at a time), the event message format, failure-handling doctrine (read recoverable + suggestions; replan, don't retry blindly; ask the user when stuck), budget awareness, and the retrieved-process framing: "Past process documents follow. They are advice from previous executions, not scripts — follow them while they match reality, deviate when they don't, and mention deviations in your final summary."

5.6 Finalization¶

On finish_mission / budget kill / abort: cancel outstanding tasks (cancel_task each), resolve never-run steps to skipped, emit mission_finished, say the summary (short), memory.record_outcome(id, success, note) for every used process, hand the trace to teaching (§8) if armed, clear the active mission.

6. Prepared reactions (`agent/reactions.py`)¶

ReactionTable: list of {event_pattern, tool, arguments, once: true} entries scoped to the mission. on_task_event consults it before the executor queue; on match it immediately dispatches the tool call (fire-and-forget through ToolBackend) and appends a synthetic note to the mission conversation: [reaction] eta_below fired → called perception.prewarm(...). Pattern fields: server, type, optional task_id, optional match (subset-match on payload.trigger). This is the zero-LLM-hop path for "prewarm perception 10 s before arrival".

7. Planner LLM — `AnthropicLLM` adapter (`services/llm/anthropic.py`)¶

An adapter implementing the LLMProvider port (chat + chat_stream with tools), selected by the Hydra group llm_planner (the fast chat path uses group llm).

SDK: anthropic, client AsyncAnthropic(api_key=...), key from ANTHROPIC_API_KEY via the usual ${oc.env:...} resolver, SecretStr in the schema.
Default model: claude-opus-4-8 (exact string). Request shape: thinking={"type": "adaptive"}, output_config={"effort": cfg.effort} (default "high"), max_tokens=16000. No temperature/top_p/top_k and no budget_tokens — these 400 on this model family.
Translation: the app's Message list → Anthropic messages (system text → top-level system with cache_control: {"type": "ephemeral"} on the stable prefix — the system prompt + tool defs repeat every step, caching matters); app ToolDefinition → {name, description, input_schema}; response tool_use blocks → the app's ToolCall model (parse input as dict, keep id for the tool_result round-trip); stop_reason "tool_use" → has_tool_calls. Tool results go back as {"type": "tool_result", "tool_use_id": …, "content": …} user-role blocks — extend the app's Message model with an optional tool_results field if it can't express that today.
Streaming (chat_stream) is required by the port: use client.messages.stream(...) and yield text deltas; the executor itself uses non-streaming chat.
Retries/timeout: reuse services/resilience.py conventions; request_timeout_s config default 120 (planner steps can think).

# config/llm_planner/anthropic.yaml
_target_: robot_voice_chat.services.llm.anthropic.AnthropicLLM
api_key: ${oc.env:ANTHROPIC_API_KEY}
model: claude-opus-4-8
max_tokens: 16000
effort: high
request_timeout_s: 120.0
# config/llm_planner/groq.yaml — reuses the existing GroqLLM target (no new code), so a
# deployment can run single-provider.

The root defaults list carries - llm_planner: anthropic. The composition root builds the adapter only when agent.enabled, and construction is lazy and failure-tolerant: if the key is missing, it logs a warning, disables missions, and keeps the app up (chat-only dev deployments boot without an Anthropic key).

8. Trace + teaching (`agent/trace.py`, `agent/teaching.py`)¶

TraceRecorder (always on): every mission appends JSONL to data/missions/{mission_id}.jsonl — one line per: instruction, retrieved processes (ids+scores), each LLM step (request size, tool calls), each tool dispatch/result, each task event, each user turn, plan snapshots, final outcome + summary + timings. This is observability and the teaching input. Retention: prune files older than agent.trace_retention_days (default 30) at startup.

Teaching: POST /api/agent/teaching {"enabled": true} (UI button) arms a one-shot flag; the next mission records teaching: true. On finalization, teaching.py:

Renders the trace into a compact text transcript (tool calls + args + result snippets + failures + fixes + user corrections; drop progress noise).
One distillation call on the planner LLM with a fixed prompt (in config/agent_prompt.yaml): produce a process document in exactly this markdown format — the robot-mcp-memory doc format (frontmatter: id slug, title, task_pattern, tags, body sections ## Steps, ## Pitfalls & fixes, ## Operator notes). Include the format spec verbatim in the prompt.
memory.save_process(markdown) via ToolBackend; emit teaching_saved {process_id}; disarm. Distillation failure: log + emit teaching_failed, never crash finalization.

9. Barge-in: safety stop → pause → confirm¶

The rule is halt motion immediately, then ask.

The stop_pipeline socket event and a spoken "stop" (fast-path classification gets a stop_mission local tool; the mission-routed turns get the same via the planner — but do NOT wait for the planner: the executor pre-scans mission-routed user turns for a config word list agent.stop_words (["stop", "halt", "stopp", "arrête", "fermati"]) and acts before the LLM).
On stop: (1) for every server flagged motion: true in mcp.yaml, cancel_task all its live tasks and call its stop_navigation-class tool if configured (safety_stop_tool per server entry); (2) output.interrupt() (drop queued speech); (3) mission → paused, steps of cancelled tasks → failed with detail "safety stop"; (4) say "Stopped. Should I continue or abort?"; (5) suspend.
Next user turn while paused goes to the planner with a system note: mission is paused after a safety stop; user reply follows; either update_plan + continue, or finish_mission(failed/ cancelled). Continue = replan from current state (the pose is whatever it is now).

10. Config¶

# config/config.yaml — defaults list
  - llm_planner: anthropic          # NEW group

agent:                              # NEW section (Pydantic AgentConfig, validated)
  enabled: true
  max_llm_steps: 40
  max_wall_clock_s: 1800
  wakeup_timeout_s: 60
  trace_retention_days: 30
  stop_words: ["stop", "halt", "stopp"]
  prompt_file: config/agent_prompt.yaml
  memory_server: memory             # name of the process-store MCP server ("" disables retrieval)

config/tools/mcp.yaml server entries gain optional keys (extend McpServerConfig):

servers:
  - name: navigation
    transport: sse
    url: "http://127.0.0.1:9101/sse"
    event_secret: ${oc.env:NAV_EVENT_SECRET}      # enables event ingress for this server
    motion: true                                   # included in safety stop
    safety_stop_tool: stop_navigation              # called unconditionally on stop
  - name: perception
    transport: sse
    url: "http://127.0.0.1:9102/sse"
    event_secret: ${oc.env:PERCEPTION_EVENT_SECRET}
  - name: memory
    transport: sse
    url: "http://127.0.0.1:9103/sse"
blocking_tools: ["perception.describe_scene"]

.env.example gains: ANTHROPIC_API_KEY=, NAV_EVENT_SECRET=, PERCEPTION_EVENT_SECRET= (generate per robot: python -c "import secrets; print(secrets.token_urlsafe(32))" — the same value goes into the capability server's .env as RVC_EVENT_SECRET).

11. Frontend: mission pipeline view + Teach button¶

11.1 Socket.IO events (server → client; declared in `types/events.ts`)¶

Event	Payload
`mission_started`	`{mission_id, instruction, teaching}`
`mission_plan`	full snapshot: `{mission_id, state, steps: [{id, title, tool, status, detail, task_id}]}` — emitted on set_plan/update_plan AND on every step change (snapshot > diffs; trivially correct UI)
`mission_event`	`{mission_id, kind: tool_call\|task_event\|reaction\|say\|user_turn, text, ts}` — the activity log line
`mission_finished`	`{mission_id, outcome, summary}`
`teaching_state`	`{armed: bool}` / `teaching_saved {process_id}` / `teaching_failed {reason}`

Client → server: none (teaching toggle is REST; stop reuses stop_pipeline).

11.2 UI¶

MissionPanel (rendered above/inline with the conversation feed whenever a mission is active or recently finished; Zustand missionStore fed by the events above):

Pipeline view — a vertical list (timeline) of plan steps, each a card with title, tool name (mono, dimmed), live detail line, and a status color:

status	color treatment
`pending`	neutral gray, hollow dot
`running`	blue, pulsing dot + subtle progress shimmer; show `detail` (ETA/%)
`succeeded`	green, check icon
`failed`	red, ✕ icon; `detail` shows `error_code` + message
`skipped`	amber/strikethrough

Use semantic Tailwind tokens consistent with the existing theme system (data-theme aware), not hard-coded hex. On update_plan the list re-renders from the snapshot — steps may appear/ reorder (replanning is visible to the user, which is the point). - Activity log — collapsible feed of mission_event lines under the pipeline (tool calls, trigger fires, reactions, says). - Header row — mission instruction, state chip (same color language), elapsed time, and a Stop button (emits stop_pipeline). When paused: highlighted "Paused — continue or abort?" with two buttons that just send_message("continue") / send_message("abort"). - Teach button in the main Header: toggles POST /api/agent/teaching; armed state shows a pill ("Teaching next mission"); on teaching_saved show a toast with the process id. - mission_finished collapses the panel into a summary card in the conversation feed.

GET /api/agent/missions/{id} lets the panel rehydrate after a page reload (fetch on mount if the store is empty but /api/agent/missions reports an active mission).

12. Robot deployment runbook (clone → running fleet node)¶

Target: fresh Jetson/Ubuntu host (or macOS for a dry run). All repos under one root.

# 0) Prereqs (Ubuntu): uv, node, audio (only for voice), as in this repo's README §Production.
sudo mkdir -p /opt/robot /opt/mcp-servers && sudo chown $USER /opt/robot /opt/mcp-servers

# 1) Clone everything
git clone git@github.com:<org>/robot-voice-chat.git      /opt/robot/robot-voice-chat
for r in robot-mcp-navigation robot-mcp-perception robot-mcp-memory; do
  git clone git@github.com:<org>/$r.git /opt/mcp-servers/$r
done
# robot-mcp-kit is a pip dependency of the servers — no separate checkout needed on the robot.

# 2) Secrets — generate the per-server HMAC secrets ONCE per robot
NAV_SECRET=$(python3 -c "import secrets;print(secrets.token_urlsafe(32))")
PERC_SECRET=$(python3 -c "import secrets;print(secrets.token_urlsafe(32))")

# 3) Capability servers: install + env + systemd  (repeat per server; ports 9101/9102/9103)
cd /opt/mcp-servers/robot-mcp-navigation && uv sync
cat > .env <<EOF
RVC_EVENT_SECRET=$NAV_SECRET
EOF
sudo cp deploy/robot-mcp-navigation.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now robot-mcp-navigation
#   perception additionally needs GEMINI_API_KEY in its .env. Without a camera, override the
#   frame source via PERCEPTION_CONFIG_OVERRIDES: "frame_source=folder
#   frame_source.directory=/opt/mcp-servers/robot-mcp-perception/sample_images"
#   memory needs ROBOT_ID=<this robot's name> in its unit/.env.

# 4) Chat app
cd /opt/robot/robot-voice-chat/backend && uv sync
cp .env.example .env   # then set: GROQ_API_KEY or OPENAI_API_KEY, ANTHROPIC_API_KEY,
                       # NAV_EVENT_SECRET=$NAV_SECRET, PERCEPTION_EVENT_SECRET=$PERC_SECRET,
                       # ADMIN_PASSWORD_HASH + ADMIN_SECRET_KEY (see README admin section)
cd ../frontend && npm ci && npm run build
sudo cp ../deploy/robot-voice-chat.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now robot-voice-chat

# 5) Smoke test — run in order; each line must pass before the next
curl -s http://127.0.0.1:9101/sse -m 2 -o /dev/null -w "%{http_code}\n"       # 200 (SSE open)
curl -s http://127.0.0.1:9112/healthz                                          # perception artifacts ok
curl -s http://127.0.0.1:8000/api/livez                                        # chat app up
curl -s http://127.0.0.1:8000/api/readyz | jq .data.mcp_servers                # navigation/perception/memory: connected
journalctl -u robot-voice-chat -o cat | grep mcp_server_connected              # 3 lines (+gesture server if present)

# 6) Functional smoke (browser → http://<robot>:8000 behind TLS per README, or :5173 in dev)
#    a) type: "drive to the mailroom and tell me when you arrive"
#       → MissionPanel appears, navigate_to step turns blue with live ETA, then green;
#         mission_finished summary is spoken/shown.
#    b) mid-drive type "stop" → motion task cancelled, panel shows Paused, robot asks
#       continue/abort.
#    c) Teach + "fetch the package from the mailroom" → after finish, toast "process saved";
#       then repeat the same instruction → trace shows the process injected
#       (journalctl … | grep processes_retrieved).

Order matters only for step 5's expectations — services can start in any order; the chat app's 30 s reconnect loop adopts late servers, and reconciliation (§5.4) handles server restarts. TLS, fleet naming, and admin hardening: unchanged, see the main README.

13. Tests¶

Unit (tests/unit/agent/), all with fake LLM/ToolBackend/clock:

Executor happy path: scripted planner (set_plan → navigate_to (async) → wait → trigger → reaction fires → task_done → finish) — assert step statuses, emitted socket events, conversation contents, summary.
Replanning: task_failed DOOR_CLOSED wakes the loop; planner's update_plan preserves terminal steps, marks removed ones skipped.
Budgets: step/wall-clock caps fail the mission and cancel live tasks.
Routing: start_mission tool call hands off; mission-active turns route to on_user_turn; second mission rejected with busy message.
Barge-in: stop word pre-scan cancels motion tasks + calls safety_stop_tool, pauses, resumes via planner.
Task registry: TaskInfo-shaped results registered; reconciliation synthesizes SERVER_RESTARTED; orphan events dropped.
Events route (integration, TestClient): valid HMAC accepted; bad signature/skewed timestamp/unknown server → 401; duplicate event_id → deduped; reaches a stub executor.
AnthropicLLM (respx/mocked SDK): message+tool translation both directions; no sampling params in requests; tool_result round-trip; streaming yields deltas.
Teaching: trace → distillation prompt contains failures/fixes; save_process called; distillation error doesn't break finalization.
Frontend: npm run typecheck + a MissionPanel render test over a snapshot fixture covering all five step states.

14. Acceptance criteria¶

All backend gates (pytest, ruff check, ruff format --check, mypy, import-linter) and frontend gates (typecheck, build) pass.
import-linter enforces that agent/ is application-layer — it may import services/**/base.py ports and models, never concrete adapters.
Runbook §12 runs end-to-end on a dev machine against the real capability servers (perception with frame_source=folder).
The mailroom scenario works: plan visible in UI with correct colors, prewarm fires via a prepared reaction at eta_below:10 before arrival, failure injection (map failures: entry) triggers visible replanning, and the teaching round-trip stores and later retrieves the process.
Chat-only RVC_DEPLOYMENT=dev (no Anthropic key, no servers) boots clean.

Appendix A — Repository & tooling conventions¶

Note

The brain (this repo) follows these conventions and is the reference implementation for them. They are listed here so the companion repos match it. The MCP servers specifically notes describe the companion servers; the brain is the MCP client/consumer.

Structure — src/<package>/ layout (never flat); config/ Hydra tree; tests/{unit,integration}/; committed README.md (quickstart + how-to table + troubleshooting table), CHANGELOG.md (Keep-a-Changelog + SemVer), .pre-commit-config.yaml, .github/ (CI + dependabot.yml), optional multi-stage non-root Dockerfile, .env.example committed / .env gitignored.

Tooling — uv for env + deps (uv sync, lockfile committed), Python ≥ 3.11, optional extras for heavy/optional deps (the --extra observability pattern). ruff (lint + format), mypy --strict, pytest + pytest-asyncio, coverage gate ≥ 80%. Pre-commit: trailing-whitespace, end-of-file-fixer, check-yaml/toml/merge-conflict/large-files, ruff (--fix), ruff-format, and a local mypy hook run via uv run (so it picks up the pydantic.mypy plugin). CI on every PR: pytest, ruff check, ruff format --check, mypy, pip-audit. Dependabot weekly for the uv ecosystem.

Architecture — ports-and-adapters: every capability with >1 implementation or that touches hardware/network gets an abc.ABC port in base.py; concrete adapters live beside it; application code imports only ports. Hydra + Pydantic config: one config-group dir per port, one YAML per implementation carrying _target_; swap implementations by changing one line in the root config.yaml defaults list — no if provider == … ladders. Validate the composed config through Pydantic dataclasses at startup (fail fast). Deployment presets under config/deployment/<name>.yaml selected via an env var.

Secrets — never in YAML or git. .env + OmegaConf ${oc.env:KEY} resolvers, typed SecretStr; startup refuses to boot if a required key is empty.

Logging — structlog JSON by default, pretty renderer toggle for dev; bind correlation IDs (request/turn/mission) via contextvars; log duration_ms per stage. Subprocess — always argv lists, never shell=True. Testing — fakes at the port boundary (don't patch internals); respx for HTTP-level mocking; integration via httpx.AsyncClient + ASGITransport.

MCP servers specifically — one repo per capability from a copy-me template; transport stdio if the consumer supervises it (1–2 co-located), else sse/streamable_http with its own systemd unit/container/logs/restart policy and the consumer just holds a URL. Tools namespaced server.tool; mark slow perception tools "blocking", else fire-and-forget; expect a ~30 s call timeout; tolerate clients health-checking via list_tools(). Resilience: the consumer reconnects on an interval — design for clean restart, never assume connection order at boot. Deploy: systemd unit with hardening (NoNewPrivileges, PrivateTmp, ProtectSystem=strict), bind localhost, dedicated user, journald.