AuthorPascal DateJuly 14, 2026 Rev1.0

Agentic Orchestration Concept — v2 (control plane / data plane)¶

Status: Redesign for discussion — supersedes v1 Date: 2026-06-13 Scope: How robot-voice-chat becomes the conversational + configuration brain of a robot that runs a ROS 2 stack, where the ROS modules talk to each other over ROS, and the brain configures and observes that stack rather than shuttling data through itself.

Why this is a redesign. v1 made every capability an MCP tool that the brain calls, and had the brain pass results (even pixels) from one capability to the next. That is wrong for a robot: the segmentation module must hand a mask to the manipulation module directly over ROS, at sensor rates, with the brain nowhere in that loop. The brain's job is to chat and to configure the ROS stack (turn capabilities on, point them at the right target, wire who listens to whom) and then watch what the stack produces. v2 encodes that split.

Decisions taken (2026-06-12 → 2026-06-13):

Question	Decision
Brain ↔ ROS boundary	MCP = control plane (brain configures + observes; brain never imports `rclpy`). ROS topics = data plane (module ↔ module).
Skill discovery + use	Two isolated MCP servers: `mcp-skill-control` (discover + instantiate skills) and `mcp-perception-buffer` ("what do you see").
Vision	One configurable ROS node = the existing `robot-mcp-perception` repo (built from scratch, not a fork): grounder Gemini Robotics-ER + segmenter SAM 3, combined per skill config, publishing to ROS topics. Absorbs the useful patterns from the `ros2-vlm-groq` sample (dynamic runtime tasks + ROS publishing); Groq llama-vision is dropped (insufficient localization). Fresh `ros2_vision_interfaces` message package.
Navigation	Dropped. `robot-mcp-navigation` repo + spec deleted (prototype under development elsewhere).
Manipulation	Designed-for, not built. It is a future ROS consumer of the vision topics; the vision output contract is manipulation-ready.
Planner / brain	Mission brain (MissionExecutor, planner LLM, teaching, mission UI) survives; its capability-interaction model shifts from "call MCP async task" to "instantiate a skill = configure ROS, then observe the buffer."
Learned-process store	Unchanged: per-robot, local-first (`robot-mcp-memory`, SQLite FTS5 + markdown).
Episodic memory	`ros2-memory` (rosbag/MCAP recorder + temporal-query MCP) available as a sample; out of immediate scope, noted as a future capability.

1. The two planes¶

  ╔══════════════════════ CONTROL PLANE (MCP) ══════════════════════╗
  ║  brain configures the stack and observes it. NO rclpy in brain.  ║
  ║                                                                  ║
  ║   robot-voice-chat (brain): chat · mission planner · skill use   ║
  ║        │                                   │                     ║
  ║        │ MCP: discover + instantiate       │ MCP: read buffer    ║
  ║        ▼                                   ▼                     ║
  ║  ┌──────────────────┐              ┌──────────────────────────┐  ║
  ║  │ mcp-skill-control│              │ mcp-perception-buffer    │  ║
  ║  │ (rclpy + MCP)    │              │ ("what do you see")      │  ║
  ║  │ list/instantiate │              │ (rclpy + MCP)            │  ║
  ║  └────────┬─────────┘              └───────────▲──────────────┘  ║
  ╚═══════════│══════════════════════════════════ │ ════════════════╝
              │ ROS service: create/manage/list    │ ROS subscribe
              │ vision task                         │ /vision/**
  ╔═══════════▼═════════════════════ DATA PLANE (ROS topics) ════════╗
  ║                                                                  ║
  ║  /camera/image_raw ─► ┌───────────────────────────┐             ║
  ║                       │ robot-mcp-perception node │             ║
  ║                       │  grounder (Gemini ER) ─┐  │             ║
  ║                       │  segmenter (SAM 3) ────┴─►│             ║
  ║                       └───────────┬───────────────┘             ║
  ║         /vision/{skill}/detections (label,box,mask,point)        ║
  ║         /vision/{skill}/{output}  (semantic: text/num/bool/list) ║
  ║                                   │                              ║
  ║                                   ▼                              ║
  ║                    (future) manipulation node  ◄── subscribes    ║
  ║                                              directly, no brain  ║
  ╚══════════════════════════════════════════════════════════════════╝

Three rules:

The brain never imports rclpy. It speaks MCP only. Every ROS interaction is brokered by a bridge server (skill-control, perception-buffer) that is a ROS node and also serves MCP.
Modules exchange data over ROS, not through the brain. Vision publishes masks; manipulation subscribes to them. The brain is not in that path — it only set it up.
A capability is configured, then it runs autonomously. "Use the camera to find the scissors" is not a request/response tool call — it is instantiating a skill that makes the vision node start publishing scissor masks on a ROS topic, which the brain (via the buffer) and other ROS modules can consume until it is stopped.

2. Skills — the core abstraction¶

A skill is a named, declarative capability configuration that, when instantiated, configures one or more ROS nodes to start producing a stream on a ROS topic. Today every skill is a vision skill (a configuration of the robot-mcp-perception node); the model generalizes to future nodes (manipulation policies, etc.).

# a skill template (parameterized; lives in the catalog)
name: find_object
kind: vision
mode: ground_and_segment          # describe | ground_and_segment | segment_all | region
params: [query]                   # filled in at instantiation
grounding: "the {query}"          # scene-understanding prompt (what/where)
segment: true                     # run SAM on the grounded target(s)
outputs:                          # semantic side-channel (optional)
  - {name: found, type: boolean, description: "whether the object is visible}
publishes: /vision/{name}/detections   # label, box, mask, grasp point

Your three examples are one node, three skill configs:

Instruction	mode	grounding	segment	publishes
"detect the scissors lying on top"	`ground_and_segment`	"the scissors on top of the pile"	yes	mask+box+point of that one object
"segment anything"	`segment_all`	—	yes	masks for every object
"where can I put the package"	`region`	"free flat space big enough for the package"	region-only	placement rectangle

Skill template — parameterized capability in the catalog (e.g. find_object(query)).
Skill instance — a template filled in and running on the ROS node = a live vision task publishing to a topic.
Skill discovery — list_skill_templates() + list_running_skills() (on mcp-skill-control).
Skill use ("dynamically make use of them") — instantiate_skill(template, params) → {skill_id, topic}; stop_skill(skill_id). Instantiation calls the vision node's ROS service.
Skills are data: built by configuring a VLM/segmenter, not by writing code. New skills = new configs. Learned-good skills can be persisted (and later synced) via robot-mcp-memory.

This is exactly the task system already proven in ros2-vlm-groq (create_vlm_task / list_vlm_tasks / manage_vlm_task + task templates), extended with a segmentation stage and a real ROS output contract.

3. The configurable vision node (`robot-mcp-perception`)¶

One ROS 2 node — the existing robot-mcp-perception repo, built from scratch (not a fork; it absorbs the patterns of the ros2-vlm-groq sample) — that does both halves of perception and is configured entirely by skills:

Grounder (what/where): Gemini Robotics-ER (gemini-robotics-er-1.6-preview), strong at pointing/boxes. Answers "which object / where" from the skill's grounding prompt and emits the semantic outputs (text/number/boolean/list). Groq llama-vision is not used — its localization is insufficient. The grounder port stays pluggable (grounded-SAM as a local fallback).
Segmenter (pixels): SAM 3 (sam3.pt; sam2.1_* faster fallbacks), prompted by the grounder's point/box, producing the mask. Local today; may move to the cloud later — so it carries a cost_class and the cost policy (§ perception spec §8) meters it the same as the grounder when it does.
Router: per skill mode, runs grounder-only (describe), grounder→segmenter (ground_and_segment), segmenter-over-everything (segment_all), or grounder→region (region). These map onto the repo's existing reason / tiled / watch pipelines.

It publishes to ROS (the data plane):

/vision/{skill}/detections — VisionDetectionArray (label, score, box, mask, grasp point)
/vision/{skill}/{output} — StampedString per semantic output (unchanged from ros2-vlm-groq)
/vision/{skill}/overlay — sensor_msgs/Image annotated frame (debug / UI)

It is configured over ROS services (the control plane reaches it through mcp-skill-control): create_vision_task / manage_vision_task / list_vision_tasks (the ros2-vlm-groq service surface, extended with mode / grounding / segment).

Masks travel as ROS messages (fresh ros2_vision_interfaces package) on the data plane — a manipulation node subscribes to /vision/{skill}/detections and gets the mask directly, no brain, no HTTP artifact fetch. (Full message + topic + config contract: specs/robot-mcp-perception.md.)

4. The two MCP bridge servers¶

Both are ROS nodes that also serve MCP (they import rclpy; the brain does not). They are the only things that translate between the brain's MCP world and the ROS data plane.

4.1 `mcp-skill-control` — discover + instantiate skills¶

The brain's hands. Owns the skill template catalog and instantiates skills by calling the robot-mcp-perception node's ROS services. Tools (control plane):

Tool	Effect
`list_skill_templates()`	The catalog — what the robot can be configured to do.
`list_running_skills()`	What is currently publishing (live vision tasks).
`instantiate_skill(template, params)` → `{skill_id, topic}`	Fill a template, create the vision task, return the ROS topic to read.
`stop_skill(skill_id)`	Tear down a running skill.

(Built from ros2-vlm-groq's vlm_mcp_server.py, which already proves the MCP→ROS-service bridge and ships task templates. Full contract: specs/mcp-skill-control.md.)

4.2 `mcp-perception-buffer` — "what do you see"¶

The brain's eyes. Subscribes to the /vision/** topics and keeps a rolling latest snapshot, so the brain can ask "what do you see?" without a round-trip into the vision pipeline (the state_collector.py pattern, but ROS-subscription-fed and exposed over MCP). Tools (read-only):

Tool	Effect
`what_do_you_see()`	Latest scene snapshot across all active skills (semantic + object list).
`get_latest(skill)`	Latest detections/outputs for one skill (label/box/score; mask by reference).
`list_streams()`	Which `/vision/**` topics are live and their last-update age.

Optional push: when a skill is configured as a watcher (e.g. "tell me when a person enters"), the buffer can POST a watch_event to the brain's existing /api/agent/events endpoint via robot-mcp-kit — the same Async Task Contract the brain already speaks. (Full contract: specs/mcp-perception-buffer.md.)

Important nuance — the buffer is free to read, but something had to run a model to fill it. what_do_you_see() returns whatever a describe-mode skill last published; if no such skill is running, the buffer is cold and a fresh answer needs a model call. So "what do you see" resolves in three tiers, by recency:

Question	Served by	Cost
"what do you see now" (warm)	buffer cached latest	free (no model call)
"look again / describe now" (cold)	a `once` `describe` skill → one grounder call	one cloud call
"what did you see N min ago / last time X"	`ros2-memory` (rosbag history)	free (reads a bag)

To keep tier 1 usually-available cheaply, the vision node can run a gated low-rate ambient scene skill (perception §8 cost policy) — the frame-diff gate means a static scene costs almost nothing. The buffer flags each value stale/age_s so the planner knows when to refresh vs. read.

Why split control and buffer into two servers (your call): clean isolation — instantiation (write/config, talks ROS services) and observation (read, subscribes ROS topics) are different jobs with different failure modes and lifecycles; either can restart without the other.

5. What the brain does with all this (mission flow)¶

The mission brain is unchanged in shape — planner LLM loop, teaching, mission UI, budgets — but a "capability use" is now configure-then-observe instead of call-and-wait:

mission: "bring me the scissors from the tool tray"
  1. discover:     mcp-skill-control.list_skill_templates()      → sees find_object, segment_all, …
  2. instantiate:  mcp-skill-control.instantiate_skill(
                       "find_object", {query:"scissors on top"})  → {skill_id, topic:/vision/find_object/detections}
  3. observe:      mcp-perception-buffer.get_latest("find_object") → label/box/score (+ mask ref)
                   (a manipulation node, when it exists, subscribes to that topic DIRECTLY and grasps)
  4. adjust/stop:  mcp-skill-control.stop_skill(skill_id) when done

The brain never carries the mask. It carries the topic name; the ROS consumer reads pixels.
Skill discovery is real: the planner lists templates at mission start (and can compose new ones), so "dynamically make use of skills" is a first-class loop step, not hardcoded tools.
Teaching still records the trace and distills a process; a learned process now reads as "instantiate find_object(scissors), watch /vision/find_object/detections, …" — i.e. the skills it configured, which is exactly the reusable artifact.
The Async Task Contract / robot-mcp-kit still applies to push events (watcher fires, long-running skill completes) but is no longer the primary interaction — most observation is the pull buffer. Navigation's eta_below-style async tasks go away with navigation.

The precise delta to the already-built brain code is in specs/robot-voice-chat-agent.md (§0, added in this redesign).

6. Component & repo map¶

Component	Repo	Plane	Status	Spec
Mission/chat brain	`robot-voice-chat` (this repo)	control (MCP client)	built; small delta	`specs/robot-voice-chat-agent.md`
Configurable vision node	`robot-mcp-perception` (scaffolded; Gemini ER + SAM 3)	data (ROS node)	building	`specs/robot-mcp-perception.md`
Vision ROS messages	`ros2_vision_interfaces` (new, fresh/clean)	data (ROS msgs/srv)	to build	(in `specs/robot-mcp-perception.md` §6.2)
Skill discovery + instantiation	`mcp-skill-control` (new, pattern from `vlm_mcp_server.py`)	control (MCP↔ROS-srv bridge)	to build	`specs/mcp-skill-control.md`
"What do you see" buffer	`mcp-perception-buffer` (new, pattern from `state_collector.py`)	control (MCP↔ROS-sub bridge)	to build	`specs/mcp-perception-buffer.md`
Async Task Contract lib	`robot-mcp-kit`	both (shared)	published (v0.1.0)	`specs/robot-mcp-kit.md`
Process/skill store	`robot-mcp-memory`	control (MCP)	implemented (v1 spec; v2 unchanged)	`specs/robot-mcp-memory.md`
Episodic ROS-bag memory	`ros2-memory` (clean reimpl of `mcp-rosbag-memory`)	control (MCP `episodic`, reads bags)	in scope	`specs/ros2-memory.md`
~~Navigation~~	~~`robot-mcp-navigation`~~	—	deleted	—
(future) Manipulation	`ros2-manipulation`	data (ROS consumer)	not yet (prototype)	—

Per-robot composition stays pure config: which ROS nodes run, and which MCP bridge servers the brain's tools/mcp.yaml lists. The brain code is identical on every robot.

7. Build order — all in parallel¶

The five repos are spec'd to be independently buildable; we build them in parallel. The only hard contract between them is ros2_vision_interfaces (define it first / early so the vision node and the two bridge servers compile against the same messages). Natural internal sequencing within each track (not a cross-repo dependency):

Vision (robot-mcp-perception): understanding skills (Gemini ER) green first → add SAM 3 segmentation + VisionDetectionArray → the three example skills on a topic.
Bridges (mcp-skill-control, mcp-perception-buffer): build against a fake ROS vision node (a stub publishing canned VisionDetectionArray), so they don't block on the real node.
Episodic memory (ros2-memory): build against fixture bags; only the recorder=ros2bag integration test needs a live ROS env.
Brain delta (robot-voice-chat): swap navigation wiring for skill-config wiring; planner discovers/instantiates skills; teaching distills skill-based processes. (Brain core already built.)

8. Open questions — RESOLVED (2026-06-13)¶

Question	Decision
Grounder	Gemini Robotics-ER (`gemini-robotics-er-1.6-preview`). Groq llama-vision removed — insufficient localization. Grounder port stays pluggable (grounded-SAM local fallback).
Segmenter	SAM 3 (`sam3.pt`); `sam2.1_*` as faster fallbacks; `none` for no-GPU dev.
Interfaces package	Fresh `ros2_vision_interfaces` (clean) — not the messy `ros2mcp_interfaces`.
Watcher push	Lives in `mcp-perception-buffer` (it already subscribes); rising-edge → `watch_event` via `robot-mcp-kit`.
Episodic ROS-bag memory	In scope. `ros2-memory` (clean reimpl of `mcp-rosbag-memory`) records the `/vision/**` + state topics to bags and answers temporal queries ("what did you see N s ago", "last time X"). Complements the buffer (latest) — see the three-tier table in §4.2.
Cost of perception	Metered: every cloud model stage is gated/budgeted (`specs/robot-mcp-perception.md` §8). Segmentation may become cloud too, so the policy keys on a per-adapter `cost_class`, not on "grounding only".
Mask transport	Full-frame `mono8` on ROS at ≤2 Hz (≈0.3 MB) — fine on-robot; RLE/compressed later behind the same message if manipulation needs it.