Per-repo build prompts¶
This page collects one paste-ready LLM prompt per repo in the stack. Each prompt names the
authoritative spec and any reference code to study, written as @… placeholders — replace
each @… with the actual file or folder you attach (the spec .md from this specs/ directory, and
the reference sample repo). The spec is authoritative; references are to study, not copy verbatim.
The repos can be built in parallel. The only hard cross-repo dependency is ros2_vision_interfaces —
build and define it first so the others compile against the same messages.
No prompt for robot-mcp-kit
robot-mcp-kit is implemented and published (v0.1.0) and needs no change: the watch_event
type is already in the contract, and EventPoster, signing, and the models are unchanged (it is
now used push-only). Leave it as-is.
1. ros2_vision_interfaces — build first¶
Attach: @robot-mcp-perception.md (only §6.2 matters).
You are building `ros2_vision_interfaces`, a small, clean ROS 2 interface package (messages +
service) that several other packages depend on. Implement it exactly to the message/service
contract in §6.2 of the attached spec @robot-mcp-perception.md.
Deliver an ament package defining:
- VisionDetection.msg, VisionDetectionArray.msg, StampedString.msg, VisionTask.srv
exactly as specified (field names, types, order). Use sensor_msgs/RegionOfInterest,
sensor_msgs/Image (mask; empty allowed), geometry_msgs/Point, std_msgs/Header as given.
This is a pure interface package — no node logic. It must `colcon build` cleanly on ROS 2 Humble+
and be importable by Python (rosidl) consumers. Add a one-paragraph README listing the messages and
who depends on them. Do not add fields beyond the spec; if something is ambiguous, ask rather than
invent.
2. robot-mcp-perception — extend into the vision ROS node¶
Attach: @robot-mcp-perception.md (spec), @ros2-vlm-groq (reference patterns), and ensure
@ros2_vision_interfaces is available as a dependency. The repo already contains the Gemini-ER /
SAM / reason-tiled-watch compute and the playground — extend it, don't restart.
You are extending the existing `robot-mcp-perception` repo into the robot's single configurable
**vision ROS node**, per the attached authoritative spec @robot-mcp-perception.md. The repo already
has the compute (Gemini Robotics-ER grounder, SAM segmenter, reason/tiled/watch pipelines,
postprocess, Hydra config, playground) — keep that and add the ROS + dynamic-skill layer.
Study @ros2-vlm-groq for the PATTERNS to fold in (do not copy its Groq code): the runtime task
system (create/manage/list tasks over ROS services), per-skill publishers created on the fly,
the prompt builder + semantic output types, and the pipelined async processing (workers,
max_in_flight, drop-oldest, per-skill rate). We DROP Groq llama-vision entirely — the grounder is
Gemini ER, the segmenter is SAM 3.
Must-haves (the spec is authoritative; these are the things most often gotten wrong):
- It is a DATA-PLANE ROS node: subscribe the camera, run skills at a rate, publish results to
ROS topics. The brain never calls it directly.
- Skills are configured at runtime via ROS services (create/manage/list_vision_task); modes
describe | ground_and_segment | segment_all | region map onto reason/tiled/watch (§5).
- Publish BOTH /vision/{skill}/detections (WITH masks, for live consumers) AND
/vision/{skill}/detections_meta (mask field empty, for recording/logging). Always publish meta.
- Implement the cost-control policy (§8): every model adapter has cost_class local|cloud; meter +
gate every CLOUD stage (global budget, per-skill rate cap, cloud_requires_gate via the
frame-diff gate, once-skills exempt); `ambient_scene` exists but defaults OFF and is a config
toggle. Local stages are never throttled.
- frame_source=folder runs the whole node camera-free for dev/CI.
- Depend on `@ros2_vision_interfaces` for the published messages + the config service.
Follow Appendix A (conventions) in the spec. Deliverables: the node + tests per §9, green
colcon build/test + ruff + mypy. Where the spec and @ros2-vlm-groq disagree on architecture, the
spec wins; for pipeline parameters/prompts, prefer the repo's existing playground values. Ask if a
contract is ambiguous; don't invent message fields or tool names.
3. mcp-skill-control¶
Attach: @mcp-skill-control.md (spec), @ros2-vlm-groq (reference: vlm_mcp_server.py). Depends
on @ros2_vision_interfaces and a running robot-mcp-perception.
You are building `mcp-skill-control`, the control-plane MCP server through which the brain
DISCOVERS what the robot can be configured to do and INSTANTIATES those capabilities ("skills") on
the ROS stack. Implement it to the attached authoritative spec @mcp-skill-control.md.
Study @ros2-vlm-groq's `vlm_mcp_server.py` for the reference pattern: an MCP server that is also a
hidden rclpy node with service clients, spinning ROS in a thread pool, bridging MCP tool calls to
ROS service calls. Generalize "task" -> "skill" and make the catalog data-driven (YAML), not a
hardcoded dict.
Must-haves:
- This server is WRITE/CONFIG ONLY. It calls `robot-mcp-perception`'s create/manage/list_vision_task
ROS services. It does NOT read results (that's the separate perception-buffer server) and never
handles pixels — tools return the ROS TOPIC NAME for the caller to read.
- Tools: list_skill_templates, list_running_skills, instantiate_skill(template, params) ->
{skill_id, topic}, instantiate_raw_skill, stop_skill, set_skill_enabled (per spec §4).
- Skill templates live in config/skills.yaml with {param} placeholders + a declared param list;
catalog.fill() substitutes + validates against the vision skill schema.
- Depend on `@ros2_vision_interfaces` for the VisionTask service type.
- Skills carry a `kind` so the model generalizes beyond vision later — don't hardcode "vision".
Follow Appendix A. Transport stdio default / sse on port 9201; MCP server name `skill_control`.
Build against a FAKE rclpy service node in CI (don't require the real vision node). Deliverables +
tests per the spec; green pytest + ruff + mypy. Ask if ambiguous; don't invent tools.
4. mcp-perception-buffer¶
Attach: @mcp-perception-buffer.md (spec), @ros2brain (reference: state_collector.py). Depends
on @ros2_vision_interfaces and @robot-mcp-kit (published).
You are building `mcp-perception-buffer`, the "what do you see" server — the brain's EYES. It
subscribes to the /vision/** ROS topics the vision node publishes, keeps a rolling latest snapshot
in memory, and exposes READ-ONLY MCP tools so the brain can ask "what do you see?" without a
round-trip into the pipeline. Implement to the attached authoritative spec @mcp-perception-buffer.md.
Study @ros2brain's `state_collector.py` for the reference pattern (rolling buffer, freshness/age
checks, compact prompt-ready rendering) — but feed it from ROS SUBSCRIPTIONS and expose it over MCP.
Must-haves:
- READ-ONLY. It never configures anything. Tools: what_do_you_see, get_latest(skill),
list_streams, get_overlay_ref (per spec §4). Masks/overlays are returned BY REFERENCE
(topic + stamp) — never pixels over MCP.
- Reconcile subscriptions against the live /vision/** topics periodically; keep latest + a short
history deque per stream; flag each value stale/age_s against a freshness window.
- Owns the watcher -> push: on a watched skill's boolean rising edge, POST a `watch_event` to the
brain's /api/agent/events using `@robot-mcp-kit` (HMAC-signed). Learn which skills are watched by
reading list_vision_tasks (the skill carries the watch block). watch_push is config-gated.
- Depend on `@ros2_vision_interfaces` (VisionDetection* msgs) and `@robot-mcp-kit` (event poster +
signing).
Follow Appendix A. Transport stdio/sse on port 9202; MCP server name `perception`. Build against
synthetic messages in CI. Deliverables + tests per the spec; green pytest + ruff + mypy. Ask if
ambiguous; don't invent tools.
5. ros2-memory — clean reimplementation¶
Attach: @ros2-memory.md (spec), @mcp-rosbag-memory (reference: the existing MCP ROS Bag Memory
Server). Depends on @ros2_vision_interfaces (for the custom-msg typestore).
You are building `ros2-memory`, the robot's EPISODIC memory: an MCP server that records configured
ROS topics into time-segmented bags and answers temporal queries about the recent past ("where was
I 10 s ago", "last time you saw scissors", "how far have I driven in the last minute"). Implement to
the attached authoritative spec @ros2-memory.md.
This is a CLEAN REIMPLEMENTATION — study @mcp-rosbag-memory for the tool set, MCAP storage, the
`rosbags` reader, the `ros2 bag record` subprocess recorder, and the schema-extraction approach, but
REBUILD it tidy to Appendix A conventions. Do not port its code verbatim.
Must-haves:
- rclpy-FREE: read bags with the pure-Python `rosbags`/`mcap` libs; record via a `ros2 bag record`
subprocess (argv list, never shell=True). Recorder and query server are decoupled (two systemd
units; `recorder=none` gives a query-only deployment).
- Record vision OUTPUTS + state, NOT pixels: the mask-free /vision/{skill}/detections_meta topic,
/odom, /tf, /cmd_vel, semantic outputs — per config/topics.yaml. Masked /detections and raw
/camera/image_raw stay commented (debugging only).
- Register custom message defs (`@ros2_vision_interfaces`) with the `rosbags` typestore so
/vision/** history deserializes without a ROS runtime.
- The ~14 grouped tools (metadata / generic value / semantic / transform / conditional / analysis)
per spec §6; out-of-range returns {"error":"no_data"}, not an exception.
- Retention GC by disk budget + age; time index over bag files.
Follow Appendix A. Transport stdio/sse on port 9203; MCP server name `episodic` (distinct from the
process store `memory`/9103). Test against FIXTURE bags in CI; the recorder subprocess is
integration-only. Deliverables + tests per the spec; green pytest + ruff + mypy. Ask if ambiguous.
6. robot-mcp-memory — verify/align only¶
Attach: @robot-mcp-memory.md (spec). Work IN the existing repo — it's already built
(docmodel/store/server + tests). No external reference; the reference is the repo's own code.
The `robot-mcp-memory` repo is ALREADY IMPLEMENTED to an earlier version of the attached spec
@robot-mcp-memory.md (it has docmodel.py, store.py, server.py with find_processes/save_process/
record_outcome/etc. + tests). Do NOT rebuild it.
The role is unchanged (still the per-robot process/skill store). Your job is a VERIFY/ALIGN pass:
- Diff the current implementation against @robot-mcp-memory.md; fix any drift (tool names,
process-doc format, FTS query sanitization, file-first-then-index write order, bm25*reliability
ranking).
- Confirm Appendix A conventions are met; ensure all gates pass (pytest, ruff, ruff-format, mypy).
- Do NOT add the "skills persisted here" feature — it is a future, additive note only.
Report what (if anything) drifted from the spec and what you changed. Make no structural changes
beyond aligning to the spec. Ask before any larger refactor.
7. robot-voice-chat brain delta — small change in this repo¶
Attach: @robot-voice-chat-agent.md (read §0 first). Work in this repo on a feature branch; the
brain core is already built.
You are applying a small delta to the already-built mission brain in this repo. Read §0 of the
attached authoritative spec @robot-voice-chat-agent.md FIRST — it lists exactly what changes; §1–§14
are the still-accurate detail for everything unchanged.
Scope (small — keep the MissionExecutor/planner/teaching/mission-UI core intact):
- Planner tool surface now comes from the new MCP servers: `skill_control`
(list_skill_templates/instantiate_skill/stop_skill) + `perception` (what_do_you_see/get_latest).
Add the planner-protocol note: discover skills -> instantiate -> observe the buffer -> stop.
- `instantiate_skill` returns a TOPIC, not a running TaskInfo — so skill use does NOT register an
async task; only watcher `watch_event` pushes flow through /api/agent/events (keep that ingress).
- Safety stop: stop_skill all active skills (+ future motion-node stop). Drop navigation wiring.
- tools/mcp.yaml: entries for skill_control (9201), perception (9202, with event_secret),
memory (9103), episodic (9203). Remove navigation.
- Teaching distillation: processes now read as "skills instantiated + topics observed".
Do not rebuild the brain. Make the minimal edits, keep all existing tests green, add tests for the
changed wiring. Run the full gate set (pytest, ruff, ruff-format, mypy, import-linter; frontend
typecheck/build). Ask before any larger refactor — the spec §0 is the boundary.