# Plumshot for agents

> Give your coding agent eyes on macOS.

Plumshot lets AI agents capture and annotate native macOS windows, menus, sheets, and the menu bar — UI that has no DOM and no accessibility tree, so Playwright and browser tools can't see it. It runs locally next to your agent: no account, no token, no upload.

This is the plain-markdown twin of <https://plumshot.app/agents>.

## The two loops

- **Loop A — the agent shows you.** The agent grabs a window, draws a red box and an arrow, numbers the problem, and hands back a marked-up PNG instead of a paragraph of coordinates. Small captures and every annotation come back **inline as a base64 PNG**, so a vision agent sees the result in one call.
- **Loop B — you show the agent.** You mark up a screenshot (in Plumshot, Preview, anything) and the agent just **reads the marks**. No capture, no permission needed.

## Three surfaces

| Surface | Invoke | Best for |
| --- | --- | --- |
| Headless CLI | `plumshot …` | Driving from a shell or a non-MCP script. Every capture prints JSON `{ path, width, height, bbox }`. The CLI is the signed `Plumshot.app` binary. |
| stdio MCP server | `plumshot-mcp` (`server.mjs`, Node ≥ 18, zero deps) | Inside Claude Code, Codex, or Cursor. Small images return inline as base64 PNG; large ones return a path. **Default for MCP clients.** |
| Claude Code skill + plugin | `/plugin install plumshot` | Easiest install + built-in guidance on when to reach for Plumshot. |

No hosted URL, no network, no API key — the inverse of every cloud MCP. Captures never leave your Mac.

## Quickstart (4 steps)

The CLI is the signed app binary, so build the app once, then wire any client to it.

```bash
# 1. Build the app (Apple Development signing so the Screen Recording grant sticks)
./scripts/run_dev.sh

# 2. Put the CLI on PATH
ln -sf "$PWD/scripts/plumshot" /usr/local/bin/plumshot

# 3. Grant Screen Recording:
#    System Settings ▸ Privacy & Security ▸ Screen Recording → enable Plumshot → relaunch

# 4. Verify
plumshot doctor
#   screen recording (TCC): granted ✓
#   accessibility (AX):     not granted (optional)
#   bundle id:              com.plumshot.app
#   binary:                 …/Plumshot.app/Contents/MacOS/Plumshot

# Smoke test before wiring a client
plumshot capture-window --app Finder --out /tmp/cap.png
plumshot list-targets
```

## Connect your agent

Every block ends with the same verify ritual: run `plumshot doctor`.

### Claude Code

The easy button — install the plugin (brings the MCP server and the skill together):

```text
/plugin marketplace add .
/plugin install plumshot
```

Or register the MCP server yourself — a `.mcp.json` at the repo root:

```json
{
  "mcpServers": {
    "plumshot": {
      "command": "node",
      "args": ["mcp/plumshot-mcp/server.mjs"]
    }
  }
}
```

…or one line from the CLI:

```bash
claude mcp add plumshot -- node mcp/plumshot-mcp/server.mjs
```

### OpenAI Codex

Add the server to `~/.codex/config.toml` with an **absolute** path to `server.mjs`:

```toml
[mcp_servers.plumshot]
command = "node"
args = ["/Users/you/Documents/GitHub/Plumshot/mcp/plumshot-mcp/server.mjs"]

[mcp_servers.plumshot.env]
PLUMSHOT_APP = "/Applications/Plumshot.app"
```

The env table is optional — drop it if your dev build resolves on its own.

### Cursor

Add the server to `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global), absolute path:

```json
{
  "mcpServers": {
    "plumshot": {
      "command": "node",
      "args": ["/Users/you/Documents/GitHub/Plumshot/mcp/plumshot-mcp/server.mjs"],
      "env": { "PLUMSHOT_APP": "/Applications/Plumshot.app" }
    }
  }
}
```

### Other MCP clients

```bash
node mcp/plumshot-mcp/server.mjs
```

For a non-default build, set `PLUMSHOT_APP` (path to the `.app`) or `PLUMSHOT_BIN` (full binary path) in the host's env block.

## Permissions & privacy

Plumshot needs just one macOS permission — **Screen Recording** — for the capture tools. Not Accessibility (except scrolling auto-scroll), not Full Disk Access, and it makes **no network requests**.

- Captures are local files you choose to share — nothing is uploaded. No account, no token.
- `annotate` needs no permission at all — it composites onto a PNG you already have. Only the `capture_*` tools touch the screen.
- Images ≤ 5 MB come back inline as base64 PNG; larger ones return a path. `capture_full` and `capture_scrolling` always return a path by design.
- `capture-scrolling`'s auto-scroll additionally needs Accessibility — run `plumshot doctor` to check, or pass `--no-auto-scroll` and scroll yourself.
- Redact before you share: cover secrets with a `blur` shape (`style: "blackOut"` for a hard cover) before surfacing an image.

## The 9 MCP tools

All coordinates are top-left origin, image pixels. Every capture also prints `{ path, width, height, bbox }`. On failure, commands print `{ "error": "…" }` to **stderr** and exit non-zero.

| Tool | Arguments | Returns |
| --- | --- | --- |
| `capture_region` | `x`, `y`, `w`, `h` (required), `display?` (0) | Inline PNG + path |
| `capture_window` | `app` (required, substring, case-insensitive), `window?` (title substring) | Inline PNG + path |
| `capture_previous_region` | none | Inline PNG + path · repeats the last region |
| `capture_full` | `display?` (default: display under cursor) | Path only (large) |
| `capture_scrolling` | `x`, `y`, `w`, `h` (required), `display?` (0), `max_seconds?` (30) | Path only · needs Accessibility for auto-scroll |
| `annotate` | `image` (path), `shapes` (array or `{shapes:[…]}`), `out?` | Inline PNG + path · no permission |
| `capture_text` | `input?` (PNG path) \| `x`,`y`,`w`,`h` \| `app` + `window?`; `keep_linebreaks?` (false) | JSON `{ text, chars }` · on-device OCR |
| `list_targets` | none | JSON `{ displays, windows }` |
| `doctor` | none | Screen Recording / Accessibility / signing status |

Note the surface idioms differ: the MCP `annotate` arg is `image` while the CLI flag is `--in`; the CLI region capture takes `--rect X,Y,W,H` while the MCP tool takes `x`/`y`/`w`/`h` fields.

## Annotate shape spec

Coordinates are top-left image pixels — the same space a capture's `bbox` reports.

```jsonc
{ "shapes": [
  { "type": "box",       "x": N, "y": N, "w": N, "h": N,
                         "color": "#RRGGBB", "lineWidth": N, "filled": false },
  { "type": "arrow",     "from": [x, y], "to": [x, y] },
  { "type": "text",      "x": N, "y": N, "text": "…", "fontSize": N },
  { "type": "number",    "x": N, "y": N, "n": N },
  { "type": "highlight", "x": N, "y": N, "w": N, "h": N },
  { "type": "blur",      "x": N, "y": N, "w": N, "h": N,
                         "style": "pixelate|secureBlur|smoothBlur|blackOut" }
] }
```

Defaults: color `#FF3B30` (red), `lineWidth` 3, text `fontSize` 28, number marker 36, blur `pixelate`. Pipe the spec via `--shapes -` (stdin) or `--shapes file.json`. On Retina, use the pixel numbers Plumshot prints — they are already device pixels.

## Example prompts

Loop A — the agent shows you:

- `Capture the Preferences window of MyApp and tell me if the Save button label is clipped.`
- `Screenshot the Settings sheet, draw a red box around the broken label with an arrow to it, number it 1, and show me.`
- `Take a scrolling capture of this list and check that the bottom rows render.`

Loop B — you show the agent:

- `Read ~/shot.png — I drew arrows on what to fix. Do what they point at.`
- `Here's a screenshot I marked up; what does box 1 point at?`

Priming:

- `Run plumshot doctor and list-targets, then capture the front window of MyApp.`

## When to reach for Plumshot

Use Plumshot for native AppKit/SwiftUI windows, `NSMenu` and the menu bar, sheets and alerts, custom-drawn views, drag previews — anything with no reliable accessibility tree.

Reach for something else for:

- Web UI → Playwright / a browser MCP (you get DOM, console, network).
- Pure text extraction → `capture_text` / `plumshot capture-text` reads the words directly (on-device OCR) — cheaper than shipping pixels to a vision model.
- Headless CI with no display or Screen Recording grant.
- Sub-pixel diffing → capture, then diff with an image tool.

## Troubleshooting

- **`plumshot: command not found`** → symlink the shim onto PATH: `ln -sf "$PWD/scripts/plumshot" /usr/local/bin/plumshot`.
- **`doctor` says Screen Recording NOT granted** → grant it and relaunch. Rebuilding with ad-hoc signing drops the grant — rebuild with Apple Development signing (`scripts/run_dev.sh`).
- **Captures are black / empty** → the classic missing-Screen-Recording symptom even when the call "succeeds." Confirm with `plumshot doctor`; ensure a logged-in GUI session.
- **Could not find the Plumshot.app binary** → build the app, or set `PLUMSHOT_APP` / `PLUMSHOT_BIN`.
- **`capture-scrolling` doesn't scroll** → grant Accessibility, or pass `--no-auto-scroll`.
- **`node` not found when the host launches the server** → Node ≥ 18 must be on the host's PATH; use an absolute path to `node`.

When in doubt, re-run `plumshot doctor` — it's the universal did-it-work check.