Consume human-verified coordinates, perform the interaction, confirm it landed

60sSixty seconds

pixelcoords
# a human freezes the screen, marks and labels regions
# → Downloads/pixelcoords-captures/<timestamp>/  session.json + crops: the ground truth

pixelactions doctor --probe
# prove input permission — move the cursor 1px and back
# a missing macOS grant makes event posting a SILENT no-op; the probe raises the dialog

pixelactions plan --session <dir> click:email type:"a@b.com" key:enter verify:success
# every coordinate after conversion, acts on nothing

pixelactions run  --session <dir> click:email type:"a@b.com" key:enter verify:success --yes
# relocate → refuse if ambiguous → act → verify;  exit 0 done, 1 failed, 2 malformed, 3 refused

pixelactions serve --session <dir>
# JSON per line on stdin/stdout — any language owns the loop

pixelactions mcp
# read-only: a model can plan and find, but not act
pixelactions mcp --yes
# ...and act. A model cannot pass this flag; you do

loopThe tool is a loop

{ "schema": 1,
  "selections": [{ "shape": "rect", "label": "submit",
    "px": { "x": 812, "y": 440, "w": 96, "h": 40 } }] }

{ "executed": false, "steps": [{ "summary": "click submit",
  "points": [{ "x": 430.0, "y": 170.0,
    "space": "logical", "monitor": 0, "scale": 2.0 }] }] }

{ "summary": "click submit", "outcome": "verified",
  "elapsed_ms": 412 }   # exit 0

→ {"id": 2, "do": "click", "target": "email"}
← {"id": 2, "result": "done", "outcome": "verified"}

refused: cursor in a screen corner — kill switch   # exit 3

mcpA model can drive it

pixelcoords serves a read-only MCP server, so a model could already ask where to click. Nothing let it say do it. Now it can — over stdio, against a session a human marked.

{
  "mcpServers": {
    "pixelactions": { "command": "pixelactions", "args": ["mcp"] }
  }
}

Acting is off unless you launch it with --yes. This surface posts real input, so unlike a read-only server it has no safe default — and a model cannot pass a command-line flag. So the consent stays with whoever wired the client. A per-call confirmation would be written by the model itself: a speed bump against a slip, not a gate against intent.

A refusal is an answer, not an error. A failed step, a region that could not be found, an act call on a read-only server — all come back as ordinary results with ok: false. Only a malformed question is a protocol error. A model that reads a refusal as a broken tool retries, and retrying something that posts input is the worst thing this could do.

It adds nothing underneath: the kill switch, re-location, verification and the audit log are the run loop's, and a model-driven run gets all four because they were already there.

reportWhat a run reports

{
  "schema": 1,
  "session": "/Users/you/Downloads/pixelcoords-captures/20260728-182121-117",
  "executed": true,
  "steps": [
    { "index": 0, "summary": "click submit", "outcome": "verified",
      "points": [{ "x": 430.0, "y": 170.0, "space": "logical", "monitor": 0, "scale": 2.0 }],
      "elapsed_ms": 412 }
  ]
}

points are the coordinates actually used, relocation corrections included — where the click went, not where the session said it would. outcome distinguishes verified from executed: the OS accepting an event is not the app reacting to one, and “nothing errored” is not “it worked”. The honesty runs deep enough that scroll always reports executed, never verified — it changes its own region on purpose, so confirm it with a wait_for on whatever it should bring into view.

To assert an outcome rather than an acceptance, name what should have moved. changed:panel proves a region stopped matching, which is the question verify cannot answer: correlation is contrast-normalized, so a region that dims uniformly behind a modal still scores ~1.0 and still reads as matching. changed compares RGB directly. And because that comparison happens where the session recorded the region, a region that merely moved would differ for the wrong reason — so when something does differ, it spends one re-location to tell a change from a departure, and refuses rather than claiming one it cannot stand behind.

The same report is appended to an audit log as the run goes — NDJSON under $XDG_STATE_HOME, %LOCALAPPDATA% on Windows, or ~/.local/state; doctor prints which. On by default. Written step by step rather than at the end, so a run the watchdog stopped or someone killed still says what it did first. It never contains typed text, and nothing strips it: steps are recorded by their summary, and a type summary is type 26 chars.

comparisonWhere it stands

refusedNon-goals

Knowing what a tool is means knowing what it isn't. These are settled:

• An embedded interpreter— your bot is written in your language; the binary speaks a line protocol instead
• A scripting language— a flow is a list of steps — loops, branching, and data live in your program
• A network surface— this process holds the permission to click and type — stdio with its caller, or nothing
• A recorder— record-and-replay makes unreviewable artifacts; regions are marked by a human, on purpose
• Browser automation— Playwright and Selenium own the web, and own it well
• Cloud, accounts, telemetry— offline by design, permanently

platformsPlatform status

Binaries ship for every platform above — shipping one that refuses to inject would imply support a build does not have. Windows placement is measured rather than assumed, but on a single-display machine: multi-monitor and mixed-DPI layouts are unit-tested and have not been run on real hardware yet. This table is kept honest — claims match runs.

Those runs were 0.2.0 through 0.4.0. Nine releases have shipped since — the pixelcoords seam, changed, the audit log, the MCP surface, and five rounds of bug fixes — and on Windows, X11 and Wayland none of them has been driven by hand. macOS has. This page will say so until someone sits at each machine.

Two slices of that are automatic now. A scenario script runs in CI against a live X server and checks that a marked region is located, that a click lands where the plan said, that a cursor in a screen corner refuses the step, and that the audit log records a refused run — a real X server and a real synthetic event, on a bare 1280×1024 virtual display with no window manager, which is not a desktop. And everything the tool does other than synthesise input now runs against a real display on macOS, Windows and Linux every push: planning against a session marked from a genuine capture, every verb, every settings key, the exit codes, the refusals, the line protocol, and the agent tools. What neither can answer is whether a click reached an application, and whether the permission model let it.