Consume human-verified coordinates, perform the interaction, confirm it landed
click · double_click · drag · scroll · type · key · verify · wait_for · wait_gone · pause — one vocabulary across three surfaces
pixelcoords freezes your screen, lets you mark labeled regions, and writes pixel-exact coordinates with crops. pixelactions reads that session and acts on it — referencing regions by label, never by raw coordinate, so a run survives the UI moving. The loop is find → act → assert.
Status: early, and macOS only. The loop works end to end; Windows and X11 are next. No crate or binaries are published yet.
session = "~/captures/checkout"
[[step]]
action = "click"
target = "email"
[[step]]
action = "type"
text = "a@b.com"
[[step]]
action = "key"
chord = "enter"
[[step]]
action = "verify"
target = "success"Sixty seconds
pixelcoords # a human freezes the screen, marks and labels regions
# → Downloads/pixelcoords-captures/<timestamp>/ session.json + crops: the ground truth
pixelactions doctor --probe # prove input permission — move the cursor 1px and back
# a missing macOS grant makes event posting a SILENT no-op; the probe raises the dialog
pixelactions plan --session <dir> click:email type:"a@b.com" key:enter verify:success
# every coordinate after conversion, acts on nothing
pixelactions run --session <dir> click:email type:"a@b.com" key:enter verify:success --yes
# relocate → refuse if ambiguous → act → verify; exit 0 done, 1 failed, 2 malformed, 3 refused
pixelactions serve --session <dir> # JSON per line on stdin/stdout — any language owns the loopThe tool is a loop
pixelcoordsA human freezes the screen, marks and labels regions. The saved session is the ground truth.
{ "schema": 1,
"selections": [{ "shape": "rect", "label": "submit",
"px": { "x": 812, "y": 440, "w": 96, "h": 40 } }] }pixelactions plan --flow checkout.tomlEvery coordinate after conversion, before anything moves. The permanent dry run, not a temporary phase.
{ "executed": false, "steps": [{ "summary": "click submit",
"points": [{ "x": 430.0, "y": 170.0,
"space": "logical", "monitor": 0, "scale": 2.0 }] }] }pixelactions run --flow checkout.toml --yesRelocate, act, verify. A crop that matches in more than one place produces no correction at all — the run refuses before anything is injected.
{ "summary": "click submit", "outcome": "verified",
"elapsed_ms": 412 } # exit 0pixelactions serve --session <dir>One JSON request per line, one response back — a program in any language owns the loop.
→ {"id": 2, "do": "click", "target": "email"}
← {"id": 2, "result": "done", "outcome": "verified"}echo $?Exit codes are the API: 0 done, 1 a step failed, 2 malformed, 3 refused. Grab the mouse into a screen corner and the run stops — a refusal is never worth retrying.
refused: cursor in a screen corner — kill switch # exit 3
What a run reports
{
"schema": 1,
"session": "/Users/you/Downloads/pixelcoords-captures/20260728-182121-117",
"executed": true,
"steps": [
{ "index": 0, "summary": "click submit", "outcome": "verified",
"points": [{ "x": 430.0, "y": 170.0, "space": "logical", "monitor": 0, "scale": 2.0 }],
"elapsed_ms": 412 }
]
}points are the coordinates actually used, relocation corrections included — where the click went, not where the session said it would. outcome distinguishes verified from executed: the OS accepting an event is not the app reacting to one, and “nothing errored” is not “it worked”. The honesty runs deep enough that scroll always reports executed, never verified — it changes its own region on purpose, so confirm it with a wait_for on whatever it should bring into view.
Where it stands
| Feature | pixelactions | PyAutoGUI | SikuliX | AutoHotkey |
|---|---|---|---|---|
| Price | Free | Free | Free | Free |
| Platforms | macOS today; Windows and X11 nextearly — no crate or binaries published yet | Windows · macOS · Linux (X11)no Wayland; primary monitor only | macOS · Windows · Linux (JVM)requires Java; development archived, continued by the OculiX fork | Windows only |
| Declarative, reviewable flows | TOML flows + chained argva pull request shows "click submit", not arithmetic | not offered | not offered | not offered |
| Language it takes to drive it | Any — JSON line protocol over stdioa 40-line stdlib Python client ships in the docs; no embedded interpreter, ever | Python | Jython, JRuby, JavaScript — inside its runtime | AutoHotkey, its own language |
| Re-location before acting | Every target re-located against a fresh capture before actinga crop that matches in more than one place stops the run — ambiguity is the test, not distance | locateOnScreen image search, in-script | Continuous visual search across the screen | ImageSearch, in-script |
| Verified execution | Distinguishes executed from verified; exit codes 0/1/2/3 | not offered | In-script image matching | ImageSearch / PixelGetColor, in-script |
| Maintenance status | In active development — v0.1.0, unpublished | Last release May 2023 | Archived upstream March 2026; the OculiX fork continues | Active — v2.0.26, May 2026 |
| License | MIT, open source | BSD-3, open source | MIT, open source | GPLv2, open source |
Non-goals
Knowing what a tool is means knowing what it isn't. These are settled:
- An embedded interpreter— your bot is written in your language; the binary speaks a line protocol instead
- A scripting language— a flow is a list of steps — loops, branching, and data live in your program
- A network surface— this process holds the permission to click and type — stdio with its caller, or nothing
- A recorder— record-and-replay makes unreviewable artifacts; regions are marked by a human, on purpose
- Browser automation— Playwright and Selenium own the web, and own it well
- Cloud, accounts, telemetry— offline by design, permanently
Platform status
| macOS | Supported — the loop works end to end; primary development platform |
|---|---|
| Windows | Next — the goal is the same flow file running unmodified |
| Linux (X11) | Next — alongside Windows |
| Linux (Wayland) | Later — after Windows and X11 |
No crate or binaries are published yet. This table is kept honest — claims match runs.
Two ways in
cargo install pixelactionsOr skip the toolchain: prebuilt binaries — download, unpack, run.
Rust 1.88+ for the cargo route. pixelactions drives the pixelcoords binary for capture-time work — install both:
cargo install pixelcoords pixelactionsmacOS asks for an Accessibility grant on first run. The grant attaches to the terminal that launches pixelactions, not the binary — doctor --probe proves the grant instead of assuming it.
No account, no network surface, no daemon — one small native binary that runs, acts, and exits. MIT-licensed, because the aim was to build the best executor in this category and give it away.