Handy

See and drive the real desktop — native mouse, keyboard and coordinate-mapped screen control for LLMs and agents.

Version 1.4.0

What it is

Handy is an MCP server and CLI that gives a model real-time native control of a desktop — mouse, keyboard, clipboard and windows — paired with a live coordinate-mapped view of the screen and Set-of-Mark detection that numbers every interactive element so the agent clicks mark 13 instead of guessing (631, 447).

It installs with npm and zero native compilation, speaks plain stdio to any MCP client, and treats permission as a product surface: a synchronous toggle checked before every effect, a one-way off switch, and a (0, 0) corner-slam failsafe that reads the real cursor each step.

It is the tool the author's own agents drive this machine with — the same code, not a demo.

Handy's six MCP tools, one per row, from handy_see to handy_permissionThe four Set-of-Mark element kinds beside a marks capture and a click-by-mark chain with verifyChangeThe handy_act step vocabulary beside a chain-DSL example with screenshotAfter and motion profilesThe five platform backends from SendInput and CGEvent to the raw X11 client, the Wayland route and the virtual workbenchThree panels on the toggle, the (0, 0) corner-slam failsafe and the no-self-restart ruleA handy doctor report beside the standalone CLI verbs with JSON receipts

What it’s for

For developers and agent builders who want their assistant to operate real desktop applications — fill forms, drive native apps and verify what happened — rather than describe them.

  • Click what you see

    handy_see numbers every interactive element on screen; the agent clicks mark 13 instead of guessing coordinates from pixels.

  • One call per interaction

    Focus a window, click, paste and wait for the UI to settle in a single chained call, with a receipt that says what ran and whether the screen changed.

  • Stop it by hand

    An on/off toggle checked before every capture and every step, plus a physical failsafe: slam the mouse into the top-left corner and the chain aborts.

Version
1.4.0
Licence
MIT
Install
npm @miisodev/handy · MCP registry io.github.miisodev/handy
Transport
stdio · npx -y @miisodev/handy mcp
Built with
TypeScript · Model Context Protocol · Zod · zero native build steps

How it’s made

  1. No native builds

    Every platform bridge is written in TypeScript down to the wire protocol: SendInput on Windows, CGEvent on macOS, a raw X11 client on Linux — npm install is the whole install.

  2. Permission before every effect

    The permission store is checked synchronously before each capture, clipboard call and every individual chain step; a turned-off Handy cannot be re-enabled by the client holding it.

  3. Marks resolve at execution time

    A mark reference resolves to the element's own centre when the step runs, so chains survive layout shift and HiDPI scaling; stale marks fail loudly with the available indices.

  4. Fail soft where it can

    Mark detection degrades to a plain screenshot instead of failing a capture, and every screenshot route is retried in order with its error named.

  5. Honest readiness

    handy doctor reports the session type, the capture routes and the input tools it actually found, and warns when a backend cannot see the real desktop.

Disclaimer

  • On GNOME Wayland, window listing and focusing see XWayland windows only; capture and input are unaffected.
  • Wayland input needs a running ydotoold daemon (uinput), which handy doctor reports on.
  • The gallery frames illustrate the tool surface and its guarantees. They are composed diagrams, not screenshots of the product.

Support

Write to miisodev@gmail.com about installing or driving Handy, or a bridge that misbehaved.