SoMatic

Most desktop automation breaks the moment the UI shifts — a new dialog, a resized window, an element that moved two pixels. SoMatic runs a local YOLO model against a screenshot of any native desktop application, detects and numbers every interactive element, and returns a JSON coordinate map the agent can act against. Elements can be targeted by mark ID, by nearest-mark offset, or by direct pixel coordinate. The public binary is `somatic`; install is via npm or pip.

The differentiating feature is the Set-of-Marks approach: instead of asking an agent to infer where a button lives, SoMatic assigns each detected element a numbered mark and hands those numbers back as structured data. The agent sees element 7 is a submit button at these coordinates — and clicks it by ID. No screen-reading heuristics, no brittle XPath, no browser-only DOM access. This makes it usable against native apps, PDFs, and any UI that does not expose a programmatic interface.

SoMatic ships an MCP server, which means Claude and other MCP-compatible agents can call it as a tool in an automation pipeline — the agent requests a screenshot, receives the mark map, decides which element to interact with, and issues the action, all within a standard MCP handoff. Headless Xvfb support is included for server environments where there is no physical display. Both paths — interactive desktop and headless server — use the same JSON-returning CLI surface, so the integration contract does not change between environments.

The project is MIT-licensed and fully open-source, hosted under a single-maintainer GitHub repository. There are no paid tiers, no hosted API, and no telemetry described in the docs. Detection runs locally, so screenshots of sensitive internal apps do not transit a third-party service. The tradeoff is that model quality and detection coverage depend entirely on what the bundled YOLO weights handle — the vendor does not describe fine-tuning paths for custom UI patterns in the available documentation.