SoMatic

The core workflow is a CLI command that takes a screenshot, runs element detection locally, and returns numbered marks with coordinates as JSON — so agents target elements by ID, not by fragile pixel hunts. Every action returns JSON, which means downstream agents can chain steps without parsing unstructured output. The self-hosted, MIT-licensed model runs on your own hardware, so no screenshot data leaves the machine. The wall appears with non-standard or highly dynamic UIs where YOLO detection misses elements or mislabels them — teams handling those cases add a fallback coordinate layer manually. At this GitHub star count, the community size is small, which means debugging edge cases happens in the codebase, not a forum.