Vercel Labs has open-sourced agent-desktop, a tool that gives AI coding agents control of the whole desktop computer. Announced on October 30, 2026, and introduced publicly the following day, it lets an agent see and operate ordinary desktop applications on macOS, Windows and Linux — not by looking at screenshots, but by reading the operating system's own structural map of what is on screen.
The release, version 1.0.2, arrives as coding agents move beyond the terminal and the browser. Until now, giving an agent "computer use" usually meant feeding it screenshots and asking a vision model to guess where to click. Vercel Labs is betting that a cheaper, more reliable route already exists inside every major operating system.
No screenshots, just the accessibility tree
Instead of parsing pixels, agent-desktop reads each application's accessibility tree — the same structured data that screen readers rely on, listing buttons, text fields, menus and other elements. It hands the calling agent a compact snapshot in which every actionable element carries a reference tag, so the agent clicks a named element rather than estimating coordinates from an image.
Actions run headless by default, meaning they do not move the on-screen pointer or steal focus. In principle, agents can therefore keep working in the background while a person uses the same machine, each with its own cursor. The tool covers macOS 13 and later, Windows 10 version 1809 and later, and Ubuntu 24.04, on both x64 and ARM64 hardware, using each platform's native accessibility API.
Built for any coding agent
Crucially, agent-desktop is not itself an agent. It embeds no language model, has no graphical interface and runs no reasoning loop. It is a Rust command-line tool, plus a C library for in-process use, that any shell-capable coding agent can call — installed with a single package-manager command and licensed under the permissive Apache-2.0 licence.
Vercel Labs pitches it as model-agnostic computer use, a counterpart to its earlier agent-browser, which does the same job for web pages through compact accessibility snapshots. The builder's own figures claim large token savings over dumping full accessibility trees: a skeleton view of Apple Maps is put at 273 tokens against 3,811 for the full tree, and a simulated 52-step task at 31,000 tokens against 198,000. Those numbers are self-reported, not independently benchmarked, and no public comparison on standard computer-use tests exists yet.
Early days, and clear limits
The project is young. Its repository was created in February 2026 by an independent developer and transferred to Vercel Labs on October 30, with version 1.0.0 reaching the npm registry only days earlier. The tool ships 60 command names, 56 of them currently operational, and runs on plain CPU hardware.
Its limits are equally plain. It cannot see an application that publishes no accessibility tree, it does not automate web pages — that remains agent-browser's job — and the quality of any automation depends on how well each app's interface is built. Rivals take different bets: Anthropic's and OpenAI's computer-use systems lean on screenshots, while Microsoft's UFO² project also uses Windows' accessibility layer but ships as a complete agent rather than a bare tool.
For now, agent-desktop is best read as infrastructure: a small, open building block that could make desktop automation cheaper and less brittle for developers building agents — if the young codebase proves itself in real applications. Vercel Labs has not announced pricing, a hosted version or a roadmap beyond the open-source release.
