Most computer use tools for AI agents work by looking at screenshots and guessing where to click. Vercel Labs agent-desktop takes a different route. It reads the operating system accessibility tree of a desktop app and hands the agent a structured snapshot, with a reference for every element it can act on. Version 1.0.2 shipped on October 30, 2026, under the Apache-2.0 license, and it runs on macOS, Windows, and Linux (MarkTechPost, October 31, 2026; vercel-labs/agent-desktop, GitHub, retrieved October 31, 2026). The agent acts on a reference, takes a fresh snapshot, and checks the result, with no pixel matching anywhere in the loop.
Key Takeaways
agent-desktop is an open source Rust command line tool and C library from Vercel Labs. Version 1.0.2 shipped on October 30, 2026, under Apache-2.0.
It is a tool that agents call, not an agent. It embeds no language model, has no graphical interface, and has no interactive prompt.
It reads desktop apps through the operating system accessibility tree, not screenshots. Each snapshot is JSON, and every actionable element gets a reference tied to that snapshot.
Actions are headless by default. They go through the accessibility interface without moving the pointer or stealing focus, so a person can keep working while an agent works.
Its limits are plain. Apps that publish no accessibility tree stay out of reach, web pages belong to the sibling agent-browser tool, and the big token savings are self-reported, with no independent benchmark yet.
The screenshot approach has a clear weakness. A button that moves, a theme that changes, or a window that scales can break a click aimed at pixels. A screenshot also costs tokens on every look, because the model must read an image to find anything. The accessibility tree avoids both problems in principle. It is the structured description of buttons, fields, and menus that screen readers use. It names elements instead of picturing them. That is why agent-desktop matters, even though it is only a building block.
What Is agent-desktop?
agent-desktop is a command line tool and C library that lets AI agents read and operate desktop apps on macOS, Windows, and Linux through the operating system accessibility tree. That is how its official documentation describes it. The definition needs two clarifications. First, agent-desktop is not an agent. It makes no decisions and it never embeds a language model. The observe, act, and verify loop stays in the calling agent or script. Second, it is not a screen reader and it is not a remote desktop tool. It is an interface between an agent and the apps already installed on one machine.
The project began in February 2026, and its repository shows how fast interest has grown. At the time of writing, on October 31, 2026, the repository listed 3,526 stars, 239 forks, and 316 commits (vercel-labs/agent-desktop, GitHub, retrieved October 31, 2026). Those counts will change as people star and fork the project, so treat them as a snapshot of launch week, not as a permanent score. What does not change is the shape of the tool. It ships 60 command names, with 56 operational commands, and it runs on plain CPU hardware, with no graphics card requirement.
How It Works: Snapshot, Reference, Verify
Every agent-desktop interaction follows the same four step loop. The agent maps an app, drills into the region it needs, acts on a reference, and checks the result with a fresh snapshot. A successful action alone does not prove the interface changed, so the loop always ends with a read. For example, the documentation shows the loop against a calculator app: map with a skeleton snapshot, drill into one region, click the 7 button, then wait until the display reads 7 (official documentation).
A snapshot is the core object. It returns JSON with a reference for every element the agent can act on. The reference carries the id of the snapshot that produced it, for example a reference that starts with a snapshot id and ends with an element id. That design is deliberate. A bare element id is rejected, and when the interface changes, an action returns a stale reference or ambiguous target error instead of acting on the wrong element. The agent must then take a new snapshot and use its references. Errors also carry a disposition, and the agent should retry only when the disposition says it is safe.
Dense apps need a second idea, because their accessibility trees can hold thousands of nodes. A skeleton snapshot reads only the shallow structure, three levels deep, and reports how many children each cut container holds. The agent then drills into one region with a root reference, instead of loading the whole tree at once. That keeps each look small enough to be useful, which is where the token argument begins.
Platform | Accessibility interface | Minimum version |
|---|---|---|
macOS | AX API | macOS 13 or later, Intel or Apple Silicon |
Windows | UI Automation | Windows 10 1809 or later, or Windows Server 2019 or later |
Linux | AT-SPI | Ubuntu 24.04 or later, x86_64 or aarch64 |
Under the surface, the design keeps platforms separate on purpose. A core crate owns the commands, the reference store, and snapshot building. It reaches the operating system only through a platform adapter trait. One platform crate is chosen at compile time for each operating system, and it talks to that system's native accessibility interface. Core never imports a platform crate, and platform crates never import each other. A binary therefore drives only the operating system it was built for.
The Token Argument, With a Caveat
The builder reports that skeleton traversal cuts tokens by 78% to 96% compared with full tree dumps in Slack, VS Code, and Notion (MarkTechPost). In Apple Maps, the same account puts a skeleton look at 273 tokens, down from 3,811 tokens for a full look (MarkTechPost). A simulated 52 step task is reported to fall from 198,000 tokens to 31,000 tokens (MarkTechPost). Every one of those numbers is self-reported by the builder. There is no independent computer use benchmark for agent-desktop yet, and no public comparison on standard computer use tests. Treat the percentages as a direction, not as a measurement you can quote against another tool.
The comparison still helps, because the cost structure is different in kind, not just in size. A screenshot based tool pays for an image on every look. MarkTechPost, citing Anthropic documentation, puts each screenshot at roughly 1,000 to 1,800 input tokens, plus about 4,500 tokens for the tool definition on each request (MarkTechPost). A structured snapshot pays only for the elements it returns, and a skeleton snapshot pays only for the structure. That is why the token story is plausible even before any benchmark exists. It is also why the missing benchmark matters. Plausible is not proven, and dense apps with poor accessibility trees may not behave like the examples.
How to Install and Run It
Installation is one npm command on all three platforms: npm install -g agent-desktop. Bun, Yarn, and pnpm can install the same package, and the package bundles native binaries for x64 and ARM64. A direct native archive is also available from the v1.0.2 release page, for use without a JavaScript launcher. Official Homebrew and Scoop packages are being submitted and are not yet in the default catalogs, according to the repository readme.
Permissions differ by platform, and they are worth reading before the first run. On macOS, the app that runs agent-desktop needs Accessibility permission, granted with the guided permissions command. On Windows, no grant is needed for apps at the terminal's integrity level. On Linux, the tool needs the user's graphical session with the at-spi2-core package installed. The status command reports the platform, permissions, and supported surfaces, so run it first when something does not work.
The working loop is short. List the apps, take an interactive snapshot of one app, act on a reference from that snapshot, then snapshot again to verify. Agents learn the tool from bundled documentation, loaded with the skills commands for the desktop guide and the platform guide. A skill is also listed on skills.sh for agent harnesses. Each command prints one JSON envelope and exits, which makes the tool easy to call from a script and easy to log, but it also means there is no interactive mode to fall back on.
What It Cannot Do, and What to Watch
The limits are stated in the documentation, and they are worth taking seriously. agent-desktop does not automate web pages. The sibling agent-browser tool handles those. For the web contents of a Chromium or Electron app it launches, agent-desktop can open a DevTools port that agent-browser connects to, but that is a bridge, not a browser of its own. It also does not match pixels, so it cannot see a user interface that publishes no accessibility tree. An app with a thin or broken accessibility implementation will give an agent little to work with, however good the tool is.
Four of the 60 command names fail closed everywhere. The key down, key up, mouse down, and mouse up commands refuse to run until a daemon can release a held key or button. That is a safety choice, and it is the right shape for a tool that can act on a person's machine. The same caution applies to privacy. The repository warns that snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content, and it advises keeping them out of public logs and uploads. An agent that can read your desktop can also leak your desktop, if its logs are treated carelessly.
Headless by default is both a strength and a risk. Reference actions go through the accessibility interface without moving the pointer or taking focus, so an agent can work while a person keeps using the machine. A headed flag allows physical input when needed. That separation is useful, but it also means an agent's work may be invisible while it happens. The verify step is the control. If a workflow skips it, an agent can believe it acted when nothing changed. The tool's error design helps, but it cannot replace a workflow that checks its own results.
How It Compares With Screenshot Computer Use
Feature | agent-desktop | Screenshot computer use |
|---|---|---|
How it sees the interface | Operating system accessibility tree | Screenshots and pixel coordinates |
Model bundled | None, works with any shell capable agent | Tied to a provider model in most products |
Platforms | macOS, Windows, Linux | Varies by product, often one hosted environment |
Pointer movement | No, headless by default | Yes, pixel clicks move the cursor |
License | Apache-2.0 | Proprietary in most commercial products |
Independent benchmark | None yet | Vendor benchmarks, not a shared yardstick |
Microsoft UFO2 is the closest open comparison, because it also uses Windows UI Automation. The difference is scope. UFO2 ships as a full Windows agent framework, while agent-desktop ships as a tool that any agent can call, on three platforms, with no model inside. If you want a complete agent, agent-desktop alone is not it. If you want to give an agent you already run reliable hands on a desktop, agent-desktop is the smaller, more composable piece.
Who Should Use It, and Who Should Not
agent-desktop fits developers who already run a coding agent and want it to operate desktop apps as part of a task. It fits teams testing how an agent handles a real application, and builders who want a repeatable, scriptable interface instead of fragile screen coordinates. The C library extends the same idea to programs that want to call the tool in process instead of starting the command line tool for each command.
Skip it if you need web automation, a complete agent, or a visual check of what an app looks like. Skip it, for now, if your target apps expose weak accessibility trees, or if you need a benchmarked comparison before you commit. And treat the token savings as a hypothesis to test on your own apps, not as a promise. The right first test is small: one app, one task, skeleton snapshots, and a verify step after every action. If the tree for that app is rich, the tool will feel fast and precise. If it is thin, you will learn that in an afternoon, before you build anything on top of it.
How This Article Was Checked
Our methodology for this article was to compare the repository readme and the official documentation, both retrieved October 31, 2026, with independent coverage from MarkTechPost, published October 31, 2026. Where the builder token figures had no independent benchmark, we labeled them self-reported instead of presenting them as measured results. Where the documentation stated a limit, such as web pages belonging to agent-browser, we kept that limit in the article instead of softening it. This article was editorially reviewed on October 31, 2026.
In our view, the most important design choice is not the accessibility tree itself, but the snapshot scoped reference. A screenshot tool can be wrong in a way that looks right, because a pixel can belong to a different element after a layout shift. A reference that fails closed forces the agent to look again. That small idea has a large effect on reliability, and it is why this tool deserves attention even before any benchmark exists.
Frequently Asked Questions
Is agent-desktop an AI agent? No. It is a tool that agents call. It embeds no language model, has no graphical interface, and has no interactive prompt. The calling agent supplies the decisions, and agent-desktop supplies the structured view of the desktop and the actions on it.
Does agent-desktop use screenshots? No, for its core loop. It reads the operating system accessibility tree and returns JSON snapshots with references. It does include a screenshot command, but the observe, act, and verify loop does not depend on pixel matching.
Which platforms does agent-desktop support? macOS 13 or later, Windows 10 1809 or later or Windows Server 2019 or later, and Ubuntu 24.04 or later, on x64 and ARM64 hardware. Each platform uses its native accessibility interface: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux.
How do I install agent-desktop? Install it with npm install -g agent-desktop. Bun, Yarn, and pnpm can install the same package. On macOS, grant Accessibility permission with the guided permissions command. On Linux, run it in a graphical session with at-spi2-core installed.
Is agent-desktop free? Yes. The software is open source under the Apache-2.0 license. Your costs are the machine you run it on and the model access of whatever agent calls it, since agent-desktop itself contains no model.
The Bottom Line
agent-desktop does one thing well, and it says so plainly. It gives any agent a structured, reference based way to read and operate desktop apps, without screenshots and without a bundled model. The token savings are still self-reported, the benchmark does not exist yet, and apps without accessibility trees stay out of reach. Those caveats do not cancel the idea. They define the test to run. Start with the GitHub repository and the official documentation. For the wider agent picture, read our Microsoft Scope article on grading agents, and our GitHub Copilot Projects article on computer use inside a product app, or read the Indonesian version of this article.