Vercel Tool Lets AI Agents Drive Desktop Apps

Vercel Labs, a project group at cloud platform Vercel, has released agent-desktop 1.0, a free command-line tool that lets AI coding agents read and operate native apps on macOS, Windows and Linux. The open-source project, licensed under Apache 2.0, reached version 1.0.2 on Oct. 30, according to its GitHub release notes, and was publicly promoted on Oct. 31, the Turkish tech site ShiftDelete reported.

Key takeaways

• agent-desktop reads each operating system's accessibility tree and hands agents element references to act on, instead of relying on screenshots.
• Release builds now cover macOS, Windows and Linux on both x86-64 and ARM64; Windows support arrived in version 0.10.0 on Oct. 18.
• The tool ships no language model. It works with any agent that can run shell commands, and installs with npm.

What is Vercel agent-desktop?

The project's README describes it in one line: "Agent Desktop gives any agent reliable computer use on the desktop." It is a Rust command-line tool that also ships as a C library, so other languages can embed it. It installs with npm install -g agent-desktop, and Bun, Yarn and pnpm also work. Official Homebrew and Scoop packages are still being submitted, the README says.

The repository sits under the vercel-labs organization on GitHub and is labeled a Vercel Labs product. ShiftDelete reported that the project was started in February by a developer known as Lahfir and was later brought under Vercel Labs. The v1.0.2 release notes say repository links were moved to the vercel-labs organization.

How accessibility-tree computer use works

Most computer-use agents look at a screenshot and estimate where to click. agent-desktop instead asks the operating system for the structured list of on-screen elements that screen readers use. A snapshot command returns that tree, with interactive elements tagged by references such as @e1 and @e2. The agent then clicks, types or reads values by naming a reference.

The README lists 60 command names, 56 of them operational, including snapshot, click, type, find, screenshot, launch and wait. Actions on references run headless by default; a --headed flag allows physical mouse and keyboard input. Errors carry a "disposition" field, and agents are told to retry only when it marks a retry as safe.

The documentation, last updated Oct. 31, also describes sessions for several agents at once. In a multi-agent session, each agent ID gets its own on-screen cursor. Actions that share a cursor take turns, so one command never clicks on another's target.

Platforms and limits

On macOS, users must grant Accessibility permission. On Windows, apps at the terminal's integrity level need no extra grant, and the Oct. 18 release named Windows 10 version 1809 and later and Windows Server 2019 and later. Linux needs a graphical session with at-spi2-core, and some features depend on whether the desktop uses X11 or Wayland.

The tool can only see what an app exposes through accessibility interfaces. The README also warns that snapshots, clipboard reads, screenshots and debug files "can contain private desktop content" and should be kept out of public logs. An optional helper called JEV sends screen descriptions to a model provider the user configures.

Context: coding agents move onto the desktop

The release lands in a busy month for desktop control. Cursor said on Oct. 23 that its coding agents can now drive native apps such as Apple's iOS Simulator. In August, Anthropic shipped a Browser Use toolset that likewise targets web page elements by reference through the accessibility tree.

Interest in the repository is climbing but figures vary. ShiftDelete reported more than 5,000 GitHub stars; the repository page showed about 1,900 stars when AI Newsway checked it on Nov. 2.

What's next

Packaging is the most visible open item. Homebrew and Scoop listings are pending, and the project had one open issue and 12 open pull requests on Nov. 2. Vercel has not announced a hosted or paid version.

FAQ

Does agent-desktop include an AI model?

No. It is a tool that agents call. Any coding agent that can run shell commands can use it.

Which operating systems does agent-desktop support?

macOS, Windows and Linux. Release builds are published for x86-64 and ARM64 on each platform.

Is agent-desktop free?

Yes. It is open source under the Apache 2.0 license.

References

• vercel-labs/agent-desktop — GitHub
• Releases · vercel-labs/agent-desktop — GitHub (Oct. 18–30, 2026)
• Sessions — agent-desktop documentation (updated Oct. 31, 2026)
• Vercel's agent-desktop Lets AI Agents Use Your Whole Computer — ShiftDelete (Nov. 1, 2026)