Anthropic Adds Screen Control to Claude 3.5 Sonnet
The new computer use API lets Claude 3.5 Sonnet move the mouse, type text, and click inside applications on your desktop. Developers send screenshots and instructions through the Anthropic SDK; the model returns coordinate-based actions that execute in a sandboxed environment. This removes the need for brittle scripts when handling data entry or browser workflows.
You stop writing one-off automation scripts and instead describe the outcome you want. The model handles the low-level clicks and keystrokes, so your workflow becomes prompt-driven rather than code-driven.
Anthropic's own research team demonstrated the feature automating spreadsheet data entry and web research tasks, cutting manual steps by roughly 80 percent in internal tests.
Step 1: Sign up at https://console.anthropic.com and enable the computer-use beta. Step 2: Install the Anthropic Python SDK and call the messages endpoint with tool choice set to computer_20241022. Step 3: Provide a screenshot and a natural-language instruction; the model returns mouse and keyboard actions that run on your machine.