Claude 3.5 Sonnet Gains Direct Desktop Control Through Anthropic API
Anthropic added a computer use feature to Claude 3.5 Sonnet that lets the model move the cursor, click interface elements, and type text inside applications. The model receives screenshots and outputs mouse coordinates and keystrokes through the Anthropic API. Users can now chain desktop actions without writing code or hiring developers.
This shifts automation from scripting languages to natural language instructions. Readers stop thinking of AI as a text tool and start treating it as a direct interface operator. Workflows that once required developers become simple prompt sequences.
The Anthropic team documented this capability in their October 2024 release. Their internal tests showed the model completing multi-step tasks like spreadsheet editing and web navigation using only the computer use API.
Step 1: Sign up for the Anthropic API at console.anthropic.com and request access to the computer use beta. Step 2: Send a prompt that describes the desktop task and include the computer use tool parameter in your API call. Step 3: Review the returned actions and confirm execution in your local environment.