Anthropic Gives Claude 3.5 Sonnet a Mouse and Keyboard
Claude 3.5 Sonnet now uses the computer use API to move the cursor, click, type, and scroll on a real desktop. The model receives screenshots at 1280x800 resolution and outputs structured actions such as left_click[x=450,y=320] or type[text='invoice_2024']. Early tests show it completing multi-step spreadsheet edits and web form submissions without custom scripts.
Users stop treating AI as a text box and start treating it as a desktop agent. The workflow changes from copying model output into software to letting the model operate the software itself, which removes the copy-paste layer and reduces context-switching time.
Anthropic released the feature in public beta on October 22, 2024. Several small agencies are already routing invoice data entry through Claude, cutting a two-hour manual task to roughly twenty minutes per batch.
Step 1: Sign up at https://console.anthropic.com and enable the computer-use beta in your account settings. Step 2: Run the provided Python SDK example that opens a visible desktop window and passes each screenshot to the model. Step 3: Watch the model complete a three-field web form; the expected outcome is a submitted form with no manual clicks from you.