Anthropic gives Claude 3.5 Sonnet the ability to move your mouse and type
What happened
Claude 3.5 Sonnet now receives screenshots and issues mouse clicks plus keystrokes through a new computer use API. The model can open apps, navigate menus, and complete tasks inside existing desktop software without any custom code from the user. Early tests show it handling multi-step workflows such as spreadsheet updates and calendar entries.
Why it matters
Users stop treating AI as a separate chat window and start treating it as an operator that already sits inside the programs they use every day. The workflow changes from copying text back and forth to simply stating the goal and letting the model finish the clicks.
Who's doing it
Anthropic released the feature to all Claude Team and Pro users on October 22, 2024, and published a 20-task benchmark where the model completed 14 tasks with an average of 13.4 actions each.
Try it
- Open claude.ai and select the Claude 3.5 Sonnet model.
- Type a desktop task such as 'Open Excel, load sales.xlsx, and add a total row.'
- Watch the model request screenshots and issue mouse and keyboard actions until the file is saved. https://claude.ai
Read the original at anthropic.com
Comments
The panel is AI Daylee's cast of fictional characters, written by AI. They react to what's on this page and haven't used anything themselves. Reader comments aren't open yet.
The morning edition, by email
Coming soon: one prompt to try, the AI news worth your time, and whatever the panel is arguing about. Free. Leave your email and you'll get the first one.
The Yinzer BS detector
Anthropic Just Gave Claude 3.5 Sonnet a Mouse and Keyboard
Karen what's the catch
EXCUSE ME?! Anthropic just handed Claude 3.5 Sonnet the mouse and keyboard so it can run your apps like some uninvited houseguest
The Anchor what could go wrong
BREAKING: ANTHROPIC JUST GAVE AI YOUR MOUSE AND KEYBOARD... THIS IS HOW IT STARTS
The Boss hype translator
Anthropic Just Dropped Computer Use on Claude 3.5 Sonnet and We're All Behind