Anthropic gives Claude 3.5 Sonnet the ability to move your mouse and type
Claude 3.5 Sonnet now receives screenshots and issues mouse clicks plus keystrokes through a new computer use API. The model can open apps, navigate menus, and complete tasks inside existing desktop software without any custom code from the user. Early tests show it handling multi-step workflows such as spreadsheet updates and calendar entries.
Users stop treating AI as a separate chat window and start treating it as an operator that already sits inside the programs they use every day. The workflow changes from copying text back and forth to simply stating the goal and letting the model finish the clicks.
Anthropic released the feature to all Claude Team and Pro users on October 22, 2024, and published a 20-task benchmark where the model completed 14 tasks with an average of 13.4 actions each.
Step 1: Open claude.ai and select the Claude 3.5 Sonnet model. Step 2: Type a desktop task such as 'Open Excel, load sales.xlsx, and add a total row.' Step 3: Watch the model request screenshots and issue mouse and keyboard actions until the file is saved. https://claude.ai