Anthropic just rolled out an upgraded version of its Claude 3.5 Sonnet and it’s safe to say that it’s revolutionary.
The product can not only control the way PCs work but also interact with any app on your computer. This is all thanks to the latest Computer Use API that’s now available as an open beta. This model imitates keystrokes, mouse gestures, and even button clicks. It’s quite like emulating people on a PC.
In a newsroom post shared by the company, Anthropic says it trained the model to see if it could decipher what’s taking place on a screen and then use software tools to carry out such tasks. When developers give the model a task, it sees what’s visible to others. It counts the pixels needed to generate a command so users click the right location.
Developers can experiment with Computer Use through Anthropic’s API. Remember, it’s not exactly a novel idea to automate apps on computers. Plenty of tech giants achieved success including startups in the industry.
Thanks to AI, this field keeps getting more and more saturated. AI automating software is what the latest model is all about. As per the company, the ability to explore the web is the right way to go about the situation. Anthropic says the latest action-execution layer adds a new 3.5 Sonnet to carry out commands at the desktop level.
Humans are in control at all times by giving rise to certain prompts that direct their actions. It’s quite like using data from different computers for filling out forms. People can access and even restrict when required.
This model will break down the prompts from the user’s end into certain computer commands where you move cursors, click, and type when doing a task. We can even confirm that several companies are already using this mode like Replit who used an older variant of Sonnet 3.5 to make autonomous verifiers. Meanwhile, Canva is working on ways by which new models can offer support for designs and edits.
Now a lot of people are asking if this model is unique and if yes, how does it differ from others in the market? The question is so reasonable and the answer is that Antrhopic promises the new model is not only stronger but more robust in design. It can do a great job in terms of task coding, better than OpenAI’s flagship model. It’s shocking because although it’s not trained to do so, the latest model self-corrects and retries when hit with challenges. It can work to fix objectives that feature hundreds of steps.
With the good does come some bad. And Anthropic did explain how the latest Sonnet model is struggling in terms of simple commands like scrolls and zooming in and out of pictures. It also tends to miss out on short-lived actions like alerts because of the manner it clicks pictures and puts them together.
Read next: Google Messages Introduces Contact Verification, Scam Filters, and More in Security Overhaul