- Microsoft is expanding Copilot’s Vision capabilities
- Soon, it will be able to view and interact with Windows as a whole and apps
- You could even use it to learn a Photoshop skill
It’s a big day for Microsoft – it’s the 50th anniversary of one of the most impactful companies ever, and it’s the day that Copilot gets taken to the next level. Microsoft’s AI assistant, Copilot, is getting more personalized and has a slew of new features.
However, Copilot Vision’s expansion is worth paying attention to as it will provide one of the most meaningful intelligent boosts for Windows. We’ve already seen that Copilot can browse the web alongside you, sharing what you’re seeing with select partners and lending a helping hand.
Courtesy of the new Copilot for Windows app, you’ll be able to let the AI view your screen when in the base operating system and in select applications. As with the web, it blends multimodal AI with agentic capabilities, and thanks to its ability to view and understand your screen, it will also be able to interact with applications.
On-screen help
This comes in the form of highlighting options on your screen, and even adding in an additional cursor but also walking you through the required steps through speech. It’s quite cool.
In one shared demo, we saw Copilot provide instructions for an edit within Photoshop, essentially walking you through how to use the tool. This could be a big game-changer as it’s not just an AI for fast facts; you can integrate Copilot into your workflow and really use it how you see fit.
Of course, we’ll need to see exactly how well this performs, but alongside a more personalized Copilot, the addition of Vision for Windows is a big step. It will be opt-in at launch, and Microsoft is taking a slow and steady approach.
Vision within the Copilot for Windows app will begin rolling out for Windows Insiders next week, and Microsoft says it will be available “more broadly afterwards.” That’s not super specific, so it might be a while before you can actually give it a go, if you’re not a Windows Insider.
For Android and iOS, too
Alongside ‘Deep Research’ capabilities and a new ‘Shopping’ function for Copilot, its Vision capabilities are expanding to your Android or iOS phone, too. Via the Copilot app, you’ll be able to fire up your camera and essentially let the AI have that view for those devices. This means you could point it at a dog and ask for the breed or even a store and ask for reviews of it. You can also upload photos from your camera roll.
This enhanced Vision capability for Copilot on mobile is available right now as well, just be sure to update the app.
This change on mobile also brings Copilot into the same playing field as Google’s capabilities with Astra on Android. The changes to Windows represent a big jump in intelligence that could change the way you use apps on the operating system.
I can’t wait to give Vision a go with a complex task I’m trying to figure out how to accomplish in Photoshop or when I need to create a complex formula in Excel.
All of these changes, though, show Microsoft’s ambition with AI, specifically to get more folks to use Copilot. None of these are category-defining or brand-new per se, but they certainly make Copilot much more compelling alongside Google Gemini or ChatGPT – after all, it has the brains, so why not more features?
Leave a comment