Microsoft‘s Copilot AI assistant will start seeing what you’re looking at online, according to a tease post on X. The tech giant gave Copilot a major makeover a month ago that included a feature called Copilot Vision for a select group that signed up to the Copilot Labs experiment hub, but giving the AI the ability to see everything on your screen is now going to be an option for everybody.
After gathering feedback for the last month or so, Copilot Vision is apparently ready for a general rollout, including integration into the Microsoft Edge browser. Once it is available, you’ll just need to click on the screen-like icon on the browser to activate the tool. Copilot Vision essentially puts an AI eye on the same things you’re looking at, from websites to documents. It can even read text, whether typed or handwritten and displayed on the screen. The AI will observe and respond to what’s on the screen in order to answer any of your questions and offer ideas on what to do next.
Instead of researching a future trip from a website about options, Copilot Vision will just offer details and recommendations and answer your questions without needing a new search. If you see a recipe online but want to change it, Copilot Vision could tell you some substitution options or offer cooking tips without you having to leave the page or pull up the chatbot.
If only your browser could see what I see… oh wait, Copilot Vision will be able to very soon 👀 pic.twitter.com/EtLGyMGPq3November 1, 2024
Vision and Visuals
This approach is what makes Copilot Vision different from typical chatbots as it doesn’t need you to explain what is happening that led you to ask for help or advice. It can see the same content as you. It’s part of how Microsoft is trying to make Copilot a useful helper that people will turn to whenever they need advice.
If you’re worried about privacy, Microsoft has said Copilot Vision data will not be carried over from previous sessions. Microsoft doesn’t save or use your browsing data after the session is over. It also won’t work across every website, at least at first. Copilot Vision will only look at some popular websites that fit Microsoft’s security levels. As ChatGPT and Claude both move into Copilot’s space with desktop apps, this kind of feature might be the key for Microsoft to compete in the broader AI assistant arena.
