Feeling Lonely? Microsoft Copilot Can Now Listen To Your Every Word, Watch Yourscreen

As if pulling support for Windows 10 was not punishment enough for long-suffering customers, Microsoft has decided to shove Copilot down everyone’s throats with a new voice activation feature and even more control over your PC. Soon, a Copilot box may even replace the search box on your taskbar.

According to Yusuf Mehdi, Microsoft’s consumer marketing chief, Copilot Voice and Copilot Vision are coming to “every Windows 11 device where Copilot is available.” Users can bark instructions with the wake word “Hey Copilot,” and have Copilot Vision analyze what’s on their screens without investing in Copilot+ hardware.

The features are on a gradual rollout and Microsoft hasn’t said exactly when everyone can expect to get them. However, The Register’s US Editor, Avram Piltch, got access this morning to the always-listening feature on a virtual machine he was running with Windows 11 Beta Channel installed. When he launched the Copilot app, Piltch saw a button allowing him to enable the feature (or not).

Enable Copilot wake word – Click to enlarge

After he had turned it on, Piltch was able to invoke Copilot by saying “Hey Copilot,” but he had to actively click a Copilot Vision button for the assistant to see what was on his screen. After clicking the button, he watched a web video about a mechanical keyboard and asked Copilot “what keyboard is that” to which it correctly replied “the Keychron K2 HE” (note that the name of the keyboard was on the web page). He then asked Copilot for pricing and the assistant responded correctly that it’s $139.

However, when Piltch asked Copilot to perform some actions, it only told him what to do, rather than doing it for him. For example, he asked Copilot to go to the website to purchase the keyboard, but instead of pointing the Edge browser toward a URL, the assistant told him to find a link on the YouTube page and click it himself. It also refused to open apps like Notepad. It wouldn’t change settings for him either.

Piltch also found that you can’t see the text output from Copilot while you are in voice mode and this can be annoying. He clicked the Copilot Vision button and went to visit his LinkedIn profile in Edge browser and then asked Copilot to “write my resume based on this LinkedIn profile,” and had to listen while it read him the resume it was writing slowly, via voice. To read and copy the actual text (which was just a summary of LinkedIn), he had to hit stop and close out of voice mode.

Yapping at a PC is not particularly new, but now Microsoft has stirred Copilot as an opt-in function into the mix. The wake word starts the AI listening, and saying “Goodbye” or tapping X will stop it. Alternatively, give Copilot the silent treatment for a few seconds, and it will also terminate the conversation.

While voice-control is a boon for accessibility, we’d have to wonder if there are that many people out there who really want to interact with their PCs in this way. Imagine the increase in volume in open-plan offices, or some jerk saying: “Hey Copilot, format C, goodbye,” as they walk past a victim’s cubicle. We jest, of course, and at least the feature is opt-in.

Copilot Vision gives the assistant access to the desktop to come up with helpful tips and “see” entire Excel, Word, or PowerPoint documents, not just what is on the screen. As a reminder, it is just over a year since Microsoft’s catastrophic launch of Recall, a tool that screenshots your every move, even financial and personal info, which caused uproar in the privacy and security communities.

Copilot Vision can only be used with voice at present, but Windows Insiders will soon be able to interact with it using text in that same chat window.

Microsoft is also tinkering with the taskbar and adding an “Ask Copilot” function. As part of an opt-in experience, the Windows giant wants the taskbar to be a “dynamic hub… transforming everyday interactions into moments of productivity and delight.”

When you opt into this experience – something Piltch was not able to test – a Copilot box replaces your search box entirely. It’s unclear what happens if you just try to search for an app or file from this Copilot box. We’re concerned that eventually this Copilot promotion could replace search as everyone’s default taskbar UI.

According to Microsoft: “This experience leverages existing Windows APIs to return your apps, files, and settings, like the Windows Search experience, and does not grant Copilot access to your content.”

Microsoft is also making good on its threat to let Copilot loose on local files, at least for Windows Insiders. Copilot Actions has already been shown off, booking tables or ordering groceries. Microsoft wants to extend this. It said: “Simply describe the task you want to complete in your own words, and the agent will attempt to complete it by interacting with desktop and web applications.”

In a demo video, Microsoft shows a user asking Copilot to change the orientation of a folder full of photos. A voiceover says you can also use Copilot to read PDFs for you.

The theory goes that a user can do other things while the magic happens. In reality, we suspect that many users will want to keep a close eye on the assistant in case something unpredictable happens.

Microsoft acknowledged this, saying: “You may see the agent make mistakes or encounter challenges with complex interfaces, which is why real-world testing of this experience is so critical to help us apply learnings to make this experience more capable and streamlined.”

That’s also why a very narrow set of use cases will be rolled out first.

Microsoft has also announced Manus, an agent that can perform several tasks on a user’s PC, such as building websites based on local documents. Zoom functionality is also to be implemented for Click To Do Copilot+ PC users.

Microsoft did not provide a timeline for when the features would roll out to end users or Windows Insiders.

It’s a scattergun approach, and it is difficult to escape the feeling that Microsoft is shoveling AI tech into Windows as soon as it emerges from the lab, hoping that something, anything will chime with users. Do users need Copilot throughout their operating system, or would they rather Microsoft focus instead on releasing updates that don’t break previously working functionality?

We suspect we know the answer, even if Microsoft is too busy talking to Copilot to hear it. ®

Original Source