Uk Government Trial Of M365 Copilot Finds No Clear Productivity Boost

A UK government department’s three-month trial of Microsoft’s M365 Copilot has revealed no discernible gain in productivity – speeding up some tasks yet making others slower due to lower quality outputs.

The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority of these allocated to volunteers and 30 percent to randomly selected participants. Some 300 of these people consented to their data being analyzed.

Microsoft promises Copilot will be a ‘moneymaker’ in the long term

READ MORE

An evaluation of time savings, quality assurance, and productivity was then calculated in the assessment.

Overall, 72 percent of users were satisfied or very satisfied with their digital assistant and voiced disappointment when the test ended. However, the reality of productivity gains was more nuanced than Microsoft’s marketing materials might suggest.

Around two-thirds of the employees in the trial used M365 at least once a week, and 30 percent used it at least once a day – which doesn’t sound like great value for money.

In the UK, commercial prices range from £4.90 per user per month to £18.10, depending on business plan. This means that across a government department, those expenses could quickly mount up.

According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user.

“Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day,” the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as “very low,” less than 1 percent and 3 percent per day, respectively.

“PowerPoint and Excel were slightly more popular; both experienced peak activity of 7 percent of license holders using M365 Copilot in a single day within those applications,” the study states.

The three most popular tasks involved transcribing or summarizing a meeting, writing an email, and summarizing written comms. These also had the highest satisfaction levels, we’re told.

Participants were asked to record the time taken for each task with M365 Copilot compared to colleagues not involved in the trial. The assessment report adds: “Observed task sessions showed that M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users. Time savings observed for writing emails were extremely small.

“However, M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users, conflicting time savings reported in the diary study for data analysis.

“PowerPoint slides [were] over 7 minutes faster on average, but to a worse quality and accuracy than non-users.” This means corrective action was required.

A cross-section of participants was asked questions in an interview – qualitative findings – and they claimed routine admin tasks could be carried out with greater efficiency with M365 Copilot, letting them “redirect time towards tasks seen as more strategic or of higher value, while others reported using these time savings to attend training sessions or take a lunchtime walk.”

Nevertheless, M365 Copilot did not necessarily make them more productive, the assessment found. This is something Microsoft has worked on with customers to quantify the benefits and justify the greater expense of a license for M365 Copilot.

“We did not find robust evidence to suggest that time savings are leading to improved productivity,” the report says. “However, this was not a key aim of the evaluation and therefore limited data was collected to identify if time savings have led to productivity gains.”

Microsoft can’t guarantee data sovereignty – OVHcloud says ‘We told you so’

READ MORE

And hallucinations? 22 percent of the Department for Business and Trade guinea pigs that responded to the assessors said they did identify hallucinations, 43 percent did not, and 11 percent were unsure.

Users reported mixed experiences with colleagues’ attitudes, with some teams embracing their AI-augmented workers while others turned decidedly frosty. Line managers’ views appeared to significantly influence adoption rates, proving that office politics remain refreshingly human.

The department is still crunching numbers on environmental costs and value for money, suggesting the full reckoning of AI’s corporate invasion remains some way off. An MIT survey published last month, for example, found that 95 percent of companies that had collectively sunk $35-40 billion into generative AI had little to show for it.

For now, it seems M365 Copilot excels at the mundane while stumbling over the complex – an apt summary of GenAI in 2024. ®

Original Source