CSIRO looks to next-gen AI agents to fulfil ‘copilot’ promise – Cloud – Software – Emerging Tech

CSIRO has emerged from its trial of M365 Copilot convinced that ‘AI agents’ have a role to play in organisations, but unconvinced the Microsoft tool represents that future.

The scientific research statutory authority is the second participant in the six-month government Copilot trial to release its own analysis of the tool and its effectiveness, after the Treasury did the same in February.

The paper, published on arXiv, is broadly reflective of the overarching findings of the government-wide technology trial: that it has utility but does not live up to the marketing, leaving some user cohorts underwhelmed.

“AI copilots are marketed as transformative technologies, but their real-world value depends on several socio-technical factors, including integration with existing workflows, user trust, and the extent to which they align with professional demands,” the researchers wrote.

CSIRO included both quantitative and qualitative assessments in its paper, with an emphasis on in-depth interviews with 27 of the trial participants at the agency.

“Findings from this study reveal that while M365 Copilot provided measurable improvements in certain areas, such as meeting summarisation, email drafting, and basic information retrieval, it fell short in areas requiring domain-specific knowledge, creative problem-solving, and nuanced decision-making,” the CSIRO researchers wrote.

“Users reported a productivity paradox, where time saved through automation was often offset by the need for extensive verification and correction of AI-generated outputs.

“As organisations weigh the return on investment for AI copilots, they must consider whether these tools genuinely enhance productivity or simply shift cognitive effort elsewhere.”

CSIRO noted that it is not a typical “corporate or administrative” environment, which made its dalliance with Copilot more challenging. 

“The integration of AI solutions in scientific research presents distinct challenges and opportunities,” the researchers wrote.

On the positive side for CSIRO, users found M365 Copilot to be “most effective when used for summarisation, drafting, meeting-related activities, and technical troubleshooting… [appreciating] the way it can condense documents or web content into actionable points, assist with writing initial drafts, and simplify workflows by generating meeting summaries and action items.”

Some users – and CSIRO itself – also emerged from the trial optimistic about the promise of AI agents or assistants, even if M365 Copilot was not the right implementation of this future.

“The rise of AGI [artificial general intelligence] and AI agents means that the current generation of copilots, including M365 Copilot, will soon be eclipsed by more advanced, autonomous AI assistants,” the researchers wrote.

“Unlike M365 Copilot, which is largely an augmentation tool for Microsoft’s ecosystem, AI agents developed by competing AI firms are pushing toward more autonomous decision-making capabilities. 

“The question for organisations is no longer whether to adopt AI copilots but how to strategically integrate AI agents in ways that align with governance, workforce dynamics, and ethical considerations. 

“The rapid development of multimodal AI systems – capable of text, image, and voice-based reasoning – means that organisations must prepare for a future where AI agents operate alongside employees in a far more embedded and autonomous manner than current copilots allow.”