Colonial First State brings AI to its FirstTech function – Emerging Tech – Financial Services – Digital Nation – Software

Colonial First State is optimising a generative AI chatbot for its FirstTech function – which answers technical questions for financial advisers – to improve accuracy and expand potential uses of the technology.

Head of technical services Craig Day told a recent CFS 10x event in Sydney that while it’s still “early days”, the bot is able to answer complicated financial questions to about 65 percent accuracy.

FirstTech was set up around 1999 to help advisers navigate complex financial rules around self-managed superannuation, aged care, tax and other issues.

Day said the team receives between 12,000 and 15,000 calls from advisers a year.

“My team needs to be skilled up to be able to understand the question and respond to it and give [advisers] some ideas about how the rules work in that situation and what to watch out for,” he said.

“One of the common types of questions [wet get] is, ‘My client recently sold their home. How will the sale proceeds be assessed for social security purposes?'”

The institution first started looking at generative AI as a way to help new FirstTech team members to onboard faster.

“It takes me about 18 months to two years for a new person to be useful for me. During that time, we’re training,” Day said.

“We thought a bot could assist one of our [team] to skill up to do the job maybe a bit quicker.”

Day said that FirstTech was also interested in whether the bot could be opened up to be adviser-facing in the future as well.

The bot was initially trained on “3000-to-5000 pages” of technical documentation, and while its initial outputs “weren’t very good”, several weeks spent “pulling different levers and organising the data in a different way” brought the accuracy of responses up to 65 percent.

While promising, the answers lacked nuance. 

“It can produce a response that if you read that, you go, ‘That’s pretty good’.  But it’s when you sit down and look at the detail, you see there’s some stuff missing in there,” Day said.

“If my team gave that [AI-generated] response [to an adviser], there’s nothing wrong. But if an advisor relied on it, [they] would go down a [path] where [they’re] missing potential opportunities or putting the client in a position where [they] wouldn’t get the outcome that they wanted.”

The next step in improving the accuracy is transcribing all calls that come into FirstTech and feeding that into the bot.

“Hopefully we’ll continue to work on this over time and we can actually begin to develop and deliver really good outcomes for [advisers] where [they] can use it as a tool just like we see the ability to use it for the FirstTech team,” Day said.

The ability for generative AI to transcribe calls and to use that content to improve accuracy when answering technical questions has also encouraged CFS to look for more internally-facing uses for the technology.

One possibility is implementing agentic AI to summarise a call, pass that by the FirstTech team for review, and add it to a file note automatically.

It could also create an accurate and auditable trail “that shows the type of questions [advisers] ask and the issues that came out of that.”

Day flagged anomaly detection as another potential use case.

“Once this bot gets good, I can have it listening to every single one of my calls  and it can spot the one that doesn’t sound right, and we can go back in and have a look at that,” he said.

“Currently, we do quality control on FirstTech calls, but we can do a tiny minority of the number of calls. 

“All of a sudden we get 100 percent of the calls being checked and then validated when something pops up, so we’re checking by exception – not looking for a needle in the haystack.”

Day said that a key milestone for the technology would be to provide a definitive and comprehensive answer to a question that advisers could singularly rely on.

“I know it’s a common thing for advisers – what they do is you call FirstTech, and then they call another technical team, and another and another, and if each says the same thing, they can be pretty confident in the answers they’re getting.

“What if they just needed to do it once?”