ANZ Banking Group has one of its Melbourne engineering teams using GitHub Copilot, an “AI pair programming” tool that has led to some work being completed up to 50 percent faster.
Copilot is intended to aid faster coding, drawing context from comments and code to suggest individual lines and whole functions to development teams.
Not wanting to rely on “anecdotal evidence around double-digit percentage increases in performance”, ANZ’s team set out to assess the performance of the AI pair programming tool itself.
“We got a large pool of engineers and we basically cut them into two halves and gave them a bunch of artificial programming problems, with and without using GitHub Copilot,” ANZ’s chief technology officer Tim Hogarth told iTnews.
“We also looked at some of the other co-generation tools as well. But this specific one we looked at and said, ‘We’re going to create some programming problems and we’re going to get some of our engineers to self-assess [efficacy], whether they’re relatively inexperienced in that technology, or are intermediate or advanced.”
About 105 different problems were generated, with the average time saving calculated from here.
“We timed how long it took engineers to solve the problems, with and without using this technology, and then we took the average,” Hogarth said.
He said with just this code generation capability alone, “we think we’re going to be able to deliver a lot more things” of value to market, faster.
So far ANZ has 1000 engineers using GitHub Copilot. While still early days for the technology, Hogarth is hopeful around 3000 engineers will use it by the end of the financial year.
Hogarth said Copilot “automatically generates boilerplate code” and could also act as “an autocorrect” for hand-coding.
“So, instead of spending six minutes on a task, you can probably spend as little as 60 seconds,” Hogarth said.
“With tools like Copilot, a lot of the routine work that people do can be significantly accelerated,” Hogarth said.
“If we’re trying to get a product feature out to market, it doesn’t matter if it’s a customer-facing application or web app, or it’s an internal application or mobile app, this should help all our engineers produce those tasks more quickly, which would give us a flywheel effect, which means we can deliver all of the value to market faster.”
ANZ is treating the expansion of GitHub Copilot “thoughtfully and carefully”.
“We’ve got engineers in lots of locations and we want to make sure we do this in accordance with all the local rules,” he added.
Some areas the bank is applying Copilot in include its ANZ Plus division, which should help the team “tremendously” across its payments divisions.
Part of its safety controls include “staying well away from confidential, restricted data” and keeping to “as internal policies and procedures for now”.
“We’re still just making sure we understand this technology … we’re still trying to make sure we get to the right level of controls so we can understand the performance of these models and therefore work out when we can trust them and when we when we actually need to be adding additional rigour.”
Z-GPT next steps
In a different area of AI-based assistance, the bank is continuing its cautious experimentation with Z-GPT, a privately hosted generative AI chatbot it has stood up in late August.
“As we start to train it or tune it using subsets of ANZ documents for very specific topics, we’re starting to prove a lot of potential value,” he said.
Applications included in specialised back-office and support functions.
“We’re anticipating doing more work in that, in those more focused areas,” he said.
Before its experimentation, Hogarth said the team created a framework around “what the problem is we’re trying to solve”, associated risks and “importantly, what does a proof point look like”.
“The objective of our AI is not to prove it can work alone – it’s got to be to prove a specific piece of learning, knowledge or something you couldn’t speculate upon to justify a large investment,” Hogarth said.
“For example, when we were looking at using generative AI to develop faster coders, the hypothesis was that people would be able to solve complex problems more quickly.
“We defined a set of proof points and criteria for the experiment rather than just say, anecdotally, ‘Yeah, that felt fast’.”
He added that “where people have come up with really grandiose ideas” it was decided not to explore these further as “success wasn’t well-defined”.
“We want to make sure we spend [time] on the things that are going to be the biggest ‘bang for buck’ and we want to make sure that we’re spending things on the right things,” Hogarth said.
“We’re not ready or interested in using this for any customer-centric activities yet.
“We want to get confident on things that are nowhere near the customer.”
When building its own generative AI, Hogarth said its Z-GPT “was essentially the first step in us exploring the value of AI” with work undertaken “relatively simply”.
The in-house tool was aided by OpenAI’s models via ANZ’s established relationship with Microsoft and some of Google’s AI tools.
Expanding AI use via existing enterprise applications
Since its August announcement, Hogarth said ANZ is exploring how it might adopt AI-driven technologies and capabilities “almost off-the-shelf” in the platforms the bank already uses today.
ANZ is also exploring AI in Microsoft 365, with active work “underway right now” and use cases identified across commodity use cases.
This work is still “confidential” Hogarth said but relates to examples of how ANZ plans to help its staff find and compare information.
“We’re going to take some commodity things off-the-shelf, and they’re the things that are in a lot of the products that you’ll hear about in the marketplace,” Hogarth said.
“A lot of the platforms we use are increasingly baking this capability in.
“But we are also going to start to take ANZ data and tune it and use some of these LLMs in partnership with narrow datasets from ANZ to focus on helping zero in and laser focusing on very specific problems, like helping people find the answers to things.”
One of the use cases ANZ has underway is the ability to compare one policy with another document, which will be a “significant augmentation”, said Hogarth.