Bendigo and Adelaide Bank tackles Kafka growth since open banking

Bendigo and Adelaide Bank will switch on chargeback for internal departments using Kafka resources from July, but is already realising efficiency gains from a period of showback of expenses incurred.

Database and middleware service owner Dom Reilly told a recent Confluent summit that the bank started using Kafka – implemented via Confluent Cloud and Confluent Platform, both in AWS – back in 2020 “as a critical component of Bendigo’s open banking solution”.

Open banking is a government program to enforce data sharing between institutions and third-parties.

Since then, the number of teams consuming Kafka resources at the bank has grown from one to 16, coupled with a larger environment and increase in resource usage.

The environment has been set up to enable technology teams to self-serve access to Kafka resources.

“This frees up application teams to deploy what they need when they need it,” Reilly said.

“The goal of the self-service is to empower our users to define and deploy their Kafka requirements, enabling them to deliver fast, reducing friction between teams, and empowering them to diagnose and remediate issues. 

“We want to give them that freedom, but we also want them to be responsible. 

“How do we do that at the same time as giving them this freedom and get them to act commercially and manage resources efficiently? Putting a dollar figure on their choices is a great way to assign that accountability. It’s a language everyone understands.

“From my team’s point of view, as providers of the Kafka capability, we just want to encourage good behaviour, manage resources efficiently, clean up resources that aren’t required, thin provision where possible, [and] ultimately get the most value out of the platform.”

Reilly noted  the self-service pipelines also “enforce the capture of metadata, ownership and data classification”, which serves two purposes – managing access to “topics”, Kafka parlance for a grouping of messages; and also “to assign costs through [a] chargeback system”, which the bank and Confluent have worked to build.

Chargeback is not conceptually new for the bank’s tech teams.

“Luckily for the middleware team, Bendigo Bank is already doing a lot of chargeback for other cloud services, so the cloud platform costs, [and for] Splunk [as] a shared service, so we didn’t have to trailblaze there, we had a precedent to build on,” Reilly said.

“That made the conversation with our Kafka customers a lot easier.”

Departments using Confluent resources are given their own dashboard to track their usage and costs.

Underpinning the dashboards is a Splunk instance fed by multiple data sources – a combination of Confluent Cloud metrics API [output], an Open Telemetry collector running in AWS, “and other reference material from Terraform and GitOps.”

Reilly said that three categories of resource usage are being tracked: resources that contribute to customer limits or thresholds such as schemas and partitions; resources that have a measurable cost such as cloud and storage costs, tracked using a “retained bytes” metric; and resources that have a measurable cost but that are shared, such as cluster costs and licensing platform components.

“Shared cost is an interesting challenge with a chargeback model,” Reilly noted.

“A key idea underpinning our solution is that all usage metrics are aggregated to provide a measure of the proportion of the cost per customer. 

“There’s a fixed monthly bill which is based on the contract commitment, [and] this amount is distributed according to proportional use that’s calculated to each customer.”

Reilly said that there could be fluctuations in the amount month-to-month due to currency exchange rates.

Additionally, “towards the end of the contract period, it’s possible that the business’s total usage will exceed the contract commitment so going into overages”, although he added it “will be clear well in advance if we’re going into overages”, allowing some advance notice.

“Lastly, it’s important to note that because the cost is based on your proportional usage, the cost can vary even though usage remains constant,” Reilly said.

“So, for example, if one customer decommissions something and reduces their usage, other customers’ proportional usage will go up and also therefore their cost, even though usage is flat. 

“This is something customers are made aware of, and with 16 different teams now consuming Kafka resources and sharing the costs, this is less of an issue.”

So far, departments have been given access to their dashboards on a showback basis – that is, their resource costs are apparent, but they aren’t yet being charged for usage.

This had resulted in some benefits for the bank already.

“It’s relatively early days but we’ve already seen benefits from showing the cost back to customers,” Reilly said.

“Metadata has been cleaned up, in particular ownership; unused resources are being cleaned up, especially schemas and topics that are not used.

“Teams are [also] more accurately forecasting their costs, [and] are being proactive, discussing plans to deploy resources.

“The initial knowledge sharing and socialisation needs to continue but pleasingly with costs distributed people are reaching out already to find out more and understand what they can do and how they can contain their costs.”

Reilly added that chargeback would begin in July.