Optus said it has traced the root cause of a near-full-day outage of its network last week to bad route data from an “international peering network” that was allowed to propagate through its own network.
The telco disclosed the apparent root cause ahead of a senate inquiry into the outage on Friday this week, which had listed the topic as one it wanted to get to the bottom of.
“At around 4.05am Wednesday morning [November 8], the Optus network received changes to routing information from an international peering network following a routine software upgrade,” Optus said.
“These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these.
“This resulted in those routers disconnecting from the Optus IP Core network to protect themselves.”
Optus confirmed industry speculation that the nature of the issues meant it needed to have technicians physically attend routers to try to reboot and/or reconnect them.
The telco described the restoration as a “large-scale” team effort, “requiring the dispatch of people across a number of sites in Australia.”
“This is why restoration was progressive over the afternoon [of November 8],” Optus said.
“Given the widespread impact of the outage, investigations into the issue took longer than we would have liked as we examined several different paths to restoration.
“The restoration of the network was at all times our priority and we subsequently established the cause working together with our partners.”
Optus said it had made changes to its network “to address this issue so that it cannot occur again”.
The potential for the same issue to re-occur, either in Optus’ network or on the networks of other major Australian telcos, is a topic also set for discussion this Friday.
Optus apologised for the lengthy outage; however it did not address a third issue the Senate has expressed interest in, which is user compensation.
Currently that is limited to extra data quota, but the telco’s response has been criticised in part because the outage resulted in financial losses for business customers and other users.
The Telecommunications Industry Ombudsman (TIO) is offering guidance on paths for claiming financial compensation.