Cloudflare’s privacy-first public DNS resolver service was hit by two simultaneous BGP issues recently, resulting in an unintentional BGP hijacking incident that highlights ongoing concerns over the security of the 35-year-old internet routing protocol.
The outage and slowdowns that affected the free Cloudflare DNS resolver service “1.1.1.1” for a few hours on June 27 affected less than 1 percent of internet traffic, but the issue is likely to bring fresh attention to BGP, dubbed the “three-napkin protocol” for the way it was drafted on a lunch break at an IETF meeting in 1989. The FCC recently voted to require ISPs to report on their BGP security progress, a preliminary vote that will go through a public comment period before it can be finalized.
Historical Use of 1.1.1.1 Hits Cloudflare DNS Resolver Service
One problem affecting the six-year-old Cloudflare DNS resolver service is the historical use of 1.1.1.1 by networks in labs or as a testing IP address, “resulting in some residual unexpected traffic or blackholed routing behavior. Because of this, Cloudflare is no stranger to dealing with the effects of BGP misrouting traffic,” Cloudflare engineers wrote in a blog post reporting the incidents.
The June 27 incident combined a routing hijack with a BGP route leak to effectively bring the service to a halt for users in affected regions.
A routing hijack of 1.1.1.1 could potentially occur if a network assigned, say, 1.1.1.1/32 to one of their routers and shared the prefix with their internal network, which would make it difficult for their customers to route to the 1.1.1.1 DNS service. “If they advertise the 1.1.1.1/32 prefix outside their immediate network, the impact can be even greater,” the engineers wrote.
The reason 1.1.1.1/32 would be selected instead of the 1.1.1.0/24 used by Cloudflare is due to Longest Prefix Matching (LPM), they said. Many prefixes in a route table could match for the 1.1.1.1 address, but 1.1.1.1/32 is considered the “longest match” by the LPM algorithm because it has the highest number of identical bits and longest subnet mask while also matching the 1.1.1.1 address. “In simple terms, we would call 1.1.1.1/32 the ‘most specific’ route available to 1.1.1.1,” Cloudflare said.
BGP route leaks occur “when a network becomes an upstream, in terms of BGP announcement, for a network it shouldn’t be an upstream provider for. … If enough networks within the Default-Free Zone (DFZ) accept a route leak, it may be used widely for forwarding traffic along the bad path. Often this will cause the network leaking the prefixes to overload, as they aren’t prepared for the amount of global traffic they are now attracting.”
These issues can happen to any IP address or prefix, but 1.1.1.1 is “is such a recognizable and historically misappropriated address that it tends to be more prone to accidental hijacks or leaks than other IP resources,” Cloudflare said.
What Happened to the Cloudflare 1.1.1.1 Service
The Cloudflare 1.1.1.1 incident began when AS267613 (Eletronet S.A.) began announcing 1.1.1.1/32 to peers, providers, and customers.
A minute later, AS262504 (Nova) leaked 1.1.1.0/24, also received from AS267613, upstream to AS1031 (PEER 1 Global Internet Exchange), which propagated 1.1.1.0/24 to various Internet Exchange peers and route-servers, widening the impact of the leak.
One tier 1 provider received the 1.1.1.1/32 announcement from AS267613 as a Remote Triggered Blackhole (RTBH, a blunt method of fighting off DDoS attacks) route, causing blackholed traffic for all of the tier 1’s customers.
All of that happened in about a minute. Cloudflare engineers responded a little more than an hour later by disabling two partner peering points and engaging with Nova and Eletronet, but the issues weren’t fully resolved until about four hours after they began.
“The problem during this incident was AS267613 was unauthorized to blackhole 1.1.1.1/32,” the engineers wrote. “Cloudflare only should have the sole right to leverage RTBH for discarding of traffic destined for AS13335, which is something we would in reality never do.”
Cloudflare said AS1031 “does not perform any extensive filtering for customer BGP sessions, and instead just matches on adjacency (in this case, AS262504) and redistributes everything that meets this criteria. Unfortunately, this is irresponsible of AS1031 and causes direct impact to 1.1.1.1 and potentially other services that fall victim to the unguarded route propagation. While the original leaking network was AS262504, impact was greatly amplified by AS1031 and others when they accepted the hijack or leak and further distributed the announcements.”
Cloudflare’s BGP Security Recommendations
Cloudflare offered several BGP security recommendations for network providers, particularly major networks with a large sum of downstream Autonomous Systems.
RPKI Adoption
Resource Public Key Infrastructure adoption “recently reached a major milestone at 50% deployment in terms of prefixes signed by Route Origin Authorization (ROA),” the Cloudflare engineers wrote. “While RPKI certainly helps limit the spread of a hijacked BGP prefix throughout the Internet, we need all networks to do their part, especially major networks with a large sum of downstream Autonomous Systems (AS’s). During the hijack of 1.1.1.1/32, multiple networks accepted and used the route announced by AS267613 for traffic forwarding.”
RPKI and Remote-Triggered Blackholing (RTBH)
Cloudflare said a significant amount of the impact from the incident was because of a Tier 1 provider accepting 1.1.1.1/32 as a blackhole route from a third party that was not Cloudflare.
“This in itself is a hijack of 1.1.1.1, and a very dangerous one,” the Cloudflare engineers said. “RTBH is a useful tool used by many networks when desperate for a mitigation against large DDoS attacks. The problem is the BGP filtering used for RTBH is loose in nature, relying often only on AS-SET objects found in Internet Routing Registries. … AS-SET filtering is not representative of authority to blackhole a route, such as 1.1.1.1/32. Only Cloudflare should be able to blackhole a destination it has the rights to operate.”
A potential fix for lenient filtering of providers on RTBH sessions would again be leveraging an RPKI. An expired IETF draft proposal specified a Discard Origin Authorization (DOA) object that would be used to authorize only specific origins to authorize a blackhole action for a prefix. “If such an object was signed, and RTBH requests validated against the object, the unauthorized blackhole attempt of 1.1.1.1/32 by AS267613 would have been invalid instead of accepted by the Tier 1 provider,” Cloudflare said.
BGP Best Practices
Simply following BGP best practices laid out by Mutually Agreed Norms for Routing Security (MANRS) and rejecting IPv4 prefixes that are longer than a /24 in the Default-Free Zone (DFZ) would have reduced impact to 1.1.1.1, Cloudflare said. “Rejecting invalid prefix lengths within the wider Internet should be part of a standard BGP policy for all networks.”
ASPA for BGP
Cloudflare has advocated for the adoption of RPKI into AS Path-based route leak prevention. Under Autonomous System Provider Authorization (ASPA) – still in its draft phase at IETF – ASPA objects are similar to ROAs, except instead of signing prefixes with an authorized origin AS, the AS itself is signed with a list of provider networks that are allowed to propagate their routes, they said. “So, in the case of Cloudflare, only valid upstream transit providers would be signed as authorized to advertise AS13335 prefixes such as 1.1.1.0/24 upstream.”
Cloudflare Expands BGP Route Leak Detection
Cloudflare has expanded its data sources for its route leak detection system to cover more networks and is “incorporating real-time data into the detection system to allow more timely response toward similar events in the future.”
“Like all approaches to solving route leaks, cooperation amongst network operators on the Internet is required for success,” Cloudflare said. “While the actions of external networks are outside of Cloudflare’s direct control, we intend to take every step within both the Internet community and internally at Cloudflare to detect impact more quickly and lessen impact to our users.”
Cloudflare also encouraged users to check isbgpsafeyet.com to see if their ISP is enforcing RPKI origin validation.