A mysterious problem impacting users of Cisco small-to-medium business switches whose devices started rebooting regularly turned out to be caused by a change in how Cloudflare’s popular domain name system (DNS) server responded to queries.
Network switches worldwide entered reboot loops every 10 to 30 minutes, with logging and core dumps from the resets suggesting it was caused by DNS lookups for domains such as www.cisco.com and network time protocol (NTP) servers.
Administrators trying to troubleshoot the problem found that turning off DNS resolution or SNTP synchronisation stopped the reboots.
They also noticed that it was devices using Cloudflare’s popular DNS server at 1.1.1.1 that crashed; this turned out to be the root cause.
Cloudflare posted an incident report message on January 9, saying it had reverted a software update to restore the standard record ordering, to resolve the problem.
“Specifically, the sequence of the CNAME and non-CNAME records in the ‘answer’ section was changed, which conflicted with the expectations of certain DNS client implementations,” Cloudlfare said.
CNAME is an abbreviation for Canonical Name, and is a record in the DNS that maps to another one.
Models in the CBS, C1200 and SG range of switches were affected by the problem.
Cisco has reportedly acknowledged the issue, but not yet released updated firmware for the affected switches.