An ongoing Cloudflare outage has taken down many of its products, including the company’s dashboard and related application programming interfaces (APIs) customers use to manage and read service configurations.
The complete list of services whose functionality is wholly or partially impacted includes the Cloudflare dashboard, the Cloudflare API, Logpush, WARP / Zero Trust device posture, Stream API, Workers API, and the Alert Notification System.
“This issue is impacting all services that rely on our API infrastructure including Alerts, Dashboard functionality, Zero Trust, WARP, Cloudflared, Waiting Room, Gateway, Stream, Magic WAN, API Shield, Pages, Workers,” Cloudflare said.
“Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.”
Customers currently have issues when attempting to log into their accounts and are seeing ‘Code: 10000’ authentication errors and internal server errors when trying to access the Cloudflare dashboard.
Cloudflare says the service issues don’t affect the cached file delivery via the Cloudflare CDN or Cloudflare Edge security features.
Data center power outage behind dashboard and API issues
Two hours into the outage, the company revealed that the ongoing issues are due to power outages at multiple data centers.
“Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. We will keep providing regular updates until the issue is resolved, thank you for your patience as we work on mitigating the problem,” an incident report update said.
This is the second large outage that has hit Cloudflare since the start of the week, with the first one taking down multiple products, including Cloudflare Sites and Services (Access, CDN Cache Purge, Dashboard, Images, Pages, Turnstile, Waiting Room, WARP, Workers KV) on Monday, October 30.
As the company explained in a post-mortem published two days later, the Monday outage was caused by a misconfiguration in the tool used to deploy a new Workers KV build.
Workers KV is “used by both customers and Cloudflare teams alike to manage configuration data, routing lookups, static asset bundles, authentication tokens, and other data that needs low-latency access,” Cloudflare’s Matt Silverlock and Kris Evans said.
“During this incident, KV returned what it believed was a valid HTTP 401 (Unauthorized) status code instead of the requested key-value pair(s) due to a bug in a new deployment tool used by KV.”
This is a developing story…