Incident history
Every incident with customer impact, with a short summary of what happened and what we changed.
Elevated purge latency Resolved
Purge requests took up to 40 seconds to propagate for 26 minutes. Root cause was a message queue consumer stuck on a malformed message. We added a dead-letter queue and a watchdog that restarts stalled consumers.
Dashboard login errors for some users Resolved
A session store configuration change caused intermittent login failures for 14 minutes. The change was rolled back. Edge traffic was not affected.
Increased 5xx rate in Singapore Resolved
A failing line card in one router caused packet loss for 9 minutes before traffic was drained to neighbouring locations. The card was replaced the same day.
Delayed analytics Resolved
Analytics in the dashboard lagged by up to 20 minutes for two hours while we rebalanced the ingestion cluster. No data was lost.
DNS resolution delays in South America Resolved
Queries from some Brazilian networks were routed to Miami for 35 minutes after an upstream provider withdrew routes. We added a second transit provider in São Paulo.
Image optimisation timeouts Resolved
Very large PNG uploads exhausted worker memory for 18 minutes. We now stream large images and cap per-request memory.