Resolved -
As of 9:05pm PT Azure APIs have been healthy and cloud sync lag is down to normal levels.
Sep 18, 21:37 PDT
Update -
We are continuing to monitor Azure APIs for sustained recovery.
Sep 18, 20:51 PDT
Identified -
We are continuing to see intermittent connection problems with Azure leading to further increases in lag. The cloud sync system has been able to ensure no data loss other than some loss between 5:30pm PT and 6:00pm PT.
Sep 18, 19:46 PDT
Monitoring -
Azure cloud sync lag has normalized starting approx 7:30pm PT. We are continuing to monitor for further increases in lag.
Sep 18, 19:40 PDT
Update -
Lag has gone down to normal momentarily at 7pm PT and is increasing again. Data between 5:30pm until 6:00pm PT may have been lost, data after approximately 6pm has been recovered. Data starting 7pm PT is experiencing lag again.
Sep 18, 19:18 PDT
Identified -
We’ve identified failures in Azure cloud metric sync due to timeout failures in Azure cloud APIs, we are continuing to investigate the issue.
Detectors and charts relying on cloud sync data from Azure are impacted and may be missing data.
Sep 18, 18:51 PDT