As of 9:05pm PT Azure APIs have been healthy and cloud sync lag is down to normal levels.
Posted Sep 18, 2021 - 21:37 PDT
Update
We are continuing to monitor Azure APIs for sustained recovery.
Posted Sep 18, 2021 - 20:51 PDT
Identified
We are continuing to see intermittent connection problems with Azure leading to further increases in lag. The cloud sync system has been able to ensure no data loss other than some loss between 5:30pm PT and 6:00pm PT.
Posted Sep 18, 2021 - 19:46 PDT
Monitoring
Azure cloud sync lag has normalized starting approx 7:30pm PT. We are continuing to monitor for further increases in lag.
Posted Sep 18, 2021 - 19:40 PDT
Update
Lag has gone down to normal momentarily at 7pm PT and is increasing again. Data between 5:30pm until 6:00pm PT may have been lost, data after approximately 6pm has been recovered. Data starting 7pm PT is experiencing lag again.
Posted Sep 18, 2021 - 19:18 PDT
Identified
We’ve identified failures in Azure cloud metric sync due to timeout failures in Azure cloud APIs, we are continuing to investigate the issue.
Detectors and charts relying on cloud sync data from Azure are impacted and may be missing data.
Posted Sep 18, 2021 - 18:51 PDT
This incident affected: Datapoint Ingest and Alerting.