Charts not Loading and detectors not starting for Some Customers

Incident Report for Splunk Observability Cloud US1

Resolved

Following the monitoring period, no further issues have been observed and the incident is now considered as Resolved.
Posted 10 months ago. Jun 05, 2024 - 05:02 PDT

Monitoring

A fix has been implemented at 04:14 PT, the services are now recovered and we are monitoring the results.
Posted 10 months ago. Jun 05, 2024 - 04:38 PDT

Update

There are also issues since 3:21a PT in the Monitoring MetricSets. A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be corrupted and represent only a portion of the traffic and activity they are meant to measure on our customer’s services, endpoints, traces and workflows.
Posted 10 months ago. Jun 05, 2024 - 04:11 PDT

Investigating

As of 3:18a PT this morning, charts for a subset of customers may not be loading. Some detectors may not be able to start. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
Posted 10 months ago. Jun 05, 2024 - 03:44 PDT
This incident affected: Splunk Observability Cloud Web Interface, Alerting and Splunk APM (Splunk APM Monitoring MetricSets).