Event alerts are delayed by 2 hours

Incident Report for Splunk Observability Cloud US1

Resolved

A fix has been implemented and full functionality has been restored. Past missing events have been filled in and will be visible as expected
Posted 9 months ago. Jul 09, 2024 - 21:07 PDT

Update

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails.
Expected resolution by 21:30 PT
Posted 9 months ago. Jul 09, 2024 - 21:01 PDT

Update

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails.
Expected resolution by 21:00 PT.
Posted 9 months ago. Jul 09, 2024 - 20:47 PDT

Update

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails.
Expected resolution by 21:30 PT.
Posted 9 months ago. Jul 09, 2024 - 20:34 PDT

Update

A fix has been implemented and we are monitoring the results.
Posted 9 months ago. Jul 09, 2024 - 20:19 PDT

Update

A fix has been implemented and we are monitoring the results.
Posted 9 months ago. Jul 09, 2024 - 20:03 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted 9 months ago. Jul 09, 2024 - 19:48 PDT

Identified

Event alerts are delayed by 2 hours with some small number of events lost. UI and ingest are unaffected. The problem has been identified and we are working on a resolution. Updates will be provided every 15 minutes.
Posted 9 months ago. Jul 09, 2024 - 19:36 PDT
This incident affected: Alerting.