Impaired Session End Performance due to Misconfigurations

Executive Summary

Between 23rd January 2023 15:30 UTC to 24th January 2023 09:00 UTC, one of the automatic functions that handle data processing at session-end was impaired. This was caused by a misconfiguration of the function, which led to webhook events not being triggered and significant delays to end-of-session data processing and payment capture.

Events Timeline

After the incident was closed, the Engineering and Customer Success teams continued to review all active sessions during this period for any adverse effects.

Closed Jan 24 10:30 UTC
Following additional monitoring by our Engineering teams, no further session calculation or webhook event delivery errors were observed and we have closed this incident.

Update and Monitoring Jan 24 09:00 UTC
The engineering team has updated the configuration.

Identified Jan 24 08:50 UTC
We declared an incident following confirmation that there was a misconfiguration on one of our systems. This was causing the webhooks and session calculations to be not appropriately handled.

Investigating Jan 24 08:30 UTC
Our Engineering team received a high-priority alert for webhooks not delivered and incorrect calculation of session energy reports.

Mitigation Actions

To prevent these types of issues from happening again in the future, we have taken or are taking the following actions:

• Improved our automated monitoring and testing systems to detect failures in automatic functions
• Significant change in how we manage and deploy configuration changes to automatic functions