AdHoc payments capture significantly delayed

Executive Summary

Between 20th February 2023 12:30 UTC and 23rd February 2023 07:00 UTC, one of the automatic functions that handle payment captures was being impeded. This led to increasing delays in payment captures.
Our investigation concluded that this was due to a steady increase in the number of charging sessions, resulting in insufficient memory allocation. Once this was addressed, functionality was restored to normal.

Events Timeline

After the incident was closed, the Engineering and Customer Success teams continued to review all active sessions during this period for any adverse effects.

Closed Feb 23 10:00 UTC
Following additional monitoring by our Engineering teams, no further sessions were affected and payment captures continued as normal. All affected sessions were fixed and the pending transactions were captured.

Update and Monitoring Jan 24 09:00 UTC
The engineering team has updated the configuration, adding sufficient resources to the component as well as extra layers of monitors and alerts.

Identified Feb 23 02:30 UTC
We declared an incident following confirmation that there was a processing function that was being throttled, resulting in payments not being captured.

Investigating Feb 23 01:00 UTC
Our Engineering team received a high-priority email communication indicating that some AdHoc payments were not being captured in real time.