OCPP Connection Errors 13-Dec-2023

Executive Summary

Between 13th December 2023 08:30 UTC and 13th December 2023 09:30 UTC, eDRV's OCPP service was impaired in its ability to accept new connections from chargestations.

Events Timeline

After the incident was closed, the Engineering and Customer Success teams continued to review all active sessions during this period for any adverse effects.

Closed Dec 13 09:30 UTC
No further handshake failures were observed, and the incident has been closed.

Update & Monitoring Dec 13 09:15 UTC
Following additional monitoring by our Engineering teams, we were still getting a small amount of handshake failures. After observation, we rolled back the deployment to the previous major release and started monitoring the OCPP service.

Update & Monitoring Dec 13 09:00 UTC
Our Engineering team deployed a fix and continued monitoring connections and message processing.

Investigating Dec 13 08:40 UTC
Errors were identified in the logs, indicating handshake failures in WebSocket connections.

Identified Dec 13 08:40 UTC
Upon completion of release, connection errors were reported by the service monitoring systems. Our Engineering team began investigating the reported errors immediately.

Minor Release Dec 13 08:30 UTC
A configuration update for duplicate WebSocket connections on the OCPP service was deployed at 08:30 UTC.

Mitigation Actions

To prevent these types of issues from happening again in the future, we have taken or are taking the following actions:

• Expanding canary deployments to include minor releases and configuration updates