Start Date: 1/3/2025 8:31am (EST) / 3rd January 2025 13:31 (UTC)
Finish Date: 1/3/2025 9:04am (EST) / 3rd January 2025 14:04 (UTC)
Description:
Onward processing delay of international mobile terminating traffic.
Impacted Services:
Impacted Customers:
Cause:
Message senders responsible for international traffic lost connection to their suppliers. These sending services automatically attempted to reconnect however due to resource limitations on the server the connections could not be re-established.
Detection:
Internal alerting notified staff of the issue who begun work immediately to investigate and resolve.
Corrective Actions:
Staff were unable to reach the service directly and so were forced to initiate a reboot of the affected server to clear the high resource consumption issue. This led to services being restored. The backlog of queued traffic quickly cleared and normal traffic processing resumed. No further instances of the issue presented themselves.
Preventative actions:
We are working on better accessibility to reach servers that have high resource usage. We have items on our roadmap to swap out the technologies that caused the high resource usage and intend to separate more services to prevent such events in the future.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2025 surveillance audit process.
External audit:
The security incident will be evaluated as part of the review cycle for the August 2025 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).