Start Date: 6/20/2025 2:00pm (EST) / 20th June 2025 18:00 (UTC)
Finish Date: 6/20/2025 8:40am (EST) / 22nd June 2025 12:40 (UTC)
Description:
Outage with SMTP traffic acceptance for onward processing
Impacted Services:
Impacted Customers:
Cause:
On Friday at 18:00 UTC a routine scheduled restart of the Email2sms service was started. This is designed to prevent memory usage spikes. This procedure completed at 18:21 UTC, however due to resource limitations on the host the startup process of Email2sms took much longer than usual causing it to fail inbox polling authentication with M365 Exchange. This prevented inbox polling from starting up and prevented the SMTP server from being able to accept connections. The next scheduled restart approximately 3 hours later correctly started up, completed authentication and resumed listening for traffic. The events of Sunday morning are the same cause and resolution.
The cause of the resource limitations on the host were due to another process that runs the SMSC server that accepts traffic via SMPP from customers. This process increased the CPU usage of the host to 100% for a prolonged period of time.
Detection:
Internal monitoring alerts were triggered when the Email2sms became unavailable.
Corrective Actions:
The automated scheduled restart resolved the issue on both occasions. Staff restarted the high CPU consuming SMSC process and available resources have normalised.
Preventative actions:
An improvement has been made to resource monitors of all hosts so that staff will be able to address high resource situations earlier. The Email2sms service will be isolated to another host to prevent other services causing disruption. We are increasing the priority of a replacement Email2sms service which will require a customer change.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2025 surveillance audit process.
External audit:
The security incident will be evaluated as part of the review cycle for the August 2025 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).