Start Date: 2/25/2025 7:03am (EST) / 25th February 2025 12:03 (UTC)
Finish Date: 2/25/2025 8:36am (EST) / 25th February 2025 13:36 (UTC)
Description:
Outage with SMTP traffic acceptance for onward processing
Impacted Services:
SMTP Email2sms
Impacted Customers:
Customers of SMTP Email2sms
Cause:
During routine monthly maintenance of the platform infrastructure, it is normal for the Email2sms SMTP service to be shifted between available hosts whilst patching is occurring. An infrastructure configuration error caused the Email2sms SMTP service to jump between services unintendedly. Due to authentication restrictions this service may only be moved infrequently. As a result, when Email2sms moved between host nodes, there was an extended period for which the service was unavailable whilst it attempt to re-authenticate and load balancers attempted to update their target configurations. The length of the impact will have varied based on our customer’s email server’s ability to update their DNS configurations and their SMTP retry intervals.
Detection:
Upon customer notification, staff begun work immediately to investigate and perform mitigations.
Corrective Actions:
Whilst investigating and attempting to mitigate by moving services to nodes with greater performance, the email2sms application was able to re-authenticate and resumed accepting customer traffic for onward processing.
Preventative actions:
As we are aware of increased customer volumes via the SMTP service, we are rapidly working on a replacement for the email2sms solution that will provide improvements in availability, performance and configuration eliminating a reoccurrence of this issue in the future.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2025 surveillance audit process.
External audit:
The security incident will be evaluated as part of the review cycle for the August 2025 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).