Start Date: 9/19/2023 07:00 AM (EST) / 19th September 2023 11:00 (UTC)
Finish Date: 9/29/2023 06:00 AM (EST) / 29th September 2023 10:00 (UTC)
Description:
Custom domain SMTP email2sms messages were stuck pending within Microsoft 365
Impacted Services:
Impacted Customers:
Cause:
The Email2sms SMTP message processor failed to accept traffic from Microsoft 365 (M365) as a secure connection could not be established. This secure connection could not be established due to missing/corrupted cipher suites for to the Emai2lsms Java application. As part of essential upgrade works on our servers, services including Email2sms were moved onto more modern infrastructure. It is during this move the error commenced.
Detection:
Due to faults with both our internal monitoring and M365 alerting, this issue was not detected until raised by an affected customer.
Corrective Actions:
2sms investigated the outage report and verified the issue details. The issue was traced to cipher mismatch, checks we made against supported M365 ciphers which led to discovery of the failing ciphers available to Email2sms. The cipher suites were restored from an available backup. Processing resumed and monitoring continued while the immediate backlog was cleared.
Preventative actions:
2sms has already corrected the alerting within M365 to notify of pending messages stuck within M365. We have increased the priority of updating internal testing for the SMTP process. We are planning to re-architect how Email2sms functions to remove multiple dependencies, simplify the process to improve reliability and performance.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2024 surveillance audit process.
External audit:
The security incident will be reported to the external accredited ISO27001:2013 auditor Certification Europe and will be part of the review cycle for the August 2024 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).