Email2sms SMTP processing delays
Incident Report for 2sms LLC
Postmortem

Start Date: 9/19/2023 07:00 AM (EST) / 19th September 2023 11:00 (UTC)

 

Finish Date: 9/29/2023 06:00 AM (EST) / 29th September 2023 10:00 (UTC)

 

Description:

 

Custom domain SMTP email2sms messages were stuck pending within Microsoft 365

 

Impacted Services:

 

  1. Email2sms messaging initiated by custom domain SMTP requests

Impacted Customers:

 

  1. All customers of SMTP

Cause:

 

The Email2sms SMTP message processor failed to accept traffic from Microsoft 365 (M365) as a secure connection could not be established. This secure connection could not be established due to missing/corrupted cipher suites for to the Emai2lsms Java application. As part of essential upgrade works on our servers, services including Email2sms were moved onto more modern infrastructure. It is during this move the error commenced.

 

Detection:

 

Due to faults with both our internal monitoring and M365 alerting, this issue was not detected until raised by an affected customer.

 

 

Corrective Actions:

 

2sms investigated the outage report and verified the issue details. The issue was traced to cipher mismatch, checks we made against supported M365 ciphers which led to discovery of the failing ciphers available to Email2sms. The cipher suites were restored from an available backup. Processing resumed and monitoring continued while the immediate backlog was cleared.

 

Preventative actions:

2sms has already corrected the alerting within M365 to notify of pending messages stuck within M365. We have increased the priority of updating internal testing for the SMTP process. We are planning to re-architect how Email2sms functions to remove multiple dependencies, simplify the process to improve reliability and performance.

 

 

Internal audit:

 

The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2024 surveillance audit process.

 

External audit:

 

The security incident will be reported to the external accredited ISO27001:2013 auditor Certification Europe and will be part of the review cycle for the August 2024 surveillance audit process.

 

GDPR:

 

This incident did not compromise PII (Personally Identifiable Information).

Posted Oct 02, 2023 - 12:46 UTC

Resolved
This Email2sms SMTP issue is resolved.

The remaining backlog of emails will be processed as Microsoft365 resubmits them to 2sms over the next 24 hours.

We will provide a full incident report early next week.
Posted Sep 29, 2023 - 16:45 UTC
Update
We are continuing to monitor the backlog catch-up; we see a 19% completion of processing of the email backlog. Customers may experience delivery of these delayed messages out of sequence due to the nature of Microsoft365's deferred retries. New requests are handled immediately in parallel with this catch-up.
Posted Sep 29, 2023 - 12:59 UTC
Monitoring
We’ve deployed a fix that corrects processing from Microsoft 365 Exchange.

Users should now see SMTP processing restored and the backlog of emails queued with M365 now processing.

We’re monitoring to ensure continued restoration.
Posted Sep 29, 2023 - 10:00 UTC
Investigating
We are currently seeing processing issues for Email2sms SMTP traffic. We are investigating the cause and will update shortly. Traffic sent to our inboxes will be queued and released when the issues are resolved.
Posted Sep 29, 2023 - 05:00 UTC
This incident affected: SMTP.