Incident Report
Start Date: 03/08/2022 01:00 AM (CT) / 08 March 2022 07:00 (UTC)
Finish Date: 03/09/2022 05:00 AM (CT) / 09 March 2022 11:00 (UTC)
Description:
Processing outage of Email2sms services.
Impacted Services:
Impacted Customers:
Cause:
When alerts came through for failing Email2sms services the service logs were investigated. The initial cause was found to be IMAP authentication failures. Credentials for all inboxes were checked and confirmed authenticating correctly using the office365 portal. Port availability was investigated but showing that the portal on the office365 email server was available. It was determined via the Microsoft Remote Connectivity Analyzer that the protocol for IMAPS had been denied by the server for the 2sms tenant. In contact with Microsoft Support it was found that IMAP is due to be blocked by default by the end of 2022. As part of a phased change Microsoft were blocking tenants that do not use the IMAP protocol and our tenant was blocked in error of their detection methods without direct warning.
Detection:
Infrastructure staff were alerted by internal monitoring systems that there were multiple Email2sms service checks failing concurrently. All Infrastructure staff were brought into an incident meeting to investigate the issues.
Scope of incident:
Customer’s inbound email requests are received into dedicated email inboxes on Office365. The Email2sms service processes traffic by retrieving these messages from the inboxes using the IMAPS protocol. Due to the issue, these connections could not be established and therefore the messages containing send request information could not be retrieved and processed. Although messages could not be processed, they remained queued within the inboxes until services resumed, upon which the message queues were processed and cleared.
Corrective Actions:
Upon discover of the IMAPS protocol denial, a support ticket was raised with Office365 and escalated with our account manager at Microsoft. 2sms worked with Microsoft to provide them with the information they required to investigate and resolve. During the waiting period it was stressed that this issue was severely impactful to our customers and although we request updates none were provided until resolution. 2sms was preparing an emergency deployment for a change of protocol which would have been deployed if it had been ready before the original protocol issue was fixed.
Preventative actions:
Given that the IMAP authentication will be phased out, work will be performed to move inbox access to more modern protocols. 2sms will be considering alternative backup suppliers for email inbox handling to prevent a reoccurring event from being impactful.