USSUP-228 Shared Shortcode Processing Outage
Incident Report for 2sms LLC
Postmortem

Start Date: 10/9/2022 8:01 PM (EST) / 10th October 2022 00:01 (UTC)

 

Finish Date: 10/9/2022 9:23 PM (EST) / 10th October 2022 01:23 (UTC)

 

Description:

 

Processing failure of MT traffic over shared short code 62346.

 

Impacted Services:

 

  1. MT traffic Shared Short Code 62346

Impacted Customers:

 

  1. All customers of shared short code 62346

Cause:

 

Our main connection for 62346 traffic stopped processing mobile terminating traffic towards our supplier’s primary API endpoint. Upon initial investigation there appeared to be errors regarding the validity of the endpoint’s TLS Certificate. This showed an expiry that matched the time that the failures began. We were not notified that this endpoint’s certificate would be expiring. We require a valid certificate to ensure the secure transmission of SMS requests.

 

Detection:

 

On call staff were automatically alerted to the traffic processing queuing on 62346 and message requests not reaching their intended destinations. Initial response actions were taken to investigate and resolve the issue. As the issue was determined to be external to 2SMS, the issue was immediately escalated to our supplier, who then investigated their systems.

 

 

Corrective Actions:

 

2sms attempted initial responses but when those failed, we routed traffic to an alternative API endpoint that was unaffected. Once the routing had taken affect this resolved the processing issue for new message requests. The backlog of delay messages was then sequenced and sent out over the alternative connection.

Once our supplier had confirmed that the endpoint certificate expiry issue had been resolved, we then tested and routed traffic back onto the original endpoint, traffic continued to process.

 

Preventative actions:

Our suppliers are reviewing why their certificate expired and did not get replaced automatically. We are reviewing our failover processes to ensure a quicker connection failover should a similar issue reoccur. We will be reviewing our connections with our suppliers to ensure we achieve maximum availability and redundancy.

 

 

Internal audit:

 

The security incident has been fed into the ISMS and will be part of the review cycle documents for the January 2023 surveillance audit process.

 

External audit:

 

The security incident will be reported to the external accredited ISO27001:2013 auditor Certification Europe and will be part of the review cycle for the January 2023 surveillance audit process.

 

GDPR:

 

This incident did not compromise PII (Personally Identifiable Information).

Posted Oct 25, 2022 - 14:30 UTC

Resolved
This incident has been resolved.
Posted Oct 25, 2022 - 01:34 UTC
Update
We are continuing to monitor for any further issues.
Posted Oct 25, 2022 - 01:20 UTC
Monitoring
Traffic has been diverted and is processing through an alternative supplier.
Posted Oct 25, 2022 - 01:19 UTC
Identified
Issue with the certificate validity of one of our suppliers API endpoints, we are working to redirect traffic. We will update shortly when this has been completed.
Posted Oct 25, 2022 - 01:01 UTC
Investigating
We are currently investigating a processing issue with our supplier on shared shortcode 62346, we will update with further information shortly.
Posted Oct 25, 2022 - 00:37 UTC
This incident affected: Shared Shortcodes.