Start Date: 10/9/2022 8:01 PM (EST) / 10th October 2022 00:01 (UTC)
Finish Date: 10/9/2022 9:23 PM (EST) / 10th October 2022 01:23 (UTC)
Description:
Processing failure of MT traffic over shared short code 62346.
Impacted Services:
Impacted Customers:
Cause:
Our main connection for 62346 traffic stopped processing mobile terminating traffic towards our supplier’s primary API endpoint. Upon initial investigation there appeared to be errors regarding the validity of the endpoint’s TLS Certificate. This showed an expiry that matched the time that the failures began. We were not notified that this endpoint’s certificate would be expiring. We require a valid certificate to ensure the secure transmission of SMS requests.
Detection:
On call staff were automatically alerted to the traffic processing queuing on 62346 and message requests not reaching their intended destinations. Initial response actions were taken to investigate and resolve the issue. As the issue was determined to be external to 2SMS, the issue was immediately escalated to our supplier, who then investigated their systems.
Corrective Actions:
2sms attempted initial responses but when those failed, we routed traffic to an alternative API endpoint that was unaffected. Once the routing had taken affect this resolved the processing issue for new message requests. The backlog of delay messages was then sequenced and sent out over the alternative connection.
Once our supplier had confirmed that the endpoint certificate expiry issue had been resolved, we then tested and routed traffic back onto the original endpoint, traffic continued to process.
Preventative actions:
Our suppliers are reviewing why their certificate expired and did not get replaced automatically. We are reviewing our failover processes to ensure a quicker connection failover should a similar issue reoccur. We will be reviewing our connections with our suppliers to ensure we achieve maximum availability and redundancy.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the January 2023 surveillance audit process.
External audit:
The security incident will be reported to the external accredited ISO27001:2013 auditor Certification Europe and will be part of the review cycle for the January 2023 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).