Incident Report

‌

Start Date: 04/11/2022 9:04 PM (CT) / 12 April 2022 02:04 (UTC)

Finish Date: 04/11/2022 10:56 PM (CT) / 12 April 2022 03:56 (UTC)

‌

Description:

Processing outage via our Supplier SAP.

‌

Impacted Services:

1. MT over SAP such as shared short code 62346

‌

Impacted Customers:

1. All customers that send over SAP

‌

Cause:

When alerts came through reporting queuing for Mobile Terminating SAP traffic immediate responses were applied, when these actions failed to resolve the traffic flow, the issue was escalated to an incident. Upon investigation of the application logs it was found that the delivery receipt engine cache had invalid data stored. Our supplier had provided a model that was accepted and stored but not understood when retrieved. This invalid data caused the bind connections to fail.

 

Detection:

 Infrastructure staff were alerted by internal monitoring systems that queues were beginning to form. The duty managers were brought online to investigate the issue.

 

Scope of incident:

Customers sending mobile terminating requests over SAP serviced numbers such as 62346 would have been affected by this issue. Messages requests would have been accepted for sending but would have been queued. Queues were then rapidly cleared once the fault had been removed.

‌

Corrective Actions:

As we are in the middle of a migration, some short codes were able to be switched over to an alternative connection and delays for such codes were brief. Troubleshooting of the supplier connection was performed by infrastructure staff. The issue was escalated with our supplier and account managers, we worked together to uncover the cause. The invalid data was removed from the cache, services then restarted, the queue then rapidly clear to resume to normal processing.

‌

Preventative actions:

 We will be improving handling of invalid data in the DLR cache to prevent a recurrence ensuring that more data validation is performed.

‌

Internal audit:

The security incident has been fed into the ISMS and will be part of the review cycle documents for the November 2022 surveillance audit process.

 

External audit:

The security incident will be reported to the external accredited ISO27001:2013 auditor SGS and will be part of the review cycle for the November 2022 surveillance audit process.

 

GDPR:

This incident did not compromise PII (Personally Identifiable Information).