Start Date: 8/5/2024 1:27pm (EST) / 5th August 2024 17:27 (UTC)
Finish Date: 8/6/2024 3:13am (EST) / 6th August 2024 07:13 (UTC)
Description:
Customer facing SNPP endpoint became unavailable.
Impacted Services:
Impacted Customers:
Cause:
At 17:27 UTC both redundant SNPP endpoints ceased passing traffic up to the internal API for authentication and message processing. At this time it remains unclear as to the original caused of the issue however rouge customer input is suspected. Endpoints should have automatically restarted upon experiencing a fault, this particular failure was not handled correctly, and an auto restart was not triggered.
Detection:
Internal system alerts were triggered however due to unforeseen circumstances on call staff were unable to act upon the alerts due to transmission channel outages.
Corrective Actions:
Day staff immediately addressed the issue and began working on the root cause analysis.
Preventative actions:
We are reviewing out of hours systems coverage with our service provider to prevent a repeat occurrence of a connectivity failure. We are working to implement better automated restarters to cover all instances of issues within the SNPP endpoint and its dependent internal gateway. We are adding external monitoring restarters to take affect if the primary restarters fail. We are implementing additional logging to capture any future occurrences that may occur.
Internal audit:
The security incident has been fed into the ISMS and will be part of the review cycle documents for the August 2024 surveillance audit process.
External audit:
The security incident will be evaluated as part of the review cycle for the August 2024 surveillance audit process.
GDPR:
This incident did not compromise PII (Personally Identifiable Information).