P0: There is currently delays/errors matching properties against the land register

Symptom
During the incident, the following symptoms were observed:
- A backlog of requests in the queue for matching real properties with the land register.
- Certain services failed to start due to an inability to establish a connection with the DB2 database on the mainframe.
- A significant number of constraint errors and transaction rollbacks were returned from the DB2 database.

Impact

Users from Financial Institutions were unable to create loans for real properties that had not been recently updated with a new land registry.

Root cause

The issue originated from a new patch applied to one of the two DB2 database instances. This patch was introduced during a scheduled restart of the database instances overnight. The patch caused:

• Failures in establishing database connections,
• Constraint errors, and
• Transaction rollbacks on the affected database instance.

The land register matcher service, which relies on DB2 to process incoming requests, was directly impacted. With the errors caused by the patch, the service could not keep up with incoming requests, leading to a congested queue of unmatched records.

Resolution

The operations team collaborated with the infrastructure provider to investigate and resolve the issue. The following steps were taken:

1. The investigation revealed differences in the state between the two DB2 database instances.
2. A review of the patch notes identified a specific change as the root cause of the issue.
3. As it was not feasible to roll back the patch during the incident, all network traffic was redirected to the healthy DB2 database instance.
4. This action successfully resolved the errors, allowing the queue of matches to be processed without further interruptions.

Actions

1. Fallback Plan: A fallback plan has been created and agreed upon, detailing the steps for rolling back a failing patch if future incidents occur. This ensures operations can continue even if only one database instance is functional.
2. Root Cause Investigation: The infrastructure provider will conduct an in-depth investigation into the patch to identify and resolve the underlying issue.
3. Problem Management Process: A problem management process has been initiated to track progress and ensure a permanent resolution is implemented.
   

Reference: IM516415 & JN INC 5515549

• Reference IDs: IM516415 & JN INC 5515549
• Event Start Time: 05:00
• Event End Time: 16:36
• Recorded Downtime: 11 hours, 36 minutes

We sincerely apologize for the inconvenience caused by this incident and appreciate your understanding as we work to improve system reliability and resilience. Should you have further questions or concerns, please do not hesitate to reach out.