ES1: Smoke tests are failing the environment

Incident Report for NRP

Postmortem

Symptom

Failing smoketest and errors on failing database transactions

Impact
Unstable environment and degradation of service quality across the whole NRP service landscape

Root cause

The underlying Oracle database ran out of resources due to a high amount of sessions.
This increase in load was introduced by four new servers that has been added to a staging environment and naturally increased the amount of sessions open at a given time.

Resolution

The allocated dedicated memory for the database was raised to account for the extra load.

Actions
No further actions will be taken
It was expected to that the database could to handle this increase in sessions.
The database was recently migrated to a new Exadata setup. Due to unforeseen behaviour of resource allocation on the new setup it, was discovered that the allocation of memory was not dedicated but shared amongst several databases.
This has been rectified with the raise of dedicated memory as the resolution.

Reference: INC0014346

Event started at: 08:00
Event ended at: 09:37

Posted Nov 19, 2025 - 13:46 CET

Resolved

The issue has now been resolved.
Posted Nov 19, 2025 - 09:45 CET

Update

We are still investigating the issue.

Next status: 10:45
Posted Nov 19, 2025 - 09:28 CET

Investigating

Symptom: Smoke tests are failing the environment

Reference: INC0014346

Technical impact: Investigating

Business impact: Investigating

Next status: 09:30
Posted Nov 19, 2025 - 08:45 CET
This incident affected: ES1 Environment (ES1 - environment smoke test status).