Experiencing issues with server
Incident Report for IDnow GmbH
Postmortem

All bigger changes, configurations and rollouts of releases on the IDnow system are done out of active working hours of the IDnow call centres to ensure there is no impact on system uptime.

However, smaller changes are also done during working hours for certain cases based on customer requests.

On 13th of May the rollout of such a small change resulted in a downtime of our system. It turned out that the script change was faulty but the error was not detected prior to rollout by our automated checks and the internal review process we have defined.

Issues: All servers were impacted.

Timeline: 13th of May, 11:50 - 12:35 CET

‌Customer Impact:

End users could not connect to IDnow agents during the period of outage as servers were becoming extremely slow and sometime non-responsive. End user however was waiting in the queue during this time as soon as system was up, they could connect to IDnow agent.

‌Actions taken:

In order to recover from the outage, we reverted the change immediately. After this the system was back online and able to handle all the incoming traffic.

Mitigation:

We are working on improving the process of the rollout of such changes by adding even more automated checks, additional testing and implementing more restrictions on what can be changed in the system especially during work hours.

‌We sincerely apologise for the trouble this has caused.

Your IDnow Team

Posted May 15, 2020 - 10:44 CEST

Resolved
This incident has been resolved.
Posted May 13, 2020 - 15:00 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 13, 2020 - 12:39 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted May 13, 2020 - 11:58 CEST
Investigating
We are currently investigating this issue.
Posted May 13, 2020 - 11:51 CEST
This incident affected: Europe - IDnow (Video-Ident, eSigning QES, eSigning AES, Photo-Ident, API, AutoIdent).