European Data Center - Service Unavailability
Incident Report for IDnow GmbH
Postmortem

On Thursday, 15. August, at 09:26 CEST, a major outage hit our Data
center in Germany.

The root cause was a failing core component within the infrastructure
of our Data center provider.

Process Description

Within the scope of a standard change of a customer environment, a
serious malfunction occurred at a central core router component.

Solution

As part of the incident processing, the malfunction was resolved in the
long term.
During the implementation of the standard change, the resulting network
loop was directly recognized and eliminated. The convergence time of
the redundancy protocols was the reason why the whole network was not
available again until 09:54 am.
Such an impact by a network loop was normally not to be expected, since
the components and protocols used are hardened against network loops.
We immediately opened a support ticket with the manufacturer, as the
affected network infrastructure unexpectedly failed.
In addition, our Change Advisory Board will analyze and, if necessary,
introduce an extension of the audit framework for future network
changes.
We apologize for this failure.

Best Regards,
Your IDnow DevOps Team

Posted Aug 22, 2019 - 14:09 CEST

Resolved
We monitored the system closely and the issue did not appear again. We are closing this incident therefore.

Incident start: 2019-08-15 09:31 UTC+2
Incident end: 2019-08-15 09:53 UTC+2
Posted Aug 15, 2019 - 12:25 CEST
Monitoring
The system is available again. We are monitoring it closely.

Incident start: 2019-08-15 09:31 UTC+2
Incident end: 2019-08-15 09:53 UTC+2
Posted Aug 15, 2019 - 10:32 CEST
Investigating
We are investigating a service unavailability of our European data center.
Posted Aug 15, 2019 - 09:57 CEST
This incident affected: IDnow Solutions (Video-Ident, eSigning QES, eSigning AES, API, AutoIdent).