However careful your design, however rigorous your validation, however vigilant your monitoring, your AI systems will sometimes fail. A model will make a series of harmful decisions before anyone notices; a drift will outpace your detection; an attack will succeed; an agent will do something it should not have. Maturity in regulated AI is not the belief that failure can be prevented entirely — it cannot — but the discipline of being ready for it: to detect it quickly, contain it, remediate the harm, discharge the obligations it triggers, and learn from it so it does not recur. This part is about that readiness.
What counts as an AI incident
The first step is recognising an incident as one. AI incidents are often less obvious than a server crashing, because a model can fail while the system around it runs perfectly. An AI incident is any event where the system causes or threatens harm through its decisions or behaviour, including:
- A model making materially wrong decisions at scale — a threshold error, a data problem, a drift that crossed a line.
- A discovered unfairness — the system has been treating a group inequitably, perhaps for some time.
- A security compromise — poisoning, evasion, injection, or an agent subverted by an attacker.
- An agent taking harmful or unauthorised actions.
- A privacy breach delivered through the model — leaked training data, an inversion attack, unlawful processing discovered.
- A use of the system outside its approved scope, intentional or accidental.
A recurring difficulty is that these failures can be silent and gradual rather than loud and sudden. A model quietly making biased decisions for months is an incident, even though nothing "broke". Part of incident readiness is cultivating the awareness — through monitoring, override signals, and complaints — to recognise a slow-motion failure as the incident it is, rather than dismissing it because no alarm sounded.
Preparing before it happens
The time to design your response is before you need it. An AI incident response capability, ideally integrated with your broader incident management rather than invented from scratch, should be in place ahead of any failure. It needs, at minimum:
- Defined roles. Who leads the response, who has authority to stop a system, who handles communications, who engages regulators — decided in advance, not improvised at 2 a.m.
- Detection routes. The monitoring signals, override patterns, complaint channels, and security alerts that surface incidents, feeding a clear path to the response team.
- The ability to stop. A reliable means to halt or roll back a misbehaving system quickly — the kill switch of Part 14 and the rollback of Part 16 — with the authority to use it residing somewhere reachable.
- An assessment method. A way to quickly determine an incident's scope: which decisions were affected, which people, over what period — which depends entirely on the audit trail and lineage built earlier.
An incident is not the moment to design your incident response. It is the moment to execute the one you already built.
Containing and remediating
When an incident occurs, the response moves through recognisable phases, though rarely as tidily as a list suggests.
- Contain. First, stop the harm from continuing — pause the system, roll back to a safe version, tighten human oversight, or disable the affected capability. Containment takes priority over diagnosis; you can investigate a stopped system, but every minute a harmful one runs adds to the harm.
- Assess scope. Determine what happened and how far it reached: which decisions, which people, what period. This is where the audit trail (Part 11) and data lineage (Part 7) prove their worth — with them, scoping is a query; without them, it is guesswork that delays everything downstream.
- Remediate the harm. Put right what was done wrong — revisit affected decisions, compensate or correct where people were harmed, fix the underlying cause. Remediation is about the people affected, not just the system; a technically fixed model does not undo the wrong decisions it already made.
- Recover. Return the system to safe operation — a corrected model, a fixed pipeline, restored controls — with heightened monitoring afterward, since a system that just failed warrants extra scrutiny.
Obligations that failure triggers
An incident is not only an operational event; it can trigger external obligations, and missing them compounds the original failure. Depending on the incident and the regimes you operate under, failure may require notifying a regulator within a defined window, informing affected individuals, reporting a data breach, or providing redress. These obligations often run on tight timelines, which is another reason to have decided in advance who assesses them and who acts. A firm that handles the technical failure well but misses a mandatory notification has turned one problem into two — and regulators tend to treat the failure to report as more serious than the underlying incident, because it speaks to whether the firm can be trusted to be candid.
Engaging regulators in a crisis
When an incident is serious enough to involve regulators, the posture that serves best is the one this course has advocated throughout: candour and evidence. Regulators respond far better to a firm that comes forward promptly, explains clearly what happened, shows it understood the cause, and demonstrates a credible remediation, than to one that minimises, delays, or is caught having concealed. The audit trail and documentation are what make candour possible — they let you give a precise, evidenced account rather than a vague and defensive one. An incident, handled with transparency and competence, can even build regulatory trust, because it demonstrates that your controls detected the problem and your processes handled it. Handled with evasion, the same incident corrodes trust that took years to build.
Learning from failure
The final and most often neglected phase is learning. Every incident is information about where your defences were weak, and a mature programme treats it as such through honest post-incident review: what happened, why, what allowed it, and what must change — in design, controls, monitoring, or process — so that this class of failure is less likely or less harmful next time. The crucial cultural condition is that this review be blameless enough to be honest. If incidents lead to punishment rather than learning, people hide them, and hidden incidents are the most dangerous of all, because they recur and accumulate in the dark. The organisations that get steadily safer are the ones that surface failures, examine them without flinching, and feed the lessons back into the framework. This is the loop that connects incident response back to the start of the lifecycle, and it is the engine by which an AI governance programme actually improves rather than merely persisting. The final part assembles all of these threads — from classification through incident learning — into the single operating model they are meant to form.
The incident that never set off an alarm
The hardest AI incidents to handle well are the ones that never trigger an alarm, because nothing "broke". A model quietly making biased decisions for months is a serious incident — arguably more serious than a loud outage, because it ran undetected and the harm accumulated. Yet it produces no crash, no error, no page in the middle of the night. It surfaces, if at all, through softer signals: a rising tide of complaints, a pattern in override data, a journalist's question, an analyst noticing an odd distribution of outcomes. Part of incident readiness is cultivating the organisational awareness to recognise a slow-motion failure as the incident it is, rather than dismissing it because no system alarm fired. This means treating the soft signals — complaints, overrides, fairness drift — as potential incident indicators, having a low enough threshold to investigate them, and resisting the comfortable assumption that an absence of technical alerts means an absence of harm. The incidents that damage institutions most are rarely the dramatic outages; they are the quiet, prolonged failures that everyone could have seen and no one was looking for.
Not every incident breaks something. The worst ones break no system at all — they just harm people, quietly, for a long time, while every dashboard stays green.
Why the audit trail is the hero of incident response
When an incident does occur, the single capability that most determines how well you respond is one built long before: the audit trail and data lineage from earlier in the course. The reason is that the first urgent question in almost any incident is scope — which decisions were affected, which people, over what period? With a complete, queryable audit trail and real lineage, answering this is a search: you identify the faulty model version or data source and trace precisely which decisions it touched and whom they affected. Containment, remediation, notification, and redress all depend on this scoping, and all of them stall until you have it. Without the audit trail, scoping becomes guesswork — you cannot say with confidence which decisions were affected, so you must treat everything potentially affected as suspect, which balloons the remediation, delays every downstream obligation, and leaves you explaining to a regulator why you cannot even establish the boundaries of your own incident. The investment in evidence, made calmly in advance, pays off most dramatically in the chaos of an incident, when the difference between a precise answer and a shrug is measured in harm and trust.
The obligations a failure triggers
An incident is not only an operational event; it can set legal clocks running, and missing those obligations compounds the original failure — often more seriously than the failure itself. Depending on the incident and the regimes you operate under, failure may require notifying a regulator within a defined and frequently short window, informing affected individuals, reporting a data breach, or providing redress. These timelines are unforgiving, which is why the assessment of what an incident triggers cannot be improvised during the crisis; it must be a prepared part of the response, with someone whose job is to determine, quickly, what obligations apply and to ensure they are met. Regulators tend to treat a failure to report as more serious than the underlying incident, because it speaks directly to whether the firm can be trusted to be candid when something goes wrong — and a firm that handles the technical failure competently but misses a mandatory notification has turned one problem into two and damaged the very trust that candour would have preserved.
The blameless review and the engine of improvement
The phase of incident response most often skipped is the one that makes the whole capability improve over time: the honest, blameless post-incident review. Every incident is information about where the defences were weak — what failed, why, what allowed it, and what must change in design, controls, monitoring, or process so that this class of failure becomes less likely or less harmful. Extracting that information requires a culture honest enough to look at failure without flinching, and that in turn requires the review to be blameless enough that people surface incidents rather than hide them. This is the crucial cultural condition: if incidents lead to punishment rather than learning, they go underground, and hidden incidents are the most dangerous of all, because they recur and accumulate unexamined. Organisations that get steadily safer are the ones that treat each failure as a lesson, feed it back into the framework, and so close the loop from incident response to the start of the lifecycle. This loop is the engine by which an AI governance programme actually matures rather than merely persisting — and it is why an incident, handled with transparency and learning, can leave an organisation stronger than it was before, while an incident met with concealment leaves it weaker and exposed.
In the next part: third-party and foundation-model risk — governing the increasingly common case where the model at the heart of your system was built by someone else.