The AI Incident Response Runbook: Severity Tiers, Response Steps, and Postmortems

What AI Systems Lack Most After Launch Is a Playbook for When Things Break

AI incidents are on the rise. The AI Incident Database logged 362 incidents in 2025, up from 233 in 2024 — a jump of more than 50% in a single year, with roughly 58% tied to generative AI. Over the same period, Gartner forecasts that more than 40% of agentic AI projects will be canceled by the end of 2027, with inadequate risk controls cited as a leading cause. Most teams pour their energy into shipping the model and never prepare a response plan for hallucinations, API timeouts, runaway costs, or data leaks. The cost of downtime is not abstract: a 2026 Splunk/Cisco study estimates the average cost of downtime for large enterprises at roughly USD 15,000 per minute, and the Uptime Institute's annual analysis found that more than half of operators' most recent major outages cost over USD 100,000, with about one in five costing over one million dollars. Without a runbook, every incident falls to whoever happens to be the most senior person online — leaving recovery slow, accountability murky, and the same mistakes free to recur.

This article offers an AI incident response runbook you can put into practice directly: first grade the incident, then define the response flow and roles, and finally use postmortems to turn every incident into systematic improvement.

Step 1: Grade AI Incidents by Severity (SEV)

The point of grading is to give "who to wake up and how fast to respond" an objective basis. We recommend following SRE convention with four tiers, anchoring each to a real AI incident:

The biggest benefit of grading is removing judgment from individual heroics: a new engineer can read the table and decide whether to escalate, instead of waiting for the most senior person to arrive. Grading also determines whether a postmortem is required — SEV1/SEV2 always get one; SEV3 and below as needed. Write this standard into the runbook in advance and get team-wide buy-in, so you are not arguing "which severity is this" at the moment of peak pressure.

Don't Overlook AI-Specific Incident Sources

Traditional runbooks usually only cover "the service went down," but the sources of AI incidents are expanding. These new risk categories all belong in your scenario list:

• Hallucination and false promises: the model confidently returns wrong information. The Air Canada case (Moffatt v. Air Canada) established that enterprises are legally responsible for what their chatbots say.
• Prompt injection and data leaks: EchoLeak was the first demonstrated zero-click data-exfiltration vulnerability in a production enterprise AI. The attack hid inside an ordinary email and bypassed Microsoft's own injection classifier; Microsoft patched it in June 2025, with no known exploitation in the wild.
• Tool-chain supply-chain risk: in 2026 OX Security disclosed a systemic flaw in the MCP ecosystem estimated to affect around 200,000 instances; a poisoned tool description can contaminate the entire context without ever being called.
• Cost blowouts: Uber burned through its annual AI coding-tool budget in four months, and one engineer running about a hundred concurrent AI agents racked up roughly USD 1.3 million in API spend in 30 days — with a single "fast mode" setting accounting for about 70% of the bill. Runaway cost usually stems from defaults, not the model itself.

Response Flow and Roles

The worst outcome on-scene is a crowd all acting at once with no one coordinating. Establish a single Incident Commander role: the commander does not necessarily fix things personally, but owns decisions, task assignment, and external communication, so accountability is clear by process. The standard flow:

• Detection and alerting: trigger automatically on monitoring signals (error rate, latency, token cost, hallucination-detection score) rather than waiting for customer complaints.
• Grading and mobilization: assign severity per the SEV table, page people via on-call rotation, and open a war room immediately for SEV1/SEV2.
• Stop the bleeding first: switch to safe mode (human review, roll back, pause autonomous execution, set token caps) to contain impact before calmly chasing root cause.
• Communication sync: update affected parties at a fixed cadence, with all internal and external messaging issued by the commander.
• Recovery and confirmation: keep watching after service is restored, and only declare closure once metrics stabilize.

Use Postmortems So Incidents Don't Recur

Recovery only stops the bleeding; the real value is in the postmortem. Google's SRE team, over many years, has found that the single most effective tool for preventing recurrence is an open, blameless postmortem. Adopt a blameless principle — focus on process and system gaps rather than blaming individuals — so engineers are willing to reconstruct the truth honestly. A useful postmortem contains at least:

• Incident timeline: key moments and decisions from detection, grading, and mitigation through recovery.
• Impact scope: affected users, duration, and data and cost losses.
• Root-cause analysis: split into technical (model, data, tool-chain integration) and process (monitoring blind spots, missing alerts) sides.
• Action items: each with an owner and deadline, tracked to closure so it doesn't become a paper exercise.

Note that AI incident root causes often span both technical and process ends: a model-side hallucination or injection is frequently amplified into a disaster by monitoring blind spots and missing alert rules. The output of a postmortem should therefore be not just a report, but new monitoring metrics, alert thresholds, and scenario playbooks fed back into the next version of the runbook.

Drill the Runbook Into Team Muscle Memory

A runbook that is written but never rehearsed is worthless. Schedule a game day each quarter to simulate SEV1/SEV2 scenarios and verify whether alerts fire, whether on-call can be reached, and whether mitigation steps actually work. After the drill, feed the sticking points back into the runbook so the document evolves alongside the system. It also helps to write common incident procedures as directly executable checklists and attach the relevant runbook link to alert messages, so on-call staff aren't flipping through docs and firefighting from memory at 2 a.m. When incident response shifts from "improvised firefighting" to "following the playbook," recovery speed no longer hinges on whether one particular person is online — and only then does an AI system truly have post-launch operational resilience.

How Nerdtechnic Helps You Build AI Incident Response Capability

Nerdtechnic has long helped Taiwanese enterprises adopt and operate AI systems. We don't just ship AI agents to production; we build monitoring and alerting, SEV grading, on-call rotation, and postmortem mechanisms together with you, giving AI services auditable, recoverable operational resilience. Whether you are evaluating AI adoption, need custom system development, or want to build automated workflows with OpenClaw, we provide end-to-end consulting from launch to operations — helping you turn "what do we do when it breaks" into standard team practice rather than starting from zero every time.

References

• Stanford HAI, "AI Index Report 2026 — Responsible AI," 2026. Source
• Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027," 2025. Source
• Cisco/Splunk, "The $600 Billion Wake-up Call: New Splunk Research Reveals Downtime is a Systemic Business Crisis," 2026. Source
• Uptime Institute, "Annual Outage Analysis 2026," 2026. Source
• Aim Labs researchers, "EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System" (arXiv:2509.10540), 2025. Source
• McCarthy Tétrault, "Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot," 2024. Source
• OX Security, "The Mother of All AI Supply Chains: Critical, Systemic Vulnerability at the Core of the MCP," 2026. Source
• TechCrunch, "Uber caps employee AI spending after blowing through budget in four months," 2026. Source
• The Next Web, "Peter Steinberger's 100 AI agents racked up $1.3 million in OpenAI tokens in 30 days building OpenClaw," 2026. Source
• Google SRE, "Site Reliability Engineering — Ch. 15: Postmortem Culture: Learning from Failure," 2016. Source