Reliability is designed in, not bolted on after an outage
AI systems are more fragile than traditional software because they place their least stable link—external LLMs and third-party APIs—directly on the critical path. This is not alarmism: even a leading provider like OpenAI does not always hold to the availability it promises, and June 2025 saw a multi-hour outage that affected users worldwide. Rate-limit errors are also one of the most common failure sources in production, and as AI workflows multiply calls to external APIs, these errors only become more frequent. Laid out this way, dependency failure is not a rare anomaly but a monthly reality; what actually determines reliability is whether the design phase decided in advance how the system holds up the moment something fails.
Convert downtime into money before deciding how much to invest
The most common mistake in reliability investment is chasing 100% uptime without limit. The pragmatic move is to price downtime first: Gartner's long-cited baseline is roughly 5,600 USD per minute, about 336,000 USD per hour; ITIC's 2024 survey found that over 90% of mid-to-large enterprises lose more than 300,000 USD per hour, with around 40% losing between 1 and 5 million USD. With that number you can work backward to the availability you need—the key insight being that each extra nine tends to cost a step-change, not a linear increase.
| Availability | Allowed downtime per year | Suitable for |
|---|
| 99% | ~3.65 days | Internal tools, non-real-time batch jobs |
| 99.9% | ~8.8 hours | General external services, internal core systems |
| 99.99% | ~52 minutes | Transactions, real-time support and other critical flows |
Multiply hourly downtime cost by your target availability and you get a number you can take to a budget sign-off. A transaction flow that loses hundreds of thousands per hour justifies extra architecture when moving from 99% to 99.9% (roughly three fewer days of outage a year); forcing 99.99% onto an internal reporting tool is just waste. Without that number, reliability investment is either too little or too much.
Three core patterns: circuit breaker, fallback, retry
Designing reliability in relies not on pricier servers but on three battle-tested failure-handling patterns. They were popularized by Netflix's Hystrix library; Hystrix is now in maintenance mode and new projects mostly adopt Resilience4j, but the patterns themselves remain the industry standard.
| Pattern | What it solves | The cost to watch |
|---|
| Retry | Brief jitter, occasional timeouts | Needs exponential backoff plus deduplication, or it amplifies traffic and double-charges |
| Circuit breaker | A dependency failing continuously | The feature pauses while open; must probe periodically to recover |
| Fallback | The primary path is entirely unavailable | Backup quality is lower; the acceptable simplified version must be defined in advance |
Retry handles transient failures where trying again usually works; the key is exponential backoff with full jitter—roughly sleep = random(0, min(cap, base × 2^attempt)), with the cap at 32–60 seconds and attempts kept to 3–5—so requests don't all retry at once and cause a thundering herd that crushes the dependency. Among 4xx errors, usually only 429 (rate limit) is worth retrying, and if the response carries Retry-After, honor it. Payment or write operations also need an idempotency key (the standard at Stripe and PayPal) so retries don't take effect twice. The circuit breaker is the opposite: when a dependency fails repeatedly, fail fast, pause calls, and probe periodically. Fallback is the last line of defense: when the primary path truly goes down, fall back to a cached older result, a simpler model, or a human.
Chain the three patterns into one line of defense per request
These are not either/or; they stack in sequence on the same request path. For an LLM call, a sensible 2026 production order is:
- Timeout: cap the wait per call so a stuck request doesn't hold resources hostage.
- Retry: on timeout or transient errors, retry once or twice with exponential backoff to absorb jitter.
- Circuit breaker: when failures exceed a threshold in a short window, open the breaker and route requests straight to the next layer.
- Fallback: once the breaker opens or retries are exhausted, switch to a backup model or cached result so users always get a response.
The value of the chain is that any failure one layer can't stop is caught by the next. It matters most for multi-agent systems: if each step succeeds 85% of the time, a ten-step end-to-end workflow succeeds only about 19.7%, because errors compound along the chain. So beyond the defense chain, pair it with keeping steps as few as possible and gating every step, or cascading failure will slip through.
The cost trap in fallback chains
Fallback sounds like pure upside, but it hides a counterintuitive trap: if the backup model is more expensive than the primary, routing all traffic to it during an outage pushes cost up exactly when the business is already hurting. That is why 2026 multi-model routing practice insists the fallback chain be ordered by cost-adjusted quality—for example primary model → cross-provider peer → cheaper same-provider model → self-hosted open model—so worst-case spend is bounded. Fallback must also be explicit and testable: silently switching to an unsuitable model is often worse than returning an outright failure.
Rollout order: start with the most painful path
You don't need to wrap every feature in full protection at once. The pragmatic order is: find the one or two critical paths with the highest downtime cost and the heaviest reliance on external services, add timeouts and retries there, then add circuit breakers and fallbacks for the shakiest dependencies, and only then extend to the rest. Every layer you add should be exercised with fault injection—manually kill a dependency and watch whether the system degrades gracefully rather than seizing up—instead of discovering on the day of a real outage that the protection never worked. Reliability is not a one-off project but a long-term investment tuned as dependencies and traffic change.
Nerdtechnic: growing reliability into the system
The hard part of reliability engineering is not knowing the terms circuit breaker, fallback, and retry, but judging which path is worth protecting, setting each layer's thresholds and exit behavior, and keeping cost under control while degrading—which takes an understanding of both business loss and system implementation. Nerdtechnic's AI system consulting and custom development services first help you quantify the downtime cost of your critical flows, then map that to the right reliability design, so the system can survive dependency failures from day one instead of scrambling to patch things after an outage has already damaged the business.
References
- Tom's Guide, "ChatGPT / OpenAI down outage — live updates, June 10, 2025," 2025. Source
- Atlassian, "Calculating the cost of downtime" (citing Gartner's downtime cost baseline). Source
- ITIC, "ITIC 2024 Hourly Cost of Downtime Report," 2024. Source
- Netflix, "Hystrix" GitHub project notice (maintenance-mode statement). Source
- Stripe, "Designing robust and predictable APIs with idempotency." Source
Technical Sharing
