Many companies put their AI employees through a probation period, only to decide their fate on gut feel at the end: "seems okay," "guess it helped." That isn't a sign-off; it's a coin toss. And the cost is now being exposed by the data. MIT's 2025 study "The GenAI Divide," which analyzed 300 enterprise deployments, found that 95% of generative-AI projects delivered no measurable P&L impact, and only 5% actually crossed the pilot-to-production cliff. Gartner likewise forecasts that more than 40% of agentic-AI projects will be canceled by the end of 2027 due to unclear value, runaway costs, or inadequate risk controls.
Most of these failures aren't because the AI wasn't good enough, but because no one defined "what counts as passing" up front. A real probation sign-off should define a set of quantifiable gates at the start, check them item by item at the end, and reach a clear verdict: go (confirm and go live), conditional (fix within a deadline), or no-go (take offline and rebuild). This article focuses on the moment of confirmation—the acceptance thresholds and go/no-go logic—not on how to design the metrics or manage the organizational transition.
Break the sign-off into four independent gates; failing any one means no unconditional confirmation. Thresholds should be set against real benchmarks rather than intuition. The widely used τ-bench shows that even top models can clear 80% success on the easier retail tasks but often land at just 54–58% on the more tightly ruled airline-service tasks—proof that the "pass line" must be adjusted for task difficulty and risk. The starting points below are for general low-to-medium-risk work; high-risk scenarios (payments, compliance, external-facing decisions) should be tightened (all figures are reference benchmarks, not guarantees):
| Gate | How to measure | Suggested pass threshold | No-go signal |
|---|---|---|---|
| Task completion rate | Successfully closed ÷ total assigned | ≥ 90% | Below 75% or steadily declining |
| Quality pass rate | Sampled human-review pass ratio | ≥ 95% | One or more critical errors |
| Human-intervention rate | Human takeovers ÷ total tasks | ≤ 15% | Above 30% |
| Performance stability | Consecutive weeks meeting targets without regression | ≥ 3 weeks | Large week-to-week swings |
These thresholds can be anchored to public data: in customer-service AI, for example, Intercom's Fin reports an average resolution rate of about 76% across roughly 12,000 customers—meaning about a quarter of conversations still require human takeover—a useful order-of-magnitude reference for the "human-intervention rate." The key is "thresholds before probation"—these numbers must be written down and agreed with the business unit before probation begins, or the end-of-term call will be swayed by favoritism or sunk cost.
Of the four gates, stability is the easiest to overlook. Many teams see one impressive demo and rush to confirm, forgetting that AI output is probabilistic—one success doesn't mean success every time—and the math here is brutal. An agent with a 75% single-run success rate has only about a 42% chance of succeeding three times in a row (0.75³). τ-bench's pass^k measurements go further: an agent with roughly 60% single-run success drops below 25% when it has to pass all eight consecutive runs. Chaining amplifies the problem: a three-step flow at 70% each yields only about 34% end to end (0.7³). In other words, an AI that "looks about 80% capable during probation" will see its errors magnified into nasty surprises once it faces hundreds or thousands of real tasks a day. Stability therefore can't be judged on a single sign-off; it must be measured over "several consecutive weeks without regression," using pass^k (the share of runs where all k in a row pass) rather than single-run success, to fend off pass-by-luck illusions.
The four gates are not summed and averaged; each is guarded individually. If any one falls in the no-go range, there's no direct confirmation—but that doesn't mean instant elimination either. Instead, you enter a three-way decision:
During sign-off, separate the "hard-decidable" items from those "requiring judgment": task completion and human-intervention rates can be computed precisely by rules and auto-compared with deterministic metrics; reply quality and tone, which need judgment, are best handled in 2026's mainstream way—with LLM-as-judge, a stronger model scoring against your defined criteria, backed by human spot checks. Guarding each gate rather than averaging avoids a dangerous mistake: letting a flashy completion rate mask failing quality, when quality lapses are usually what truly cost the business.
Hitting the numbers is only a necessary condition; before confirmation you must also verify the AI employee can be safely entrusted. The list below should be jointly signed off by the business owner and the technical owner—missing any one means no green light:
The root of most confirmation disputes isn't that the AI fell short, but that no one spelled out "what counts as passing" before rollout. Nerdtechnic's AI adoption advisory service works with you at project kickoff to set the thresholds for all four quantitative gates, the sign-off checklist, and the go/no-go decision logic, and helps establish post-launch monitoring and rollback mechanisms—so that when probation ends, the data does the talking instead of a show of hands. Rather than discovering six months in that the AI can't hold up, write the acceptance standard into the contract and the process on day one.
Want to bring these practices into your own company?
Free consultation on LINE
AI Research
Memory makes AI understand you better the more you use it, but ungoverned memory leaves your company more exposed. Using actual provisions from Taiwan's PDPA, the GDPR, and the EU AI Act, this article lays out a practical AI memory governance framework: tiering, retention and forgetting, PII compliance, and audit.
AI Research
Singapore and the UAE have ordered banks to retire SMS OTP on a deadline while India and Taiwan accelerate the transition, even as SIM swap fraud losses climb. This article uses each market's timeline and hard data on FIDO/passkeys to map out this authentication paradigm shift and a pragmatic upgrade path.
AI Research
Most companies treat producing an AI usage policy as governance done, yet 81% of organizations remain in the first two maturity stages. This article offers a five-level maturity model and a seven-element scorecard, aligned to NIST AI RMF and ISO/IEC 42001, to help you self-assess, see the gaps, and find the upgrade path.
Need Help?
Click here to contact us!
