Handling Ambiguous Instructions: Prompt Design for Intent Recognition and Clarification

Why Ambiguous Instructions Make AI Systems Fail Silently

"Tidy up last week's data for me" leaves a system with several undefined parameters: which data, in what format, output to where. The tricky part is not that the model fails to "understand"—it is that the model detects the ambiguity yet still tends to guess a single interpretation and run with it. Cornell's 2026 study Knowing but Not Showing quantified this: ask a model directly "is this sentence ambiguous?" and it usually judges correctly, but drop it into an actual QA setting and it overwhelmingly answers directly instead of asking back—and adding retrieved context makes that tendency even stronger (arXiv 2605.25284). Another 2026 coding-agent study, Ask or Assume, put a price on it: on the same task set, the resolve rate was 70.8% when instructions were complete but fell to 54.8% when instructions were deliberately underspecified and no interaction was allowed—a roughly sixteen-point gap coming entirely from a system treating assumptions as facts. The failure from ambiguous instructions is therefore not a model-capability problem but a missing layer that makes uncertainty explicit and clarifies proactively when confidence is low. We break this into four deployable parts.

Intent Classification and Slot Filling

The first step is mapping free text to a finite intent set: predefine the enterprise's task types—"query report," "schedule meeting," "compile document"—each intent mapped to a set of required slots. On receiving an instruction the system does two things at once: intent classification (outputting the intent and a confidence score) and missing-slot detection (comparing against the required list and listing fields not yet provided). For "schedule meeting," the required slots are participants, time, and duration; if the user only says "set up a meeting for me," the system should detect all three slots empty rather than fill in defaults.

Filling slots accurately is itself hard; you cannot assume the model gets it right in one pass. On the speech-noise benchmark SpokenWOZ, even the strongest dialogue-state-tracking model reaches only 25.65% joint goal accuracy—the whole-turn state must be entirely correct to count (arXiv 2305.13040). So slot detection and validation must be gated by code, not left to the model to decide whether to skip a required field; researchers also use conformal prediction to produce a "credible intent set," executing directly when the set is a singleton and triggering clarification only when it holds multiple candidates (arXiv 2403.18973).

Confidence Thresholds and an Escalation Decision Table

Whether to clarify or hand off to a human is decided jointly by the confidence score and the slot state. The key insight from practice is that no single percentage works universally; the threshold must be set to the business's error tolerance, and it is best to use "two thresholds"—can the model give a grounded answer (can), and given this risk level should it (should). One medical voice agent sets a default threshold of 0.75; below it, the call is handed to a human with the transcript attached and the inferred intent labeled as a "suggestion" rather than a "decision."

A single threshold is not enough: mature systems add multi-signal escalation—several turns of low confidence, or a conversation stuck in a repetition loop, hands off to a human—and log each classification's confidence and action as the basis for later threshold tuning and few-shot examples.

Designing the Clarification Loop

The clarification loop is a finite state machine: classify intent → check missing slots → if any are missing, generate one focused question → refill slots → recheck. Two principles decide whether it works well. First, "ask only the one or two most critical missing slots at a time"—dumping five questions at once makes users give up; ICLR 2025's Modeling Future Conversation Turns shows that a well-posed clarifying question not only disambiguates but also raises final accuracy even on seemingly unambiguous queries (arXiv 2410.13788). Second, "do not over-ask": the loop must cap its rounds (say, three), beyond which it falls back to defaults or hands off. The 2026 clarification benchmark ClarEval writes this into its metrics, using Average Turns to Clarify (ATC) and Efficiency-Adjusted Recall (EAR) to penalize over-questioning and quantify the accuracy-versus-friction trade-off (arXiv 2603.00187). To continue the example: the user says "arrange tomorrow's review meeting," intent confidence 0.9 but participants and duration missing, so the system asks only those two, creates the meeting only after they are filled, and never guesses the attendee list.

A Deployable Clarification Prompt and Architecture Pattern

Merely writing "ask first when unsure" in the system prompt has limited effect, because the model's default is to answer directly. A more reliable approach splits "deciding whether to ask" and "executing the task" into two roles. The two-agent design in Ask or Assume is the example: an Intent Agent only analyzes the conversation and reports the intent, what is missing, and where the assumptions are, emitting structured JSON with a needs_clarification boolean flag; the Main Agent is constrained by that flag and may only ask back—not act—when it is true. This configuration lifts the resolve rate of OpenHands + Claude Sonnet 4.5 from 61.2% for a single agent to 69.4%, nearly matching the 70.8% seen with fully specified instructions. Three points for deployment:

• Structured output: require the model to return three fields—intent, confidence, missing_slots—rather than natural language, so the backend decides to execute or ask back with program logic instead of betting on another LLM judgment.
• Generate a question only when slots are missing: produce a clarify_question only when missing_slots is non-empty, and restrict it to the missing fields.
• Demonstrate correct behavior with few-shot: include "ambiguous input → ask only for missing slots, do not execute" examples to counter the model's default to answer directly; EMNLP 2025's Ask-when-Needed framework confirms that explicitly prompting models to "ask only when needed" raises both question quality and API-call accuracy (arXiv 2409.00557).

Rollout Order: Start with the Flow Where One Mistake Costs the Most

You do not need to add a full clarification mechanism to every feature at once. The pragmatic order is to first find the one or two flows where "one misunderstanding costs the most"—irreversible actions such as payment, sending mail, or deleting/modifying data take priority—add intent classification, slot validation, and confidence thresholds there, then extend to the rest. Measure with real numbers: mis-execution rate, first-time completion rate after clarification, and how many clarifications were necessary rather than intrusive. Gartner predicted in March 2025 that by 2029 agentic AI will autonomously resolve 80% of common customer-service issues and drive roughly a 30% cut in operating costs; but the premise of that ceiling is precisely a system that asks when it should and hands back to people when it is beyond its confidence—rather than treating a vague sentence as a clear instruction to execute.

How Nerdtechnic Can Help

Getting clarification right is hard not because of the terms—intent classification, slot filling, the clarification loop—but because of judging which flow is worth the investment, how to set each confidence threshold and exit behavior, and how to block mis-execution without over-bothering the user; this requires understanding business tolerance and system implementation at the same time. Nerdtechnic's AI adoption consulting and custom development services first help you inventory your task types, define your intent and slot structures, and design confidence thresholds and a two-agent clarification architecture, then keep tuning thresholds and few-shot examples after launch. If your AI system often produces useless output from misread instructions, talk to us about making "ask when you should" a stable, built-in behavior.

References

• Jinyan Su, Claire Cardie (Cornell University), "Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions" (arXiv:2605.25284), 2026. Source
• Nicholas Edwards, Sebastian Schuster, "Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents" (arXiv:2603.26233), 2026. Source
• S. Si et al., "SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents" (arXiv:2305.13040), 2023. Source
• F. den Hengst et al., "Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition" (arXiv:2403.18973, NAACL Findings 2024). Source
• Lucid AI Labs, "Why confidence thresholds beat intent classifiers for medical voice agents." Source
• M.J.Q. Zhang et al., "Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions" (arXiv:2410.13788, ICLR 2025). Source
• J. Li et al., "ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions" (arXiv:2603.00187), 2026. Source
• W. Wang et al., "Learning to Ask: When LLM Agents Meet Unclear Instruction" (arXiv:2409.00557, EMNLP 2025). Source
• CX Today, "Gartner Predicts that Agentic AI Will Solve 80 Percent of Customer Service Issues by 2029," 2025. Source