Diagnosing Agent Failures: Retrieval, Generation or Tool
A decision procedure that classifies a bad agent answer as a retrieval fault, a generation fault or a tool fault — because each has a completely different fix.
"The agent gave a wrong answer" is not a bug report you can act on. Almost every wrong answer from a retrieval-augmented, tool-using agent falls into one of three buckets, and the fixes have nothing in common. Diagnosing the bucket before you touch anything is the single highest-value habit in agent operations, because the default reaction — editing the prompt — is the correct fix in only one of the three cases.
The decision procedure
- Was the information needed to answer correctly present in the model's context? Pull the retrieved chunks and tool observations from the trace and read them. If the correct fact is not there, stop: this is a retrieval fault. No prompt wording can make a model recall a passage it never saw.
- If it was present, did every tool return correct data? Compare each tool's inputs and outputs against what the underlying system holds. If a tool errored, timed out, or returned something wrong, it is a tool fault.
- If the right information was in context and the tools behaved, it is a generation fault. The model had what it needed and produced the wrong text anyway.
Retrieval faults
Symptoms: vague answers, "I don't have information about that" when you know the document exists, citations to the wrong document, confident invention. Causes and fixes live in the ingestion and query path, not the prompt — chunks split mid-sentence or mid-table, a chunking strategy that ignores document structure, an embedding model mismatched to the domain, numberOfResults too small, metadata filters excluding the right documents, semantic search where hybrid would have matched an exact product code, or an index that simply has not been synced since the source changed. Reproduce these in isolation: call Retrieve on the knowledge base with the same query, without any generation, and look at the scores.
Generation faults
Symptoms: the cited passage says one thing and the answer says another; the answer ignores an instruction; format is wrong; the model contradicts a tool result. Causes: conflicting or duplicated sources in the context, the key fact buried in the middle of a very long prompt, instructions that fight each other, temperature too high for an extraction task, or a model too small for the reasoning required. Fixes are prompt-side and model-side: reduce and order the context, restate the constraint near the end, drop the temperature, enforce structure with tool schemas rather than pleading in prose, or move up a model size.
Tool faults — and the distinction that matters
Split these in two. A broken tool is ordinary engineering: 5xx responses, Lambda timeouts, throttling, a missing IAM permission on the agent's execution role, a schema change upstream. Your existing observability catches it and your existing on-call fixes it. Tool misuse is different: the tool worked perfectly and the model called the wrong one, or passed a malformed argument, or ignored the result. That is not a model defect, it is a description defect. Action-group and tool schemas are prompt: the tool's description, its parameter descriptions and its examples are what the model reasons over. Vague descriptions, two tools with overlapping purposes, and error messages that say "invalid request" instead of "order_id must be eight digits" all produce misuse, and all are fixed by rewriting the schema.
Instrument so triage takes minutes
The procedure above only works if the evidence is already recorded: retrieved document ids and their scores, full tool inputs and outputs, the model's rationale from the orchestration trace, and the model and prompt version in force — all under one correlation id. If you have to reproduce the session to diagnose it, you have already lost, because the next run will take a different path.
The common trap
Most incidents reported as "hallucination" are retrieval faults. The model behaved reasonably given an empty or misleading context. Check the context before you touch the prompt, and log enough that you can.
Example
import boto3, json
agent_rt = boto3.client("bedrock-agent-runtime")
# Step 1 of triage: reproduce RETRIEVAL alone, with no model generation at all.
# If the right passage is not in here, the bug is not in your prompt.
resp = agent_rt.retrieve(
knowledgeBaseId="KB1234567",
retrievalQuery={"text": "what is the return window for clearance items"},
retrievalConfiguration={
"vectorSearchConfiguration": {
"numberOfResults": 10,
"overrideSearchType": "HYBRID", # compare against SEMANTIC
"filter": {"equals": {"key": "docType", "value": "policy"}},
}
},
)
for r in resp["retrievalResults"]:
print(round(r.get("score", 0), 4),
r["location"].get("s3Location", {}).get("uri"),
r["content"]["text"][:160].replace("\n", " "))
# Triage outcome:
# correct passage absent, or present with a low score ....... RETRIEVAL fault
# -> chunking, embeddings, filters, search type, sync state
# correct passage present and high, answer still wrong ...... GENERATION fault
# -> context length/order, temperature, model size, instructions
# passage fine but a tool errored or was called wrongly ..... TOOL fault
# -> broken tool: IAM, timeouts, upstream schema
# -> misused tool: rewrite the tool/parameter descriptionsWhen to use it
- A user reports invented shipping dates; the trace shows the knowledge base returned nothing above the score threshold, so the team fixes chunking rather than rewriting the prompt.
- An agent keeps calling get_invoice when it should call get_credit_note; the fix is disambiguating the two tool descriptions, not changing the model.
- Latency and error spikes are traced to a tool Lambda missing an IAM permission, producing repeated retries that also tripled loop length and cost.
Discussion