Why an Agent Needs Retrieval

A foundation model is a fixed set of weights. Everything it "knows" was compressed into those weights during training, and nothing has been added since. That gives you three permanent gaps that no amount of prompting will close.

The three gaps

• Knowledge cutoff. The model was trained on data up to a point in time. Ask about anything after that and it will either say it does not know, or — worse — produce a confident, plausible, wrong answer.
• Private data. Your staff handbook, your ticket history, your product catalogue and your contracts were never in the training set and never will be. The model cannot know what your refund policy says.
• Freshness. Even data the model saw once goes stale. Prices change, policies get revised, people leave. An answer that was right last quarter is a liability now.

Retrieval closes all three the same way: at request time you search a corpus you control, put the matching passages into the prompt, and instruct the model to answer from those passages. This is Retrieval-Augmented Generation (RAG). The model contributes language and reasoning; the corpus contributes facts.

What retrieval is not

Three distinctions the exam likes to test.

• Retrieval is not memory. A knowledge base is a shared, mostly read-only corpus that every user queries. Memory is per-conversation or per-user state — what this person said three turns ago, what they prefer. "Remember that I fly economy" belongs in agent memory or session state, not in a knowledge base. Writing per-user facts into a shared index leaks them to everyone.
• Retrieval is not an action. A knowledge base answers "what does the document say?" An action group or tool answers "what is the live value?" and "do this thing". A customer's current balance is an API call, not a chunk of text. If the answer changes minute by minute, it is a tool; if it lives in a document, it is a knowledge base.
• Retrieval is not fine-tuning. Fine-tuning changes how the model behaves — tone, format, domain vocabulary. It is a poor and expensive way to teach facts, because facts change and the model cannot cite them.

How it fits the agent loop

In a hand-rolled RAG pipeline, your code always retrieves and then always generates. In an agentic pipeline, retrieval is one of several things the model can choose to do: it sees a description of the knowledge base alongside its tools, decides whether the question needs it, may reformulate the query, may search more than once, and may combine a retrieved policy with a live API result before answering. That choice is the difference between a fixed workflow and an agent.

The failure mode

The dangerous case is not retrieval failing loudly — it is retrieval returning nothing relevant and the model answering anyway from its training data. The answer looks authoritative and carries no warning. Guard against it in two places: instruct the model explicitly to say it cannot find an answer when the retrieved passages do not contain one, and check citations on the way out. An answer with no supporting reference is an answer the corpus did not produce.

{
  "toolConfig": {
    "tools": [
      {
        "toolSpec": {
          "name": "search_hr_policy",
          "description": "Search the published HR handbook for policy wording: leave, expenses, notice periods, benefits. Returns document passages, not live data. Use for questions about what the policy says.",
          "inputSchema": {
            "json": {
              "type": "object",
              "properties": {
                "query": { "type": "string", "description": "Natural-language question to search for" }
              },
              "required": ["query"]
            }
          }
        }
      },
      {
        "toolSpec": {
          "name": "get_leave_balance",
          "description": "Return the live remaining leave balance in days for one employee from the HR system. Use for 'how many days do I have left' - never answer this from the handbook.",
          "inputSchema": {
            "json": {
              "type": "object",
              "properties": {
                "employee_id": { "type": "string" }
              },
              "required": ["employee_id"]
            }
          }
        }
      }
    ]
  }
}