Standard RAG Should Be a Tool, Not the Architecture

Query ambiguity is what changed my mind

I started from a very simple question: what happens when the user’s query is ambiguous?

This is not a theoretical corner case. AmbigQA showed that more than half of the questions in NQ-open were ambiguous, with ambiguity coming from things such as entity references and events.[1] Later work on clarification-based QA treated ambiguity detection and clarification-question generation as explicit steps before answering.[2]

That observation made me look again at the architecture we usually call “standard RAG.”

A typical pipeline looks roughly like this:

query → embedding → vector search → top-k chunks → LLM → answer

The weakness is not simply chunk size. The deeper assumption is that the original query is already the right retrieval query.

That assumption breaks surprisingly often.

A user may be vague. The question may contain multiple possible interpretations. It may require two facts that cannot be found with the same search phrase. The first retrieved document may reveal a term, entity, date, exception, or relationship that should change the next search. In those cases, retrieval is not one operation. It is a process.

This is exactly why I think agentic retrieval is a more useful architecture.

flowchart TD
    A["User query"] --> B["Interpret / detect ambiguity"]
    B --> C["Decompose or rewrite"]
    C --> D["Retrieve"]
    D --> E["Inspect evidence"]
    E --> F{"Missing / contradictory / insufficient?"}
    F -->|Yes| G["Reformulate query"]
    G --> D
    F -->|No| H["Select / rerank / verify"]
    H --> I["Answer"]

The key difference is simple: standard RAG treats the query as an input; agentic RAG treats retrieval as a trajectory.

The research already points in this direction

IRCoT is one of the clearest examples. The paper argues that one-step retrieve-and-read is insufficient for multi-step questions because what should be retrieved next depends on what has already been derived from previous evidence.[3] On HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC, IRCoT reported improvements of up to 21 points in retrieval and up to 15 points in downstream QA by interleaving retrieval with reasoning.[3]

FLARE reached a similar conclusion from a different direction. Instead of retrieving once at the beginning, it actively decides when more information is needed during generation and issues new retrieval queries based on predicted upcoming content.[4] Across four long-form knowledge-intensive generation tasks, FLARE achieved superior or competitive results against its baselines.[4]

Adaptive-RAG makes the architectural point even more directly: not every query deserves the same retrieval strategy.[5] It dynamically chooses among no retrieval, single-step retrieval, and iterative retrieval according to question complexity, and the authors report a better balance of accuracy and efficiency than relevant baselines.[5]

SELF-RAG attacks another weakness of fixed RAG: blindly retrieving a fixed number of passages whether they are necessary or relevant can reduce generation quality.[6] Its solution is to let the model retrieve and critique evidence more selectively.[6]

So the direction across these systems is consistent: retrieval should adapt to the query and to what the system learns while solving it.

A 2026 comparison makes the trade-off very concrete

NVIDIA’s NeMo Retriever team published an agentic retrieval pipeline on the ViDoRe v3 Pipeline benchmark.[7] Their agent can rewrite complex queries, call retrieval multiple times, answer sub-questions, and finally select the top documents.[7]

The numbers are interesting:

NVIDIA reports the agentic pipeline at 69.2 nDCG@10, versus 64.4 for the simple dense pipeline — a relative gain of about 7.4%.[7] But the search latency rises from 3.61 seconds to 136 seconds because the agent is allowed to rewrite and retrieve repeatedly.[7]

This is exactly where many people stop and say: “Agentic retrieval is too slow.”

I think that conclusion is too shallow.

Latency is real, but latency is also a product-design problem

First, streaming does not magically turn 136 seconds of computation into 3.6 seconds. Total compute latency is still total compute latency.

But there are two different user experiences:

136 seconds of silence

and

Understanding the request…
Searching the primary corpus…
Found 8 candidate sources…
Checking conflicting evidence…
Rewriting the query…
Searching again…
Verifying the final evidence set…
Generating the answer…

They are not equivalent.

Streaming partial output, tool calls, retrieval progress, source discovery, and verification updates turns an opaque wait into an observable process. It changes perceived latency, even when it does not change the physical runtime.

More importantly, an agentic system does not need to run the expensive path for every question. That is the lesson I take from Adaptive-RAG.[5]

A good production architecture can route like this:

simple / explicit query
    → one-shot retrieval

ambiguous query
    → clarify or generate multiple interpretations

multi-hop query
    → iterative retrieval

weak evidence
    → rewrite + retrieve again

conflicting evidence
    → retrieve + compare + verify

In other words, the answer to agentic latency is not “go back to standard RAG.” The answer is budgeted agentic retrieval.

Agentic systems can fail too — and that matters

There is useful negative evidence as well.

A reproducible open-source HotpotQA benchmark compared multiple RAG patterns while holding the dataset, embeddings, chunking, vector store, and 8B Llama model fixed.[8] In that setup, Basic RAG achieved Recall@5 of 0.609, while its ReAct-style Agentic RAG achieved 0.498 and roughly doubled median latency.[8]

I do not read this as evidence that agentic retrieval is useless. I read it as evidence that agency amplifies the quality of the reasoning and control loop.

If a small model makes poor search decisions, rewrites the query badly, or stops at the wrong time, iterative retrieval compounds those mistakes. A bad agent can absolutely be worse than a good retriever.

That is an engineering constraint, not an argument for freezing retrieval into one shot.

A 2026 IEEE paper on Agentic Self-RAG shows the other side. Its architecture separates query analysis, retrieval criticism, and answer verification into specialized control steps.[9] It reports 54.3% exact match vs. 45.2% for Vanilla RAG on HotpotQA, and 68.5% vs. 59.2% accuracy on StrategyQA.[9]

The useful question is therefore not:

“Does agentic RAG always beat standard RAG?”

Of course not.

The useful question is:

“Which architecture can adapt when the first retrieval attempt is not enough?”

On that question, standard one-shot RAG has a structural limitation.

Cursor is a good production signal

Coding agents make this point very visible.

Cursor documents that its codebase indexing pipeline splits changed files into syntactic chunks and converts those chunks into embeddings for semantic search.[10] Cursor also describes an agentic search setup in which agents can combine semantic search with exact search tools such as grep to locate the right code.[11]

In Cursor’s own evaluation, adding semantic search improved agent response accuracy by an average of 12.5%, with gains varying by model.[11] Cursor calls semantic search one of the major drivers of agent performance.[10]

This is not a controlled scientific comparison proving that every agentic RAG design beats every standard RAG design. It is production evidence of something more practical: high-performing agents benefit from being able to decide how to search, what to search for, and which retrieval tool to use.

That matters to me.

If adaptive retrieval were fundamentally useless, it would be difficult to explain why serious coding-agent products invest so much engineering effort in indexing, semantic retrieval, exact search, query choice, and repeated tool use.

My conclusion

I do not think standard RAG should disappear.

I think it should be demoted from architecture to tool.

Dense top-k retrieval is useful. BM25 is useful. Hybrid retrieval is useful. Rerankers are useful. Fixed chunk search is useful.

But none of those should decide in advance that every user question deserves exactly one retrieval pass.

The stronger architecture is an agent that can decide:

• whether retrieval is needed;
• whether the query is ambiguous;
• whether the query should be rewritten;
• whether it should be decomposed;
• which retriever or search tool to use;
• whether the evidence is sufficient;
• whether another retrieval pass is justified;
• and when to stop.

For simple questions, that agent may choose the exact same fast retrieval path we call standard RAG.

For harder questions, it can do something standard RAG fundamentally cannot do: adapt.

That is why, after starting with query ambiguity, I ended up much more convinced by agentic retrieval.

The future architecture I would bet on is not “agentic RAG instead of retrieval.” It is agentic orchestration with retrieval as a first-class tool — and standard RAG as the fast path inside it.

References

1. Min, S. et al. (2020). AmbigQA: Answering Ambiguous Open-domain Questions. EMNLP. Source
2. Lee, D. et al. (2023). Asking Clarification Questions to Handle Ambiguity in Open-Domain QA. Findings of EMNLP. Source
3. Trivedi, H. et al. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL. Source
4. Jiang, Z. et al. (2023). Active Retrieval Augmented Generation. EMNLP. Source
5. Jeong, S. et al. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. NAACL. Source
6. Asai, A. et al. (2024). SELF-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR. Source
7. NVIDIA NeMo Retriever Team (2026). NeMo Retriever’s Agentic Retrieval Pipeline — ViDoRe V3 Pipeline benchmark. Source
8. Pradhan, S. (2026). RAG Architecture Benchmark — HotpotQA. Open-source reproducible benchmark; not peer reviewed. Source
9. Agentic Self-RAG: Multi-Agent Reasoning for Self-Correcting Retrieval-Augmented Generation (2026). IEEE ICIT. DOI: 10.1109/ICIT68548.2026.11577708. Source
10. Cursor (2026). Securely indexing large codebases. Source
11. Cursor (2025). Improving agent with semantic search. Source