← All posts

Scaling Enterprise RAG: More Documents Are Not the Hard Part

10 November 2026 10 min read Experience #RAG#Enterprise AI#Retrieval#Inference#MLOps

A RAG system can work extremely well for one team and still fail when the rest of the company arrives.

During a pilot, we usually have a limited collection, a handful of enthusiastic users and enough time to investigate every incorrect answer. The index is rebuilt manually. A failed document is uploaded again. If the model becomes slow, somebody restarts the service. It feels as if the hard part is already behind us.

Then a second department joins. The document count grows, permissions start to differ, files change without notice and several users ask long questions at the same time. Suddenly, a system that looked like an AI application starts behaving like a distributed production system.

This is the main idea of this article: scaling RAG is not the act of uploading more documents. It is the work of managing more users, versions, permissions, failure modes and operational responsibility without losing answer quality.

Where the pilot actually starts to fail

Consider an internal policy assistant. During the pilot, it answers questions using one team's approved documents. A second department joins with different access rules, a new source connector and frequently changing files. A former team member loses access to a document while an old indexed copy still exists. Meanwhile, interactive requests compete with an overnight batch summarisation job.

These are not problems a larger context window or a higher top_k can solve. They need system boundaries: permissions enforced before retrieval, predictable deletion and reindexing, recoverable ingestion, admission control and fair inference scheduling.

This is an illustrative scenario, not a measured benchmark. Its point is that the first scaling failures often occur between services. The useful question is not just how many documents the vector database can store, but what happens when a source changes, a permission is revoked or many users ask at once.

Document count is the wrong unit of scale

A one-page memo and a 500-page annual report both count as one document. They do not create the same work. The useful units are pages, extracted elements, chunks, changed versions, embedding operations, authorised retrieval candidates and generated tokens.

At minimum, we measure four separate scale planes:

PlaneWhat growsTypical bottleneckMetric to watch
DataPages, versions, tables and imagesParsing, OCR and embeddingsQueue age, pages/minute, failed-page ratio
RetrievalChunks, filters and candidatesShard fan-out, recall and rerankingRecall@k, p95 search latency, candidates/query
InferenceConcurrent prompts and output tokensQueueing, memory and model throughputQueue wait, TTFT, tokens/second, p95 end-to-end latency
ControlUsers, roles, tenants and policy changesAuthorisation drift and auditabilityACL propagation time, denied-query tests, audit coverage

If these planes are collapsed into a single “number of documents” metric, the capacity plan will be wrong before production begins.

1. Scale ingestion as a queue, not as a request

A production ingestion pipeline should not keep an HTTP request open while a 600-page PDF is parsed, OCR is applied and embeddings are generated. We accept the document, create a durable job and process it asynchronously with bounded workers.

Each job needs an idempotency key derived from values such as source_id, source_version, content_hash, parser_version, chunking_version and embedding_version. If a worker crashes after indexing half of a document, a retry must resume or safely overwrite the same logical version rather than duplicating every chunk.

We keep the original file immutable and write a processing manifest beside it. That manifest records which pages succeeded, which parser handled them, whether OCR or a multimodal fallback was used, how many chunks were produced and which index version received them. Failed jobs go to a dead-letter queue instead of silently disappearing.

For search indexing, batch writes usually provide better throughput than one request per chunk, but bigger batches are not automatically better. Elastic recommends benchmarking bulk sizes, using multiple workers only until the cluster is saturated and applying randomized exponential backoff when the cluster returns 429 TOO_MANY_REQUESTS.Elastic indexing guidance

A simplified job contract looks like this:

job_id: ingest-01J...
tenant_id: acme
source_id: sharepoint:contract-1842
source_version: "2026-09-03T10:42:00Z"
content_hash: sha256:...
parser_version: doc-service-4.2
chunking_version: legal-v3
embedding_version: multilingual-e5-v2
target_index_version: knowledge-2026-09-v4

Versioned physical indexes behind stable aliases make re-embedding and rollback manageable. We build the new index in the background, run retrieval and permission tests, then move the read alias. A new embedding model should be a controlled migration, not an in-place surprise.

2. Do not scale retrieval by increasing top-k

As a corpus grows, increasing top_k and filling a larger context window feels like the quickest solution. It also increases vector-search work, reranking cost, prompt tokens and the amount of irrelevant context the model must distinguish.

Long context does not remove the need for retrieval discipline. The peer-reviewed “Lost in the Middle” study found that language models can use long contexts unevenly and often perform worse when relevant information appears in the middle.Lost in the Middle

Our retrieval path therefore narrows the search before it expands the context:

  1. Resolve the authenticated tenant, user groups and allowed logical indexes.
  2. Route the question to the relevant domain or document collection.
  3. Apply tenant, ACL, time, document-type and lifecycle filters before candidate generation.
  4. Combine lexical and vector candidates when the domain requires exact terms, identifiers or acronyms.
  5. Rerank a bounded candidate set.
  6. Send only the smallest sufficient, cited context to the model.

This is also where hierarchical retrieval becomes valuable. A small document-level or section-level index can first locate the relevant region; paragraph-level retrieval can then search only inside that region. We do not ask every shard and every chunk to compete for every question.

Elasticsearch makes an important operational point: search runs on a single CPU thread per shard, so a very large number of shards can exhaust the search thread pool, while oversized shards make recovery and search slower. Its current 10–50 GB and fewer-than-200-million-documents guidance is a starting range, not a universal rule; production data, filters, hardware and query concurrency must be benchmarked.Elastic shard sizing

3. Treat inference capacity separately from search capacity

A retrieval cluster can return results in 80 milliseconds while users still wait 25 seconds for an answer. Search scale and generation scale are different problems.

For inference, we separate interactive traffic from batch summarisation and offline enrichment. Each class has its own queue, concurrency limit, timeout and priority. We use admission control so one user cannot occupy the entire model pool with dozens of long requests. Cancellation must propagate when the user closes the request, and streaming should expose useful progress without hiding queue latency.

Model routing also matters. Query classification, metadata normalisation and rewrite tasks often do not need the largest model. A smaller model can handle those steps, while complex synthesis is sent to a stronger model. The decision must be evaluated against quality; routing is not simply a cost switch.

For self-hosted inference, engines such as vLLM provide continuous batching, PagedAttention, quantisation, prefix caching and several parallelism strategies. These features improve the available design space, but they do not replace workload-specific benchmarks. vLLM exposes separate latency, throughput and serving benchmarks for that reason.vLLM documentation and vLLM benchmarking

We measure at least queue wait, time to first token, time per output token, total input/output tokens, end-to-end p50/p95/p99 latency and rejection rate. Average latency alone hides the exact concurrency failures that employees notice.

4. Cache work, not security mistakes

Caching can remove enormous amounts of repeated work:

  • Parsed pages and OCR results can be keyed by content hash and parser version.
  • Embeddings can be keyed by normalised chunk hash and embedding version.
  • Query rewrites can be cached for low-risk deterministic operations.
  • Retrieval results can be cached when index and permission versions are part of the key.

A retrieval cache key that contains only the user’s query is dangerous. At minimum it should include tenant_id, the authorised scope or ACL version, logical index version, retrieval configuration and query. Response caching requires even stricter isolation and invalidation because the generated text may contain confidential context. If permission state cannot be proved current, the cache must fail closed.

5. The control plane must grow with the corpus

At enterprise scale, SSO and RBAC are not UI features added after retrieval. Access rules from the source system must travel with the document and be enforced before candidates are returned. Deactivated users, removed group memberships, document deletion and legal-retention rules need measurable propagation times.

Every returned chunk should retain source_id, version, page or section and index version. Every request should record the authenticated principal, selected logical index, policy decision, retrieved source IDs, model and prompt versions, token use and latency. Sensitive document text does not need to be copied into every log; provenance and policy decisions do.

The NIST Generative AI Profile recommends recording data provenance, model versions, access modes and human-oversight responsibilities, and calls for post-deployment monitoring, incident response, recovery and change management.NIST AI 600-1 In a RAG system, these requirements apply to the whole pipeline, not only to the final LLM.

6. Use evaluation as a release gate

A green infrastructure dashboard does not mean the answers are correct. Scaling changes retrieval behaviour: new documents compete with old ones, ACL filters shrink candidate pools, index settings change recall and a new embedding version moves neighbourhoods.

We maintain representative question sets by department, language, document type and permission profile. Retrieval is evaluated separately from generation. Useful retrieval metrics include Recall@k, MRR or nDCG; answer tests cover groundedness, citation correctness, completeness and appropriate abstention. RAGAS formalises a related separation between relevant retrieval context, faithful use of that context and generation quality.RAGAS paper

The Microsoft RAG architecture guide similarly recommends evaluating representative media and queries, measuring both retrieval and end-to-end quality, and documenting the results.Microsoft RAG design and evaluation guide

Security tests belong in the same gate. We ask whether a user can retrieve another tenant’s document, whether a removed group still sees cached results, whether prompt injection can select an unauthorised index and whether an answer without sufficient evidence refuses correctly. A release that improves average answer quality but breaks one of these tests does not ship.

A practical set of scaling gates

StageRequired gateWhat is still allowed to be manual
PilotRepresentative documents, agreed questions, source citations and basic tracesCorpus approval and failure investigation
BetaSSO/RBAC, asynchronous ingestion, golden set, quotas and user feedbackSelected document-quality review
ProductionAvailability target, p95/p99 latency, backups, alerts, rollback and incident ownerEscalation for low-confidence/high-impact answers
Organisation-wideTenant controls, cost allocation, audit coverage, lifecycle automation and adoption metricsRisk-based human approval

The transition is based on evidence, not on an arbitrary number of documents.

Reference architecture

flowchart TD
    S["Enterprise sources"] --> Q["Durable ingestion queue"]
    Q --> W["Parser / OCR / embedding workers"]
    W --> I["Versioned search index"]
    W --> P["Processing manifest and error tracking"]
    U["Authenticated user"] --> G["Authorisation + ACL pre-filter gateway"]
    G --> I
    I --> R["Candidate retrieval + reranker"]
    R --> M["Model router + admission control"]
    M --> L["Inference pool"]
    L --> A["Cited answer"]
    G --> O["Quality, security and latency traces"]
    P --> O
    L --> O

The important boundary is the retrieval gateway. The model may choose a tool, but it may not choose its own identity, tenant or raw physical index. Authorisation remains deterministic and server-side.

Capacity planning before buying hardware

We begin with a workload sheet rather than a GPU catalogue:

  • Peak queries per second and concurrent sessions
  • p50 and p95 retrieved context tokens
  • p50 and p95 generated output tokens
  • Pages and changed source tokens ingested per day
  • Chunk expansion factor and active embedding versions
  • Candidate count, rerank count and filter selectivity
  • Required ACL-update and deletion propagation time
  • Target p95 latency, availability and recovery time

Two simple calculations expose many hidden costs: daily generation work is approximately queries multiplied by average input-plus-output tokens; a reindex is approximately changed source tokens multiplied by the number of embedding versions being built. We then benchmark the actual models, parsers and search cluster with realistic input lengths and arrival patterns. Vendor maximums are not a capacity plan.

Conclusion

Enterprise RAG does not scale by making one box larger. It scales when ingestion can retry safely, indexes can be versioned and rolled back, retrieval remains permission-aware, inference queues survive concurrency and every release is evaluated against real questions.

The number of documents is visible, so it attracts attention. The difficult work is less visible: ACL changes, stale caches, failed pages, overloaded queues, drifting retrieval quality and unclear incident ownership.

A production RAG system is not a model connected to a vector database. It is a governed information system with an LLM at the end of the path.

LMXAI designs enterprise AI platforms across document intelligence, retrieval, inference, security and observability. If your RAG pilot is ready to become a reliable company-wide service, we can design the scaling path and its release gates together.

Sources

  1. Microsoft — Design and develop a RAG solution
  2. Elastic — Optimize indexing speed
  3. Elastic — Size your Elasticsearch shards
  4. vLLM — Benchmarking
  5. Liu et al. — Lost in the Middle
  6. Es et al. — RAGAS: Automated Evaluation of Retrieval Augmented Generation
  7. NIST — AI RMF Generative AI Profile (AI 600-1)