Document Processing for Enterprise RAG

A company rarely has one document to connect to an AI system. It has hundreds of policies, contracts, scanned PDFs, technical reports, presentations, tables and Excel workbooks. From the outside, the task looks straightforward: upload the file, extract the text, split it into chunks and send it to a vector database.

This is where the first major pitfall begins.

Files cannot all be read in the same way. Even two PDFs with the same extension can be technically very different. One may contain a clean text layer, while the other consists entirely of scanned page images. One report may use a single-column layout; another may combine multiple columns, footnotes, tables, charts and annotations placed across the page. A person understands the reading order almost intuitively. A parser has to reconstruct it.

DocLayNet required 80,863 manually annotated pages and 11 layout classes to represent the diversity of real-world document layouts.[1] The TableFormer research also shows that recovering rows, columns, merged cells, missing values and complex headers from a table image is not a simple text-extraction problem.[2]

Document processing is therefore not a minor preprocessing step in a RAG pipeline. It is a service that must be designed and evaluated in its own right.

Saying “we can read PDFs” is not enough

PDF is a presentation format rather than a reliable logical content model. The reading order visible on a page is not always encoded explicitly in the file. All of the following can arrive under the same PDF extension:

Machine-readable structured text
Unstructured text with broken reading order
Scanned printed text
Multi-column layouts
Tables with merged cells
Charts and visual annotations
Forms, checkboxes and signatures
Pages where text and visual structure carry meaning together

Docling provides a strong starting point. The Docling Technical Report explains that it uses models based on DocLayNet for layout analysis and TableFormer for table-structure recognition, while targeting commodity hardware and a modest resource budget.[3] It can export a document as Markdown or as a structured representation.

from pathlib import Path
from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert(Path("documents/report.pdf"))

document = result.document
markdown = document.export_to_markdown()
print(markdown[:1500])

A production environment, however, should not depend on a single parser. In our CPU-only document-processing service, we begin with standard parsing through Docling. If the output is empty, contains far less text than expected or fails layout-quality checks, we do not silently mark the file as successful. We route it to a second processing path.

OCR fallback for pages Docling cannot read

Some PDFs have no text layer. Others contain broken character maps, corrupted fonts or unusable reading order. In those cases, we render the relevant pages as images and apply OCR. PyMuPDF can render pages as Pixmaps, while Tesseract can extract printed text from page images.[4][5]

This fallback is useful, but it is not free. Rasterising an entire archive at high DPI and running OCR across every page creates substantial latency on a CPU-only system. We therefore changed the workflow instead of treating the problem purely as an OCR-speed problem.

We designed the document service so an agent can call it as a tool. The agent first inspects metadata, headings, the available text layer or previously extracted content. When answering a question requires specific pages, it invokes the read_pdf tool only for those pages. The service renders the selected pages, performs OCR and writes the result to sidecar storage. The same page does not need to be processed again for the next query.

from io import BytesIO

import pymupdf
import pytesseract
from PIL import Image
from langchain_core.tools import tool

@tool
def read_pdf(file_id: str, pages: list[int]) -> str:
    """Read selected pages from an authorised PDF using OCR fallback."""
    path = resolve_authorized_file(file_id)  # Server-side access check
    extracted = []

    with pymupdf.open(path) as pdf:
        for page_number in pages[:10]:
            page = pdf.load_page(page_number - 1)
            pixmap = page.get_pixmap(dpi=200, alpha=False)
            image = Image.open(BytesIO(pixmap.tobytes("png")))

            text = pytesseract.image_to_string(image, lang="eng+nld")
            sidecar.put(file_id, page_number, text)
            extracted.append(
                f"[Page {page_number}]\n{text}"
            )

    return "\n\n".join(extracted)

The example is deliberately simplified. In production, file_id must resolve to a real file through server-side authorisation. The agent must never be allowed to submit arbitrary file paths. Page count, DPI, execution time and output size also require hard limits.

On-demand processing does not remove OCR cost. It pays that cost at the right moment. Instead of creating latency across thousands of pages during initial ingestion, the system processes only the pages needed for the active question. Document processing becomes an active, controlled knowledge tool rather than a passive ETL step.

With large Excel files, row count is not the real problem

The second important challenge we encountered was large Excel workbooks. A dataset containing 1,000 or 10,000 rows of sales, operations, inventory or finance data can be extremely valuable to employees.

An important distinction is needed here: 1,000 rows is not large for pandas. The problem begins when all rows are serialised as text and placed inside a language model’s context.

Sending the complete workbook to the model rapidly increases token usage. The model may lose relationships between columns, make aggregation errors or invent a pattern that is not supported by the data. A long context window does not automatically solve this problem. “Lost in the Middle” found that models do not use information at every position in a long context with equal reliability; performance can decline significantly when the relevant information is positioned in the middle.[6]

We therefore treated an Excel workbook as a queryable data source rather than as prompt context. pandas can load Excel sheets into DataFrames and supports selecting columns, filtering, aggregation and statistical operations.[7] We exposed those capabilities to the agent through an allowlisted data-analysis tool.

from typing import Literal

import pandas as pd
from langchain_core.tools import tool
from pydantic import BaseModel, Field

class ExcelRequest(BaseModel):
    file_id: str
    sheet_name: str
    operation: Literal["summary", "sum", "mean", "top"]
    column: str | None = None
    top_n: int = Field(default=10, ge=1, le=50)

@tool(args_schema=ExcelRequest)
def analyse_excel(
    file_id: str,
    sheet_name: str,
    operation: str,
    column: str | None = None,
    top_n: int = 10,
) -> str:
    """Run an allowlisted analysis without placing the workbook in context."""
    path = resolve_authorized_file(file_id)
    frame = pd.read_excel(path, sheet_name=sheet_name)

    if operation == "summary":
        result = frame.describe(include="all").transpose().reset_index()
    elif operation == "sum" and column:
        result = {"column": column, "sum": float(frame[column].sum())}
    elif operation == "mean" and column:
        result = {"column": column, "mean": float(frame[column].mean())}
    elif operation == "top" and column:
        result = frame.nlargest(top_n, column).head(50)
    else:
        raise ValueError("Unsupported operation or missing column")

    if isinstance(result, pd.DataFrame):
        return result.head(50).to_json(orient="records", date_format="iso")
    return str(result)

When asked to identify the ten customers with the highest revenue in 2025, the agent does not place the whole workbook in the prompt. It sends a structured request describing the sheet, column and required operation. pandas performs the calculation deterministically, and only the answer plus a limited number of supporting rows enters the context.

This protects the context window and moves calculation away from the language model to a library designed for data analysis. The agent should not be allowed to run arbitrary Python or eval expressions. Operations should be constrained by explicit, validated tool schemas.

Multimodal models for documents whose meaning is visual

OCR converts characters in an image into text. Some pages, however, carry meaning through visual relationships. The direction of a line in a chart, the mapping between colours and a legend, the position of a checkbox or the relationship between a table cell and its parent header may disappear in plain text.

A multimodal vision-language model can serve as a second fallback or enrichment layer for such pages. Qwen2.5-VL-3B-Instruct is one compact example. Qwen positions the 3B model for edge AI and highlights the family’s ability to understand documents and diagrams.[8] Its technical report describes dynamic-resolution processing and window attention for handling images of different sizes efficiently.[9]

from transformers import pipeline

vision_model = pipeline(
    task="image-text-to-text",
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    device_map="auto",
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "file:///tmp/page-12.png"},
        {
            "type": "text",
            "text": (
                "Extract the headings, printed text, table structure and "
                "chart meaning. Return valid JSON and do not infer "
                "values that are not visible."
            ),
        },
    ],
}]

result = vision_model(text=messages, max_new_tokens=800)
print(result[0]["generated_text"][-1]["content"])

A multimodal model should not be applied to every document. It can be more expensive and less deterministic than OCR. The strongest use case is a selected page where classical parsing and OCR have low confidence and visual structure is essential to the answer. Requesting JSON does not guarantee correctness. Outputs still require schema validation, and numerical values should be cross-checked against the source image or a deterministic parser whenever possible.

Additional techniques for a reliable document pipeline

1. Classify before parsing

Do not trust the extension alone. Inspect MIME type, file signature, encryption, page count, text-layer density and image ratio. Route the file to a native parser, layout-aware parser, OCR engine, spreadsheet engine or multimodal model accordingly.

2. Use confidence-aware fallback

A parser that returns no exception has not necessarily produced a good document. Character count, empty-page ratio, corrupted-character ratio, table-cell coverage and reading-order consistency can be used as quality signals. Low-quality output should trigger a second method automatically.

3. Preserve layout before chunking

Character-count chunking can separate a heading from its paragraph, a table header from its rows or a footnote from its reference. Preserve relationships among headings, paragraphs, lists, tables, figures and captions before creating chunks.

4. Preserve provenance

Each chunk should retain file_id, document version, page number, section heading, parser version and, where possible, bounding-box coordinates. A RAG answer that cannot return to its source is difficult to verify in an enterprise environment.

5. Make ingestion failure visible

The most dangerous outcome is a damaged PDF indexed as empty content while the pipeline reports success. Employees believe the document is available, but retrieval can never find it. Parsing coverage, failed pages and fallback results must be observable.

flowchart TD
    F["Incoming file"] --> R["Type and quality router"]
    R --> D["Docling parser"]
    D --> Q{"Quality sufficient?"}
    Q -->|Yes| S["Structured sidecar"]
    Q -->|No| O["Page render + OCR or VLM"]
    O --> S
    A["Agent"] --> T["read_pdf / analyse_excel tools"]
    T --> S

Conclusion: document processing is not file conversion

Retrieval quality in an enterprise RAG system does not begin after a document reaches the vector database. It starts much earlier, when the file is read, its structure is preserved and the system decides which extracted elements are trustworthy.

A robust document-processing service does not force every file through the same parser. It identifies the file, selects the appropriate method, measures quality and applies OCR or multimodal fallback when necessary. It queries large tables through tools instead of filling the context window. It runs expensive operations on demand instead of applying them upfront to the entire archive.

A larger context window, a better embedding model or a more powerful LLM cannot rescue a badly parsed document.

LMXAI designs enterprise AI systems across the full stack: ingestion, document intelligence, retrieval, security and inference.

Previous: Enterprise AI Integration: From Prototype to Production

Next: Choosing the Right Vector Database