Why AI Projects Die Between the Demo and Production

Confusion around enterprise AI integration is no longer unusual. Executives develop high expectations, a technical team produces an impressive demo within weeks, early users see the potential—and then the project gets close to production. Costs rise, latency appears, security questions surface, retrieval quality becomes inconsistent, and an architecture that worked for a handful of developers behaves very differently when hundreds of employees arrive.

The pattern is visible in industry data. Gartner reports that, on average, only 41% of generative AI prototypes reached production in its 2024 enterprise survey.[1] Its later analysis identifies poor data quality, inadequate risk controls, escalating costs and unclear business value as major reasons GenAI initiatives are abandoned after proof of concept.[2] McKinsey, drawing on work with more than 150 companies, similarly describes risk, compliance friction, rework and cost overruns as recurring barriers between prototype and scale.[3]

In many cases, the model is not the main problem. The system around the model has not been designed for production.

A demo asks whether it works. Production asks whether it survives reality.

A pattern I repeatedly see is bottom-up experimentation. Early-adopter developers combine documentation, example repositories, videos and AI-assisted coding to produce a functioning MVP quickly. There is nothing inherently wrong with this; experimentation is often how innovation starts. The problem begins when the demo architecture quietly becomes the production architecture.

A demo can be judged by one question: “Does it work?” Production requires many more: How many concurrent users can it support? What is the true cost per successful task? Which documents may each user access? How will retrieval failures be measured? What happens when an embedding model changes? How are updates and deletions synchronized with the index? What latency target is acceptable at p95 or p99? Where do auditability and observability live?

NIST’s Generative AI Profile explicitly treats AI as a lifecycle risk-management problem—govern, map, measure and manage—not simply as a model-selection problem.[4]

If you build RAG, the vector database is only one decision

RAG is a natural choice when an enterprise wants an AI system to work with internal knowledge. The foundational RAG paper describes the architecture as a combination of parametric model memory and an external non-parametric memory retrieved at generation time.[5]

But “embed documents and put them in a vector database” is not a production RAG strategy. Before indexing starts, an engineering team must decide how documents are parsed, how chunks are formed, whether retrieval is dense, sparse or hybrid, how metadata filters and tenant boundaries are enforced, whether reranking is needed, which embedding model is appropriate, how indexes are versioned, and how retrieval quality is evaluated.

Modern RAG surveys divide the system into pre-retrieval, retrieval, post-retrieval and generation stages precisely because quality emerges from the interaction of these layers rather than from one database choice.[6]

“The model suggested this architecture” is not an architecture decision

Ask a general-purpose model how to build document Q&A and it will often return a reasonable default recipe: parse, chunk, embed, store in a vector database, retrieve similar chunks and send them to an LLM. The recipe is useful. It is not evidence that the same design is optimal for your company.

With a hosted embedding API, the pipeline does not end after document processing. Once parsing and chunking are complete, those text chunks are sent to an embedding service through separate API calls and converted into vectors. This is a distinct billed step from the later LLM calls made by the agent or generation layer: embedding providers can charge separately for the number of tokens embedded. OpenAI prices its embedding models by input-token volume.[7] Cohere likewise describes embedding pricing in terms of embedded tokens.[8] In other words, the ingestion pipeline can already create an embedding cost before a user sends the first production query. At scale, the choice between a hosted API, local embeddings, batch inference and incremental indexing directly affects total cost.

The engineering task is not to choose the most popular tool. It is to optimize the trade-off among quality, latency, throughput, data sensitivity, operational complexity and total cost of ownership.

The expensive anti-pattern: asking an LLM to make every intermediate decision

A large share of avoidable cost comes from orchestration rather than model price.

A common pipeline sends the user message to one LLM for intent detection, a second call for schema selection, another for query rewriting, another to summarize retrieved documents, and finally the main agent call. One user request has silently become four or five generations.

Some systems genuinely need these stages. But every decision does not deserve an LLM call. If intent categories, tool contracts, data structures and decision boundaries can be described to the main model—or handled deterministically in code—separate generation steps may only duplicate reasoning.

The cost effect of this chain is direct. Every LLM invocation processes its own input and output tokens and is billed as a separate inference call.[9] If one user request triggers three comparable calls—for example intent detection, schema selection and the final answer—the generation-layer cost can be roughly three times that of a one-call design. Larger contexts or longer outputs can widen the difference further. The impact is not only financial: each invoke adds another network round trip, increases tail latency and reduces concurrency headroom. Serving research demonstrates how sensitive LLM throughput is to memory management and batching. The vLLM PagedAttention paper reported 2–4× throughput improvements at comparable latency through better KV-cache management.[10] The practical implication is the inverse: unnecessary generations directly consume production capacity.

The right default is usually to start with the smallest architecture that can satisfy the requirement, measure it, and add routers, rerankers, validators or specialist agents only when evidence shows they improve the system.

Why cost explosions rarely show up in the demo

An expensive architecture can look cheap when one developer sends 30 requests a day. Economics change when hundreds of employees use the system, long contexts are sent repeatedly and every user request triggers several model calls.

This is why AI cost should be tracked as cost per successful task, not simply as the monthly API bill. A cheap answer that forces the user to retry three times can be more expensive than a more capable model that completes the task correctly once. Likewise, a system with low token cost but poor retrieval may create significant human verification cost.

Field research reinforces the importance of context. In a study of 5,179 customer-support agents, Brynjolfsson, Li and Raymond found that access to a generative AI assistant increased productivity by 14% on average and by 34% for novice and lower-skilled workers, while gains for highly experienced workers were much smaller.[11] There is no universal “AI productivity percentage.” Value depends on the workflow, users and implementation.

The other architecture problem is expectation architecture

Frontier-model narratives around AGI, reasoning and autonomous agents can create an image of AI as a general-purpose superpower rather than a probabilistic inference system with specific strengths and failure modes. A polished demo reinforces that perception because demos operate on curated data and controlled behavior.

Production is made of edge cases: incomplete questions, contradictory sources, changing permissions, tool timeouts, overconfident outputs, prompt injection attempts and unusual workloads. A system that appears 95% correct can still be unusable when the remaining 5% occurs in a high-impact business process.

Before implementation starts, C-level expectations need to be aligned with three things: what current AI can technically do, what the company’s own data and processes allow, and what level of reliability is economically worth achieving. NIST’s focus on mapping use context and measuring system behavior provides a useful foundation for this alignment.[12]

Real AI engineering begins in the first customer meeting

A capable AI engineer or architect should not leave the first discovery meetings with only a feature list. They should be able to make the gap between expectations and reality visible.

At minimum, the team should define the business problem, the metric representing value, the readiness of the data, the cost of an incorrect answer, the minimum sufficient architecture, the scaling economics and explicit criteria for moving from pilot to production.

This is where technical design becomes business design.

Do feasibility before implementation

The most expensive failure is not a failed AI project. It is discovering too late that the project was never economically or operationally viable.

A useful feasibility study should cover at least four layers:

• Value feasibility: Is the problem valuable enough, and can success be measured?
• Data feasibility: Is the required data accessible, current, high-quality and permissionable?
• Technical feasibility: Can the required accuracy, latency, throughput and integration constraints realistically be met?
• Economic feasibility: At target scale, are inference, retrieval, storage, observability and human-oversight costs acceptable?

For regulated and high-impact environments, governance and compliance feasibility must be evaluated as well.

Gartner’s findings point in the same direction: the reasons pilots fail to reach production are frequently not model benchmark scores, but data, risk, cost and unclear business value.[13]

Conclusion: design the system before choosing the model

Enterprise AI projects usually do not die because the model is not intelligent enough. They die because the wrong problem was selected, a demo architecture was promoted to production, unnecessary LLM calls multiplied cost and latency, retrieval and authorization were treated as afterthoughts, or executive expectations were never translated into a measurable business case.

A good demo tells you: this is possible.

A good feasibility study answers the harder questions: Should we build it? What is the smallest architecture that will work? What will it cost at scale? How will we know it is successful?

That is where production-grade AI engineering begins.

References

1. Gartner, AI Maturity Matters: Proportion of AI and GenAI Prototypes Making It Into Production (2025). Source
2. Gartner, Why 50% of GenAI Projects Fail — And How to Beat the Odds (2026). Source
3. McKinsey, Overcoming two issues that are sinking gen AI programs (2025). Source
4. Autio et al., NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (2024). Source
5. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020. Source
6. Huang & Huang, A Survey on Retrieval-Augmented Text Generation for Large Language Models (2024). Source
7. OpenAI, text-embedding-3-small model documentation. Source
8. Cohere, How does Cohere pricing work? Source
9. OpenAI, GPT-5.4 model documentation. Source
10. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023). Source
11. Brynjolfsson, Li & Raymond, Generative AI at Work, NBER Working Paper 31161; published in QJE (2025). Source
12. NIST, AI Measurement and Evaluation. Source
13. Gartner, Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept by End of 2025 (2024). Source