Most companies we talk to have run three or four AI pilots and shipped zero of them. The pattern behind that is consistent, fixable & almost never about the model.
The demo went well. Leadership was impressed. Someone said "we should roll this out." That was five months ago, and the pilot is still a pilot (or quietly dead in a Slack channel nobody opens).
We hear a version of this story in most first conversations about AI work. The details vary, but the failure modes repeat. Here are the six we see, in rough order of frequency.
1. The demo was the deliverable
A pilot built to impress in a 30-minute meeting is optimized for the happy path: clean inputs, cherry-picked examples, a presenter steering around the rough edges. Production is the opposite: adversarial inputs, edge cases, users who type things you didn't anticipate. Teams that treat the demo as 80% done discover it was closer to 20%, get discouraged by the remaining grind, and stall.
The fix is scoping the pilot as a thin slice of the production system (real data access, real auth, one real workflow end to end) rather than a facsimile of it. That version takes longer to demo and months less to ship.
2. No evals, so nobody can say whether it works
Ask "is the AI good enough to roll out?" and the room goes quiet, because "good enough" was never defined. Without an eval set (a few hundred representative inputs with known-good outputs, scored automatically), every quality conversation runs on anecdotes. One bad example from an exec kills the project, one good one over-inflates it, and neither is evidence.
Teams skip eval sets because building them is tedious, and skipping them is the most common technical reason pilots die. Build the eval set before scaling the pilot. If you can't define what a good output looks like, that's not an eval problem; the use case isn't ready.
3. It can't reach the data that would make it useful
The pilot ran on ten uploaded PDFs. The production version needs the warehouse, the docs wiki, the ticketing system & the CRM, with the user's actual permissions enforced. That's an integration and access-control project, and it's routinely discovered late because the pilot sidestepped it.
This is also where AI projects turn out to be data projects in disguise: if the warehouse is untrusted or undocumented, the AI inherits every one of those problems and adds hallucination risk on top. We've redirected more than one AI engagement into a transformation engagement first, because a system that confidently answers from bad data is worse than no system.
4. Nobody owns it after the demo
Pilots get built by an enthusiast (an engineer with spare-time energy, an innovation team, an intern). Production systems need an owner with a budget line: someone accountable for accuracy drift, prompt updates when the model version changes, cost monitoring & the support queue. When the enthusiast moves on and no owner was named, the pilot doesn't fail so much as stop.
Name the owner before the build starts. If no team wants to own it, that tells you the organization doesn't actually value the use case, which is cheaper to learn before the build.
5. Cost & latency were never budgeted
At demo scale, inference cost rounds to zero and nobody times the responses. At production scale, both matter: a use case that costs $0.40 a query is viable for contract review and absurd for autocomplete, and a 20-second response that felt fine in a demo is unusable inside a support workflow. Set the per-query cost and latency targets in week one and engineer to them; model choice, caching strategy & how much context goes into each call all follow from that budget.
6. Security review arrived at the end instead of the beginning
The pilot ran on someone's laptop with an API key in a notebook. Then, at rollout, security asks the reasonable questions: where does the data go, what's the retention policy, who can query what, what happens when the model is prompted to leak? Answering these late can force a re-architecture; answering them early is mostly a set of known patterns (zero-retention API terms, scoped access, injection defenses, audit logging). Invite security to the kickoff and your longest-lead blocker becomes a checklist.
What the successful ones do differently
The through-line isn't talent or budget. Teams that ship treat the pilot as the first increment of a production system: real data, an eval set, a named owner, cost and latency budgets, security in the room, all from week one. That front-loads the effort, and it's why the project is still alive in month four.
That's how we run AI engagements: an audit-plus-prototype phase (typically 2–4 weeks) that produces the eval set and a real go/no-go, before anyone commits to a full build. Sometimes the answer coming out of it is "not this use case, but here's a better one" (cheaper to hear in week three than month six).
If you've got a stalled pilot (or a list of them), send us two or three sentences about it. We'll tell you within a business day whether it looks recoverable and what we'd check first.