Is your company ready for AI? A technical readiness checklist

Most of the AI projects we are asked to rescue did not fail as engineering. The model did what the demo promised. What failed was everything around the model: pulling clean data out of the warehouse took a three-week ticket, nobody could say whether "seems accurate" meant 95 percent or 60, and the compliance officer — meeting the system for the first time in month eight — asked one question about PHI in the logs that ended the meeting. Projects like that do not die cleanly. They do something more expensive: they stay pilots forever.

We build LLM systems for a living, which gives us an obvious incentive to tell you that you are ready. This checklist points the other way. It is the assessment we run in the first two weeks of every AI engagement, and it ends prospective builds about as often as it starts them — because the cheapest AI failure is the one caught before the budget is committed. Run it yourself, honestly, before you talk to us or to anyone else.

Can you get your own data, and is it any good?

Every useful LLM system is a data pipeline with a model in the middle, and the pipeline fails first. Before any architecture conversation, run two tests.

The one-day pull test

Ask an engineer to assemble 500 representative examples of the real input — referral faxes, support tickets, contracts, whatever the system will actually read — by tomorrow. Not a curated sample: the true distribution, including the photographed forms, the scans at an angle, and the emails where the answer hides four replies deep. If the pull requires a ticket queue, two access reviews, and a legal opinion, you have found your first project, and it is not an AI project.

The someone-who-knows test

Now put those 500 examples in front of a person who can explain them: what each field means, which values are plausible, which source systems lie. Schemas and validation rules get written from exactly this knowledge. If it lives in nobody's head, or in one head with no time for you, accuracy has a ceiling that no model choice can raise. Note what is missing from this section: volume. Most systems we ship train on nothing and run on retrieval, schemas, and validation. Meaning is the constraint, not size.

An eval harness before the first feature

The strongest readiness signal we know is boring: a hand-labeled evaluation set — a few hundred real examples with correct answers, agreed by the people who own the workflow. It converts "the model seems good" into a sentence like "the model is right on 96 percent of insurance-ID fields and 71 percent of diagnosis codes" — a sentence you can act on: it says what ships, what routes to review, and what needs redesign.

Build the harness before the first feature, because it is the only honest way to answer the go/no-go question. Two weeks of labeling with your domain experts measures the accuracy ceiling on your documents rather than on a vendor's benchmark. If the ceiling is too low to ship, you have spent two weeks finding out. Skip the eval and you will find out in production, two quarters and one executive review too late.

A demo proves the model can do the task once. An eval proves it can do the task ten thousand times — and tells you the week it stops.

The harness keeps paying after launch. Every prompt change, model upgrade, and threshold adjustment runs against it before production does, and a monthly audit against freshly labeled samples surfaces drift in a report instead of in a customer incident.

Name the human who checks the model's work

Human-in-the-loop is not an admission that the AI failed. It is the design decision that lets a probabilistic system touch consequential records at all, and it has to be designed — a person rubber-stamping a queue at 4pm is not a control, it is a liability with a login.

The mechanics that make review real: every extracted value carries a confidence score and a pointer back to its exact location in the source, low-confidence outputs demand attention before anything else, and the review screen puts each field beside its highlighted source snippet so confirming a value takes one glance instead of a hunt. In the intake build for a multi-clinic healthcare provider, nothing posts to the EHR without a named human approval — and because review was designed as the product, not bolted on, handling time still fell 71 percent.

The readiness questions: who are your reviewers, what does one review cost in minutes, and does the workflow owner accept that cost? If reviewing an output takes as long as producing it by hand, the design has failed before the model runs. Done well, review is a small fraction of the original task — and that fraction is where the ROI lives, so model the review cost before you model the savings.

Draw the compliance and cost boundaries before the architecture

Where PHI and PII may travel

If your data includes PHI, PII, or anything a regulator has a name for, the data boundary is an input to the architecture, not a review gate at the end. Write one page with your compliance lead before any code exists: which fields may leave your infrastructure and under which agreements (BAA, DPA), whether anything may enter a model provider's training set (the correct answer is no), and what logs and traces may contain — prompts leak PHI by default, and excluding it is deliberate engineering, not a setting. A system designed inside those lines passes review in weeks. A system that meets its compliance officer in month eight becomes a pilot that cannot legally graduate.

A cost ceiling per request, in writing

Token costs look like rounding errors until they are multiplied by production volume. Set a ceiling per unit of work — per document, per ticket, per conversation — and tie it to the money that unit saves. A pipeline that costs $0.04 per document at pilot volume can land at several times that once retries, long outliers, and reranking arrive. Model routing, caching, and batching are the engineering responses to a ceiling; without one, "we'll optimize later" quietly becomes the unit economics. The discipline is the same one that keeps any build honest — we wrote up what custom software actually costs in 2026 — but AI adds a variable cost per request that no launch date makes retroactively affordable.

Mind the pilot-to-production gap

Between a working pilot and a production system sits most of the real budget, and it is where AI projects go quiet. The pilot handled the happy path on curated inputs. Production handles malformed documents and 40-page outliers, provider outages and retries, prompts and model versions pinned and change-controlled so that Tuesday's output can be explained on Thursday, accuracy and cost on a dashboard someone reads, and an on-call rotation that knows what "broken" means for a system that is probabilistic on its best day.

Then ask the question pilots never answer: who owns this in month 13? Somebody adjusts thresholds, reads the drift report, and approves the next model upgrade through change control. If the answer is "the vendor, forever," price that dependency honestly. If the answer is a shrug, the pilot is a sunk cost in progress.

Signs you are not ready

The honest list, drawn from assessments where we advised against building:

• The sponsor is enthusiasm, not a workflow owner. Ready projects start from a metric that hurts — a queue measured in days, a cost per case everyone knows. "The board asked about AI" is not a workflow.
• Data access requires a committee. If your own engineers cannot reach the inputs, an outside team certainly cannot.
• Nobody can name a reviewer. No named humans for the loop means no loop, which means no path past legal for anything consequential.
• "Seems good" is the acceptance criterion. No labeled baseline means every executive review reopens the question of whether the system works, forever.
• Compliance has not heard of the project. Every month of silence is converting into rework at an unfavorable rate.
• The plan has no month 13. No owner for thresholds, drift, and upgrades means you are building a demo with production infrastructure costs.

None of these is a verdict. Most are six-to-twelve week fixes, they cost far less than a stalled pilot, and every one of them — data access, an eval habit, a compliance relationship that starts early — pays off even if you never ship a model. Scored honestly, the checklist fits on one page:

AI readiness scorecard — one point per yes

[ ] 500 representative real inputs pulled in one day, legally
[ ] A named person who can explain every field in that sample
[ ] A hand-labeled eval set and a per-field accuracy number
[ ] Named reviewers, and the cost of one review in minutes
[ ] A one-page data boundary your compliance lead has signed
[ ] A cost ceiling per request, tied to what the unit saves
[ ] A named owner for the system in month 13

6-7  fund the build
4-5  fix the gaps first — weeks now instead of quarters later
0-3  run a feasibility phase, not a project

A high score does not guarantee the model clears your accuracy bar — that still has to be measured, on your documents. It guarantees something rarer: that if the model works, your company can actually ship it. If you land in the middle and want the measurement done properly, our approach to AI development starts every engagement with a two-week, fixed-price feasibility phase — a labeled eval set and a measured accuracy ceiling on your own data, priced so that "not yet" is an affordable answer.