Your AI pilot demoed well. The slide said success. Three months later nobody owns an SLA, and the spend is still on the calendar.
That is not a product. That is a demo with a budget.
A product has an owner, a service level, and a path to kill it when it fails. Most AI pilots have none of those. They have a vendor deck, a sample set someone cleaned by hand, and a review date that slips.
I have sat on both sides of the pilot table. The pattern does not change with the model name.
If the pilot cannot say what it will own in production, do not fund another quarter.
What a pilot owes you
A pilot is a time-boxed test. It answers one question: should we run this in production under a real owner.
It is not a soft launch. It is not a product with training wheels. It is not a reason to keep a vendor on retainer while the team "learns."
Write the question on one line before the kickoff. If you cannot write the question, you do not have a pilot. You have a science fair.
The SLA test
Ask this early: what service level will this own if we keep it.
Uptime. Latency. Freshness of the data it uses. Accuracy against a labeled set you already trust. A human review rate. Pick the ones that matter for that use case. Write the number. Write who gets paged when it misses.
If the room says "we will figure the SLA out after the pilot," you already know the ending. There will be no SLA. There will be another demo.
A pilot that never names an SLA is not preparing for production. It is avoiding it.
Kill list before another quarter of demo spend
I use this list as a fractional CDO. Fail 2 or more and I recommend you stop. Fail the SLA item alone and I recommend you stop.
1. No named production owner
Who on your payroll will own this on day 91. Not the vendor. Not the pilot squad that was "temporary."
If the answer is "the AI team" with no person, you do not have an owner. You have a committee.
2. No SLA draft
Even a draft SLA counts. "We will answer 95% of cases in under 30 seconds with human review on the rest" is a draft. "It felt accurate in the demo" is not.
3. Success is a vibe
Success metrics that only live in a slide are not metrics. You need a number you can recompute after the room leaves.
If success is "stakeholders liked it," you bought theater.
4. The data path is a one-off
The pilot ran on a file someone exported once. Or on a sandbox that does not match production filters, rights, or delay.
If you cannot point to the production pipeline that will feed the same feature, you are not piloting the real system. You are piloting a photograph of it.
5. No kill criteria
Write the fail conditions on day one. Example: if precision on the holdout set stays below the baseline for 2 review cycles, we stop. If the human review load stays above X hours a week, we stop.
If you have no kill criteria, the pilot cannot end. It can only ask for more time.
6. The vendor needs the next phase to "make it real"
Some vendors sell a pilot that only works if you buy the platform next. That is not a test of your use case. That is a funnel.
A fair pilot can fail without a follow-on contract. If failure is not an allowed outcome, you are not running a pilot.
7. Nobody can explain the miss modes
Ask what wrong looks like. Hallucinated citations. Wrong customer segment. Stale inventory. A polite answer that skips the constraint.
If the team cannot list 3 miss modes and how they will catch them, they are not ready for users who trust the output.
What good looks like in 90 days
A useful pilot ends with one of three outcomes.
- Promote. There is an owner, a draft SLA, a production data path, and a launch date.
- Kill. You learned the use case does not pay for the cost. You stop the spend.
- Narrow. You cut scope to the part that cleared the bar, and you kill the rest.
"Extend the pilot" is not a fourth outcome. It is a delay. One short extension with new kill criteria can be real. A third quarter of demos is not.
What I would do this week
Pick your loudest AI pilot. Do not start with a new tool meeting.
Open a one-page doc. Write the question the pilot is supposed to answer. Write the draft SLA. Write the owner on your payroll. Write the kill criteria. Write the production data path in one paragraph.
If any of those lines stay blank after one working session, you have your answer. Stop the next quarter of demo spend. Keep the lesson. Move the budget to a use case that can own an SLA.
Your AI pilot is not a product. Treat it like a test with an end, or it will own your calendar instead.