Shipping AI Features in Production: 12 Months of Lessons

Why "AI" is the easy part

Most teams discover quickly that the model is the easy part. The hard part is everything around it — eval pipelines, caching, prompt versioning, observability, fallback strategies, and the cost curves that surprise you in week three.

This post is a frank look at what we learned building three production-grade AI features for clients between Q2 2025 and Q1 2026.

The three features

1. Smart triage — a customer-support classifier routing 14k tickets/day
2. Document Q&A — RAG over legal contracts for a fintech client
3. Synthesis — auto-summarisation that ships drafted emails to operators

Each one taught us a different lesson.

Lesson 1 — Latency budgets are product decisions

A 1.4-second average response felt fast in isolation, but our P95 was 4.8 seconds, and the P99 was over 12. Operators started bypassing the AI entirely.

The fix wasn't a faster model — it was a streaming UI + a tighter prompt + a small-model fallback for the 80% case.

"If the AI can't answer in two seconds, it's not an AI feature, it's a chore." — Engineering lead on the triage team

Lesson 2 — Evals are the spec

We learned to treat the evaluation set as the source of truth for what the feature does. Every prompt change ran:

results = run_eval_suite(
    suite="triage_v3",
    sample_size=500,
    judges=["accuracy", "tone", "no_hallucination"],
)

A change that didn't move the metrics didn't ship.

Lesson 3 — Cost is a feature

Token costs ballooned from ~$280/day to $11k/day in one quarter. The fix was a four-step pipeline:

• Cheap classifier first
• Cache embeddings + retrieval results
• Stream early-exit on confident outputs
• Reserve frontier models for the long tail

We brought it back to $1.9k/day with zero quality regression.

Where we go next

The next frontier is agents that act, not just answer. We're prototyping tool-using agents for back-office workflows — early signs are promising, but the eval surface is enormous.

If you're building something similar, we'd love to compare notes.