Shipping AI Features in Production: 12 Months of Lessons

Why "AI" is the easy part

Most teams discover quickly that the model is the easy part. The hard part is everything around it, eval pipelines, caching, prompt versioning, observability, fallback strategies, and the cost curves that surprise you in week three.

This post is a frank look at what we learned building three production-grade AI features for clients between Q2 2025 and Q1 2026.

The three features

1. Smart triage, a customer-support classifier routing 14k tickets/day
2. Document Q&A, RAG over legal contracts for a fintech client
3. Synthesis, auto-summarisation that ships drafted emails to operators

Each one taught us a different lesson.

Lesson 1, Latency budgets are product decisions

A 1.4-second average response felt fast in isolation, but our P95 was 4.8 seconds, and the P99 was over 12. Operators started bypassing the AI entirely.

The fix wasn't a faster model, it was a streaming UI + a tighter prompt + a small-model fallback for the 80% case.

"If the AI can't answer in two seconds, it's not an AI feature, it's a chore." , Engineering lead on the triage team

Lesson 2, Evals are the spec

We learned to treat the evaluation set as the source of truth for what the feature does. Every prompt change ran:

results = run_eval_suite(
    suite="triage_v3",
    sample_size=500,
    judges=["accuracy", "tone", "no_hallucination"],
)

A change that didn't move the metrics didn't ship.

Lesson 3, Cost is a feature

Token costs ballooned from ~$280/day to $11k/day in one quarter. The fix was a four-step pipeline:

• Cheap classifier first
• Cache embeddings + retrieval results
• Stream early-exit on confident outputs
• Reserve frontier models for the long tail

We brought it back to $1.9k/day with zero quality regression.

Where we go next

The next frontier is agents that act, not just answer. We're prototyping tool-using agents for back-office workflows, early signs are promising, but the eval surface is enormous.

If you're building something similar, we'd love to compare notes.