Practice · 2023–now · AI Present
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
Why LLM eval pipelines stuck
LLM evals stuck because shipping prompts without measurement is shipping bugs with confidence intervals. Early adopters treated eyeball checks as QA; by 2026 enforcement in CI is the difference from eval theater. What remains is boring gates for model behavior — and that is the point.
Compare with
Related
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem — then "just stuff the window" as the counter-fad. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.
Framework · 2023–now
LLM app frameworks
LangChain-class glue mutated into MCP clients and thin wrappers around vendor APIs. The durable pieces are still boring: evals, retrieval, and product UX.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features — mostly constrained tool loops, not autonomous agents. Datadog for tokens.
Practice · 2024–now
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy. The foil to real eval pipelines in CI.
$ Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.