Practice · 2024–now · AI Present
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy.
Eval theater failed because measurement without enforcement is theater. Real LLM evals stuck when they lived in CI with owners. The fad was buying an eval product to decorate a demo; the stuck practice is regression tests that can fail the build.
Cost of the fad
Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.
Patterns
Context
The stack gets a co-pilot
AI pair programming is already changing how code is written. Agent frameworks and vector stores are still sorting winners from demos. The durable layer will look familiar: evals, retrieval quality, product UX, and ownership. Autopilot rewrites without tests are just big-bang migrations with better slides.
Compare with
Related
Practice · 2023–now
LLM eval pipelines
Regression tests for nondeterministic models. The unglamorous infrastructure that separates demos from products.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features. Datadog for tokens.
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.