Practice · 2023–now · AI Present
LLM eval pipelines
Regression tests for nondeterministic models. The unglamorous infrastructure that separates demos from products.
LLM evals stuck because shipping prompts without measurement is shipping bugs with confidence intervals. The practice failed early adopters who treated eyeball checks as QA. What remains is boring CI for model behavior — and that is the point.
Context
The stack gets a co-pilot
AI pair programming is already changing how code is written. Agent frameworks and vector stores are still sorting winners from demos. The durable layer will look familiar: evals, retrieval quality, product UX, and ownership. Autopilot rewrites without tests are just big-bang migrations with better slides.
Compare with
Related
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.
Framework · 2023–now
LLM app frameworks
Chains, agents, and prompt wrappers shipping weekly. The durable pieces will be boring: evals, retrieval, and product UX.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features. Datadog for tokens.