Practice · 2024–now · AI Present
Long-context cargo cult
Million-token windows as a substitute for retrieval design. Stuffing PDFs until the model shrugs.
Why Long-context cargo cult became a costly fad
Long-context cargo cult failed because attention is not free and relevance still decays. Bigger windows help when used deliberately; they fail when they excuse skipping evals and chunk quality. What sticks is measured context budgets; what fades is context as a silver bullet.
Cost of the fad
What Long-context cargo cult cost
"Just put it in the window" replaced chunking, RAG, and information architecture until token bills and lost-in-the-middle hallucinations arrived.
Patterns
Compare with
Related
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem — then "just stuff the window" as the counter-fad. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.
Practice · 2023–now
Embed everything
If it is text, vectorize it. Semantic search cosplay for problems that needed a better filter.
$ Embedding pipelines for tickets, PDFs, Slack, and the cafeteria menu — then nobody measured retrieval quality. Vector bills and reindex jobs became the product.
Practice · 2023–now
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
Practice · 2024–now
Reasoning models for grep tasks
o1-class deliberation for tickets that needed a filter. Resume-driven inference.
$ Thinking tokens burned on problems a SQL query and a unit test would settle. Latency and invoices grew; correctness did not.