The pitch and the reality
Model vendors keep racing the window out to 200K, 1M, 10M. It looks great: throw all your material in at once and stop thinking about it.
In production, a long context is not a free lunch.
Four real costs
1. The “lost in the middle” effect
Research keeps confirming it: models are most sensitive to what sits at the beginning and the end of the context, and the middle gets “compressed.”
Put the critical line in paragraph 4 of page 80 and the model may look right past it.
2. Price scales linearly with tokens
Fill a 1M window and a single call can cost 100x a normal one. That number belongs in your P&L.
3. Latency rises with window length
Both time to first token (TTFT) and total response time go up. Users feel it, especially when you’re streaming.
4. Cache hit rate drops
If you use prompt caching, stuffing a different long document into the window every time breaks the cache. Hits are the entire reason prompt caching saves you money.
What to do instead
The goal isn’t the longest window, it’s the right length for the job:
| Knowledge size | Recommended strategy |
|---|---|
| < 10K tokens | Put all of it in the context |
| 10K – 200K | Long context plus prompt caching |
| > 200K | RAG, chunked processing, or summarize then inject |
Long context is a tool, not an answer. Next time you hear “the context isn’t big enough,” ask first: is it really too small, or is the context engineering just not good enough?