Context Engineering Replaced Prompt Engineering

For a couple of years the folk wisdom was that you got better results by wording the prompt more cleverly. Add 'you are an expert', promise a tip, threaten the model. Some of it did work on weaker models. Almost none of it matters now. On a current frontier model, rephrasing your instruction is worth a rounding error compared to changing what information the model can actually see.

The real discipline is context engineering: deciding, for every single request, which tokens occupy the window. Not the most tokens you can fit, but the right ones. That reframing changes how you build the whole feature.

More context is not better context

The naive move once you have a large window is to fill it. Dump the whole document, the last hundred messages, every row the query returned. This reliably makes output worse, for two reasons that compound.

• Attention dilution: the relevant tokens now compete with thousands of irrelevant ones, and retrieval accuracy in the middle of a long context degrades measurably.
• Contradiction: large dumps almost always contain stale or conflicting statements, and the model has no reliable way to know which version you consider current.
• Cost and latency scale with every token, on every call, forever. A padded prompt is a permanent tax, not a one-off.

I treat the context window as a curated brief for a competent contractor who has no memory of yesterday. You would not hand them the entire company wiki. You would hand them the three documents that bear on today's job, plus the constraints they must not violate.

Structure beats prose

Models parse delimited structure far more reliably than they parse flowing paragraphs. When a request carries several distinct kinds of information, label them explicitly instead of blending them into narrative.

<task>
  Classify the ticket and draft one reply.
</task>

<customer_record>
  plan: enterprise
  open_tickets: 2
  renewal_date: 2026-11-01
</customer_record>

<ticket>
  Billing charged us twice for October.
</ticket>

<constraints>
  - Never promise a refund; escalate billing disputes to #finance.
  - Reply in the customer's language.
</constraints>

This version is not more eloquent than a paragraph saying the same thing. It is more reliable, because the boundaries are unambiguous and because you can now change one section programmatically without rewriting a sentence.

Compaction is a product decision

Any long-running session eventually exceeds the window. What you do at that moment is not an implementation detail. It determines what your product forgets. Truncating the oldest turns is the default and usually the worst option, because the earliest messages are often where the user stated their actual goal and constraints.

A better pattern is tiered retention: pin the durable facts (the goal, the constraints, decisions already made) into a compact running summary that never gets evicted, and let the verbose middle age out: tool output, superseded drafts, exploratory tangents. The summary is cheap. The transcript is not.

Ask what the model must still know in twenty turns. That set is small, and it is almost never the most recent twenty messages.

What I check when quality drops

When an LLM feature starts degrading, my first instinct is no longer to rewrite the instruction. It is to log the exact, fully-assembled context of a failing request and read it as if I were the model. The bug is nearly always visible there: a retrieval step that returned nothing and failed silently, a stale cached document, two instructions that flatly contradict, or a tool result formatted so poorly it reads as noise.

• Is every block in this window actually load-bearing for this request?
• Is anything here contradicted by something else here?
• Did a retrieval or tool step fail quietly and leave an empty section?
• If I removed the largest block, would the answer get worse, and have I actually tested that?

That last question is the one people skip. I have deleted more context than I have added in the last year, and quality went up almost every time.