A support agent asks your bot: “What is our current refund window for annual plans?”

The bot has a 200k-token context window. So your pipeline does what most pipelines do now. It dumps in the last 40 chat messages. It embeds the question and retrieves the top 20 documents. It pastes the tool output from the billing API. It adds the user’s profile, their timezone, their plan tier, and a block of “brand voice” guidance. The prompt is 18,000 tokens. The answer comes back: “Our refund window is 14 days.”

That’s wrong. The 14-day rule was retired in 2023. The current policy says 30 days. But the stale document and the fresh one both went in, the model picked the shorter number, and nobody noticed because the prompt was too big to read.

This is the part nobody tells you when they brag about context size. A bigger window is not a better window. It’s a bigger bucket you have to fill responsibly, or you end up paying more to get worse answers.

Bigger context is not better context

When context windows were 4k tokens, we were forced to be careful. We picked the few things that mattered. Now that they’re 100k and up, teams treat the window like a junk drawer. Everything goes in: full chat history, retrieved documents, tool output, user metadata, preference blobs.

Three things break when you do that.

Noise. The model doesn’t magically know what’s important. When relevant content competes with irrelevant content for attention, the irrelevant stuff wins more often than you’d like. A 2024 study from Stanford and UC Berkeley (Liu et al., “Lost in the Middle”) showed that models often ignore information placed in the middle of a long context, even when it’s the only correct answer. So the more you stuff in, the more you risk burying the one sentence that matters.

Cost and latency. Tokens cost money on the way in and on the way out, and they slow everything down. Sending 18,000 tokens when 740 would do is not free. On a busy support bot, that difference is real money every month.

Hidden conflict. Two sources that disagree both sit in the prompt, and the model has to guess. You’ve created a fight and given it no referee.

The fix isn’t a better prompt. The fix is a decision about what belongs in the prompt at all. Anthropic put a name on this. They call it context engineering: the practice of curating and maintaining the best set of tokens during inference, not just writing better instructions. That’s a bigger job than prompt writing, and it’s the job this article is about.

My take, stated plainly: the model should receive the smallest useful packet of information, not the largest possible one. “Context selection” is now a first-class engineering task. If you don’t own it, the model owns it by accident, and accidents are rarely in your favor.

Define a “context budget”

Start with a simple rule. Before you build anything, decide how the prompt is allowed to be split. Not in tokens yet. In percentages.

Here’s a budget I’ve used as a starting point for a RAG-style assistant:

Block	Share
System instructions	10%
User request	10%
Retrieved evidence	50%
Conversation memory	15%
Tool output	10%
Safety and policy hints	5%

If your context budget is 4,096 tokens, that gives you about 2,048 tokens for retrieved evidence, 614 for conversation memory, and so on. The numbers are not sacred. A coding agent might give tool output 40% and retrieved evidence 20%. A chat friend might give memory 30%. The point is that you decide on purpose instead of letting retrieved documents eat the whole window by default.

Two things matter about this table.

First, retrieved evidence gets the biggest slice because it’s usually the reason the answer is correct. But it still only gets half. The other half is reserved for the request, the memory, and the guardrails. That reservation is what stops one source type from crowding out the rest.

Second, the safety and policy block gets its own 5% even when nothing is “wrong.” You don’t want the model’s behavior rules competing for scraps at the end of a giant prompt. Give them a fixed seat.

How big should the total budget be? Smaller than you think. I start at one quarter of the model’s real context window and only grow it when I can show the extra tokens earned their place. A 200k window doesn’t mean you should plan for 200k of input. It means you can afford to be wrong about your estimate without falling over. The budget is a target, not a ceiling you sprint toward. If you find yourself needing 80% of the window just to answer a support question, the problem is usually your retrieval, not your budget.

Build a context priority model

Percentages tell you how much room each block gets. They don’t tell you which chunk inside a block earns a seat. For that you need a score.

When your retriever returns 20 documents, you don’t send 20. You rank them. I score each chunk on six signals:

• Relevance to the current user request.
• Recency (how old is the content).
• Source trust level (official policy doc vs. a forum post).
• User-specific importance (does this user’s plan tier make it matter?).
• Conflict risk (does it disagree with something else you’re sending?).
• Token cost (how much room does it eat?).

And evidence freshness deserves its own line. A price list from last month is not the same as one from two years ago, and the model can’t tell them apart unless you weight it.

Here’s the scoring formula I use. It’s a weighted sum, and the weights are the opinion part:

final_score =
    relevance     * 0.40
  + trust         * 0.20
  + freshness     * 0.15
  + user_need     * 0.15
  - conflict_risk * 0.10

Relevance leads because if a chunk doesn’t answer the question, nothing else saves it. Trust and freshness come next because a confident, recent source beats a vague, old one. User need nudges in the specifics of this request. Conflict risk is a penalty: a chunk that fights with another source should cost you, not help you.

Notice what’s missing. Token cost isn’t in the score directly. That’s on purpose. Cost shows up later, when you pack to the budget. A chunk can score high and still get cut because there wasn’t room. That’s a different decision from “this chunk is low quality,” and it’s worth keeping the two separate so your debug log stays honest.

The weights are a starting point, not a religion. If your app keeps misciting stale data, push freshness up. If it ignores the user’s actual plan, push user_need up. Tune it like you’d tune any other model, because that’s what it is.

Add context tiers

Scores are great for ranking, but some content should never lose a popularity contest. System rules and hard constraints don’t get scored. They get forced in. That’s where tiers help.

• Tier 0 — system rules and hard constraints. Always in. “Never reveal the customer’s full card number.”
• Tier 1 — the direct user request. Always in. It’s the whole point.
• Tier 2 — retrieved evidence. Ranked and budgeted.
• Tier 3 — short-term conversation memory. The last few turns.
• Tier 4 — long-term profile or preferences. Plan tier, locale, units.
• Tier 5 — optional examples or style hints. Cut first when room runs low.

The trick is that tiers and scores work together. Tiers 0 and 1 bypass the scorer. Tier 2 is where the scorer does its job. Tiers 3 through 5 are filled in what’s left, in order, until the budget is gone.

This is also why “just give the model more context” fails. Without tiers, your hard constraint and a random style hint are competing for the same attention, and the model has no idea which one wins. Tiers make the ranking explicit.

Trim before summarizing

Here’s a mistake I see constantly. Teams retrieve 20 chunks, immediately summarize all 20 into one paragraph, and send that. They call it “compression.” It’s actually lossy flattening of noise.

Summarizing too early destroys information you might have wanted and preserves information you didn’t. You take 20 chunks, including 4 that are stale and 3 that are off-topic, and you boil them into a summary that now contains a averaged, mushy version of the bad parts.

Do it in this order instead:

1. Remove low-value content first. Drop the stale, the off-topic, the duplicated.
2. Rank what’s left by score.
3. Pack to the budget by score.
4. Compress only the survivors, and only if they still don’t fit.

You trim with a scalpel, then you compress the few things worth keeping. By the time you summarize, you’re summarizing good material, so the summary is good too. The order matters more than the summarizer you pick.

One honest caveat: compression isn’t free. A summary can drop a number, soften a “never,” or merge two sources into one claim that neither made. If the chunk is a hard rule, don’t summarize it. Keep the original text. Tiers 0 and 1 should almost never be compressed.

Detect context conflicts

The worst failures aren’t missing data. They’re two pieces of data that disagree, both sitting in the prompt, with the model picking one and you not knowing which.

A few conflicts I’ve actually seen:

• Two prices. A 2023 price list says $9.99. A 2026 list says $12.99. Both retrieved. The bot quotes the old one.
• Old policy vs new policy. The refund window changed from 14 to 30 days. Both docs are in. The answer is a coin flip.
• Units mismatch. The user says “use metric.” The profile says “use imperial.” The answer mixes them.
• Tool vs text. The billing API returns “active.” A retrieved doc says “canceled pending.” The model trusts the doc.

The budget layer should catch these, not the model. Two cheap moves help.

First, when two chunks share a topic and disagree, raise the conflict_risk on both. That lowers their score, which pushes the fight lower in the ranking where you can see it. Second, keep a “conflict log” in your debug output: “chunk A and chunk B disagree on refund_window.” Then either pick the higher-trust source and drop the other, or surface both to the user with the source ids so a human decides.

I’d rather the bot say “the docs disagree, here are both” than guess. Guessing quietly is how you ship a wrong refund policy to 10,000 customers.

Add context observability

If you can’t see what went into the prompt, you can’t debug the answer. And in a GenAI system, “the answer was wrong” almost always means “the wrong context went in.” So log the context the same way you’d log a database query.

At minimum, record:

• What was included — the source id and the score for each chunk.
• What was excluded — and the reason (stale, low relevance, over budget, conflict).
• Token count per block — system, request, evidence, memory, tool, safety.
• Source IDs for everything that made it in.
• Relevance scores so you can trace a bad answer to a bad ranking.
• Citation coverage — did the final answer actually cite the sources you sent?

This last one connects to risk, not just debugging. NIST published a Generative AI profile (NIST AI 600-1) that lists governance, pre-deployment testing, provenance, and incident disclosure as the core things you need to manage GenAI risk. A context log is how you make provenance real. When an answer is wrong, you can show exactly which sources backed it, which were dropped, and why. That’s the difference between “the AI hallucinated” and “we sent it a stale doc and our tiering didn’t catch it.” One of those is fixable in code.

None of this needs a fancy platform. A JSON line per request is enough to start. The shape matters more than the storage.

A working Python example

The whole idea fits in about 120 lines. Here’s the core: a list of retrieved chunks, a scoring function, a token budget, a packing function, and a debug report that explains each decision. The full, runnable version (with scorer.py, packer.py, main.py, and sample_chunks.json) is in the linked repo.

The scorer is just the weighted formula:

WEIGHTS = {
    "relevance": 0.40,
    "trust": 0.20,
    "freshness": 0.15,
    "user_need": 0.15,
    "conflict_risk": 0.10,  # applied as a penalty
}

def score_chunk(chunk: dict) -> float:
    score = (
        chunk["relevance"] * WEIGHTS["relevance"]
        + chunk["trust"] * WEIGHTS["trust"]
        + chunk["freshness"] * WEIGHTS["freshness"]
        + chunk["user_need"] * WEIGHTS["user_need"]
        - chunk["conflict_risk"] * WEIGHTS["conflict_risk"]
    )
    return round(score, 2)

The packer drops stale and low-relevance chunks first, then fills the budget by score:

def pack(chunks, budget=EVIDENCE_BUDGET):
    scored = [{**c, "score": score_chunk(c)} for c in chunks]
    evidence = [c for c in scored if c.get("tier", 2) >= 2]

    included, excluded = [], []
    used = 0
    for c in sorted(evidence, key=lambda x: x["score"], reverse=True):
        if c["freshness"] < FRESHNESS_FLOOR or c["relevance"] < RELEVANCE_FLOOR:
            c["reason"] = _exclude_reason(c)
            excluded.append(c)
        elif used + c["tokens"] <= budget:
            included.append(c)
            used += c["tokens"]
        else:
            c["reason"] = "over budget"
            excluded.append(c)
    return included, excluded, used

Run it against the four sample chunks and the debug report reads like this:

Context budget: 740/2000 tokens used

Included:
- pricing_policy_2026.md, score 0.83, 430 tokens
- refund_rules.md, score 0.78, 310 tokens

Excluded:
- brand_voice.md, score 0.42, reason: low relevance
- pricing_policy_2023.md, score 0.39, reason: stale

The stale 2023 policy never reaches the model. Neither does the brand-voice hint, which is Tier 5 and just didn’t earn its place. The fresh 2026 policy and the refund rules go in. That’s the whole point in four lines: the model gets the smallest useful packet, and you can prove why.

Where teams get this wrong

A few habits I’d drop:

• Treating the context window as infinite. It isn’t, even at 200k. And “infinite” doesn’t mean “free” or “safe.”
• Summarizing before trimming. You compress the noise along with the signal. Trim first.
• No observability. If you can’t see what went in, every wrong answer is a mystery. Log it.
• Scoring everything equally. System rules and hard constraints shouldn’t lose a ranking contest. Use tiers.
• Letting one block eat the window. Give each block a fixed share. Retrieved evidence gets the most, not all.

Context budgets aren’t glamorous. They’re plumbing. But plumbing is what keeps the water clean. The teams shipping reliable GenAI apps in 2026 are the ones treating context as a thing they design, rank, trim, and audit — not a bucket they fill because the window got big.