02 – Tokens and Context Window¶

An LLM does not process characters or words directly in the traditional sense. Text is first broken into tokens, and the model operates on those tokens.

Understanding tokens matters because context size, cost, and often latency are tied to token usage.

What is a token?¶

A token can be:

a whole word;
part of a word;
punctuation;
a whitespace pattern;
a code fragment.

For example, a simple sentence does not necessarily tokenize as one word = one token.

"authentication failed"

One tokenizer may split it into a few tokens, another into more. It depends on the language and the model.

Therefore, the exact token count is always determined by the concrete tokenizer. Rough mental estimates are useful, but in system design the real token count is what matters.

What goes into the context?¶

Context is not only the user's current question.

A real AI application may include:

system instructions
+ developer/application instructions
+ conversation history
+ user question
+ retrieved documents
+ tool results
+ examples
+ generated reasoning/state depending on runtime
= effective context

The model works from this information during a specific call.

Context window¶

The context window is the maximum number of tokens a model can handle in a run. Depending on the provider and model implementation, both input and generated output typically consume that budget in some form.

A useful practical model is:

max context
- system instructions
- conversation
- retrieved data
- tool results
- reserved output
= usable context budget

A large context window does not mean that filling it is always a good idea.

More context is not automatically better¶

This is one of the most important points.

Suppose three documents are relevant to a question, but we provide 400 files.

3 relevant documents
+ 397 irrelevant documents

It may technically fit, but:

it is more expensive;
it can be slower;
important information gets buried in noise;
the chance of conflicting or stale information increases;
it is harder for the model to know what to focus on.

So the right question is not:

How much context can we fit?

but:

What is the smallest relevant context from which the task can be solved well?

Context pollution¶

Context pollution occurs when the model receives information that does not help the current decision or actively makes it worse.

Example for a coding agent:

Task: modify payment retry logic

Useful context:

retry service;
its interface;
relevant tests;
coding rules.

Less useful context:

the entire frontend;
migrations unused for five years;
unrelated READMEs;
every repository file just because it fits.

Lost in the middle¶

With long contexts, models may not use every part of the context equally well. In practice, simply dumping a huge document set into the context and hoping the model always finds the critical sentence is not a robust strategy.

A better strategy is:

retrieve relevant chunks
        ↓
rank / filter
        ↓
compact context
        ↓
LLM

This becomes a foundation for RAG and context engineering later.

Thinking in terms of a context budget¶

It is useful to treat context as a system resource, similar to CPU or memory.

Example:

Total budget: 100%

15% system/application instructions
20% recent conversation
45% retrieved knowledge
5% tool results
15% reserved output

These are not recommended percentages; they only illustrate that context can be allocated deliberately.

Conversation history: should everything be carried forward?¶

Not necessarily.

Three common strategies for a long conversation are:

1. Full history¶

Simple, but increasingly expensive and noisy.

message 1
message 2
...
message 300
current question

2. Sliding window¶

Only the latest N messages are included.

last 10 messages
+ current question

The benefit is simplicity. The downside is that an important older decision may disappear.

3. Summary + relevant memory¶

conversation summary
+ relevant durable facts
+ recent messages
+ current question

This is generally more scalable.

Example: knowledge base as a context source¶

For a repository-backed knowledge base, we do not want to provide the entire repository in every new session.

A better flow is:

START_HERE.md
      ↓
knowledge/status.md
      ↓
AI status
      ↓
relevant foundation topic
      ↓
LLM

The structure itself helps select context.

Example: RAG question¶

User:

“What is the refund policy for enterprise customers?”

Weak approach:

all company documents → LLM

Better approach:

query
  ↓
retrieval
  ↓
relevant enterprise refund policy sections
  ↓
LLM

The model now works from a smaller, cleaner, and more relevant context.

Input and output both consume budget¶

If the model needs to generate a long response, room must be reserved for it.