01 – LLM Mental Model¶
The first important goal is not to understand transformer architecture in mathematical detail, but to build a correct mental model of what an LLM does and does not do. This matters because a wrong mental model leads to bad system architecture.
What does an LLM do at runtime?¶
At a very simplified level, an LLM estimates the probability of the next token from the context it receives, then builds the response step by step.
input/context
↓
model
↓
next-token probabilities
↓
selected token
↓
new context
↓
next token...
This does not mean an LLM is merely “autocomplete”. During training, modern models learn very complex language, reasoning, programming, and problem-solving patterns. The important distinction is that at runtime they are not executing a classic deterministic program, and they are not simply reading the answer from a database.
Training knowledge vs runtime context¶
It is useful to separate two things.
Training knowledge: patterns and relationships learned into the model parameters during training.
Runtime context: information the model actually receives for the current request: system instructions, user messages, previous messages, documents, tool results, and so on.
Example:
During training:
- the model may have learned what a Kubernetes Deployment looks like
- it knows the general structure of YAML
At runtime:
- it only knows that your current deployment has replicas=7
if that information is provided in the context or through a tool
This is why the assumption that “the model surely knows” is dangerous. It may understand the topic, while still not knowing the current state.
An LLM is fundamentally stateless¶
A standalone API call does not automatically remember a previous call. If we want an ongoing conversation, the application has to provide the relevant history again.
request #1
User: The project is called Atlas.
request #2
User: What is the project called?
If the second request does not include the first message, that information does not exist for the model.
“Memory” is therefore usually not a magical LLM feature, but an application-level mechanism:
conversation / DB / files / memory store
↓
relevant information
↓
LLM
This is also why a repository-backed knowledge base is useful: in a new session, the repository can restore durable state.
Why can an LLM hallucinate?¶
The primary task of an LLM is to generate a coherent and probable continuation, not to guarantee retrieval of a fact record from an internal database.
If the context is incomplete, the model may still generate a plausible answer even when it does not have reliable information.
Example:
Question:
“What timeout did we configure yesterday for the payment service?”
If this is not present in the context, the model has several common options:
- say that it does not know;
- infer from general patterns;
- in the bad case, confidently invent a value.
A production system should therefore not rely on “the model probably will not make something up”. Instead, design an architecture where the source of truth is available:
LLM → tool → configuration service → actual timeout
Reasoning is not the same as a deterministic algorithm¶
An LLM can be very good at analyzing complex problems, but that does not make its result guaranteed in the same way as a deterministic function.
Example:
calculate_tax(invoice)
If the tax rule can be expressed clearly in program logic, there is little reason to ask the LLM to calculate it every time.
A better split is:
LLM:
- interpret the natural-language request
- decide which operation is needed
Application code:
- calculate the tax
- validate the data
- execute the transaction
General rule:
If something can be calculated or verified deterministically, prefer having the software guarantee it instead of relying on the prompt.
Model capability vs application capability¶
The capability of the model and the capability of the application are different things.
An LLM by itself does not necessarily know or cannot necessarily do things such as:
- what is in your database today;
- which GitHub issue was opened five minutes ago;
- whether the user is authorized to delete a record;
- actually send an email;
- guarantee that a business rule is enforced.
The application adds those capabilities through tools and surrounding components.
AI application
|
+----------------+----------------+
| | |
LLM Tools deterministic code
| | |
reasoning/text external world rules/validation
This is one of the most important AI engineering principles: we are not “building an LLM”; we are building a system around an LLM.
Example: support assistant¶
Suppose a support assistant has to answer:
“Why was my invoice rejected yesterday?”
Weak design:
User question → LLM → answer
The model can only list generic causes and may start guessing.
Better design:
User question
↓
LLM identifies: invoice lookup is needed
↓
get_invoice(user_id, date)
↓
actual invoice status + reason code
↓
LLM explains it in human language
Here the fact is supplied by the application/tool, while the LLM interprets and communicates it.
Example: coding assistant¶
A model may know in general what the repository pattern or dependency injection is. But it can reliably follow conventions in a specific repository only if it can see the relevant files and rules.
General model knowledge
+
AGENTS.md / project rules
+
relevant source files
+
current task
↓
better change
This leads directly into the later topic of context engineering.
What should you use an LLM for?¶
LLMs are typically strong where:
- natural language must be understood or generated;
- meaning must be extracted from semi-structured or unstructured input;
- complex context needs to be summarized;
- multiple possible solutions need to be weighed;
- semantic similarity, classification, or extraction is needed;
- human-like communication is useful.
They are weaker as a standalone mechanism where:
- 100% mathematical correctness is required;
- a strict authorization decision must be made;
- a financial transaction must be executed with guaranteed correctness;
- exact current state is required from an external system;
- a deterministic rule is easy to implement in code.
Key takeaways¶
- An LLM is a probabilistic component, not a classic database or deterministic function.
- Training knowledge and current runtime context are different things.
- Durable memory is usually an application responsibility.
- Hallucination is mitigated not only with better prompts, but with better architecture.
- Model capability is not the same as application capability.
- In an AI application, the LLM is one component alongside tools, data, and deterministic code.