14. AI Application Mental Model¶

The final foundation is to stop thinking about an AI application as "a prompt connected to a model" and instead see it as a normal software system that contains one or more probabilistic components.

The LLM is important, but it is not the application.

The core architecture¶

User / Client
      ↓
Application API
      ↓
Orchestration / Application Logic
      ↓
+----------------+----------------+----------------+
|                |                |                |
LLM          Retrieval         Tools / APIs     Memory / State
|                |                |                |
+----------------+----------------+----------------+
      ↓
Validation / Policy / Business Rules
      ↓
Response / Side Effect

The application decides:

what the model sees
which model is used
which tools are available
what data can be retrieved
what state is persisted
what output shape is accepted
what actions are authorized
when the process should stop
how failures are handled

Probabilistic core, deterministic shell¶

A useful mental model is:

Deterministic application shell
        ↓
Probabilistic AI capability
        ↓
Deterministic validation / execution

The LLM is strong at:

understanding natural language
classification
extraction
synthesis
ambiguity resolution
planning
generating alternatives
choosing among tools

Deterministic code is stronger at:

authorization
arithmetic that must be exact
persistence
constraints
transaction boundaries
idempotency
schema validation
rate limits
workflow invariants

The architecture should exploit both.

End-to-end request lifecycle¶

Consider a support assistant answering:

"Can you refund the headphones I bought last week?"

A production-style flow might be:

1. HTTP request
        ↓
2. Authentication
        ↓
3. Load conversation / relevant state
        ↓
4. Retrieve authorized customer/order context
        ↓
5. Build model context
        ↓
6. Call LLM
        ↓
7. Model requests get_order(order_id)
        ↓
8. Application validates tool arguments
        ↓
9. Application authorizes the read
        ↓
10. Execute tool
        ↓
11. Return result to model
        ↓
12. Model requests create_refund(...)
        ↓
13. Application checks refund policy
        ↓
14. Execute side effect or require approval
        ↓
15. Return result to model
        ↓
16. Generate final user response
        ↓
17. Log metrics / evaluation data

There may be several model calls, but the application owns the workflow.

The main components¶

1. Interface layer¶

Examples:

REST API
chat UI
CLI
background job
webhook

This is normal application infrastructure. It handles transport, authentication, request IDs, limits, and response delivery.

2. Application / orchestration layer¶

This layer coordinates the use case.

It decides things such as:

Which model?
Which prompt version?
Which context?
Which tools?
Which retrieval scope?
How many loop iterations?
What timeout?
What fallback?

This is where agentic behavior eventually lives.

3. Model gateway¶

Instead of spreading provider-specific calls throughout the codebase, a model gateway can centralize:

provider/model selection
request configuration
retries
timeouts
usage metrics
logging
fallback
structured output handling

Example abstraction:

Application
    ↓
ModelGateway
    ↓
OpenAI / Anthropic / local model / future provider

The goal is not to over-abstract immediately, but to prevent provider mechanics from becoming domain logic.

4. Prompt / instruction layer¶

Prompts are application artifacts.

Treat them similarly to code or configuration:

version them
review changes
test them
evaluate regressions
separate stable instructions from runtime data

Prompt engineering sits inside the architecture; it is not the whole architecture.

5. Context builder¶

The context builder assembles only the information needed for the current model call.

Possible inputs:

system instructions
conversation summary
current request
retrieved documents
application state
tool results
memory

This is where context engineering becomes executable application logic.

6. Retrieval layer¶

Retrieval gives the application access to external knowledge.

Possible sources:

SQL database
vector database
search engine
document store
repository
internal API

Important distinction:

retrieval finds candidate information
LLM interprets / synthesizes it

Retrieval is not memory and it is not reasoning.

7. Tool layer¶

Tools expose controlled capabilities to the model.

Examples:

search_orders
create_ticket
run_tests
read_repository_file
get_weather
calculate_price

The tool layer should handle:

schema
validation
authorization
execution
errors
auditing

The model requests the capability; application code owns the capability.

8. State and memory¶

State is information needed to continue a workflow.

Examples:

current step
completed tool calls
approval state
conversation summary
pending action

Memory is persisted information intended to influence later interactions.

They are related but not identical.

A long-running agent needs explicit state management; otherwise important state exists only implicitly inside a growing prompt.

9. Validation and policy¶

This is the deterministic boundary around probabilistic behavior.

Examples:

schema validation
business constraints
authorization
tenant isolation
safety rules
transaction limits

This layer often determines whether an AI demo becomes a production system.

10. Observability¶

Traditional metrics are still needed:

latency
error rate
throughput
CPU/memory where relevant

AI-specific observability adds:

model
model version
prompt version
token usage
cost
tool calls
retrieval results
number of loop iterations
evaluation scores

A request should ideally be traceable across the whole AI workflow.

Separate domain logic from AI integration¶

Bad architecture:

Controller
  ↓
500-line prompt
  ↓
LLM
  ↓
parse random text
  ↓
update database

Better:

Controller
  ↓
Application Use Case
  ↓
AI Service / Orchestrator
  ├── Model Gateway
  ├── Context Builder
  ├── Retrieval
  └── Tools
  ↓
Validated result / domain command
  ↓
Domain / Infrastructure

This keeps the AI-specific pieces replaceable and testable.

LLM output DTO vs domain model¶

The model-facing contract does not have to become your domain model.

Example AI output:

{
  "intent": "refund_request",
  "order_id": "A123",
  "reason": "damaged"
}

Application mapping:

LLM output DTO
      ↓
validation
      ↓
RefundRequestCommand
      ↓
Refund domain service

This prevents the LLM contract from leaking through the whole system.

Retrieval, tools, and model knowledge answer different questions¶

A useful separation:

Model knowledge:
"What do I generally know?"

Retrieval:
"What relevant information exists right now?"

Tool:
"What action or authoritative query can the application execute?"

Example:

User asks:

"Where is my package?"

The model's training knowledge is irrelevant to the current package state.

Correct flow:

LLM understands intent
        ↓
get_shipment_status(order_id)
        ↓
authoritative logistics API
        ↓
LLM explains result

Not every AI application needs an agent¶

Many useful systems are a single controlled pipeline:

input
  ↓
retrieval
  ↓
one model call
  ↓
structured output
  ↓
validation

Use an agentic loop only when the task genuinely needs iterative decisions.

A more complex agent might be:

observe
  ↓
reason / choose action
  ↓
tool
  ↓
observe result
  ↓
repeat until stop condition

The foundations developed so far are what make that loop safe and reliable.

Workflow vs agent¶

A deterministic workflow defines the path in code:

A → B → C → D

An agent decides the next step dynamically:

state
  ↓
LLM chooses A / B / C
  ↓
new state
  ↓
choose again

Prefer deterministic workflows when the sequence is known.

Use agentic decision-making where flexibility provides real value.

This avoids building agents merely because they are fashionable.

Example: document analysis application¶

A non-agentic architecture:

Upload PDF
   ↓
Document parser
   ↓
Chunking
   ↓
Embedding/indexing
   ↓
User question
   ↓
Retrieval
   ↓
Context builder
   ↓
LLM
   ↓
Structured answer + citations
   ↓
UI

The LLM is only one stage of the pipeline.

Example: coding agent¶

A more agentic architecture:

User task
   ↓
Repository context
   ↓
Agent state
   ↓
LLM chooses action
   ├── read file
   ├── search code
   ├── edit file
   └── run tests
   ↓
Tool result
   ↓
state update
   ↓
next decision
   ↓
stop condition
   ↓
final summary

Around this loop the application still needs:

permissions
sandbox
maximum iterations
timeout
test validation
git boundaries
logging

Failure is part of the architecture¶

Plan for:

model timeout
provider outage
malformed output
retrieval returns nothing
tool failure
rate limit
context overflow
loop exceeds budget
low-confidence result

Possible responses:

retry
fallback model
fallback deterministic path
ask user for missing information
human escalation
partial response
stop safely

Production AI design is largely about controlled failure modes.

Cost and latency are architectural constraints¶

A design with one model call is very different from one with 20 sequential agent steps.

Request latency
≈
retrieval
+ model calls
+ tool calls
+ retries
+ queueing

Request cost
≈
Σ model token cost
+ retrieval/reranking cost
+ tool/infrastructure cost

This is why model selection, context size, caching, and stop conditions belong to architecture.

Evaluation closes the loop¶

Architecture should support evaluation from the beginning.

Production request
      ↓
trace
      ↓
input + retrieval + model + tools + output
      ↓
evaluation
      ↓
feedback / regression dataset
      ↓
next version

Without evaluation, prompt and model changes become guesswork.

A practical reference architecture¶

                    ┌─────────────────────┐
                    │    Client / UI      │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Application API   │
                    │ auth / rate limits  │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ Orchestrator / Use  │
                    │       Case          │
                    └─────┬────┬────┬─────┘
                          │    │    │
              ┌───────────┘    │    └───────────┐
              │                │                │
      ┌───────▼───────┐ ┌─────▼─────┐ ┌────────▼────────┐
      │ Model Gateway │ │ Retrieval │ │ Tool Registry   │
      └───────┬───────┘ └─────┬─────┘ └────────┬────────┘
              │               │                │
              ▼               ▼                ▼
            LLMs        Knowledge Stores    APIs / DB / OS
              │               │                │
              └───────────────┴────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │ Validation/Policy │
                    └─────────┬─────────┘
                              │
                    ┌─────────▼─────────┐
                    │ Domain / Response │
                    └───────────────────┘

Cross-cutting:
- state / memory
- tracing / metrics
- evaluation
- cost controls
- security

How the previous foundation topics fit together¶

01 LLM Mental Model
        ↓
understand the probabilistic component

02 Tokens & Context
03 Inputs & Structured Outputs
04 Prompt Engineering
05 Context Engineering
        ↓
control communication with the model

06 Sampling
07 Model Selection
08 Embeddings
        ↓
choose and supply the right AI capabilities

09 Tool Calling
10 Reliability
        ↓
connect AI decisions to real software safely

11 Evaluation
12 Cost & Latency
        ↓
operate and improve the system

13 Safety & Trust Boundaries
        ↓
protect data and side effects

14 Application Mental Model
        ↓
assemble everything into architecture

Key design principles¶

The LLM is a component, not the application.
Keep probabilistic decisions inside deterministic boundaries.
Use structured contracts where software consumes model output.
Build context deliberately rather than sending everything.
Retrieve current/private knowledge instead of expecting the model to know it.
Put real-world capabilities behind narrow tools.
Enforce authorization outside the model.
Prefer deterministic workflows when the path is known.
Use agentic loops only where dynamic decisions add value.
Treat evaluation, observability, cost, latency, and failure handling as architecture from the start.

Where to go next¶

With these foundations established, deeper topics can be studied without treating them as isolated buzzwords.

Natural next areas are:

RAG and retrieval architecture
MCP
AI / agent skills
agentic loops
agent state and memory
workflow engines and durable execution
single-agent and multi-agent architecture
production evaluation and observability

The foundation goal is complete when these concepts are understood as parts of one software system rather than separate AI features.