Evaluation, Observability and Architecture Patterns¶

A production agent architecture is not complete until failures can be reconstructed and changes can be evaluated before and after deployment. Because behavior emerges from model + context + skill + tool + state + policy + environment, observability and evaluation must cover the whole execution path rather than only the final generated text.

A useful trace hierarchy is:

Run
├── Step
│   ├── Context build
│   ├── Model call
│   ├── Policy decision
│   ├── Capability/tool call
│   ├── Retrieval call
│   └── State transition
├── Step
└── Final outcome

The architecture should make this structure observable by design.

What should be observable?¶

For each run, it should be possible to answer:

What goal was requested?
Which runtime/skill/policy/model versions were used?
What state did the run start from?
Which observations/evidence were available?
Which action was proposed?
Was it allowed/denied/approved?
Which tool/capability actually executed?
What result came back?
How did state change?
Why did the run stop?
How much time/cost did it consume?

This does not require storing hidden chain-of-thought. Structured action decisions, inputs/outputs, evidence references and state transitions are more useful and safer operationally.

Trace identifiers¶

Use stable IDs throughout the system:

trace_id
run_id
step_id
model_call_id
capability_call_id
observation_id
approval_id
artifact_id

Propagate them across queues, workers and external adapters when possible.

This allows a log line from an MCP adapter to be connected to the agent step that caused it.

Structured event model¶

Instead of free-form logging only, emit events such as:

RunStarted
ContextBuilt
ModelDecisionReceived
ActionRejectedByPolicy
ActionApproved
CapabilityCallStarted
CapabilityCallSucceeded
ObservationRecorded
RunCheckpointed
RunCompleted
RunFailed

Each event can have typed fields.

Example:

{
  "event": "CapabilityCallSucceeded",
  "run_id": "run_123",
  "step_id": "step_8",
  "capability": "pull_request.create",
  "version": "1",
  "latency_ms": 840,
  "effect": "WRITE",
  "result_status": "SUCCESS"
}

Sensitive-data logging¶

Observability must not become a new data leak.

Avoid blindly logging:

secrets
full customer records
entire prompts
private documents
raw tool responses
access tokens

Prefer references, hashes, redaction and structured summaries where appropriate.

Trace design should include data classification and retention policy.

Model call observability¶

Useful metadata:

provider
model
model profile/version
input token count
output token count
latency
cost estimate
structured-output validity
tool-choice outcome
retry count
fallback used

For debugging, store the exact allowed/sanitized context when policy permits, or enough references to reconstruct it from canonical sources.

Context observability¶

Context quality strongly affects model behavior.

Record:

context strategy version
included source types
evidence IDs
memory IDs
freshness metadata
token allocation per section
items dropped due to budget
trust classifications

This makes questions such as "Did the model miss the policy because retrieval failed, or because the context builder dropped it?" answerable.

Capability/tool observability¶

For each action:

requested capability
validated arguments (redacted as needed)
policy result
approval result
adapter/provider
latency
retry/reconciliation behavior
normalized result/failure
side-effect identifier

Side-effect IDs are especially valuable for reconciling uncertain outcomes.

State-transition observability¶

A run is effectively a state machine, so transitions should be explicit:

RUNNING
→ WAITING_FOR_APPROVAL
→ RUNNING
→ COMPLETED

Record:

old state
new state
state version
trigger event
step/run ID

Unexpected transition attempts should be visible as errors.

Evaluation levels¶

Do not use one monolithic "agent accuracy" score.

Evaluate layers separately.

Model/decision evaluation¶

did it classify correctly?
did it choose an appropriate action?
did structured output match schema?

Skill evaluation¶

did the skill follow its contract?
did it find relevant issues?
did it use allowed capabilities efficiently?

Retrieval evaluation¶

Recall@k
Precision@k
ranking quality
grounding coverage
ACL leakage

Tool/capability evaluation¶

schema correctness
failure normalization
idempotency behavior
authorization enforcement

Loop/trajectory evaluation¶

success rate
steps to completion
unnecessary calls
replans
retries
premature stop
loop explosion

End-to-end outcome evaluation¶

was the user/business goal actually achieved?
was the result correct, safe and useful?

Layered evaluation localizes regressions.

Offline eval pipeline¶

A useful release flow:

Code / prompt / skill change
        ↓
Unit + contract tests
        ↓
Deterministic integration tests
        ↓
Offline agent eval set
        ↓
Safety / policy evals
        ↓
Cost + latency comparison
        ↓
Regression gate
        ↓
Canary / production

Agent changes should be reviewed with the same discipline as code changes.

Eval datasets¶

Use representative cases, including failures and edge cases.

happy paths
ambiguous requests
missing information
conflicting evidence
provider failures
stale observations
prompt injection attempts
permission denial
approval required
long-running/resume cases
budget exhaustion

A dataset made only of clean demo prompts will not predict production reliability.

Deterministic vs model-based graders¶

Prefer deterministic graders when the property is deterministic.

Examples:

JSON schema valid
correct API endpoint selected
forbidden tool not called
file exists
tests pass
no cross-tenant document retrieved
budget not exceeded

Use model-based graders for semantic qualities such as:

response usefulness
reasoning quality summary
review completeness
tone
semantic relevance

Model graders should themselves be calibrated and versioned.

Trajectory evaluation¶

Two agents can produce the same final answer but have very different operational quality.

Agent A:
3 steps, 2 reads, success

Agent B:
18 steps, 9 redundant reads, 3 retries, success

Outcome-only evaluation calls both successful. Trajectory evaluation reveals the architecture problem.

Useful trajectory metrics:

steps per successful run
model calls per success
tool calls per success
repeated-action rate
replan rate
no-progress rate
human escalation rate
cost per success

Production metrics¶

Architecture-level metrics can include:

run success/failure/block rate
p95 run latency
queue wait time
cost per successful run
provider fallback rate
capability error rate
approval rate
retrieval miss rate
budget exhaustion rate
resume-after-failure success
stuck-run count
security/policy denial count

Break them down by:

workload
skill
model profile
runtime version
tenant
capability

without exposing sensitive data.

Regression gates¶

A production change should be blocked if key thresholds regress materially.

Example:

critical-task success >= 95%
forbidden-write rate = 0
schema validity >= 99.9%
p95 cost increase <= 15%
retrieval ACL leakage = 0

Not every metric must improve, but trade-offs should be explicit.

Shadow and replay testing¶

Historical traces can be replayed against a new model/skill/runtime without executing real side effects.

recorded input + observations
        ↓
new runtime/model
        ↓
proposed trajectory
        ↓
compare

For write actions, replace real adapters with simulators/mocks.

This is useful for model upgrades and prompt/skill changes.

Failure injection¶

Test architecture under controlled failures:

model timeout
rate limit
tool 503
queue redelivery
worker crash after side effect
stale state conflict
retrieval unavailable
approval expires
sandbox crash

A system that only works when every dependency succeeds is not production-ready.

Architecture patterns that repeatedly work¶

Deterministic shell around probabilistic decisions¶

validate → model decides → validate → authorize → execute → verify

Ports and adapters¶

business capability
→ port
→ provider adapter

Durable orchestration¶

canonical run state + checkpoints + event-driven resume

Explicit context builder¶

canonical sources
→ trust/freshness/budget filters
→ temporary model context

Capability registry with policy filtering¶

all capabilities
→ identity/task/risk filter
→ effective tool set

Deterministic workflows with agentic islands¶

Autonomy stays bounded inside known business processes.

Evidence-first verification¶

Use tests, tools and systems of record rather than repeated self-reflection.

Architecture anti-patterns¶

Giant prompt architecture¶

Business rules, security, workflow and provider configuration are all embedded in one prompt.

Symptoms:

prompt changes break unrelated features
rules are duplicated
hard to test individual responsibilities

Monolithic Agent class¶

Agent
├── prompts
├── database
├── GitHub
├── billing
├── memory
├── RAG
├── retries
└── authorization

This is a normal God Object with an AI name.

Hidden state¶

Correct execution depends on facts stored only in conversation/context.

Provider SDK everywhere¶

OpenAI/vector DB/MCP-specific types leak into domain and application code.

Unrestricted tool access¶

Every run sees every capability and credentials are overly broad.

Agent everywhere¶

Simple deterministic operations are wrapped in LLM decisions, increasing cost and failure rate.

Vector database as application database¶

Operational state, memory and document retrieval are collapsed into one similarity-search store.

Multi-agent by default¶

Coordination complexity is introduced without measurable quality gain.

Observability as raw transcript dumping¶

Huge logs are expensive, sensitive and still fail to show structured state transitions.

Architecture review questions¶

When reviewing an AI system, ask:

Where is canonical state?
Who owns authorization?
What can the model actually decide?
Which side effects require approval?
Where are provider-specific dependencies?
How are capabilities typed and versioned?
How does context get built?
How is untrusted evidence separated from instruction authority?
Can a run resume after worker failure?
How are retries/idempotency handled?
How is tenant isolation enforced?
Can we trace one run end to end?
Can we evaluate a change before production?
Why is each service/agent boundary necessary?

If these answers are unclear, the system likely has hidden coupling.

Completion mental model¶

A maintainable production agent architecture can be summarized as:

Business capabilities and domain rules
          ↓
Application use cases / ports
          ↓
Durable agent runtime
  ├── canonical state
  ├── explicit context builder
  ├── policy / authorization
  ├── skills / capabilities
  ├── budgets / recovery
  └── observability
          ↓
Infrastructure adapters
  ├── models
  ├── retrieval
  ├── MCP/connectors
  ├── queues
  ├── databases
  └── external services

The AI-specific parts fit into a normal software architecture instead of replacing it.

Engineering takeaways¶

Trace the entire run, not only model calls or final text.
Use structured events and stable IDs across workers, tools, retrieval and state transitions.
Evaluate layers independently so regressions can be localized.
Prefer deterministic graders for deterministic properties and calibrated model graders for semantic ones.
Evaluate trajectory efficiency as well as final outcome.
Gate production changes on safety, quality, cost and latency regressions.
Use replay, shadow testing and failure injection for model/runtime upgrades.
Watch for giant prompts, God-agent classes, hidden state, provider leakage, unrestricted tools and unjustified multi-agent complexity.
Observability must respect privacy and secret boundaries.
Good agent architecture is recognizable as good software architecture with explicit probabilistic components.