Core Agent Runtime and Orchestrator¶

The agent runtime is the software subsystem that turns a goal into controlled multi-step execution. The orchestrator is a central coordination component inside that runtime, but it should not become a god object that owns every concern.

A useful mental model is:

Create Run
   ↓
Load canonical state
   ↓
Build context
   ↓
Ask model for next decision
   ↓
Validate / authorize
   ↓
Execute capability
   ↓
Record observation
   ↓
Update state + budgets
   ↓
Continue / wait / stop

The runtime owns the loop lifecycle; the model does not.

Runtime responsibilities¶

A production runtime typically needs to coordinate:

run creation and identifiers,
lifecycle state,
canonical execution state,
context assembly,
model invocation,
action/skill/tool routing,
policy and authorization checks,
budget accounting,
retries and recovery,
checkpoints,
cancellation,
timeout handling,
observability and audit.

These responsibilities may live in several modules/services. The runtime is a conceptual subsystem, not necessarily one class.

Run as an explicit domain/application concept¶

Treat an agent execution as a first-class object.

Example:

AgentRun
├── run_id
├── tenant_id
├── goal
├── status
├── current_step
├── state_version
├── plan
├── observations
├── approvals
├── budgets
├── artifacts
├── failure_history
├── started_at
└── updated_at

Typical statuses:

CREATED
RUNNING
WAITING_FOR_HUMAN
WAITING_FOR_EXTERNAL_EVENT
COMPLETED
FAILED
CANCELLED
BUDGET_EXHAUSTED

Do not infer lifecycle status from the latest chat message.

Execution IDs and step IDs¶

Use stable identifiers.

run_id = one logical execution
step_id = one state transition / decision step
tool_call_id = one capability invocation

These enable:

tracing,
idempotency,
replay,
debugging,
correlating tool results,
resuming after failure.

Without them, observability quickly becomes a collection of unrelated model/tool logs.

Orchestrator as coordinator, not owner of everything¶

A clean orchestrator should delegate specialized concerns.

AgentOrchestrator
├── RunRepository
├── ContextBuilder
├── DecisionEngine / ModelGateway
├── CapabilityRegistry
├── PolicyService
├── BudgetService
├── ActionExecutor
├── CheckpointService
└── Telemetry

Pseudo-flow:

def execute_step(run_id):
    run = run_repository.load(run_id)
    policy.assert_runnable(run)
    budget.assert_available(run)

    context = context_builder.build(run)
    decision = decision_engine.decide(context)

    validated = action_validator.validate(decision, run)
    policy.authorize(validated, run)

    result = action_executor.execute(validated)

    updated = state_transition.apply(run, validated, result)
    budget.record(updated, result)
    run_repository.save(updated)

The exact implementation can differ, but the important property is explicit ownership.

State machine mental model¶

The runtime should be understandable as a state machine even if it is implemented using ordinary application code.

Example:

RUNNING
  ├─ model selects tool ─────────→ RUNNING
  ├─ asks user ──────────────────→ WAITING_FOR_HUMAN
  ├─ waits for job/event ────────→ WAITING_FOR_EXTERNAL_EVENT
  ├─ success conditions met ─────→ COMPLETED
  ├─ terminal failure ───────────→ FAILED
  ├─ budget exceeded ────────────→ BUDGET_EXHAUSTED
  └─ cancellation requested ─────→ CANCELLED

Transitions should be validated in code.

A model output saying status=COMPLETED is a proposal, not an authoritative state transition.

Model gateway¶

Centralize provider interaction behind a model gateway/port.

The gateway may own:

provider selection,
model selection,
structured-output handling,
timeout/retry for inference calls,
token/cost accounting,
provider-specific error normalization,
tracing metadata,
fallback/routing policy.

Example interface:

class AgentDecisionPort(Protocol):
    def decide(self, request: AgentDecisionRequest) -> AgentDecision: ...

The runtime should consume typed decisions rather than vendor response objects.

Capability registry¶

The runtime needs to know which actions are available for a run.

A registry can represent:

capability id
name
description
input schema
output schema
required permissions
risk level
side-effect classification
handler / adapter
version

The available capability set should be filtered before model selection when possible.

all registered tools
      ↓
tenant policy
      ↓
user permissions
      ↓
run-specific allow-list
      ↓
model-visible capabilities

This is safer than showing everything to the model and hoping it never selects a forbidden tool.

Policy coordination¶

Policy belongs outside the model.

The orchestrator can call a policy service for:

permission checks,
action risk classification,
approval requirements,
tenant restrictions,
data-access rules,
maximum autonomy level.

For example:

Decision: SEND_EMAIL
        ↓
Policy check
        ↓
low risk internal draft? → execute
external recipient?      → approval required
restricted recipient?    → reject

Action executor¶

The executor turns a validated action into a capability invocation.

Responsibilities may include:

input validation,
idempotency key generation,
credential scoping,
timeout,
retry policy,
normalized result envelope,
audit metadata.

The executor should not allow arbitrary capability names/arguments to flow unchecked from model output into infrastructure calls.

Long-running runs¶

Agentic execution often outlives a single HTTP request.

For long-running tasks:

API
 ↓
Create AgentRun
 ↓
Queue / Scheduler
 ↓
Worker executes step
 ↓
Checkpoint
 ↓
reschedule next step / wait

Benefits:

crash recovery,
pause/resume,
human approval,
external-event waiting,
controlled concurrency,
horizontal scaling.

Do not keep an HTTP connection open for a 30-minute autonomous run unless there is a deliberate streaming design behind it.

Checkpointing¶

Checkpoint after meaningful state transitions.

Examples:

plan accepted,
external side effect completed,
approval requested,
tool observation recorded,
subtask completed.

A checkpoint should contain enough canonical information to resume without reconstructing the run from model messages.

checkpoint =
  state_version
  run_status
  current_goal
  plan
  completed_work
  observations
  pending_actions
  budgets
  approvals
  artifact references

Optimistic concurrency¶

Two workers should not silently update the same run from the same old state.

Use a state version or equivalent concurrency control:

load version 17
compute transition
save WHERE version = 17
      ↓
version becomes 18

If another worker already wrote version 18, reject/reload/reconcile.

This is especially important when:

external callbacks arrive,
human approval resumes a run,
parallel subtasks execute,
queues redeliver messages.

Cancellation¶

Cancellation should be a runtime feature, not a polite instruction inserted into the prompt.

The runtime should:

mark cancellation requested,
prevent new actions,
attempt to cancel interruptible external work,
reconcile in-flight side effects,
persist final state.

For irreversible actions already completed, cancellation means "stop further work", not time travel.

Timeout layers¶

Different timeouts belong at different levels:

model call timeout
capability/tool timeout
step timeout
run deadline
human approval expiry
external wait deadline

One global timeout usually obscures the real failure boundary.

Retry ownership¶

Retries should happen at the layer that understands the failure.

Examples:

HTTP 503 from provider → model gateway may retry,
idempotent API timeout → action executor may retry/reconcile,
invalid business input → do not transport-retry; repair/replan,
whole strategy failed → orchestrator may replan.

Avoid nested invisible retries across every layer, because latency and cost multiply unpredictably.

Synchronous vs asynchronous orchestrator¶

A simple runtime can be synchronous:

request → several bounded steps → response

An advanced runtime may be durable/asynchronous:

run → queue → worker → checkpoint → event → worker → completion

The architecture should evolve based on run duration, reliability and scaling needs.

Orchestrator anti-patterns¶

God orchestrator¶

One class contains prompts, SQL, provider SDK calls, authorization, tool implementations and state updates.

Stateless loop over chat history¶

Every iteration reconstructs execution state from messages.

Model-controlled lifecycle¶

The model freely decides that a run is complete, cancelled or approved.

Registry without policy filtering¶

Every registered capability is visible to every run.

Retry everywhere¶

Gateway, executor, orchestrator and queue all independently retry without shared budgets/idempotency.

Non-durable long runs¶

A process crash loses all progress because state existed only in memory.

Minimal architecture¶

Do not overbuild the runtime on day one.

A useful progression:

Stage 1
bounded synchronous orchestrator
+ typed state
+ explicit tools

Stage 2
persistent run state
+ checkpoints
+ approvals

Stage 3
queue/workers
+ durable resume
+ parallel subtasks

Stage 4
specialized scaling / multi-agent only if justified

Takeaways¶

The runtime owns the agent lifecycle; the model does not.
Represent a run explicitly with stable IDs and canonical state.
Treat orchestration as a state machine with validated transitions.
Keep model gateway, context building, policy, execution and persistence as separate responsibilities.
Filter capabilities before they reach the model.
Make cancellation, timeouts, retries, checkpoints and concurrency deterministic runtime features.
Introduce queues/durable workers when run characteristics justify them, not automatically.