Reliability, Failure Handling and Deterministic Boundaries¶

A skill is not reliable just because “the prompt is good”¶

A production skill has many failure modes:

invalid input
model error
bad semantic decision
tool timeout
authorization failure
rate limit
partial external failure
duplicate side effect
infinite loop
cost/latency budget exhaustion

Reliability does not mean these never happen.

The goal is:

Failures should be detectable, bounded, represented by explicit outcomes, and handled by deterministic software boundaries wherever possible.

Reliability layers¶

A useful mental model:

Input validation
      ↓
Skill execution
      ↓
Tool/runtime guards
      ↓
Output validation
      ↓
Business validation
      ↓
Side-effect execution
      ↓
Postcondition / audit

No single layer solves everything.

Explicit success and failure semantics¶

A reusable skill should define its possible outcomes.

For example:

SUCCESS
PARTIAL_RESULT
INVALID_INPUT
INSUFFICIENT_CONTEXT
NOT_AUTHORIZED
TOOL_UNAVAILABLE
TIMEOUT
BUDGET_EXCEEDED
FAILED

This is better than:

try:
    run_skill()
except Exception:
    return "Something went wrong"

The caller can then apply policy:

INSUFFICIENT_CONTEXT → ask for input
TOOL_UNAVAILABLE → retry/fallback
NOT_AUTHORIZED → stop
PARTIAL_RESULT → continue with warning

Recoverable vs terminal failure¶

Not every failure should be retried.

Often recoverable¶

network timeout
429 rate limit
transient 503
short-lived tool outage

Often terminal for the current request¶

invalid input
not authorized
resource permanently not found
business rule violation
unsafe requested action

Weak pattern:

retry everything 3 times

This only:

increases latency,
increases cost,
creates unnecessary load,
may be dangerous for write operations.

Retry policy should be explicit¶

For example:

TOOL_TIMEOUT:
  max_attempts: 3
  backoff: exponential

RATE_LIMITED:
  respect_retry_after: true

NOT_AUTHORIZED:
  max_attempts: 0

INVALID_INPUT:
  max_attempts: 0

Retry policy is typically a runtime/application concern, not prompt text.

Model retry is a separate problem¶

If the model returns invalid structured output, a schema-constrained platform may handle some repair automatically.

For semantic failure such as:

model gives unsupported conclusion

resending the same prompt may not be a good strategy.

Possible repair flow:

original result
 ↓
deterministic validation fails
 ↓
repair prompt with exact validation error
 ↓
new result

Even then, enforce maximum attempts and budgets.

Preconditions and postconditions¶

Classic design-by-contract thinking works well.

Preconditions¶

Before the skill runs:

input schema valid
required context available
required capability present
permission scope valid

Postconditions¶

After the skill runs:

output schema valid
required evidence present
forbidden action not requested
business invariants preserved

For example, a Refund Recommendation Skill may require:

amount >= 0
currency supported
recommendation in allowed enum

Further business validation is still needed before an actual refund.

Deterministic boundary: model recommendation vs business decision¶

Suppose the model returns:

{
  "recommendation": "APPROVE",
  "amount": 500
}

while business policy says:

amount > 100 EUR → manager approval required

Correct flow:

model recommendation
      ↓
policy engine
      ↓
manager approval required

not:

model said APPROVE → refund

The model can provide semantic judgment; hard invariants must be enforced in code.

Timeouts at every external boundary¶

An agentic skill may block on:

model call
tool call
retrieval
database
external API
human approval

Each needs a timeout or explicit waiting state.

For example:

model timeout: 60s
tool timeout: 10s
overall skill timeout: 90s

Exact numbers are domain-specific, but the principle is:

Do not allow implicit infinite waiting.

Execution budgets¶

Multi-step skill/agent execution may need several budgets:

max model calls
max tool calls
max iterations
max tokens
max wall-clock time
max cost

For example:

max_steps = 12
max_tool_calls = 20
max_runtime = 2 minutes

When exhausted, return an explicit:

BUDGET_EXCEEDED

outcome.

Stop conditions¶

Good execution knows not only how to continue, but when to stop.

Stop conditions can include:

goal satisfied
terminal failure
user cancellation
approval denied
budget exceeded
no new information
repeated action detected

This is especially important in agentic loops.

Loop detection¶

The runtime can track patterns such as:

same tool + same args repeated N times
same plan repeated
no state change across steps

For example:

query_logs(service=A, last=5m)
query_logs(service=A, last=5m)
query_logs(service=A, last=5m)

If no new information appears, stop or change strategy.

Idempotency for writes¶

Retry is most dangerous around side effects.

Example:

create refund
 ↓
server processed it
 ↓
network response lost
 ↓
client retries

Without idempotency:

duplicate refund

Use:

operation_id / idempotency_key

so the external boundary guarantees:

same key → same logical operation

The model should not generate a new operation ID for every retry.

At-most-once vs at-least-once thinking¶

Agentic writes are still classical distributed-systems problems.

The key question is:

What happens if the request succeeded but we never received the response?

If the system has no answer, its retry policy is unsafe.

Compensation¶

Not every side effect can be rolled back simply.

For example:

1. create GitHub issue
2. send Slack notification
3. update tracking DB

If step 3 fails, deleting the first two effects may not be desirable.

Saga/compensation thinking can help:

record partial state
retry failed step
or execute compensating action

Skill orchestration can become a distributed-workflow problem like any other application.

Fallbacks¶

If a tool is unavailable, a fallback may exist:

primary metrics provider unavailable
 ↓
secondary provider

or:

live data unavailable
 ↓
return partial result with explicit warning

Dangerous fallback:

live data unavailable
 ↓
use model training memory as current truth

Never use hallucination as a fallback for current external facts.

Graceful degradation¶

For example, in PR Review:

diff available ✅
source files available ✅
CI results unavailable ❌

The system can return:

PARTIAL_RESULT

and explain:

"Code review completed; CI/test-result verification was unavailable."

This is often better than total failure.

Output validation at several levels¶

Schema validation¶

required fields
enums
types

Referential validation¶

For example, for a finding location:

file exists?
line exists?

Evidence validation¶

finding cites retrieved evidence?

Business validation¶

recommended action permitted?

Many of these checks are deterministic.

Example: deployment rollback recommendation¶

Skill output:

{
  "recommendation": "ROLLBACK",
  "evidence": [
    "5xx increased from 0.2% to 18% after version 7.5.0"
  ]
}

Runtime:

validate deployment still at 7.5.0
 ↓
validate rollback target exists
 ↓
check authorization
 ↓
require production approval
 ↓
execute rollback with operation id

This protects against stale reasoning.

The world may change between the skill's decision and action execution, so revalidate current state before side effects.

Time-of-check vs time-of-use¶

AI workflows can have classic TOCTOU problems:

Step 1: skill reads account status
Step 2: user/account changes
Step 3: skill executes action based on stale status

Mutating actions need fresh authoritative validation.

Human fallback¶

If the system cannot decide safely because of:

low confidence
conflicting evidence
high-risk action
policy ambiguity

support an explicit:

ESCALATE_TO_HUMAN

Human review is not necessarily a failure; it is a valid control path.

Observability is part of reliability¶

If we cannot answer:

which skill version ran?
which model?
which tools?
which retry?
where did it timeout?

then reliability problems are difficult to improve.

Anti-pattern: catch-all retry¶

except Exception:
    retry()

is especially dangerous around writes.

Classify errors first.

Anti-pattern: prompt-only invariant¶

"Never refund more than 100 EUR."

If this is a hard business rule, enforce it in code too.

Anti-pattern: unlimited autonomous retry¶

try until it works

can cause cost explosions, infinite loops, or repeated side effects.

Anti-pattern: silent fallback to guessed data¶

After a tool failure, do not answer current factual questions with:

"probably..."

Use an explicit unavailable state.

Takeaways¶

Reliability is layered; it is not just prompt quality.
Define an explicit success/failure taxonomy.
Separate recoverable and terminal failures.
Retry policies should be error-aware, bounded, and side-effect-aware.
Surround model execution with preconditions, postconditions, and deterministic validation.
Hard business invariants must not be enforced only by prompts.
Use timeouts and execution budgets.
Agentic execution needs explicit stop conditions.
Write retries need idempotency and operation state.
Revalidate current authoritative state before mutating actions.
Partial results and human escalation are legitimate outcomes.
Missing current data must never fall back to hallucinated values.