Applying Distributed System Patterns to Design Highly Reliable LLM Agents

Sat, 13 Jun 2026 10:09:32 +0900

In production operations of LLM (Large Language Model) agents, encountering non-deterministic errors when integrating with external APIs and databases is inevitable. For example, if a payment API call succeeds but a subsequent inventory reservation API call returns a 503 error, the system falls into an inconsistent state (partial success). Agents built with simple loop processing alone cannot handle these failures unique to distributed systems, resulting in either abnormal termination or leaving the system in an inconsistent state.

This article explains design methodologies for building robust agent systems by applying reliability patterns developed in distributed systems (circuit breakers, the Saga pattern, exponential backoff, and structured validation) to LLM orchestration.

1. Three Failure Modes in Agent Execution

When agents interact with the external environment, the following three main failure modes occur:

Failure Mode 1: Tool Exceptions Rate limits (HTTP 429), temporary network disconnections, timeouts, etc. Without appropriate retry logic, the entire agent loop crashes, and the execution context is lost.
Failure Mode 2: Garbage Tool Outputs If a tool fails to handle errors and returns an invalid payload disguised as a successful response, the LLM will make subsequent decisions based on that incorrect information.
Failure Mode 3: State Inconsistency due to Partial Success In a multi-step workflow, when only some processes succeed and subsequent processes fail. Due to the lack of a rollback mechanism, the system state is left undefined.

These challenges cannot be solved solely by improving the reasoning capabilities of LLMs. A design that wraps the probabilistic behavior of LLMs in a deterministic state machine (orchestrator) is required.

2. Five-Layer Reliability Architecture for Defense

To ensure the reliability of tool calls, we construct an architecture that applies the following five nested layers:

[Agent Loop]
 |
 v
+----------------------------------------------------------------------------+
| 1. traced_call (Execution logging, latency measurement, credential masking) |
| +-----------------------------------------------------------------------+
| | 2. Circuit Breaker (Immediate trip on downstream service failure) |
| | +------------------------------------------------------------------+
| | | 3. with_retry (Retries with exponential backoff and jitter) |
| | | +-------------------------------------------------------------+
| | | | 4. validated_call (Strict schema and type validation) |
| | | | +--------------------------------------------------------+
| | | | | 5. call_tool (Execution of actual tool logic) |
+----+----+----+----+--------------------------------------------------------+

Layer 1: Exponential Backoff and Jitter (`with_retry`)

To recover from transient network errors, the retry interval is increased exponentially. Additionally, to prevent the “Thundering Herd” phenomenon where multiple agents retry simultaneously and overwhelm downstream services, random fluctuations (jitter) are added.

Layer 2: Circuit Breaker

Repeatedly retrying against a service that is completely down wastes resources and hinders the recovery of the target service. If failures occur $N$ times consecutively, the circuit transitions to “OPEN,” immediately blocking subsequent calls (fail-fast). After a certain period, it transitions to the “HALF-OPEN” state, and if a test request succeeds, the circuit returns to “CLOSED.”

Layer 3: Saga Pattern and Idempotency Keys

In distributed transactions where rollbacks are impossible, a corresponding “Compensating Action” is defined for each step. If a failure occurs at step $N$, the compensating actions for the previously executed steps $1$ to $N-1$ are executed in reverse order to return the system to a consistent state. Additionally, to prevent double payments during retries, a unique “Idempotency Key” is assigned to all write operations.

Layer 4: Structured Validation (`validated_call`)

Tool arguments generated by LLMs frequently suffer from type errors or missing required parameters. Before execution, schema validation is performed using tools like Pydantic. If an error occurs, the details are fed back to the LLM to trigger autonomous self-correction.

Layer 5: Observability and Tracing (`traced_call`)

To prevent the agent’s behavior from becoming a black box, the arguments, execution time, and success/failure of all tool calls are recorded as structured logs. Sensitive information such as API keys and passwords is automatically masked during this process.

3. Implementation Details of the Reliability Layers

The Python implementation code integrating these patterns provides robust error handling and state management in an integrated manner.

import time
import random
import logging
import json
import uuid
from typing import Callable, Optional, Any
from pydantic import BaseModel, create_model, ValidationError

_log = logging.getLogger(__name__)

# --- Layer 1: Exponential Backoff and Jitter ---
def with_retry(
 fn: Callable[..., str],
 args: dict,
 max_attempts: int = 3,
 base_delay: float = 1.0,
) -&gt; str:
 for attempt in range(max_attempts):
 try:
 return fn(**args)
 except Exception as e:
 if attempt == max_attempts - 1:
 raise
 delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
 _log.warning("Attempt %d failed (%s) - retrying in %.1fs", attempt + 1, e, delay)
 time.sleep(delay)

# --- Layer 2: Circuit Breaker ---
class CircuitBreaker:
 CLOSED, OPEN, HALF_OPEN = "closed", "open", "half-open"

 def __init__(self, failure_threshold: int = 3, reset_timeout: float = 30.0) -&gt; None:
 self.failure_threshold = failure_threshold
 self.reset_timeout = reset_timeout
 self._failures = 0
 self._state = self.CLOSED
 self._opened_at: Optional[float] = None

 def call(self, fn: Callable[..., str], args: dict) -&gt; str:
 if self._state == self.OPEN:
 elapsed = time.time() - (self._opened_at or 0.0)
 if elapsed &lt; self.reset_timeout:
 raise RuntimeError(
 f"Circuit open - service unavailable (resets in {self.reset_timeout - elapsed:.0f}s)"
 )
 self._state = self.HALF_OPEN

 try:
 result = fn(**args)
 if self._state == self.HALF_OPEN:
 self._reset()
 return result
 except Exception:
 self._failures += 1
 if self._failures &gt;= self.failure_threshold:
 self._state = self.OPEN
 self._opened_at = time.time()
 raise

 def _reset(self) -&gt; None:
 self._failures = 0
 self._state = self.CLOSED
 self._opened_at = None

# --- Layer 4: Structured Validation ---
_VALIDATORS: dict[str, type[BaseModel]] = {}
_TYPE_MAP = {
 "string": str,
 "integer": int,
 "number": float,
 "boolean": bool,
}

def call_tool(name: str, args: dict) -&gt; str:
 # Actual tool execution placeholder
 return f"Success: {name} executed with {args}"

def validated_call(name: str, args: dict) -&gt; str:
 validator = _VALIDATORS.get(name)
 if validator is None:
 return call_tool(name, args)
 try:
 validated = validator(**args)
 return call_tool(name, validated.model_dump(exclude_none=True))
 except ValidationError as e:
 return f"Invalid arguments for '{name}': {e}"

# --- Layer 5: Tracing and Masking ---
def traced_call(name: str, args: dict, fn: Callable[..., str]) -&gt; str:
 sanitized = {
 k: "***" if any(w in k.lower() for w in ("key", "secret", "token", "password")) else v
 for k, v in args.items()
 }
 start = time.time()
 try:
 result = fn(**args)
 _log.info(
 "tool=%s args=%s result=%r duration=%.3fs",
 name, json.dumps(sanitized), str(result)[:120], time.time() - start,
 )
 return result
 except Exception as e:
 _log.error(
 "tool=%s args=%s error=%s duration=%.3fs",
 name, json.dumps(sanitized), e, time.time() - start,
 )
 raise

# --- Dispatcher Composition ---
def _make_dispatcher(
 breakers: dict[str, CircuitBreaker],
 max_retries: int,
) -&gt; Callable[[str, dict], str]:
 def dispatch(name: str, args: dict) -&gt; str:
 def core(**kw) -&gt; str:
 return validated_call(name, kw)

 def retried(**kw) -&gt; str:
 return with_retry(core, kw, max_attempts=max_retries)

 def guarded(**kw) -&gt; str:
 return breakers[name].call(retried, kw)

 return traced_call(name, args, guarded)

 return dispatch

4. Mitigation Strategies for Semantic Hallucinations

While structured validation prevents “syntactic” errors, it cannot prevent semantic hallucinations where the LLM generates “logically incorrect values.” These correspond to Byzantine faults in distributed systems (a state where a node appears to be operating normally but transmits incorrect data).

To address this issue, the following four approaches are applied depending on the use case.

Method	Overview	Academic Background	Trade-offs
Chain-of-Verification (CoVe)	The model itself generates verification questions for its generated answer, answers them, and performs self-correction.	Dhuliawala et al. (2024)	Low Cost: Minimal additional LLM calls are required, making it practical.
Self-Consistency	Samples multiple reasoning paths and determines the final output by majority vote.	Wang et al. (2023)	High Cost: High response latency, making it unsuitable for real-time processing.
LLM-as-a-Judge	An independent verification LLM evaluates and validates the output of the main model.	Zheng et al. (2023)	Medium Cost: Recommended for phases immediately preceding critical write operations.
Output Grounding (RAG)	Mandates strict citations to external knowledge sources and verifies the grounding.	Es et al. (2024)	Low to Medium Cost: Requires search tool design and building evaluation pipelines.

5. Troubleshooting

⚠️ This section outlines common friction points and their solutions when introducing this architecture into a production environment.

Friction Point 1: Circuit Breaker State Drift in Distributed Environments

When agents operate in parallel across multiple container instances, maintaining the circuit breaker state in-memory causes state inconsistencies between instances. While the circuit may be “OPEN” on one node, another node might keep it “CLOSED” and continue sending requests to downstream services, exacerbating the failure.

Solution: Externalize the circuit breaker state (failure count, last error time, current state) to a shared data store such as Redis, and synchronize it using distributed locks or atomic increment/decrement operations.

Friction Point 2: Infinite Substitution in LLM Self-Correction Loops

When returning structured validation errors to the LLM for retries, ambiguous prompt constraints can cause the LLM to repeatedly generate the same incorrect parameters, leading to an infinite loop.

Solution: Strictly count the maximum number of self-correction attempts for the same tool on the dispatcher side (Max Self-Correction Limits, recommended: 3 times). If the limit is reached, immediately throw an exception and transition to the higher-level Saga compensation flow.

6. Verification

🛠️ The execution log protocol of an agent applying the highly reliable dispatcher demonstrates the process of schema error self-correction, retries for transient errors, and circuit breaker activation.

# 1. Self-correction triggered by structured validation
2026-06-13 10:42:01,102 [INFO] tool=charge_card args={"amount": "forty-nine", "card_token": "tok_123"} result='Invalid arguments for "charge_card": amount must be a float' duration=0.012s
2026-06-13 10:42:02,450 [INFO] LLM detected validation error. Retrying with corrected arguments...
2026-06-13 10:42:03,115 [INFO] tool=charge_card args={"amount": 49.00, "card_token": "tok_123"} result='Success: charge_card executed' duration=0.189s

# 2. Exponential backoff triggered by downstream service failure
2026-06-13 10:42:05,201 [WARNING] Attempt 1 failed (503 Service Unavailable) - retrying in 1.2s
2026-06-13 10:42:07,412 [WARNING] Attempt 2 failed (503 Service Unavailable) - retrying in 2.5s
2026-06-13 10:42:10,920 [ERROR] tool=send_notification args={"email": "user@example.com"} error=503 Service Unavailable duration=1.002s

# 3. Circuit breaker tripped due to consecutive failures (transition to OPEN state)
2026-06-13 10:42:11,005 [ERROR] tool=send_notification args={"email": "user@example.com"} error=Circuit open - service unavailable (resets in 30s) duration=0.001s

Operational Notes

💡 The reliability design of LLM agents has moved beyond the realm of prompt engineering and is returning to the domain of classical distributed system design. To deploy agents as autonomous actors in production environments, it is essential to wrap probabilistic reasoning engines in deterministic safety nets. By applying the five defense layers presented in this article, it is possible to build truly autonomous agent systems capable of withstanding API downtime and structural LLM errors.

Llm-Agent on K-Life Hack | Systems Architecture & DevOps