Large language model APIs fail more often than traditional REST endpoints. Token limits hit without warning. Rate limits trigger mid-conversation. Model responses arrive malformed or incomplete. Networks drop connections during long-running inference.
Unlike a database query that either succeeds or raises a clear error, LLM calls introduce probabilistic failures across multiple dimensions. This article walks through four architectural patterns that keep LLM-integrated systems stable when the underlying API isn't: retry logic that doesn't waste tokens, circuit breakers that prevent cascade failures, validation layers that catch malformed responses, and rate limiting that respects API constraints.
Retry logic and exponential backoff
Transient failures dominate LLM API errors. A 429 rate limit, a 503 service unavailable, or a network timeout often resolves within seconds. Naive retry logic burns through token budgets and amplifies load during outages.
Exponential backoff spaces retries with increasing delays. Wait one second, then two, then four, then eight. This gives the API time to recover while preventing retry storms from worsening outages. Add jitter (random variance) to each delay so thousands of clients don't retry in lockstep. A typical implementation: delay = base_delay * (2^attempt) + random(0, 1000)ms.
Not all errors deserve retries. A 400 bad request or 401 authentication error won't fix itself. Build an error classifier that retries 429, 500, 502, 503, 504, and network timeouts, but fails fast on client errors. Track retry counts per request. After three to five attempts, surface the failure rather than looping indefinitely.
Retry logic without backoff and error classification turns isolated API hiccups into cascading failures that take down entire services.
LLM APIs often return partial responses before timing out. If the API supports streaming, checkpoint the tokens already received. On retry, include context from the partial response rather than starting from scratch. This saves tokens and reduces latency on the second attempt.
Circuit breakers and graceful degradation
Think of a circuit breaker like an electrical breaker that trips to prevent fire. A circuit breaker monitors failure rates to an external service and stops sending traffic when failures cross a threshold. When the LLM API returns errors on 50% of requests over a one-minute window, the circuit opens—requests fail immediately without calling the API.
The circuit has three states. Closed means normal operation. Open means failures exceeded the threshold and all requests fail fast without calling the API. Half-open means the circuit allows a few test requests through after a cooldown period to check if the API recovered. If test requests succeed, the circuit closes. If they fail, it opens again.
Implement thresholds carefully. A 50% failure rate over ten requests might be noise. A 50% failure rate over one thousand requests signals a real outage. Use a sliding window (last N requests or last M seconds) rather than a fixed bucket. Tune threshold, window size, and cooldown period based on typical API behavior.
Graceful degradation defines what happens when the circuit opens. Instead of showing an error page, serve cached responses, fall back to simpler models, or route to human operators. A customer support chatbot might switch from GPT-4 to a rule-based system. A code completion tool might show recent suggestions from local cache. Design these fallback paths during normal operation, not during an outage.
Circuit breakers prevent cascading failures. Without one, every request waits for the LLM API timeout (often 30 to 60 seconds), threads pile up, memory fills, and services crash even though the problem originated externally. The circuit breaker fails fast (milliseconds) and keeps services responsive.
Validation layers and fallback mechanisms
LLM responses are probabilistic. Even with careful prompting, models produce JSON with missing fields, hallucinate data structures, or return markdown when plaintext was expected. Architectures must treat every response as untrusted input.
Build a validation layer between the API and business logic. Define schemas for expected outputs using JSON Schema, Pydantic models, or similar tools. Parse the response, validate structure and types, then check business constraints (email format, date ranges, enum values). If validation fails, log the raw response for debugging and either retry with a refined prompt or invoke a fallback.
Structured output modes (like OpenAI's function calling or Anthropic's tool use) improve reliability but don't guarantee correctness. Models still produce invalid enum values or violate constraints. Validate even structured outputs.
| Validation Strategy | When to Use | Trade-offs |
|---|---|---|
| Strict schema enforcement | Financial transactions, code generation | Rejects potentially usable responses; requires fallback logic |
| Lenient parsing with defaults | Content summarization, chat responses | Masks model failures; harder to debug prompt issues |
| Iterative refinement | Complex structured extraction | Uses more tokens; adds latency but improves success rate |
Iterative refinement handles validation failures by feeding the error back to the model. If the model returns invalid JSON, send a second prompt: "Your previous response was malformed. The error was: [specific issue]. Please provide valid JSON." This often succeeds on the second attempt. Limit refinement loops to two or three iterations to avoid token waste.
Fallback mechanisms depend on use case. A code completion feature might fall back to simpler template expansion. A content moderation system might route borderline cases to human review. A chatbot might respond with "I need more information about that" rather than surfacing an error. Design fallbacks that degrade functionality without breaking user experience.
Rate limiting and timeout management
LLM APIs enforce rate limits at multiple levels: requests per minute, tokens per minute, and concurrent requests. These limits vary by pricing tier and can change without notice. Exceeding them triggers 429 errors that block applications.
Implement client-side rate limiting before requests hit the API. Track token usage with a token bucket or leaky bucket algorithm. A token bucket accumulates credits at a fixed rate (e.g., 90,000 tokens per minute) and spends credits on each request. If the bucket is empty, queue the request or return an error immediately. This prevents discovering rate limits through failed API calls.
Estimate token counts before calling the API using tokenization libraries (tiktoken for OpenAI models, similar tools for other providers). Track both prompt and estimated completion tokens against limits. Under-estimating burns through limits; over-estimating wastes throughput. Leave 10% to 20% headroom for estimation error.
Timeout management requires per-layer timeouts, not just a single HTTP timeout. Set a timeout for the entire user-facing operation (e.g., ten seconds for a chat response), a shorter timeout for each API call (e.g., 30 seconds), and shortest timeout for network operations (e.g., five seconds for connection establishment). If any timeout fires, cancel in-flight requests to avoid wasting tokens on responses that will be discarded.
LLM API timeouts should propagate cancellation signals all the way to the API provider to stop inference and avoid token charges for responses that will never be received.
Use request prioritization when demand exceeds capacity. Serve interactive user requests before background batch jobs. Implement separate rate limit buckets for different priority levels. A customer-facing chatbot gets 70% of token budget while document processing gets 30%. When limits hit, queue or reject low-priority work first.
Request coalescing reduces API calls for duplicate or similar prompts. If ten users ask "What is Bitcoin?" within five seconds, deduplicate the requests and fan out the single response. This works best for deterministic queries where temperature is zero. Cache responses keyed by prompt hash, but invalidate caches based on model version and prompt updates.
What this means for builders
Treat LLM APIs like unreliable external dependencies, not local function calls. Wrap every API interaction in retry logic, circuit breakers, and validation layers from day one. These patterns aren't optional polish—they're prerequisites for production stability.
Start with observability before building complex resilience patterns. Log every API call with latency, token counts, and error codes. Track failure rates, timeout frequencies, and token consumption in metrics systems. Retry thresholds and circuit breaker parameters can't be tuned without baseline data.
Design fallback paths that maintain core functionality when LLM APIs degrade. If the entire value proposition depends on LLM responses, a single point of failure has been architected. Identify which features can fall back to simpler models, cached results, or human intervention.
Budget for token waste in cost models. Retries, validation failures, and malformed responses that need refinement all consume tokens without producing useful output. Plan for 20% to 40% overhead in token usage compared to naive implementations.
Test resilience patterns in staging with fault injection. Simulate 429 rate limits, 503 service errors, slow responses, and malformed JSON. Verify circuit breakers trip at the right thresholds and fallbacks activate correctly. Load testing against production APIs is expensive and often violates terms of service.
Conclusion
LLM APIs introduce failure modes that traditional REST APIs don't exhibit—probabilistic outputs, token-based rate limits, long inference times, and frequent service degradation. Systems that ignore these characteristics fail unpredictably under load and during provider outages.
Resilient architectures combine multiple defensive layers: exponential backoff prevents retry storms, circuit breakers stop cascading failures, validation catches malformed responses, and rate limiting prevents quota exhaustion. Each layer addresses a specific failure mode while keeping the overall system responsive.
The patterns outlined here aren't unique to LLMs—they apply broadly to any unreliable external dependency. But LLM APIs make these patterns mandatory rather than optional because their failure rates, costs, and latencies exceed traditional APIs by orders of magnitude.