How to build reliable LLM applications: the harness engineering pattern for production systems

Large language models excel at reasoning but break when you ask them to also manage state, handle failures, and orchestrate complex workflows. This creates a gap between a working prototype and a system you can deploy at scale. This article shows you the harness engineering pattern: a design approach that wraps LLMs in infrastructure separating reasoning from execution, handling failures gracefully, and integrating safely with external services. You'll learn how to move from prompt experiments to hardened production architectures.

The harness as infrastructure

A harness is the infrastructure layer that surrounds an LLM. It manages the boundary between what the model does (generate text, choose actions) and what your code does (execute those actions, manage state, handle errors). Think of the LLM as an advisor and the harness as the staff that implements its recommendations.

The core principle: never let the LLM directly execute actions or hold state. The model returns structured decisions—tool calls, query parameters, routing choices—and the harness interprets and executes them. This separation means you can test, version, and rollback your execution logic independently from your prompts.

# Anti-pattern: LLM generates code that runs directly
response = llm.complete("Write Python to query the database for user X")
exec(response.text)  # Dangerous, untestable

# Harness pattern: LLM returns structured intent
response = llm.complete(
    "Given user request, return JSON with {action, params}",
    response_format={"type": "json_object"}
)
intent = json.loads(response.text)
# Your code validates and executes
if intent["action"] == "query_user":
    result = db.query_user(sanitize(intent["params"]["user_id"]))

The harness owns retry logic, rate limiting, authentication, and state transitions. The LLM owns reasoning about which action to take next. This boundary makes your system debuggable: when something fails, you know whether it's a reasoning error (bad prompt, model behavior) or an execution error (API timeout, validation failure).

The harness is not a wrapper around the LLM—it's the production system, and the LLM is one component inside it.

Tool calling and error handling at scale

Tool calling is the mechanism where an LLM signals which function to execute and with what arguments. OpenAI's function calling, Anthropic's tool use, and open-source frameworks all follow the same pattern: the model returns structured JSON describing a tool and its parameters, and your harness validates and executes it.

Reliable tool calling requires three layers of validation. First, schema validation: does the returned JSON match your tool definition? Second, business logic validation: are the parameters safe and within allowed ranges? Third, authorization: is this LLM session allowed to call this tool with these parameters? Each layer logs failures separately so you can diagnose whether the model is confused or your validation is too strict.

class ToolHarness:
    def __init__(self, tools, max_retries=3):
        self.tools = {t.name: t for t in tools}
        self.max_retries = max_retries

    def execute_tool_call(self, call, context):
        # Layer 1: Schema validation
        tool = self.tools.get(call["name"])
        if not tool:
            return {"error": "unknown_tool", "retry": False}

        # Layer 2: Business logic validation
        validation = tool.validate_params(call["params"])
        if validation.error:
            return {"error": validation.error, "retry": True}

        # Layer 3: Authorization
        if not context.can_execute(tool.name, call["params"]):
            return {"error": "unauthorized", "retry": False}

        # Execute with retry on transient failures
        for attempt in range(self.max_retries):
            try:
                return tool.execute(call["params"])
            except TransientError as e:
                if attempt == self.max_retries - 1:
                    raise
                time.sleep(2 ** attempt)

Error handling in LLM applications differs from traditional error handling because errors come in two flavors: execution errors (network timeout, database lock) and reasoning errors (model called the wrong tool, malformed output). Execution errors get retries and fallbacks. Reasoning errors get fed back to the model with context about what went wrong, giving it a chance to self-correct.

Context management at scale means controlling how much state you send to the model and how often. Long conversations exhaust token budgets. The harness manages a context window by summarizing old messages, extracting key facts to a structured store, and rehydrating relevant context when needed. This keeps your prompts focused and your costs predictable.

Connecting to external systems safely

Connecting an LLM to external systems requires a capability model: define exactly which actions are possible, then grant the LLM access to a subset based on the session's permissions. Think of it like a sandboxed environment. The LLM can request actions, but the harness enforces what's actually allowed.

Direct database access is an anti-pattern. Instead, expose specific queries or mutations as tools. A tool called get_order_status that takes an order ID is safer and more testable than letting the model construct arbitrary SQL. The harness validates the order ID, checks that the current user owns the order, executes a parameterized query, and returns sanitized results.

class DatabaseTool:
    def __init__(self, db, allowed_tables):
        self.db = db
        self.allowed_tables = allowed_tables

    def get_order_status(self, order_id: str, user_id: str):
        # Harness validates before touching database
        if not self.allowed_tables.get("orders"):
            raise PermissionError("orders table not accessible")

        # Parameterized query prevents injection
        result = self.db.execute(
            "SELECT status FROM orders WHERE id = ? AND user_id = ?",
            (order_id, user_id)
        )

        if not result:
            return {"error": "not_found"}

        # Return structured, sanitized data
        return {"status": result[0]["status"]}

API integration follows the same principle: wrap each external call in a tool with explicit rate limits, timeouts, and retry logic. If you're calling a payment API, the tool handles idempotency keys, validates response signatures, and converts API errors into structured messages the LLM can reason about. The harness logs every external call with timing, status, and payload fingerprints (not full payloads, for privacy).

Authentication flows through the harness, not the LLM. The model never sees API keys or tokens. When the LLM decides to call a tool that requires authentication, the harness injects credentials from a secure store, makes the call, and returns sanitized results. This keeps secrets out of prompt logs and limits blast radius if a prompt injection attack occurs.

Observability beyond basic logging

Traditional logging captures what happened. LLM observability captures what the model thought, what it decided, and why. You need structured traces that link a user request through multiple model calls, tool executions, and retries, all stamped with latencies and token counts.

Instrument every LLM call with metadata: prompt template version, model version, token usage, finish reason (completed vs. truncated vs. error), and a hash of the input. This lets you correlate production issues with specific prompts. When a user reports bad output, you can replay the exact conversation state and see if the issue is in the prompt logic, the model's reasoning, or downstream execution.

class ObservableHarness:
    def __init__(self, llm, tracer):
        self.llm = llm
        self.tracer = tracer

    def call_llm(self, prompt, session_id, turn_number):
        span = self.tracer.start_span("llm.call", {
            "session_id": session_id,
            "turn": turn_number,
            "prompt_hash": hash(prompt),
            "model": self.llm.model_name
        })

        start = time.time()
        response = self.llm.complete(prompt)
        latency = time.time() - start

        span.set_attributes({
            "latency_ms": latency * 1000,
            "tokens_input": response.usage.prompt_tokens,
            "tokens_output": response.usage.completion_tokens,
            "finish_reason": response.finish_reason
        })
        span.end()

        # Emit custom metrics
        self.tracer.record_histogram("llm.latency", latency)
        self.tracer.record_counter("llm.tokens",
            response.usage.total_tokens,
            {"model": self.llm.model_name}
        )

        return response

Monitor reasoning quality by tracking how often the model chooses each tool, how often tool calls fail validation, and how often the model self-corrects after an error. A sudden spike in validation failures might mean a prompt change broke the model's understanding of a tool's schema. A drop in self-correction rate might mean you need to improve error messages fed back to the model.

Cost monitoring matters because LLM costs scale with tokens, not compute time. Track cost per session, per user, and per tool call. Set budget alerts. If a single user's session runs up 10x the normal token count, your harness should gracefully degrade (summarize aggressively, limit retries) or halt the session before costs spiral.

Latency percentiles (p50, p95, p99) tell you more than averages because LLM latency is multimodal: some calls are fast, some hit model queues, some retry multiple times. Percentiles show when users hit the slow path. Trace correlations between high latency and specific tools, long prompts, or rate limit backoff.

What this means for builders

Start with the harness boundary. Don't let the model execute actions directly. Build a tool registry where each tool has a schema, validation logic, and authorization rules. This upfront structure lets you iterate on prompts without rewriting your entire execution layer.

Instrument early. Add tracing before you have scaling problems. When your first production incident happens, you'll need to replay conversations and correlate model decisions with downstream failures. Logs alone won't give you that visibility.

Design your system so you can swap models without changing business logic. The harness should abstract model-specific quirks (OpenAI vs. Anthropic vs. open-source) behind a consistent interface. This makes A/B testing models straightforward and reduces vendor lock-in.

Think about cost as a first-class constraint. Unlike traditional compute, LLM costs are unpredictable and user-driven. Your harness needs circuit breakers: max tokens per session, max retries per tool call, session timeouts. These aren't just cost controls—they also protect against runaway loops where the model and a buggy tool get stuck in an infinite retry cycle.

Test your harness like you test any critical infrastructure. Unit test tool validation. Integration test retries and timeouts. Chaos test what happens when the model returns garbage or an external API goes down. LLMs add nondeterminism, but your harness should be deterministic and testable.

Conclusion

The gap between a clever LLM prototype and a reliable production system is infrastructure. The harness engineering pattern fills that gap by separating reasoning from execution, handling failures explicitly, and integrating safely with the rest of your stack. This isn't about constraining the model—it's about building around it so the unpredictable parts (LLM outputs) are contained within predictable systems (validation, retries, authorization). As LLMs become commodity reasoning engines, the harness becomes your competitive advantage: the system that makes reasoning useful, safe, and scalable.