Steply
 Book a briefing
← Back to blog
steply / blog · como-um-agente-funciona-por-baixo-dos-panos-harness-loop.md
$ steply blog open como-um-agente-funciona-por-baixo-dos-panos-harness-loop
▸ loading article…
✓ ready

How an agent works under the hood: harness, loop, context, no detail left out

bySteply10 min read

The previous post defined what an agent is: an LLM controlling its own flow via tool calling inside a loop. This post opens the black box. It shows exactly what happens between a prompt going in and the response coming out, with a focus on the harness (the non-LLM code that makes everything spin). All the real complexity lives here. A good model with a bad harness makes a bad agent. The reverse is also true, and more often than people admit.

We are going to cover: the loop structure, how the prompt is assembled each turn, how tool calls are executed (serial vs parallel), how to return structured errors, how to manage a context that keeps growing, how to use prompt caching so you do not go broke, how to persist state, how to decide the stopping criterion, and how to instrument everything for observability. No shortcuts.

1. The loop in pseudocode, no fluff

messages = [{role: 'user', content: user_input}]
turn = 0
while turn < MAX_TURNS:
 response = llm.create(
 model=MODEL,
 system=SYSTEM_PROMPT,
 messages=messages,
 tools=TOOL_DEFINITIONS,
 max_tokens=4096,
 )
 messages.append({role: 'assistant', content: response.content})

 if response.stop_reason == 'end_turn':
 return response.content

 if response.stop_reason == 'tool_use':
 tool_results = execute_tools(response.content.tool_uses)
 messages.append({role: 'user', content: tool_results})
 turn += 1
 continue

 if response.stop_reason == 'max_tokens':
 raise BudgetExceeded()

raise MaxTurnsExceeded()

That is the complete loop. Every agent implementation, from Claude Code to Cursor to your MVP, is an elaboration of this. The difference between a toy and production lives in the details of each line. Let us open each one.

2. Assembling the prompt: what goes in system, what goes in messages

The system prompt is static throughout the run. It holds: the agent identity, invariant rules, the expected output format, the security policy, the environment description. Everything that does not change between turns goes in system, because it is cached.

The messages array grows every turn. Typical structure in Anthropic format:

[
 { role: 'user', content: 'pedido inicial' },
 { role: 'assistant', content: [
 { type: 'text', text: 'vou consultar X' },
 { type: 'tool_use', id: 'toolu_01', name: 'search', input: {...} }
 ]},
 { role: 'user', content: [
 { type: 'tool_result', tool_use_id: 'toolu_01', content: '...' }
 ]},
 { role: 'assistant', content: [
 { type: 'text', text: 'achei. agora vou processar' },
 { type: 'tool_use', id: 'toolu_02', name: 'process', input: {...} }
 ]},
 ...
]

A detail many implementations get wrong: tool_result goes with role: 'user', not 'tool'. Anthropic has no 'tool' role. OpenAI does (role: 'tool'). Mixing them breaks. Each provider has its own wire format, and hiding it behind an adapter is basic hygiene.

3. Tool definitions: the schema is a contract

Each tool is declared with name, description and input_schema (JSON Schema). The LLM reads this every turn (it is part of the prompt sent, and it costs tokens). Three practical rules:

  • The description is the tool's UI for the LLM: write it as if it were a docstring read by a junior dev. Include when to use it, when not to use it, an example of valid input, and the expected output format. The difference between a generic description and a rich description is the difference between the LLM getting it right 60% or 95% of the time.
  • The schema validates on input, always: the LLM will generate malformed input at some point. Validate with Zod/Pydantic and return a structured error, not a raw exception.
  • Do not expose a tool that can be composed in code: if two tools are always called in sequence, make it one. Every extra tool means a larger prompt, higher latency, and one more chance for the LLM to get it wrong.

4. Executing tool calls: parallel or serial

Modern LLMs can emit multiple tool calls in a single turn (parallel tool use). Anthropic has supported it since Claude 3, OpenAI since GPT-4o. The harness receives an array and decides how to execute it.

If the tools are independent (querying 3 different APIs), run them in parallel with Promise.all/asyncio.gather. Direct latency gain. If there is a dependency (rare in the same turn, because the LLM would have already asked in separate turns), run them serially. When in doubt, go parallel, and design the tools to be safe under concurrency.

async def execute_tools(tool_uses):
 results = await asyncio.gather(*[
 execute_one(t) for t in tool_uses
 ], return_exceptions=True)
 return [format_result(t, r) for t, r in zip(tool_uses, results)]

return_exceptions=True is critical. Without it, one tool that fails cancels the others and the LLM loses the partial result.

5. Tool errors: never propagate, always return

A tool error is not a program error. It is an observation that goes to the LLM to reason about. Anthropic format:

{
 type: 'tool_result',
 tool_use_id: 'toolu_01',
 is_error: true,
 content: 'API returned 503: service unavailable. Retry recommended.'
}

With is_error: true, the LLM understands that the call failed and decides: retry (same input, same tool), try an alternative path, ask the user for help, or give up. If you raise an exception and break the loop, you lose the LLM's chance to recover.

Errors that should become exceptions (not tool_result): the tool does not exist (the LLM hallucinated the name), the input schema is invalid after sanitization, an expired credential (needs external intervention). Everything that is a transient state of the outside world becomes a tool_result with an error.

6. Context management: the problem that kills agents in production

Every turn, the context grows. Each tool result adds tokens. In 10 turns you can have 50k tokens. In 30, 200k. Three things go wrong.

Cost: you pay for input tokens on every call. A 20-turn loop averaging 100k tokens costs 20x more than a single 100k call. Without prompt caching, it is ruinous.

Latency: time-to-first-token grows with input size. The agent gets progressively slower over the loop.

Context rot: the model loses precision as the context gets huge, even within the supported window. Performance degrades well before the theoretical limit.

Four techniques to mitigate it.

  • Prompt caching: the provider caches the stable prefix. Anthropic charges 10% of the normal price for cached tokens (1h TTL with extended cache). Put the system prompt, tool defs and stable messages AT THE START of the prompt. Dynamic messages at the end. A decent hit rate: 90%+. It cuts cost by 5x to 10x.
  • Summarization: every N turns, compress the old messages into a summary. You lose detail, you gain space. Used in Claude Code with automatic compaction.
  • Sub-agents: parallelizable tasks go to a sub-agent with its own context. The result comes back as a short string to the main agent. A pattern used by Claude with the Task tool, OpenAI's deep research, and almost every modern agent framework.
  • External memory: state that does not fit in the context goes to storage (Redis, database, filesystem). A read/write tool exposes access. A pattern used in long-running agents.

7. Prompt caching in detail: the one feature that decides whether you can afford it

Without prompt caching, an agent is economically unviable at volume. How it works in Anthropic: you mark a breakpoint on the message with cache_control: { type: 'ephemeral' }. Everything before the breakpoint is cached for 5 minutes (default) or 1 hour (extended). The next request that starts with the same prefix pays 10% of the price.

{
 system: [{
 type: 'text',
 text: SYSTEM_PROMPT,
 cache_control: { type: 'ephemeral' }
 }],
 tools: TOOL_DEFINITIONS, // tools entram no cache automaticamente com system
 messages: [
 ...static_messages,
 { ..., cache_control: { type: 'ephemeral' } },
 ...dynamic_messages
 ]
}

Standard strategy: two breakpoints. One after system+tools (caches the entire loop). Another after the last stable assistant message (cached between nearby turns). The hit rate climbs to 95%+ on long loops.

8. Stop reason: how the loop knows it can finish

Anthropic returns stop_reason with values: end_turn (the LLM finished speaking, no tool call), tool_use (the LLM asked for a tool), max_tokens (blew the budget), stop_sequence (found a defined sequence), refusal (refused for policy reasons).

Harness logic:

  • end_turn: return the response to the user, close the loop.
  • tool_use: execute, return the result, continue the loop.
  • max_tokens: error. Increase max_tokens on the next call, or give up with a message to the user.
  • refusal: error. Show it to the user, do not retry.

The harness's hard stopping criterion: MAX_TURNS (usually 30 to 50) and MAX_TOTAL_TOKENS (the execution budget). Without it, an agent in a loop can cost tens of dollars per run.

9. State management and durability

An agent that lasts seconds is stateless: state lives in memory and dies with the process. An agent that lasts minutes or hours (deep research, codebase refactoring) needs to be durable: it can crash midway and resume.

Strategies:

  • Event log: each turn (user message, assistant response, tool result) is appended to a persistent log (database, Kafka, file). A restart rebuilds messages from the log.
  • Checkpoint: every N turns, serialize the entire messages array. A restart loads the last checkpoint. Simpler, less granular.
  • Workflow engine: Temporal, Inngest, Restate. The agent becomes a durable workflow: each tool call is an activity, retry and resume are framework primitives. A pattern used in serious production agents.

10. Streaming: why it always pays off

The LLM generates token by token. Without streaming, you wait for the complete response before seeing anything. Perceived latency explodes. With streaming, you show text as it comes out, and you detect tool_use before the end so you can start preparing execution.

Anthropic streaming uses SSE with typed events: message_start, content_block_start, content_block_delta (text deltas), content_block_stop, message_delta, message_stop. Tool use arrives with a partial input_json_delta, aggregated into a string that becomes JSON at the end.

A serious harness streams by default. A latency of 4 seconds becomes a perceived latency of 800ms.

11. Observability: trace, span, metrics

Each agent run is a tree. Root: the initial request. Children: each turn. Grandchildren: each tool call within the turn. Useful instrumentation:

  • Per turn: input/output tokens, latency, model used, cache hit ratio, stop_reason.
  • Per tool call: name, input, truncated output, latency, success/failure, retry count.
  • Per run: total turns, total tokens, cost in USD, total time, outcome (success/budget/error).

Tools: LangSmith, Langfuse, Helicone, Phoenix Arize, or OTEL straight to Datadog/Honeycomb. Build vs buy depends on volume. Without a trace, debugging in production is archaeology in the logs.

12. Sub-agents: when and how

A sub-agent is an agent invoked as a tool by the parent agent. The pattern: the parent decides to subdivide, calls spawn_agent(prompt, tools), the sub-agent runs its own loop with an isolated context, and returns a string. The parent consumes the string as a tool_result.

Advantages: parallelism (several sub-agents at once), context isolation (the sub does not pollute the parent), specialization (a sub with restricted tools and a focused system prompt). Cost: each sub-agent is a complete loop with its own overhead.

Anti-pattern: a deep chain of sub-agents (parent calls child calls grandchild calls great-grandchild). Latency stacks up, debugging becomes impossible. Keep it to 2 levels at most, except in a very justified case.

13. Structures that matter: ReAct, Reflexion, Plan-and-Execute

Three variations of the basic loop that show up in production.

ReAct (Reasoning + Acting): the standard loop described above. The LLM alternates between reasoning (text) and acting (tool). It is the default.

Reflexion: after each action, the LLM evaluates the result and generates a 'reflection' that goes into memory. Next iterations consult the reflections to avoid repeating an error. Useful in trial-and-error tasks (debug, exploit, optimization).

Plan-and-Execute: it splits into two phases. Phase 1 (one call): the LLM generates a plan in a structured format. Phase 2 (loop): the executor follows the plan step by step, with the LLM reviewing whether the step worked. It replans if needed. Fewer calls than pure ReAct for tasks with a clear plan.

14. What will go wrong in production

An honest list of the bugs that show up early:

  • The LLM hallucinates a tool name that does not exist. The harness should return a structured error and log it.
  • The LLM passes input that fails the schema. Validate, and return with a clear message of what is wrong.
  • A tool has a side effect and the LLM retries after a timeout, doubling the action. An idempotency key is mandatory.
  • An infinite loop when the LLM insists on retrying a tool that fails in a deterministic way. Detect the repetition and break it.
  • Context overflow in the middle of a run. Compact proactively based on token count, not just when it blows up.
  • A cache miss because some message changed order or content between turns. Audit it with a hit rate metric.
  • A tool with an infinite default timeout freezes the whole agent. Always set an explicit timeout.
  • A network error during streaming abandons the partial response. Implement resume or retry the whole thing.

15. What the small harness does NOT need to do

To close with some calibration: to get started, you do not need a framework. anthropic-sdk + 200 lines of code solve 80% of cases. Frameworks like LangGraph, LlamaIndex, CrewAI add value when you have scale (>100 agents, multi-tenant) or when you need serious durability.

This entire post describes what you implement from scratch. Understanding this layer is a prerequisite for choosing a framework with judgment, instead of adopting one and finding out 6 months later that you are stuck with an abstraction that does not fit your case. The harness is your point of leverage. Treat it as critical infrastructure.