Putting an expert AI agent in production takes much more than signing up for an LLM API. You need infrastructure that sustains low latency, predictable cost, security, observability, continuous evolution, failure recovery, and governance. In 2026 this stack is reasonably stabilized, but the wrong choices cost months of rework and five-figure monthly bills wasted.
This guide shows the full stack layer by layer, the critical architectural decisions, the typical costs, and the pitfalls you only discover when the agent is going strong and something breaks at 11 p.m.
The 8 layers of a production agent's infra
1. Model (LLM): managed API (Anthropic, OpenAI, Google) or self-hosted model (Llama, Mistral, Qwen). A managed API wins on time-to-market and average quality. Self-hosted wins on strict compliance and, from very high volume onward, on cost.
2. AI gateway / proxy: a layer that centralizes calls to LLMs. It handles routing across providers, automatic fallback, rate limiting, response caching, observability, and policy enforcement. LiteLLM, Helicone, OpenRouter, and in-house gateways are common options.
3. Harness and orchestration: the framework that runs the agent loop. Mastra, LangGraph, Claude Agent SDK, Vercel AI SDK, or an in-house harness. It runs on serverless compute or in a container depending on the usage profile.
4. MCP server and tools: MCP servers that expose internal and external tools to the agent. They usually live as microservices or as modules of the agent's own deploy.
5. RAG and vector DB: the ingestion pipeline (extraction, chunking, embedding), the vector DB (pgvector, Qdrant, Pinecone), and the retrieval layer with filters and re-ranking.
6. Cache and messaging: Redis (response cache, rate limit, sessions), Postgres or Kafka for the async task queue and events.
7. Observability and evals: execution tracing (LangSmith, Phoenix, Helicone), structured logs, metrics (Prometheus + Grafana), alerts (PagerDuty, Slack), evaluation datasets running in CI.
8. Security and governance: secret manager (Vault, AWS Secrets), access policies, auditing, per-tenant isolation, blast radius control over actions.
Critical decisions: serverless vs container vs bare-metal
Serverless (Vercel, Cloudflare Workers, Lambda) shines for irregular traffic and low cold-start latency. Limitation: short timeouts can kill agents with long loops. Container on Kubernetes wins for sustained traffic, fine-grained resource control, and integration with internal microservices. Bare-metal or dedicated GPU only makes sense if you run your own model at high volume and want to optimize cost aggressively.
For most enterprise agents in 2026, the combination that works is: Cloudflare Workers or Vercel at the edge for low latency, with workers on Kubernetes or long-running serverless for the heavy steps of the loop.
Cost: how to model and control it
The AI bill gets out of hand easily. Model cost on four fronts. 1. LLM tokens: input × output × loop steps × number of calls. 2. Embeddings: initial ingestion + occasional reindexing. 3. Vector DB: storage + requests. 4. Compute and bandwidth: harness, MCP, integrations.
Practices to keep it under control. Caching on common queries cuts LLM cost and latency. A smaller model for light tasks (classification, parsing) saves 5-10x. A token limit per session prevents an expensive runaway loop. A quota per tenant or per user keeps a single client from consuming all your margin. A cost dashboard per feature shows where the money is really going.
Latency: the metric that turns a good product into a bad one
Users tolerate 1-2 seconds of waiting. Above 5 seconds, the product becomes frustrating. Agent latency comes from four sources: the LLM call (200ms to a few seconds), tool calls (network and backend), retrieval (vector DB and re-ranker), harness overhead.
Strategies to reduce it. Streaming the LLM response to the client improves perception even without reducing total time. Tools in parallel when the agent decides to call several. Retrieval caching on repeated queries. Edge deploy close to the user. Pre-computation of predictable answers (for example, greetings, frequent FAQ).
Infra security: beyond the agent's guardrail
Four critical fronts. 1. Credential segregation: each tool has its own credential, with minimal scope. No "god" credential exposed to the LLM. 2. Per-tenant isolation: client A's data and context never leak into client B's session; ACL in the vector DB, in the MCP, and in the log. 3. Immutable auditing: execution log in append-only storage, with retention that meets legal requirements. 4. Periodic pen test and red team focused on prompt injection, context exfiltration, and tool abuse.
Observability: trace, metric, log, and eval
Without decent observability, debugging an agent is archaeology. Minimum viable. Trace: each execution has a trace_id; each loop step is a span with prompt, response, tool, latency, and cost. Metrics: p50/p95/p99 latency, cost per execution, error rate, fallback usage. Structured logs: JSON with a correlation_id across the whole stack. Evals in CI: a pipeline that runs the evaluation dataset on every prompt/model/tool change, with an automated quality gate.
Failure recovery and resilience
Models go down. APIs rate-limit. A tool goes offline. The vector DB gets slow. All of this happens. Model fallback across providers. Retry with backoff on transient errors. Circuit breaker on tools that fail repeatedly. Graceful degradation (answer without RAG if the vector DB goes down, escalating to a human when needed). Idempotency on actions with side effects. These patterns turn isolated failures into events invisible to the user.
Reference stack for an enterprise agent
A combination that works in 2026 for most companies. LLM: Claude via API, with a fallback to GPT on rate limit. Gateway: a lightweight in-house gateway or LiteLLM. Harness: Mastra or Claude Agent SDK. MCP: in-house servers for internal systems + a marketplace of community servers for generic integrations. RAG: pgvector or Qdrant + Cohere Rerank. Cache and queue: Redis + BullMQ. Observability: LangSmith or Phoenix + Grafana + Loki. Deploy: Kubernetes or serverless on Vercel/Cloudflare depending on the case.
It is not the only one, it is not "the best", it is a combination of defensible choices with good documentation and a living ecosystem. What matters is internal coherence: each layer needs to talk well to the others, with observability that runs across everything.
The human factor
Perfect infra without a team does not deliver. An agent in production needs a product owner who champions the use case, an engineer in charge who understands the stack and answers for incidents, a quality analyst who maintains the evals and monitors drift, and an operations function that triages incidents and drives continuous improvement. Without these roles, the best infra in the world rots within months.