LLM Infrastructure
Building production LLM systems requires understanding deployment options, scaling patterns, and operational concerns. This chapter covers the infrastructure layer.
CI/CD for LLM Applications
Deploying LLM applications requires adapting traditional CI/CD practices for AI-specific concerns like model evaluation, prompt testing, and quality gates.
AI Gateways and Model Routing
The case for a routing layer is now empirical, not aspirational. Per Datadog's 2026 State of AI Engineering, 69% of companies run three or more models in production, and rate-limit / capacity errors are the single largest production failure mode (about 5% of requests fail, and roughly 60% of those failures are capacity-driven). Datadog's own framing: operational complexity, not model intelligence, is now the primary barrier to reliable AI at scale.
FinOps and Token Economics
This chapter is about the economics and discipline of running LLMs at scale: how to model, attribute, budget, and structurally reduce AI spend. It is not a rehash of inference internals; for the tactical levers (quantization, batching, speculative decoding, KV cache) see Cost Optimization Playbook and the inference chapters.