Orchestrating Datacenter-Scale LLM Inference with Dynamo

Coordinating Multi-Node Inference Beyond Single-Engine Optimization

Inference engines such as SGLang, TensorRT-LLM, and vLLM are optimized for single-GPU and single-node execution. They deliver high kernel-level throughput, continuous batching, and tensor parallelism within tight hardware boundaries. However, running large language models (LLMs), reasoning models, and multimodal pipelines at datacenter scale presents distinct distributed systems challenges: uncoordinated prefill and decode bottlenecks, cache misses across worker nodes, and rigid cluster provisioning that inflates operational expenditure.

Dynamo is an open-source, datacenter-scale inference orchestration framework developed by NVIDIA and the open-source community (dynamo README). Rather than acting as a standalone kernel runtime or replacing existing engines, Dynamo operates as an orchestration layer directly above SGLang, TensorRT-LLM, and vLLM. It transforms standalone execution engines into a unified, multi-node inference cluster using a core written in Rust for high throughput and low routing latency, paired with Python for extensibility.

Disaggregated Prefill and Decode Architectures

Serving autoregressive transformer models involves two asymmetric operational stages:

Prefill Phase: Compute-intensive batch processing of prompt tokens to initialize key-value (KV) activations.
Decode Phase: Memory-bandwidth-bound sequential generation of subsequent output tokens.

When prefill and decode operations share the same GPU instances, compute stalls and inter-token latency spikes frequently occur. Dynamo addresses this through Disaggregated Serving, separating prefill and decode stages into independently scalable GPU worker pools. Each pool can be mapped to hardware profiles tailored to its computational characteristics—allocating high-compute instances to prefill workers while dedicating memory-bandwidth-optimized instances to decode workers.

KV-Aware Request Routing and Multi-Tier Cache Offloading

Repeated prefixes and multi-turn conversations often generate redundant prefill computations across distributed clusters. To mitigate this overhead, Dynamo provides KV-Aware Routing. The router evaluates live worker loads alongside prefix cache state across worker nodes, dispatching requests to workers that already hold matching KV cache entries. In published benchmarks on Qwen3-Coder 480B, KV-aware routing delivered a 2x faster time to first token (TTFT).

To accommodate extended context windows without exhausting high-bandwidth GPU memory (HBM), Dynamo incorporates the KV Block Manager (KVBM). KVBM coordinates hierarchical KV cache tiering across GPU HBM, host CPU memory, local NVMe SSDs, and remote object storage (including Amazon S3 and Azure Blob Storage), allowing clusters to retain active session state across long conversations.

Dual Kubernetes Routing Topologies: Native Frontend and Gateway API

Dynamo supports two standard Kubernetes request routing topologies, both presenting OpenAI-compatible endpoints:

Dynamo-Native Frontend Routing: External traffic reaches the Dynamo Frontend, which directly queries the integrated Dynamo Router to select the target worker (client -> Frontend -> Router -> workers). This architecture suits standalone installations, edge clusters, or environments where Dynamo manages request dispatching end-to-end (dynamo README).
Gateway API Routing with GAIE: Standardized on the Kubernetes Gateway API Inference Extension (GAIE). The cluster gateway delegates routing decisions to Dynamo's Endpoint Picker Plugin (EPP), which directs traffic directly to worker Frontend sidecars running in direct routing mode (client -> Gateway -> EPP -> Frontend sidecar -> workers). This design preserves central platform governance, rate limiting, authentication, and observability at the cluster ingress boundary.

SLA-Driven Cluster Autoscaling and Fast Cold Starts

Operating multi-node inference clusters under strict Service Level Agreements (SLAs) requires dynamic, telemetry-driven orchestration. Dynamo integrates several specialized infrastructure subsystems:

The Planner: An SLA-driven autoscaler that monitors runtime latency metrics (such as TTFT and inter-token latency) to scale prefill and decode pools independently. In production deployments, the Planner reduced SLA breaches by 80% while lowering total cost of ownership by 5% (Planner Guide).
ModelExpress: An interconnect-aware weight streaming system that transfers model weights directly GPU-to-GPU via NIXL and NVLink, achieving 7x faster cold starts for new replicas compared to standard network storage pulling (ModelExpress Repository).
Grove: A Kubernetes operator providing topology-aware gang scheduling across dense NVLink networks, such as GB200 and NVL72 racks, placing worker pods across NUMA, PCIe, and rack boundaries (Grove Repository).
AIConfigurator: A simulation tool that models over 10,000 deployment configurations rapidly to discover optimal parallelism and pool ratios without consuming GPU execution hours (AIConfigurator Repository).

Deployment Pathways and Production Execution Options

Teams evaluating Dynamo can deploy via containerized runtimes or declarative Kubernetes controllers.

Containerized Local Runtime

Official container images package Dynamo with supported inference engines, such as SGLang, TensorRT-LLM, and vLLM (Release Artifacts). A local multi-process instance can be launched inside a runtime container:

# Launch an SGLang runtime container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.1

# Start frontend router and worker processes within the container
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

# Send a test completion request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

Declarative Kubernetes Orchestration (DGDR)

For production Kubernetes environments running the Dynamo Platform, deployments are managed declaratively using the DynamoGraphDeploymentRequest (DGDR) Custom Resource Definition (DGDR Reference). Platform operators specify the target model identifier, backend runtime (such as vLLM or SGLang), and latency objectives (target TTFT and inter-token latency). Dynamo's AIConfigurator and Planner automatically analyze the requirements, synthesize the optimal cluster topology, and provision the prefill and decode worker pools.

Hardware Requirements, Scope Boundaries, and Licensing

Workload Scope: Dynamo provides orchestration value in multi-GPU and multi-node environments. Deployments running a single model instance on a single GPU do not require Dynamo's coordination layer, as the underlying inference runtime is sufficient.
Hardware Dependencies: High-throughput streaming and scheduling components (such as ModelExpress weight streaming and Grove gang scheduling) depend on high-speed hardware interconnects like NVLink and NIXL on NVIDIA H200, GB200, and GB300 systems (InferenceX).
Licensing Audit: Primary repository metadata lists the repository license as NOASSERTION / Other (ai-dynamo/dynamo metadata), while the codebase and source repository README state that the software is licensed under the Apache License 2.0 (dynamo README). Organizations requiring formal compliance clearance should audit specific source headers during onboarding.