llm-d 1.0: The Open Source Platform for Production LLM Inference at Scale

Running large language models in production is now an infrastructure problem, not a model problem. On October 20, 2026, contributors to the llm-d project announced llm-d 1.0 at PyTorch Conference North America in San Jose (Red Hat press release, published October 20, 2026, retrieved 2026-10-23). It is a major release of the open source stack for deploying and scaling LLM inference. The release brings day-two operations: multi-tenant quality of service, zero-downtime upgrades, a stable v1 configuration API, and agent-aware serving. IBM Research demonstrations served an approximately 753-billion-parameter open-weight model at more than 6.6 million output tokens per minute at peak, with up to 3,000 concurrent coding agents and zero preemptions (Red Hat press release). Per-token costs ran 5 to 10 times lower than equivalent commercial API pricing.

Key Takeaways

• llm-d 1.0 is an open source, Kubernetes-native stack for distributed LLM inference, governed by the CNCF and licensed under Apache 2.0.

• Version 1.0 adds multi-tenant quality-of-service flow control, a high-availability router with hot standby, prefill and decode replicas managed as one logical unit, a stable v1 API, and agent-aware serving with session affinity.

• llm-d sits above inference engines such as vLLM and SGLang. It orchestrates intelligent routing, caching, and scaling across them instead of replacing them.

• IBM Research demonstrations served GLM-5.2, an approximately 753-billion-parameter open-weight model, on 544 NVIDIA H100 GPUs at more than 6.6 million output tokens per minute at peak, with up to 3,000 concurrent coding agents and zero preemptions.

• The project counts more than 500 contributors from over 25 organizations, with production adopters including DigitalOcean, JetBrains, Mistral, Roblox, and Cohere, and hardware support across NVIDIA, AMD, and Intel GPUs plus Google TPU.

What Is llm-d?

llm-d is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes, as its GitHub README puts it (llm-d/llm-d, GitHub, retrieved 2026-10-23). LLM inference is the process of running a trained language model to generate responses. When one model serves thousands of users and agents at once, it becomes a distributed systems problem. Requests must be routed to machines that already hold their context. Memory caches must move between GPU, CPU, and storage. And tenants must share the same fleet without starving one another. llm-d provides the orchestration and optimizations above model servers to serve that traffic efficiently and reliably.

Engines like vLLM and SGLang handle running a model efficiently on accelerators. llm-d adds the layer above them: intelligent routing, KV-cache management with tiered offloading, serving large models through prefill/decode disaggregation and wide expert parallelism, flow control with autoscaling, and batch processing. KV cache is the stored attention state a model keeps while generating text; reusing it avoids recomputing everything for each new token, which is why cache-aware routing matters.

The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, and joined the Cloud Native Computing Foundation as a Sandbox project in March 2026 (Red Hat press release; llm-d/llm-d README, retrieved 2026-10-23). The Red Hat press release reports more than 500 contributors from over 25 organizations. At the time of writing, the GitHub repository carries 4,980 stars, 880 forks, and 1,721 commits across 13 releases (llm-d/llm-d, GitHub, retrieved 2026-10-23).

What Version 1.0 Adds

The five core techniques, prefix-cache-aware routing, KV-cache offloading, batch inference optimization, prefill/decode disaggregation, and wide expert parallelism, predate 1.0. What is new is the integration and operational maturity around them: holding performance when traffic changes, hardware fails, software is updated, or different users compete for the same resources. Prefill/decode disaggregation is the separation of the two phases of generation: reading the prompt and producing the response. Splitting them lets each phase scale on the hardware it suits best.

• Multi-tenant quality of service. Priority- and fairness-aware flow control with controlled load shedding. Latency-sensitive requests and bursty agent workloads share one fleet without starving one another.

• Zero-downtime operations. High-availability router with hot standby and safe rolling updates, plus a DisaggregatedSet controller that manages prefill and decode replicas as one logical unit.

• A stable foundation to build on. The router configuration API graduates to v1, and the endpoint-picker request path is hardened for reliability, performance, and scalability. Adopters get a release they can standardize on.

• Agent-aware serving. A session-state-producer plugin and evolving session affinity keep multi-turn conversations close to their cached context, while turn-priority scheduling and native Responses API state support, including previous_response_id, help agents reuse context across turns and reduce recomputation.

The Benchmarks That Matter

The headline demonstration comes from IBM Research. Using llm-d, researchers deployed GLM-5.2, an approximately 753-billion-parameter open-weight mixture-of-experts model, on 544 NVIDIA H100 GPUs. On a workload with hundreds of agentic sessions running in parallel, the deployment delivered more than 6.6 million output tokens per minute at peak. It served up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, IBM states, this led to costs 5 to 10 times less per token than equivalent commercial API pricing. The largest savings came on the input-heavy traffic patterns typical of agentic workloads (Red Hat press release, published October 20, 2026, retrieved 2026-10-23). The disaggregated data plane completed 6.2 million KV-cache transfers averaging 2.71 GiB per transfer with zero failures. All benchmarks completed with zero serving errors (Unite.AI coverage, retrieved 2026-10-23).

Those numbers land harder with context about what agentic workloads look like. A July 22, 2026 benchmark post on llm-d.ai, summarized by Unite.AI, drew on 219 production Claude Code sessions. The median request carried 195,000 input tokens and returned 317. And 96% of turns reused at least 90% of their input (Unite.AI). Prefix-aware routing captured 96 to 97% of that reuse (Unite.AI). At an assumed H200 rental rate of $2.71 per GPU-hour, self-hosted serving on two 64-GPU layouts came to $0.05 to $0.08 per million input tokens and $1.92 to $3.55 per million output tokens. That is GPU rental only, assuming sustained load.

A second IBM Research use case, quantum error correction hardware exploration, reported a prefix-cache hit rate of approximately 98%, a blended serving cost of about $0.11 per million tokens, and an inference cost 11.5 times lower than an all-proprietary configuration (Red Hat press release). The project README adds partner-validated results: 3 times higher output throughput and 2 times faster time to first token with prefix-cache-aware routing versus round-robin on Llama 3.1 70B (Tesla and Red Hat); a 40% reduction in time to first token and inter-token latency with predicted-latency scheduling versus heuristics (Google) (llm-d/llm-d README); up to 70% higher tokens per second with prefill/decode disaggregation versus standard vLLM on GPT-OSS (AWS) (llm-d/llm-d README); 50,000 tokens per second of cluster throughput with wide expert parallelism on 16 by 16 NVIDIA B200 GPUs; and a 13.9 times throughput improvement with hierarchical KV offloading versus GPU-only serving on 4 NVIDIA H100 GPUs (llm-d/llm-d README, GitHub, retrieved 2026-10-23). Time to first token (TTFT) is how long a user waits before the first word of a response appears. Inter-token latency is the pace of the words that follow.

Who Builds and Uses llm-d

The contributor list reads like a map of the AI infrastructure industry. Red Hat, IBM, and Google are the lead maintainers; AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI are among the contributors (Unite.AI). Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine across its TPU and NVIDIA GPU portfolio; CoreWeave runs it alongside NVIDIA Dynamo; NVIDIA is bringing it to Vera Rubin; and AMD is advancing it on Instinct GPUs.

Adoption is already concrete. DigitalOcean relies on llm-d for its Serverless Inference and Managed Agents products, serving trillions of tokens. JetBrains runs it in production behind its Junie Lite plan. Mistral uses prefix-cache-aware routing and flow control to enforce strict priority bands across its commercial tiers. Roblox runs it across cloud and on-premises GPUs, and Cohere points to it as a fit for its sovereign AI commitments (Red Hat press release, all quotes retrieved 2026-10-23).

Why This Release Matters

As organizations race to put AI into production, many are discovering that running AI at scale can be as challenging and expensive as building the models themselves. The bottleneck has shifted from generating answers to managing the economics and extreme infrastructure demands of complex agentic workloads. With agentic AI, new workloads continually access and reuse vast amounts of context. They support parallel agents that generate unpredictable bursts of activity. Managing that context often consumes more resources than generating the final output (Red Hat press release). Remember the median Claude Code session: 195,000 input tokens in, 317 out. The cost center is context management. That is exactly what llm-d optimizes.

The strategic bet is vendor neutrality: any model, any accelerator, any cloud, with a standardized hardened core adopters can run themselves instead of paying per token to a provider. Independent coverage frames it the same way: the release moves llm-d beyond demonstrating individual techniques and is meant to help organizations bring model deployment in-house as inference costs rise (Unite.AI, retrieved 2026-10-23).

Two caveats keep the picture honest: the headline benchmarks are vendor-reported demonstrations, and the dollar figures assume sustained load and cover GPU rental only. You still need a Kubernetes cluster, accelerator capacity, and the team to run them; if Kubernetes is new to you, our Kubernetes explainer covers what it is and why it runs the cloud.

What Independent Coverage Says

Unite.AI's launch coverage is detailed and measured. It confirms the October 20 announcement. It walks through the v1.0.0 release notes published on GitHub with signed and verified tags. It spells out the new machinery: consolidated flowControl configuration, the DisaggregatedSet controller, the router API graduating to v1, Responses API support for previous_response_id, and the session-state-producer plugin. The verdict it relays from llm-d co-founder Carlos Costa of IBM Research: version 1.0 adds the maturity to run llm-d like any other critical service (Unite.AI, retrieved 2026-10-23). The coverage also makes the technique boundaries clear: the five core approaches existed before 1.0. The release is about operational maturity. That is a less flashy story than a model launch, but it is the one that determines whether open source inference survives contact with production.

How to Get Started

Start with the Quickstart Guide on llm-d.ai, which deploys a first optimized inference service on Kubernetes: setting up the stack, configuring the router, and validating performance with production-ready benchmarks (llm-d/llm-d README, GitHub, retrieved 2026-10-23). Expect to bring your own cluster, accelerators, and model access: llm-d is software you run, not a hosted service you sign up for. There is no llm-d bill, but there is no free GPU time either.

What is llm-d? llm-d is an open source, Kubernetes-native distributed inference stack, founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, and governed by the CNCF as a Sandbox project. It adds routing, KV-cache management, and operational tooling above engines such as vLLM and SGLang, and reached version 1.0 on October 20, 2026.

Is llm-d a replacement for vLLM or SGLang? No. Model servers like vLLM and SGLang handle running models efficiently on accelerators. llm-d provides orchestration above them: routing requests to the right hardware, managing caches across GPU, CPU, and storage, splitting prefill and decode work, and controlling multi-tenant traffic.

Which accelerators and clouds does llm-d support? NVIDIA, AMD, and Intel GPUs plus Google TPU, on any cloud or on premises, anywhere Kubernetes runs. Adopters are not tied to one model provider, cloud, or hardware platform.

Is llm-d related to DeepSeek's inference engine? They are separate projects. DeepSeek's inference work appeared in our October 21 AI news roundup as one news item; llm-d is a distinct CNCF-governed, vendor-neutral open source project with its own architecture, community, and 1.0 release.

What does it cost to run llm-d? The software is Apache 2.0 licensed and free to use. You supply the Kubernetes cluster, the accelerators, and the model access. IBM's demonstrations suggest the economics can be favorable: up to 5 to 10 times lower per-token cost than equivalent commercial API pricing on input-heavy agentic workloads, at current cloud rental rates and sustained load. Treat those as vendor-reported demonstrations, not guarantees.

The Bottom Line

llm-d 1.0 is the moment an ambitious open source project becomes infrastructure you can run. The techniques were already impressive; the release adds what production needs: fair sharing between tenants, upgrades that do not interrupt traffic, a stable API, and serving that understands multi-turn agents. With more than 500 contributors, adopters from DigitalOcean to JetBrains, and benchmarks showing up to 10 times lower per-token cost than commercial APIs, the open alternative to proprietary inference economics has a hardened core. Read the source on GitHub, work through the guides on llm-d.ai, and compare notes with our Kubernetes explainer if you are new to the platform it runs on. This article is also available in Indonesian.