llm-d 1.0 lab: distributed inference on Kubernetes and production readiness for agents

An enterprise agent may alternate between document retrieval, tool calls, and additional model requests. This produces growing contexts, repeated prefixes, and uneven concurrency. Routing requests without accounting for those properties can waste cached work and overload individual replicas. The llm-d optimized baseline is designed to address that mismatch. (llm-d.ai)

llm-d 1.0 was announced on October 20, 2026. This article, dated October 22, 2026, uses release v1.0.0 as its reference rather than assuming a future release exists. The lab is a reproducible proposal based on official artifacts—not an executed benchmark or a production-certified configuration. (llm-d.ai)

The operational question is straightforward: what evidence should a platform team require before moving agent traffic onto this infrastructure?

1. What llm-d provides—and where its responsibility ends

llm-d is an open inference-serving stack that connects engines such as vLLM and SGLang with distributed inference infrastructure. Its Kubernetes deployment paths combine routing, model execution, cache management, and operational components. It is not a new language model. (llm-d.ai)

For this lab design, llm-d owns the inference layer. Agent orchestration, tool authorization, task persistence, and approval workflows remain application responsibilities. Treat that separation as an architectural requirement, not as functionality implicitly provided by a model server.

It also helps to distinguish three ways to distribute inference:

• Independent replicas: multiple servers handle requests for the same model.
• Model parallelism: one logical instance uses multiple accelerators; tensor parallelism splits model operations across devices.
• Prefill/decode disaggregation: initial context processing and token generation run as separate serving roles.

These are different dimensions and can be combined. The baseline below uses replicas with tensor parallelism; it does not automatically enable disaggregated serving. (llm-d.ai)

2. Components and the request path

Router, proxy, and Endpoint Picker

The llm-d Router consists of an application-layer proxy and an Endpoint Picker—EPP. The proxy receives and forwards traffic; the EPP chooses a model server using inference signals and policies. Compatible integrations use the ext-proc external-processing protocol. Request scheduling here is distinct from the Kubernetes scheduler placing a Pod on a node. (llm-d.ai)

A simplified flow is:

Agent → Proxy → Selected model server → Response tokens
          ↕
         EPP ← Server state and policies

InferencePool and Gateway API

InferencePool is a custom resource from Gateway API Inference Extension. It selects Pods through labels, declares serving ports, and can reference an EPP Service. Its schema is installed through a Custom Resource Definition—CRD. A pool does not create the model servers. (gateway-api-inference-extension.sigs.k8s.io)

In Gateway mode, a compatible controller connects resources such as Gateway and HTTPRoute to inference backends. This lab uses standalone mode with an integrated proxy, so an external gateway is not another prerequisite for the first serving experiment. (llm-d.ai)

Model engines and KV cache

The engine loads weights and executes inference. The key-value—KV—cache retains intermediate attention state. Prefix-aware routing attempts to reuse existing state; precise indexing follows cache events; offloading extends capacity into other memory or storage tiers. These mechanisms solve different problems and should be enabled according to the selected deployment path. (llm-d.ai)

Controllers for distributed topologies

LeaderWorkerSet organizes Pod groups belonging to a distributed replica. DisaggregatedSet coordinates multi-role topologies, including prefill and decode, and their revisions. These controllers become relevant when the experiment moves beyond independently managed serving replicas. (llm-d.ai)

What to evaluate in version 1.0

The release emphasizes the router's v1 configuration API, multi-tenant quality of service, high-availability operations, and agent session capabilities. Priority and fairness require configuration; a basic installation should not be mistaken for automatic tenant isolation. (llm-d.ai)

3. Build the baseline before introducing disaggregation

Prerequisites and capacity

Use a dedicated test cluster, Bash, Git, kubectl, Helm, and access to image registries and model weights. For the NVIDIA path below, verify working drivers and a device plugin, with allocatable nvidia.com/gpu resources on the nodes. (llm-d.ai)

The selected official overlay serves Qwen/Qwen3-32B with two replicas and tensor parallelism of two: four GPUs in total. The published reference uses 80 GB H100 GPUs. Each model Pod also requests 8 CPUs and 96 GiB of host memory; the router and cluster services require additional capacity. This is not a lightweight laptop exercise. (llm-d.ai)

For different hardware, select another documented overlay or develop a measured adaptation. Reducing the GPU count alone does not establish that model weights, context, and cache will fit in available memory.

Pin artifacts and install prerequisites

The following commands adapt the official quickstart while checking out a release tag instead of tracking a mutable branch. guides/env.sh supplies chart coordinates and component versions. Installing CRDs changes cluster-scoped resources; obtain approval before running this against shared infrastructure. (llm-d.ai)

git clone --branch v1.0.0 --depth 1 https://github.com/llm-d/llm-d.git
cd llm-d
export REPO_ROOT="$PWD"
source guides/env.sh
export NAMESPACE=llm-d-lab
export GUIDE_NAME=llm-d-lab

git rev-parse HEAD
kubectl get nodes
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml"

Record the commit, resolved container images, and rendered manifests in the experiment report. For production, the proposed approach is to promote images by digest and manage approved artifacts through GitOps rather than relying solely on tags.

Create the Secret and deploy the stack

The overlay expects a Secret named llm-d-hf-token containing the HF_TOKEN key. The commands below read the token without terminal echo and avoid embedding its literal value in shell history. Run them on a trusted workstation without set -x, and remove the temporary file even if a command fails. Replace this manual workflow with managed secrets and rotation for production. (raw.githubusercontent.com)

umask 077
read -rsp 'Hugging Face token: ' HF_TOKEN
printf '\n'
TOKEN_FILE=$(mktemp)
printf '%s' "$HF_TOKEN" > "$TOKEN_FILE"
kubectl create secret generic llm-d-hf-token -n "$NAMESPACE" --from-file="HF_TOKEN=$TOKEN_FILE" --dry-run=client -o yaml | kubectl apply -f -
rm -f "$TOKEN_FILE"
unset HF_TOKEN TOKEN_FILE

helm upgrade --install "$GUIDE_NAME" "$ROUTER_STANDALONE_CHART" --version "$ROUTER_CHART_VERSION" -n "$NAMESPACE" -f guides/recipes/router/base.values.yaml -f guides/optimized-baseline/router/optimized-baseline.values.yaml

kubectl kustomize guides/optimized-baseline/modelserver/gpu/vllm/base/ > modelserver.rendered.yaml
kubectl apply --dry-run=server -n "$NAMESPACE" -f modelserver.rendered.yaml
kubectl apply -n "$NAMESPACE" -f modelserver.rendered.yaml
kubectl wait -n "$NAMESPACE" --for=condition=Ready pod -l llm-d.ai/guide=optimized-baseline --timeout=3600s
kubectl get pods,svc -n "$NAMESPACE"

The Helm coordinates and Kustomize path follow the documented baseline. Review modelserver.rendered.yaml before applying it: resources, volumes, images, probes, and permissions must match your cluster. Server-side dry-run checks admission, not GPU availability or successful model initialization. (llm-d.ai)

Exercise the endpoint

Forward the standalone Service to your workstation:

kubectl port-forward -n "$NAMESPACE" svc/llm-d-lab-epp 8000:80

In another terminal, run:

curl --fail-with-body -N http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' -d '{"model":"Qwen/Qwen3-32B","prompt":"Explain Kubernetes readiness probes briefly.","max_tokens":128,"stream":true}'

This checks the client → proxy → model path using the quickstart's completions interface. It does not establish balanced routing, cache benefits, or serving capacity. There is no reason to make the endpoint publicly accessible for this smoke test. (llm-d.ai)

4. Kubernetes YAML: protect against voluntary disruptions

Save this complete manifest as modelserver-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: optimized-baseline-modelservers
  namespace: llm-d-lab
spec:
  minAvailable: 1
  selector:
    matchLabels:
      llm-d.ai/guide: optimized-baseline
      llm-d.ai/model: Qwen3-32B
      llm-d.ai/accelerator-vendor: nvidia

This PodDisruptionBudget—PDB selects model servers using the overlay's labels and requires at least one available Pod during voluntary evictions. Adapt the namespace and selector if you change the model. With one replica, it may block a drain. It does not protect against hardware failure or replace the workload controller's rollout strategy. (raw.githubusercontent.com)

kubectl apply --dry-run=server -f modelserver-pdb.yaml
kubectl apply -f modelserver-pdb.yaml
kubectl get pdb -n llm-d-lab

The PDB supplements the lab; it does not make the architecture highly available. Production design must still address placement, surviving capacity, and router failures.

5. Gateway API integration: understand the pool contract

The following complete resource illustrates a separate Gateway-mode integration. Do not apply it over the standalone lab as a mandatory installation step. It assumes an EPP configured for this pool, exposed through an agents-epp Service, and an already installed compatible gateway:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: agents-pool
  namespace: llm-d-lab
spec:
  selector:
    matchLabels:
      llm-d.ai/guide: optimized-baseline
      llm-d.ai/model: Qwen3-32B
      llm-d.ai/accelerator-vendor: nvidia
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: agents-epp
    port:
      number: 9002
    failureMode: FailOpen

selector identifies model servers, targetPorts declares their serving port, and endpointPickerRef connects the pool to its EPP. Match names, ports, and labels to actual resources. This object alone does not create a Service, EPP, TLS listener, or HTTPRoute. (gateway-api-inference-extension.sigs.k8s.io)

FailOpen permits forwarding without optimized endpoint selection when the EPP fails. Make that an explicit design decision. For this project, authentication and authorization must remain independent of scheduling, and failure tests must identify which policies no longer apply. Transport availability is not equivalent to preserving every quality-of-service guarantee. (llm-d.ai)

6. A realistic experiment: internal support agents

Consider this hypothetical scenario, with no measured results implied:

Problem → agents retrieve documentation and interact with a ticketing system. Each step extends the context, while interactive teams compete with batch processing.

Proposed solution → keep durable application state separate from serving. PostgreSQL stores tasks and audit records, workers execute authorized tools, and llm-d provides inference. No component receives administrative credentials for convenience.

Proposed implementation → compare three configurations using equivalent hardware, model settings, and request datasets:

1. A routing baseline without prefix affinity.
2. llm-d's load- and prefix-aware baseline.
3. Prefill/decode disaggregation, but only after measuring the previous configuration.

Expected experiment output → evidence rather than a promised percentage improvement. Report latency, errors, resource consumption, and task completion for each configuration.

Disaggregation adds KV transfer between serving roles and transport dependencies. The official path documents particular engine, accelerator, and networking combinations; creating Deployments named prefill and decode is not sufficient. (llm-d.ai)

The proposed test matrix includes cold and warm cache, shared and unrelated prefixes, long contexts, cancellations, and bursts. Add server loss, EPP restarts, and rollouts during streaming. Record input and output token distributions for every run—not just requests per second.

7. Observability and acceptance criteria

The documentation provides Prometheus, Grafana, and OpenTelemetry integration. For this lab, correlate metrics and traces from the application through the serving engine without recording sensitive prompts by default. (llm-d.ai)

Define Service Level Objectives—SLOs before testing. The proposed measurement plan is:

Evaluate percentiles by traffic class and context profile. Proposed rejection criteria include higher throughput achieved through tenant starvation, interrupted streams, or duplicated actions.

For alerting, prioritize error-budget consumption, sustained queue growth, and insufficient ready capacity. Avoid session identifiers as metric labels; use traces and access-controlled logs for request-level correlation.

8. Production design: capacity, availability, and security

Scale on meaningful demand signals

The documented Kubernetes Event-driven Autoscaling—KEDA path uses EPP signals such as queue depth, token backlog, and saturation. These better reflect inference demand than a CPU-only policy. Increasing replicas still cannot solve a shortage of accelerator nodes by itself. (llm-d.ai)

For capacity planning, measure the following separately:

• Prefill and decode capacity for each request profile.
• Node provisioning, download, initialization, and warm-up time.
• Memory required for weights, cache, and concurrency.
• Remaining capacity after losing a failure domain.
• Temporary headroom required during upgrades.

The proposed initial policy keeps warm capacity for critical traffic and bounds admission while new replicas are becoming ready.

High availability requires more than replicas: 2

Router standby and active-active modes behave differently. In active-active mode, flow-control state is local to each EPP, and plugins have compatibility restrictions. Increasing replicas without reviewing these semantics can change fairness and cache affinity. (llm-d.ai)

For the proposed architecture, spread independent replicas across nodes and, where practical, zones. Preserve network locality within tightly coupled groups. Do not automatically distribute members of a single model instance across zones just to satisfy generic anti-affinity guidance. Placement should follow interconnect topology and failure-test results. (llm-d.ai)

Use startupProbe for initialization, readinessProbe for serving eligibility, and livenessProbe for failures recoverable through restart. Aggressive liveness checks during model loading can cause restart loops. Set their budgets using measured startup behavior. (kubernetes.io)

Security and isolation

The production proposal requires least-privilege Role-Based Access Control—RBAC, separate ServiceAccounts, and no administrative permissions for agent workers. Disable automatic ServiceAccount token mounting when the process does not need the Kubernetes API. (kubernetes.io)

Use network policies to allow only necessary flows among the proxy, EPP, engines, DNS, telemetry, and storage. NetworkPolicy requires a supporting cluster network implementation and does not replace application authentication. (kubernetes.io)

Additional project requirements should include TLS at ingress, encrypted persistent state, credential rotation, trusted tenant identity, and image review. A client-supplied priority value must not be accepted as proof of authorization.

For agents, require bounded tool access, isolated execution, human approval for sensitive actions, and idempotency keys. Explicitly test a timeout occurring after a tool has already changed an external system.

Session state is not a backup strategy

The Responses API deployment path supports continuity through previous_response_id. That does not justify assuming durability, cross-replica recovery, or business-compliant retention. Verify those properties in the deployed configuration. (llm-d.ai)

For this design, back up durable task and audit state and test restoration. Retain approved model artifacts and configuration revisions. The recovery plan must specify whether a session can be reconstructed and when an agent must restart its workflow.

9. Upgrades, rollback, and FinOps

Treat the chart, CRDs, controllers, engine, model, tokenizer, and transport configuration as a compatibility unit. Release 1.0 documents specific LeaderWorkerSet requirements and DisaggregatedSet enablement for the disaggregated path. Do not combine older tutorial versions indiscriminately. (github.com)

The proposed promotion workflow is:

1. Render and validate manifests in CI.
2. Run functional, load, and failure tests in staging.
3. Canary a bounded amount of traffic with identifiable sessions.
4. Drain requests before removing the old revision.
5. Roll back the compatible set, not just one worker image.

Decide what happens to open sessions before the rollback begins. A release's zero-downtime operating goals are not a guarantee for every deployment configuration.

For FinOps, compare cost per successfully completed task within its SLO as well as cost per token. Include idle accelerator capacity, router CPU, weight storage, cross-zone networking, load balancers, and telemetry.

Evaluate Spot capacity only for workloads that tolerate interruption and recovery. The proposed design keeps its critical minimum capacity on non-interruptible infrastructure. Account for context recomputation, model reloads, and temporary duplication during canaries. No current prices or percentage savings are assumed here.

10. Development versus production: avoid accidental promotion

Avoid committed credentials, uncontrolled mutable images, missing probes, excessive privileges, unbounded queues, unlimited retries, and unnecessary public endpoints. A successful first completion is not a reason to promote lab defaults unchanged.

11. Troubleshooting by symptom

Start with:

kubectl get nodes
kubectl get pods -n llm-d-lab -o wide
kubectl get deployments -n llm-d-lab
kubectl get events -n llm-d-lab --sort-by=.metadata.creationTimestamp
kubectl describe pod <pod> -n llm-d-lab
kubectl logs <pod> -n llm-d-lab --all-containers=true
kubectl logs <pod> -n llm-d-lab -c <container> --previous
kubectl get inferencepools.inference.networking.k8s.io -n llm-d-lab

Use the following as an investigation plan, not an automatic diagnosis:

These checks follow the documented GPU, probe, pool, and disaggregation mechanisms. Increasing every timeout is not a general remedy. (kubernetes.io)

When finished, remove only the dedicated lab resources:

helm uninstall llm-d-lab -n llm-d-lab
kubectl delete namespace llm-d-lab

Do not delete shared CRDs as part of local cleanup. Separately verify whether external infrastructure or provisioned storage created for the experiment also needs to be decommissioned.

12. When to use llm-d—and what else to compare

Evaluate llm-d when self-hosted inference demand is substantial, contexts are reusable, multiple consumers share capacity, and the team can operate Kubernetes and accelerators.

At low scale or with simpler requirements, compare a serving engine behind conventional load balancing first. A smaller operational footprint may matter more than distributed optimizations. Managed APIs should also be considered, with an explicit review of data handling, contracts, cost, and provider dependence.

KServe provides a serving platform with separate control and execution layers, and llm-d's documentation also describes KServe integration. These are not necessarily mutually exclusive choices: a serving management layer can sit above a distributed inference stack. (kserve.github.io)

The proposed decision rule is incremental: add disaggregation, distributed caching, or further control mechanisms only when an experiment demonstrates enough benefit to justify their operational cost.

Practical summary

• Pin the release and record the actual component matrix.
• Establish a controlled baseline before adding complexity.
• Treat caching, affinity, and disaggregation as separate decisions.
• Evaluate agent task outcomes, not just tokens per second.
• Test failures, overload, sessions, and rollback before promotion.
• Separate durable state and tool authorization from inference serving.

Conclusion

This lab is intended to turn distributed inference into a verifiable engineering decision. Success is not the number of components running in the cluster. It is evidence that the architecture meets latency, reliability, security, and cost objectives under representative traffic.

Within Platform Engineering, the outcome should become a standardized deployment path with versioned artifacts, operating limits, and clear ownership. DevOps automates delivery; SRE demands evidence of recovery and predictable failure behavior. Only after that validation should an agent workload move from experiment to production service.