Open-weight models on Google Cloud: architecture with llm-d, vLLM, and NVIDIA Dynamo

Serving large open-weight models on Google Cloud: what llm-d, vLLM, and NVIDIA Dynamo change in your architecture

An inference endpoint can pass its health checks and return HTTP 200 while still delivering a poor experience. The first token arrives late, generation stalls, and a few large requests dominate the accelerators. Serving a large language model is not simply a stateless API deployment with a larger memory allocation: prompt length, incremental generation, and attention state all influence scheduling. (cloud.google.com)

On Google Cloud, the architectural question extends beyond choosing a GPU. Teams must decide which engine runs the model, how requests reach a replica, where cached state lives, and whether prompt processing should run separately from generation. Google Kubernetes Engine, or GKE, provides a foundation for these designs, including integrations with vLLM, llm-d, and NVIDIA Dynamo. (docs.cloud.google.com)

The approach recommended here is evidence-driven: establish an aggregated baseline, identify its bottlenecks, and introduce specialized routing or disaggregation only when measurements justify it. The examples are instructional, not benchmarks or a certification that arbitrary project releases work together.

1. Three projects with different responsibilities

vLLM: the execution engine

vLLM loads model weights, executes inference, and exposes completion and chat interfaces. Its capabilities include continuous batching, which updates the set of active requests throughout generation, and paged KV-cache management associated with PagedAttention. These mechanisms improve how the engine uses memory and compute resources. (docs.vllm.ai)

vLLM also supports distributed execution. It is not restricted to one GPU or one server. However, distributing computation within a model replica is a different problem from selecting which replica should receive an incoming request. (docs.vllm.ai)

llm-d: Kubernetes-native distributed inference composition

llm-d brings together model servers, routing, and distributed inference capabilities. In its Gateway API Inference Extension integration, an InferencePool identifies backends, while the Endpoint Picker, or EPP, helps select a destination using signals such as load and cache affinity. This is request scheduling, not the Kubernetes scheduler placing Pods on nodes. (llm-d.ai)

vLLM can provide the execution layer. Backend support and feature coverage depend on the release; llm-d is neither another name for vLLM nor a guarantee that all supported engines expose identical capabilities. (llm-d.ai)

NVIDIA Dynamo: distributed coordination and state movement

Dynamo combines a frontend, KV-cache-aware routing, planning mechanisms, and data-transfer components. It can use engines including vLLM, SGLang, and TensorRT-LLM. Its architecture supports separating prefill from decode and moving state through components such as NVIDIA Inference Xfer Library, or NIXL. (nvidia.com)

The useful comparison is therefore not simply vLLM versus Dynamo. Possible designs include standalone vLLM, llm-d with vLLM, and Dynamo with vLLM. My recommendation is to select a primary coordination layer rather than stack routers and autoscalers without explicit ownership boundaries.

2. Prefill, decode, and the KV cache

Prefill processes input tokens and builds the initial attention state. Decode uses that state to generate additional tokens incrementally. Prefill tends to be compute-intensive, while decode is often constrained by memory access. The actual bottleneck still depends on the model, batch shape, and hardware. (llm-d.ai)

The key/value cache, or KV cache, stores intermediate attention tensors. It is not a response cache or the application's durable conversation history. Automatic Prefix Caching can reuse computation for compatible prefixes that have already been processed. It reduces prefill work; it does not remove the cost of generating the answer. (docs.vllm.ai)

Separate the main performance signals:

• TTFT — time to first token: how long the client waits for generation to begin.
• ITL — inter-token latency: the interval between delivered tokens.
• End-to-end latency: the duration of the complete request.
• Throughput: requests or tokens processed per unit of time.

TTFT and ITL help distinguish prefill pressure from decode pressure. Define targets for input and output length classes rather than relying on one endpoint-wide average. (docs.nvidia.com)

Parallelism does not automatically provide redundancy

Tensor parallelism distributes operations across accelerators; pipeline parallelism distributes model stages. Independent replicas serve separate requests. A replica that requires several accelerators is still one logical serving unit: additional GPUs do not automatically create additional failure tolerance. (docs.vllm.ai)

As a simple arithmetic estimate, 70 billion parameters at two bytes per parameter require approximately 140 decimal GB for weights alone. This is not a complete sizing estimate. KV cache, execution buffers, and runtime overhead require additional memory. Evaluate quantization against quality and compatibility rather than treating it as a free memory reduction.

3. How the request path changes

Aggregated serving

Client → authenticated gateway → vLLM replica → prefill and decode → streaming response

Each replica performs both phases. This is a useful baseline for determining whether the bottleneck lies in engine capacity, request distribution, or traffic characteristics. vLLM can run behind a Kubernetes Service without introducing a disaggregation framework. (docs.vllm.ai)

Inference-aware routing

Client → Gateway → EPP consultation → selected replica → streaming response

With llm-d and a compatible gateway, backend selection can go beyond generic load balancing. The gateway consults the decision layer and forwards the request to the selected backend. GKE provides a documented integration path through GKE Inference Gateway, with prerequisites for the gateway, extensions, and installed resources. (docs.cloud.google.com)

Separate prefill and decode pools

Request → coordination → prefill pool → KV transfer → decode pool → streaming response

This is a logical data flow, not a universal description of every control-plane handshake. llm-d's disaggregated path includes worker selection and a routing proxy. Dynamo can also deploy specialized workers in independently managed pools. (llm-d.ai)

Separating the phases allows different resource allocation, but adds KV transport, coordination, and failure modes. At low concurrency or without an appropriate network, aggregated serving may be simpler and faster. Disaggregation should be a benchmark-driven choice, not a mandatory maturity milestone. (docs.dynamo.nvidia.com)

4. Google Cloud infrastructure decisions

Google documents a Dynamo recipe combining GKE, vLLM, and A3 Ultra machines with H200 GPUs. This establishes an integrated deployment path; it does not guarantee performance for other models or hardware combinations. (cloud.google.com)

For platform design, I recommend four decisions before implementation:

1. Choose the replica topology: can one replica fit on a node, or does it require multiple hosts?
2. Confirm regional capacity: evaluate quotas, zonal availability, and reservations before scheduling a rollout.
3. Validate the accelerator network: machine families use different mechanisms, including GPUDirect-TCPX, TCPXO, and remote direct memory access, or RDMA.
4. Maintain a compatibility matrix: qualify GKE, drivers, images, engines, connectors, and controllers together.

GPU availability varies by zone, and Google distinguishes networking requirements by machine family. Installing a connector does not enable RDMA on arbitrary nodes. (cloud.google.com)

As an architectural policy, keep tightly coupled communication within an appropriate low-latency topology and place complete replicas in separate failure domains. Do not spread model shards across zones simply to satisfy a generic availability checklist.

5. Kubernetes example: an aggregated 72B baseline

The following manifest uses Qwen/Qwen2.5-72B-Instruct, two replicas, and four 80 GB A100 GPUs per Pod. The model has its own license: access to weights does not remove usage conditions. It is an illustrative choice, not a universal model recommendation. (huggingface.co)

Prerequisites: GKE Standard with minDomains support, nodes with four available GPUs, working drivers, two eligible zones, access to the model repository, and sufficient ephemeral storage. The nvidia-a100-80gb selector matches the documented GKE accelerator type. (cloud.google.com)

The v0.10.2 image is a fixed historical reference for documented syntax, not a production release recommendation for October 2026. Before deployment, select a patched and qualified release, pin its digest, and pin the model revision. The example combines native Kubernetes APIs with documented vLLM arguments. (docs.vllm.ai)

apiVersion: v1
kind: Namespace
metadata:
  name: model-serving
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vllm
  namespace: model-serving
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  namespace: model-serving
spec:
  replicas: 2
  minReadySeconds: 30
  progressDeadlineSeconds: 3600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: vllm-qwen
  template:
    metadata:
      labels:
        app: vllm-qwen
    spec:
      serviceAccountName: vllm
      terminationGracePeriodSeconds: 300
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-a100-80gb
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      topologySpreadConstraints:
        - maxSkew: 1
          minDomains: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: vllm-qwen
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.10.2
          command: ["vllm", "serve"]
          args:
            - Qwen/Qwen2.5-72B-Instruct
            - --host=0.0.0.0
            - --port=8000
            - --served-model-name=qwen-72b
            - --tensor-parallel-size=4
            - --dtype=bfloat16
            - --max-model-len=8192
            - --max-num-seqs=16
            - --gpu-memory-utilization=0.90
            - --enable-prefix-caching
            - --disable-log-requests
          env:
            - name: HF_HOME
              value: /models
          ports:
            - name: http
              containerPort: 8000
          resources:
            requests:
              cpu: "16"
              memory: 128Gi
              ephemeral-storage: 300Gi
              nvidia.com/gpu: "4"
            limits:
              cpu: "32"
              memory: 192Gi
              ephemeral-storage: 350Gi
              nvidia.com/gpu: "4"
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 180
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 20
            timeoutSeconds: 5
            failureThreshold: 6
          volumeMounts:
            - name: model-cache
              mountPath: /models
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: model-cache
          emptyDir:
            sizeLimit: 275Gi
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen
  namespace: model-serving
spec:
  type: ClusterIP
  selector:
    app: vllm-qwen
  ports:
    - name: http
      port: 8000
      targetPort: http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-qwen
  namespace: model-serving
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-qwen

Understanding the configuration

• tensor-parallel-size=4 divides each replica across four GPUs. The nominal total is eight GPUs, not four.
• max-model-len limits the combined input and output context. max-num-seqs limits sequences processed per iteration; it does not independently implement a bounded HTTP queue.
• gpu-memory-utilization controls the engine's memory budget, not GPU compute utilization. (docs.vllm.ai)

The startup probe provides roughly 30 minutes for initialization before triggering restarts. Readiness then governs traffic eligibility, while liveness detects persistent failures. Tune these settings from measurements: aggressive probes can turn temporary slowness into repeated restarts. (kubernetes.io)

The spread constraint requires two eligible domains to accommodate both replicas. The PodDisruptionBudget limits voluntary disruptions that use the eviction API; it cannot prevent a zone failure. Deployment rollout behavior is controlled separately by maxUnavailable and maxSurge. (kubernetes.io)

Production gaps: this example does not include authentication, TLS, NetworkPolicy, metrics collection, or a hardened image. A ClusterIP is not a security boundary. Qualify non-root execution and required device access before promotion. (docs.vllm.ai)

The emptyDir cache is disposable; plan for versioned weight storage and predictable staging. The resource requests require corresponding allocatable node capacity. Also, maxSurge: 1 can require four additional GPUs. Without that headroom, an upgrade may remain blocked waiting for capacity. (docs.vllm.ai)

Save the manifest as serving.yaml and run it in an isolated environment:

kubectl apply -f serving.yaml
kubectl rollout status deployment/vllm-qwen -n model-serving --timeout=3600s
kubectl get pods -n model-serving -o wide
kubectl port-forward -n model-serving service/vllm-qwen 8000:8000

From another terminal:

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen-72b","messages":[{"role":"user","content":"Explain readiness probes."}],"max_tokens":128,"temperature":0}'

This checks the basic API path. It does not validate replica balancing, fault tolerance, or performance under load.

6. When to introduce llm-d or Dynamo

For llm-d, begin with the documented GKE Inference Gateway integration and recipes compatible with your selected release. Inference Quickstart can also generate configurations for the llm-d serving stack. Review generated manifests before admitting them into GitOps. (docs.cloud.google.com)

For Dynamo, select a recipe matching the hardware and backend version. Examples cover aggregated and disaggregated vLLM workers, but their connector and startup requirements should not be inferred from an ordinary vLLM Deployment. (docs.dynamo.nvidia.com)

My evaluation sequence would be:

1. Measure aggregated serving with cold and warm caches.
2. Test load-aware and prefix-aware routing.
3. Determine whether improvements come from less recomputation or simply more capacity.
4. Introduce separate prefill and decode workers when interference is measurable.
5. Preserve the aggregated baseline as a rollback option.

Avoid mixing Custom Resource Definitions, or CRDs, from tutorials targeting different releases. Kubernetes accepting a resource does not prove that the engine, connector, and router share a compatible operational contract.

7. Enterprise example: a document assistant

Problem: consider a hypothetical Retrieval-Augmented Generation, or RAG, service where short questions and extensive document analysis share one inference endpoint.

Proposed solution: introduce service classes and explicit limits for tokens, concurrency, and queue time. The hypothesis is that long requests degrade the responsiveness of short interactions.

Implementation: retain a vLLM baseline, replay sanitized traffic, and compare three configurations: aggregated serving, aggregated serving with specialized routing, and disaggregated serving. Keep the model revision, workload distribution, and accelerator budget consistent.

Expected outcome, not a measured result: select the design that completes more requests within service objectives per unit of cost. If distributed coordination does not deliver enough value, keeping the simpler design is a sound engineering decision.

8. Production capacity, availability, and autoscaling

Treat autoscaling as two separate concerns: requesting additional workers and obtaining infrastructure on which they can run. The Horizontal Pod Autoscaler, or HPA, supports custom metrics when the required metrics infrastructure is configured. Increasing the replica count does not guarantee immediate GPU availability. (kubernetes.io)

Consider queue depth, incoming tokens, active generations, and cache pressure as scaling signals. Avoid relying exclusively on CPU utilization. If the serving stack includes a planner, establish which controller owns each resource to prevent conflicting reconciliation loops.

For critical endpoints, my production checklist includes:

• Warm capacity to absorb failures and model-loading delays.
• Bounded queues, backpressure, and explicit overload rejection.
• Drain deadlines aligned with the longest accepted streams.
• Enough remaining capacity after losing a replica or zone.
• Failure tests covering workers, gateways, and transport paths.

Do not assume a stream can resume transparently after its worker fails. Agree with application owners on when retries are safe, especially when an agent can trigger external side effects.

A single replica and simplified telemetry may be reasonable in development. Production approval should cover redundancy, limits, isolation, alerts, recovery, and an exercised rollback plan—not merely a Pod in the Running state.

9. Observe the user experience, not just the GPUs

vLLM exposes Prometheus metrics through /metrics, including signals for running and waiting requests. Available metric names vary by release, so inspect the deployed endpoint before building dashboards. (docs.vllm.ai)

I recommend four telemetry groups:

• User experience: TTFT, ITL, total duration, errors, and cancellations.
• Engine behavior: queueing, processed tokens, KV-cache pressure, and recomputation.
• Infrastructure: GPU memory, CPU, container memory, disk, and restarts.
• Distributed execution: KV-transfer duration, pool imbalance, and communication failures.

Use Prometheus and Grafana, or equivalent services, to correlate these signals. Instrument application and gateway traces to distinguish document retrieval, queueing, and inference time. Avoid logging full prompts by default.

Define a Service Level Objective, or SLO, for each request class and alert on its degradation. A busy GPU is a capacity signal; it is not evidence that users are receiving an acceptable service.

10. Security and governance

Use Workload Identity Federation for GKE to authorize access to Google Cloud services without distributing static service-account keys. Keep the distinction between IAM permissions for cloud resources and Kubernetes RBAC permissions for the Kubernetes API. (cloud.google.com)

The trust boundary must include internal communication. vLLM's security documentation warns that distributed communication should not be considered secure by default. Internal ports and KV transfers must remain restricted to trusted networks. TLS at the gateway does not automatically protect these paths. (docs.vllm.ai)

As platform policy, I recommend:

• TLS and authentication at ingress, with authorization and consumer quotas.
• NetworkPolicy and firewall rules appropriate for every network interface in use.
• Rotated Secrets with least-privilege access and no embedded credentials.
• Verified images, patched dependencies, and integrity-checked model weights.
• Explicit cache-isolation rules between tenants.
• Appropriate encryption and retention for weights, logs, and persistent data.

Open-weight does not automatically mean permissively licensed. Record the model terms and revision during approval; the example model's card explicitly identifies its Qwen license. (huggingface.co)

11. FinOps, upgrades, and recovery

For FinOps, calculate cost per useful unit of work delivered within the SLO. Include idle capacity, networking, storage, load balancing, and observability. A test that maximizes tokens per second while missing the required latency does not establish product-level efficiency.

Compare regular capacity with Spot only for workloads whose interruption behavior has been validated. Do not make interruptible capacity the sole availability strategy for a critical service. Include rollout surge capacity and cache warming in cost projections.

Version the complete release: image, weights, tokenizer, chat template, runtime arguments, and routing configuration. Use a canary with representative traffic before promoting changes.

A Deployment rollback restores a previous Pod template; it does not automatically reverse external changes to weights, CRDs, or dependent services. Prefer immutable artifacts and retain the complete previous configuration. (kubernetes.io)

Back up the artifacts and business state required for recovery. Do not treat transient KV cache as a database: plan for rebuilding it and for the cold-cache performance impact during service restoration.

12. Troubleshooting

Start with a short investigation sequence:

kubectl get nodes
kubectl get pods -n model-serving -o wide
kubectl describe deployment vllm-qwen -n model-serving
kubectl describe pod <pod> -n model-serving
kubectl logs <pod> -n model-serving -c vllm
kubectl logs <pod> -n model-serving -c vllm --previous
kubectl get events -n model-serving --sort-by=.metadata.creationTimestamp

These are diagnostic hypotheses, not exclusive mappings. Kubernetes, vLLM, and Dynamo documentation provides initialization, memory, and transport requirements that help confirm the cause. (docs.vllm.ai)

13. When to use each approach—and when to simplify

The following is a decision framework rather than a universal ranking:

Vertex AI provides deployment paths for open models and custom vLLM containers. Actual availability depends on the model and deployment mode. (docs.cloud.google.com)

Do not choose disaggregation solely because a model is large. Likewise, avoid building a complex serving platform when traffic volume, team capacity, and control requirements do not justify maintaining it.

Practical takeaways

• Start with a reproducible baseline and a clear definition of acceptable service.
• Separate model execution, replica selection, and infrastructure placement.
• Evaluate caching and routing before adding more components.
• Treat networking and KV movement as part of the critical path.
• Promote only configurations with validated security, recovery, and cost behavior.

Conclusion

The most important architectural decision is not how many projects to adopt. It is determining which layer owns each bottleneck and what evidence justifies additional complexity.

For DevOps and Platform Engineering, that means turning versions, topologies, and policies into reproducible deployment paths. For SRE, it means operating service objectives, capacity, and failure behavior—not merely healthy containers. A mature platform should be able to explain why it uses llm-d or Dynamo, demonstrate the benefit, and safely return to a simpler design when the evidence calls for it.