Kamaji and HAMi: testing multi-tenant Kubernetes for GPU workloads

An accelerated computing platform must answer two different questions: who controls each Kubernetes environment, and how workloads consume the available GPUs. Combining those questions often leads to misleading tests. Two concurrent Pods do not demonstrate isolation between customers, and two independent Kubernetes APIs do not demonstrate safe sharing of a physical GPU.

Kamaji manages hosted control planes; HAMi manages accelerator sharing within the cluster where it runs. Together, they do not automatically create a GPU pool spanning independent clusters. Kamaji explicitly assigns each worker node to a single tenant cluster. (kamaji.clastix.io)

This article presents a verifiable lab and a set of production acceptance criteria. Resource values are illustrative. The commands are a proposed procedure, not a record of an executed environment or measured benchmark results.

1. Concepts and responsibility boundaries

Kamaji: hosted Kubernetes control planes

Kamaji is an operator that reconciles TenantControlPlane resources through a Custom Resource Definition, or CRD. Each resource represents a Kubernetes control plane running as Pods in a management cluster, with its own API server, scheduler, and controller manager. Worker nodes connect to that tenant's endpoint. (kamaji.clastix.io)

A DataStore represents the backing storage for control-plane state. Sharing its infrastructure does not merge tenants into one Kubernetes API. It does, however, introduce an operational dependency that belongs in the failure and recovery model. (clastix.io)

HAMi: GPU allocation and resource enforcement

HAMi stands for Heterogeneous AI Computing Virtualization Middleware. For the NVIDIA path using HAMi-core, the main components are:

• Webhook: participates in admission and prepares workloads for the scheduling path.
• Scheduler and extender: select nodes and devices based on requested resources.
• Device plugin: integrates device allocation with the kubelet.
• HAMi-core: intercepts CUDA calls inside the container to control memory and compute utilization. (project-hami.io)

Do not treat this as a security boundary equivalent to a virtual machine or hardware partitioning. For this design, approve software-based sharing only after assessing workload trust, privileges, and compatibility. The enforcement mechanism itself relies on intercepting application calls. (github.com)

Test three dimensions independently

1. Administrative isolation: identities, APIs, Secrets, and permissions.
2. Execution isolation: workers, networking, storage, and runtime.
3. Resource isolation: GPU memory, concurrency, and performance interference.

Kubernetes multi-tenancy covers both the control plane and the data plane. Namespaces alone do not address every dimension. (kubernetes.io)

2. Lab architecture

Use the following proposed topology:

Management cluster
├── Kamaji + datastore
├── TenantControlPlane tenant-a
└── TenantControlPlane tenant-b

Tenant A: API A → HAMi A → GPU worker A → Pods A1 and A2
Tenant B: API B → HAMi B → GPU worker B → Pods B1 and B2

A1 and A2 can share the GPU on worker A. B1 and B2 can share the GPU on worker B. A and B do not automatically compete for the same physical GPU through Kamaji or HAMi. That follows from the exclusive association between workers and tenant clusters. (kamaji.clastix.io)

With only one GPU, use one tenant cluster and two namespaces to evaluate internal sharing. A second tenant can initially run without a GPU to test API separation, but that does not validate isolation between two accelerated worker pools.

3. Prerequisites and version control

Prepare these components before starting:

• A management cluster with Kamaji, its CRDs, cert-manager, and a healthy datastore.
• A suitable datastore StorageClass and a mechanism for exposing tenant endpoints.
• Separate Linux workers for A and B.
• kubectl, Helm, kubeadm, and envsubst on the administration workstation.
• A validated Container Network Interface, or CNI, plugin for each tenant.

The official Kamaji guide covers this bootstrap. This procedure starts with an operational management cluster; it is not a complete infrastructure installation guide. (kamaji.clastix.io)

GPU workers need an NVIDIA driver compatible with the hardware and CUDA image, NVIDIA Container Toolkit, and a configured runtime. This lab assumes host-managed drivers and the NVIDIA runtime as the default. Do not run competing device plugins registering nvidia.com/gpu on the same nodes. If GPU Operator is present, disable its device plugin for this pool. (project-hami.io)

Before running the lab, commit a version matrix covering management Kubernetes, tenant Kubernetes, Kamaji, CNI, HAMi, driver, Toolkit, and the test image. Pin validated artifacts and digests. Historical examples in documentation are not production patch recommendations.

4. Create and inspect the tenant control planes

Save this as tenant.yaml.tpl:

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: ${TENANT_NAME}
  namespace: tenants
spec:
  dataStore: default
  controlPlane:
    deployment:
      replicas: 1
    service:
      serviceType: LoadBalancer
  kubernetes:
    version: "${TENANT_K8S_VERSION}"
    kubelet:
      cgroupfs: systemd
    admissionControllers:
      - ResourceQuota
      - LimitRanger
  networkProfile:
    port: 6443
    serviceCidr: "${TENANT_SERVICE_CIDR}"
    podCidr: "${TENANT_POD_CIDR}"
    dnsServiceIPs:
      - "${TENANT_DNS_IP}"
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: 8132

The template uses Kamaji API fields. dataStore must reference an existing datastore; replicas: 1 is a lab choice. Adapt the CIDRs, version, and Service exposure. A LoadBalancer is not inherently private: configure the infrastructure implementation before applying it. (kamaji.clastix.io)

Set MGMT_CONTEXT to the management context and TENANT_K8S_VERSION to an exact validated Kubernetes version, including its v prefix:

: "${MGMT_CONTEXT:?Set the management cluster context}"
: "${TENANT_K8S_VERSION:?Set the validated Kubernetes version}"
export TENANT_K8S_VERSION

kubectl --context "$MGMT_CONTEXT" create namespace tenants \
  --dry-run=client -o yaml | kubectl --context "$MGMT_CONTEXT" apply -f -

export TENANT_NAME=tenant-a
export TENANT_SERVICE_CIDR=10.96.0.0/16
export TENANT_POD_CIDR=10.244.0.0/16
export TENANT_DNS_IP=10.96.0.10

envsubst '${TENANT_NAME} ${TENANT_K8S_VERSION} ${TENANT_SERVICE_CIDR} ${TENANT_POD_CIDR} ${TENANT_DNS_IP}' \
  < tenant.yaml.tpl > tenant-a.yaml

kubectl --context "$MGMT_CONTEXT" apply --dry-run=server -f tenant-a.yaml
kubectl --context "$MGMT_CONTEXT" apply -f tenant-a.yaml
kubectl --context "$MGMT_CONTEXT" -n tenants get tenantcontrolplanes,pods,svc

Repeat for tenant-b, using, for example, 10.97.0.0/16, 10.245.0.0/16, and 10.97.0.10. These are sample ranges. Check them against management networking, corporate networks, and future routing requirements.

Once the control plane is ready, retrieve its administrative kubeconfig:

umask 077
kubectl --context "$MGMT_CONTEXT" -n tenants \
  get secret tenant-a-admin-kubeconfig \
  -o jsonpath='{.data.admin\.conf}' | base64 --decode > tenant-a.kubeconfig

kubectl --kubeconfig tenant-a.kubeconfig cluster-info
kubeadm --kubeconfig tenant-a.kubeconfig token create --print-join-command

Run the generated join command only on A's workers. Repeat kubeconfig retrieval and joining for B. Install the CNI in each tenant using the corresponding CIDRs, then check node readiness. Administrative kubeconfigs are operational secrets, not files to commit. (kamaji.clastix.io)

kubectl --kubeconfig tenant-a.kubeconfig get nodes -o wide
kubectl --kubeconfig tenant-b.kubeconfig get nodes -o wide

5. Install HAMi in each tenant

Save hami-values.yaml for the host-managed driver and Toolkit setup:

devicePlugin:
  deviceListStrategy: envvar
  nvidiaDriverRoot: /
  gpuOperatorToolkitReady:
    enabled: false

These settings select environment-variable allocation, the host driver root, and no wait for a GPU Operator readiness marker. They assume NVIDIA is the default runtime. Do not reuse them unchanged with containerized drivers or a different runtime arrangement. (project-hami.io)

Choose a chart version whose documentation includes these settings:

: "${HAMI_CHART_VERSION:?Set the validated chart version}"
: "${GPU_NODE_A:?Set the name of tenant A GPU worker}"

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
helm show values hami-charts/hami --version "$HAMI_CHART_VERSION" > hami-defaults.yaml

kubectl --kubeconfig tenant-a.kubeconfig label node "$GPU_NODE_A" gpu=on --overwrite

helm upgrade --install hami hami-charts/hami \
  --kubeconfig tenant-a.kubeconfig \
  --namespace kube-system \
  --version "$HAMI_CHART_VERSION" \
  --values hami-values.yaml \
  --wait --timeout 10m

kubectl --kubeconfig tenant-a.kubeconfig -n kube-system get pods,daemonsets,deployments
kubectl --kubeconfig tenant-a.kubeconfig describe node "$GPU_NODE_A"

The chart, gpu=on node selection, and component checks follow the official installation workflow. Repeat with B's kubeconfig and GPU worker. Installing HAMi only in the management cluster does not automatically manage tenant workers. (project-hami.io)

For the first test round, disable memory overcommit and configure at least two shareable slots per GPU. Archive the effective Helm values so the experiment can be reproduced.

6. Run an actual CUDA workload

Namespace and quota

Save this as gpu-namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: gpu-lab
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-budget
  namespace: gpu-lab
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    requests.nvidia.com/gpu: "2"

This quota caps aggregate namespace requests. Here, requests.nvidia.com/gpu counts requested units, not necessarily two exclusive physical cards. A quota is not a physical capacity reservation. Kubernetes supports extended-resource quotas through the requests. prefix. (kubernetes.io)

HAMi also documents an extended quota mechanism using limits.nvidia.com/..., including multi-GPU accounting. That is HAMi-specific behavior, not an interchangeable form of the native quota above. Validate the implementation in your selected release before adopting it. (project-hami.io)

A Job with two concurrent executions

Save this as gpu-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-smoke
  namespace: gpu-lab
spec:
  completions: 2
  parallelism: 2
  backoffLimit: 0
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: gpu-smoke
      annotations:
        hami.io/node-scheduler-policy: binpack
        hami.io/gpu-scheduler-policy: binpack
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      nodeSelector:
        gpu: "on"
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: cuda-test
          image: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
          command: [python, -u, -c]
          args:
            - |
              import json
              import time
              import torch

              assert torch.cuda.is_available(), 'CUDA unavailable'
              torch.manual_seed(42)
              a = torch.randn((2048, 2048), device='cuda')
              b = torch.randn((2048, 2048), device='cuda')
              for _ in range(10):
                  c = a @ b
              torch.cuda.synchronize()
              start = time.monotonic()
              iterations = 0
              while time.monotonic() - start < 120:
                  c = a @ b
                  torch.cuda.synchronize()
                  iterations += 1
              elapsed = time.monotonic() - start
              print(json.dumps({
                  'device': torch.cuda.get_device_name(0),
                  'iterations': iterations,
                  'seconds': elapsed,
                  'iterations_per_second': iterations / elapsed,
                  'allocated_bytes': torch.cuda.memory_allocated(),
                  'reserved_bytes': torch.cuda.memory_reserved()
              }))
          env:
            - name: HOME
              value: /tmp
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
            limits:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "4096"
              nvidia.com/gpucores: "25"

The image is published in the PyTorch repository. Its fixed tag is an educational reference, not a security or upgrade recommendation for October 2026. Validate a supported image and pin its digest before production. (hub.docker.com)

With memoryFactor: 1, gpumem: 4096 requests 4096 MiB per requested GPU; use an integer without a Mi suffix. gpucores: 25 expresses a compute-control percentage, not 25 dedicated physical cores. HAMi-core uses time slicing for this control, so utilization can fluctuate. (project-hami.io)

Extended resources are specified under limits; Kubernetes uses those values as requests when requests are omitted. Host CPU and memory remain separate resource dimensions. (kubernetes.io)

The Job bounds execution time and retries to preserve failures as evidence. It does not serve traffic, so an HTTP readiness probe is unnecessary. CUDA synchronization prevents the timer from measuring only asynchronous operation submission. (kubernetes.io)

kubectl --kubeconfig tenant-a.kubeconfig apply -f gpu-namespace.yaml
kubectl --kubeconfig tenant-a.kubeconfig apply -f gpu-job.yaml
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab get pods -o wide
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab \
  wait --for=condition=complete job/gpu-smoke --timeout=650s
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab \
  logs -l app=gpu-smoke --prefix=true

Repeat in tenant B. Within each cluster, verify overlapping execution and the physical GPU UUID assigned to each Pod. binpack favors consolidation, but the annotation alone does not prove that both Pods shared the same card. (project-hami.io)

7. Test plan and acceptance criteria

Treat this matrix as a proposed validation plan, not observed results:

For memory pressure, create a copy of the Job with a different name and replace its code with a CUDA allocation larger than 4096 MiB. Run it only in the disposable pool. Distinguish a CUDA allocation error from host-memory OOMKilled. Do not infer memory isolation solely from the number displayed by nvidia-smi.

Include a warm-up round and repeated measurements. Compare identical workloads, images, and policies. For inference, measure p50, p95, and p99 latency, errors, throughput, and queue depth. Define the Service Level Objective, or SLO, before deciding how much sharing is acceptable.

8. Security and network isolation

For the test namespace, save this initial policy as deny-all.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: gpu-lab
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

The policy blocks ingress and egress for selected Pods when the CNI implements NetworkPolicy. The numerical test needs no network access. Real applications require explicit allowances for DNS, storage, queues, and services. This policy does not replace host controls or isolate GPU memory. (kubernetes.io)

Production acceptance should require least-privilege Role-Based Access Control, or RBAC; separate tenant identities; private APIs where practical; TLS; Secrets outside Git; encryption; and credential rotation. Restrict privileged Pods, hostPath, and consumers' ability to change enforcement policies.

A tenant user with cluster-admin should not be considered constrained by quotas and policies they can modify themselves. Enforce corporate budgets at worker provisioning and infrastructure layers as well. The distinction between trust and isolation is fundamental to multi-tenant design. (kubernetes.io)

9. From a business scenario to production

Problem: computer vision teams need independent Kubernetes APIs, while their smaller services cannot justify one exclusive GPU per process.

Proposed solution: one tenant per administrative domain, separate tenant workers, and HAMi sharing within each approved pool.

Implementation: publish GitOps templates, quotas, GPU profiles, and automated compatibility tests.

Expected outcome: greater autonomy with evidence of utilization and interference. Measure the savings; installing the components does not guarantee them.

For a production-oriented design, I recommend:

• Redundant control planes distributed across nodes and availability zones where available.
• A datastore with quorum, reliable storage, and reserved capacity.
• Spare GPU workers compatible with critical workloads.
• For inference services: multiple replicas, startupProbe, readinessProbe, livenessProbe, PodDisruptionBudget, and topology-aware placement.
• Queue- and demand-driven autoscaling, including validation of GPU worker provisioning and readiness.
• Checkpointing and restart procedures for interruptible processing.

These are design criteria. Two replicas on one GPU must not be accepted as evidence of resilience to that GPU failing.

Development versus production

10. Observability, cost, and maintenance

Organize dashboards by tenant, namespace, workload, worker, and physical UUID. Recommended signals include:

• Control plane: API availability, latency, errors, restarts, and datastore health.
• Scheduling: Pending duration, rejected requests, quota, and available capacity.
• GPU: allocated and used memory, utilization, temperature, and device failures.
• Application: throughput, latency, queue depth, CUDA errors, and loading time.

Prometheus and Grafana are possible building blocks. Centralize logs and connect API traces to queued work and processing. Before implementing per-Pod chargeback, validate metric attribution: sharing can restrict the mapping between physical-device metrics and containers, as documented for NVIDIA device-plugin time slicing. (docs.nvidia.com)

For FinOps, compare cost per completed task that meets the SLO, not just provisioned GPU count. Include idle workers, storage, cross-zone traffic, load balancers, datastore resources, and observability. Consider Spot only for workloads with tested recovery; use more predictable capacity for critical services.

Promote maintenance changes through a canary tenant first. Back up datastore state, management-cluster resources, and the material required for recovery. An isolated snapshot is not a replacement for the complete Kamaji restore procedure. (kamaji.clastix.io)

Tenant upgrades require separate handling of control planes and workers. Reserve temporary rollout capacity. Do not assume reversing spec.kubernetes.version is a safe rollback; use supported procedures, rehearsed restoration, or migration to another cluster. (kamaji.clastix.io)

11. Symptom-driven troubleshooting

Useful investigation commands:

kubectl --kubeconfig tenant-a.kubeconfig get pods -A
kubectl --kubeconfig tenant-a.kubeconfig get deployments -A
kubectl --kubeconfig tenant-a.kubeconfig get nodes
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab describe job gpu-smoke
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab describe resourcequota gpu-budget
kubectl --kubeconfig tenant-a.kubeconfig get events -A --sort-by=.metadata.creationTimestamp

# Replace POD with an observed Pod name.
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab describe pod POD
kubectl --kubeconfig tenant-a.kubeconfig -n gpu-lab logs POD

Use logs --previous only when a previous container instance exists. This Job uses restartPolicy: Never, so previous-container logs should not be assumed to exist within the same Pod. (kubernetes.io)

Avoid missing requests, excessive privileges, hardcoded credentials, premature log deletion, and copying lab settings directly into production. Before teardown, export the evidence and verify the kubeconfig for every destructive operation.

12. When to use this approach—and alternatives

Consider Kamaji with HAMi when independent APIs, multi-cluster automation, and workloads suitable for internal GPU sharing are genuine requirements. Avoid the combination when a small team can meet its requirements with one cluster, namespaces, and appropriate policies.

NVIDIA documents the isolation differences between time slicing and MIG. MIG can be part of a HAMi-based design, but it needs a separate validation profile; it is not the backend tested in this article. (docs.nvidia.com)

Practical summary

• Separate API isolation tests from GPU sharing tests.
• Install HAMi in the tenant that owns the accelerated workers.
• Prove sharing through physical-device identity and simultaneous execution.
• Test quotas, memory pressure, interference, and recovery.
• Preserve versions, manifests, metrics, and logs as evidence.
• Move to production only after demonstrating appropriate isolation and SLO compliance.

Conclusion

The value of this architecture is not the installation of two projects. It is the ability to turn autonomy, utilization, and reliability into verifiable operating contracts. Platform Engineering provides standardized paths, DevOps automates their delivery, and SRE defines the objectives and failure exercises.

A sound outcome may involve sharing some GPUs, dedicating others, and keeping selected tenants on separate infrastructure. The lab should produce evidence for that decision—not merely a screenshot of Pods in Running state.