OpenKruise Agents: managing agent sandboxes on Kubernetes

An agent platform must execute tools without turning every tool invocation into unrestricted infrastructure access. Consider an assistant that accepts files, installs dependencies and runs Python: the platform must manage identity, network access, session lifetime and data ownership, not just CPU and memory.

OpenKruise Agents addresses these execution environments through Kubernetes resources and pre-provisioned capacity. Its scope includes tool environments, notebooks and workspaces; it is not a framework for defining an agent's reasoning workflow. (github.com)

This guide separates three concerns that deserve independent evaluation: lifecycle management, available capacity and security isolation. The walkthrough demonstrates the first two. The production recommendations address the additional controls needed for the third.

What OpenKruise Agents provides

OpenKruise Agents is an OpenKruise subproject focused on agent workloads. It provides Kubernetes APIs, compatibility with portions of the E2B protocol, pooling and session-management capabilities. Protocol compatibility should not be interpreted as complete feature parity with the E2B service. (github.com)

The examples use agents.kruise.io/v1alpha1. Do not confuse this API with other projects exposing a resource named Sandbox, including Kubernetes SIG Agent Sandbox. Matching resource names do not make their manifests interchangeable. (openkruise.io)

Version scope: the installation documentation reviewed for this article presents the 0.6.0 line. That is the reference baseline, not a compatibility guarantee for every subsequent release. Inspect the installed CRD schemas and pin approved charts and images before deployment. (openkruise.io)

Core resources

A CustomResourceDefinition, or CRD, extends the Kubernetes API. The central OpenKruise Agents resources are:

• Sandbox: an instance and its lifecycle.
• SandboxSet: a managed collection of pre-created sandboxes.
• SandboxClaim: a request to allocate an available sandbox.
• SandboxTemplate: an immutable template revision.
• Checkpoint: captured intermediate state, subject to platform capabilities. (openkruise.io)

A warm pool is execution capacity prepared before a request arrives. Distinguish unused inventory from active sessions: SandboxSet.spec.replicas describes unclaimed sandboxes, including available instances and those still being created. It is not a global session limit. (raw.githubusercontent.com)

The controller replenishes inventory after allocation. Consequently, sizing the pool alone underestimates infrastructure requirements when many sessions are already running. (openkruise.io)

Architecture and request flow

The main components have distinct responsibilities:

• sandbox-controller: resource reconciliation and admission webhooks.
• sandbox-manager: E2B-compatible management requests.
• sandbox-gateway: traffic forwarding to sandboxes.
• agent-runtime: in-sandbox operations, including commands and files. (openkruise.io)

A conceptual Kubernetes API flow is:

Platform backend → SandboxClaim → controller → available Sandbox → Pod

An E2B integration adds these paths:

Application → sandbox-manager → Kubernetes resources

Authorized client → sandbox-gateway → sandbox

These flows summarize the documented architecture rather than prescribing one mandatory network topology. (openkruise.io)

For production, keep business authorization in the platform backend: who may start a session, which template they may use and which data they may access. Do not give code running inside a sandbox broad permissions to create Pods.

Walkthrough: a pool and a declarative claim

This example deliberately has a narrow scope. It demonstrates provisioning, readiness and allocation using a Python HTTP server. It does not implement E2B remote execution and must not be used to run untrusted code.

1. Install the controller

In a laboratory cluster with Helm 3 configured:

helm repo add openkruise https://openkruise.github.io/charts/
helm repo update
helm upgrade --install agents-sandbox-controller openkruise/agents-sandbox-controller --namespace sandbox-system --create-namespace --version 0.6.0
kubectl get crd sandboxsets.agents.kruise.io sandboxclaims.agents.kruise.io

The controller is sufficient for this CRD-based exercise. E2B access requires the management and access components as well. The documented agent-runtime injection requires Kubernetes 1.29 or later; this is a minimum prerequisite, not a recommendation to operate an old cluster release. (openkruise.io)

2. Create the namespace, identity and pool

Save this manifest as sandbox-pool.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: agent-lab
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sandbox-workload
  namespace: agent-lab
automountServiceAccountToken: false
---
apiVersion: agents.kruise.io/v1alpha1
kind: SandboxSet
metadata:
  name: python-tools
  namespace: agent-lab
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: python-tools
    spec:
      serviceAccountName: sandbox-workload
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: sandbox
          image: python:3.12-slim-bookworm
          command: [python, -m, http.server, '8080', --bind, 0.0.0.0]
          workingDir: /workspace
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
              ephemeral-storage: 128Mi
            limits:
              cpu: '1'
              memory: 512Mi
              ephemeral-storage: 1Gi
          startupProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
            failureThreshold: 3
          volumeMounts:
            - name: workspace
              mountPath: /workspace
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: workspace
          emptyDir:
            sizeLimit: 512Mi
        - name: tmp
          emptyDir:
            sizeLimit: 128Mi

The template describes each sandbox Pod. replicas: 2 sets the initial target for unused inventory, while the label identifies the example Pods. This follows the documented SandboxSet structure. (openkruise.io)

The resource values are illustrative. Requests inform scheduling; limits constrain consumption. Tune them using actual startup and execution measurements, including temporary storage usage. (kubernetes.io)

The startup probe allows initialization time, readiness checks availability, and liveness can trigger a container restart. A restart can interrupt an agent session, so unstable external dependencies should not determine liveness. (kubernetes.io)

The root filesystem is read-only, with writable /workspace and /tmp mounts. An emptyDir is not durable storage: deleting the Pod removes its contents. For production, replace the demonstration image with an approved digest-pinned image and define how results are persisted. (kubernetes.io)

Apply the configuration and inspect the resources:

kubectl apply -f sandbox-pool.yaml
kubectl get sandboxsets.agents.kruise.io -n agent-lab
kubectl get sandboxes.agents.kruise.io -n agent-lab
kubectl get pods -n agent-lab

3. Allocate a sandbox

Save this manifest as sandbox-claim.yaml:

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxClaim
metadata:
  name: analysis-session-001
  namespace: agent-lab
spec:
  templateName: python-tools
  replicas: 1
  claimTimeout: 2m
  skipInitRuntime: true

templateName identifies the pool in the same namespace. This exercise needs skipInitRuntime because it does not inject agent-runtime. The timeout governs allocation attempts, not session lifetime. (openkruise.io)

kubectl apply -f sandbox-claim.yaml
kubectl get sandboxclaims.agents.kruise.io -n agent-lab
kubectl get sandboxes.agents.kruise.io -n agent-lab -l agents.kruise.io/claim-name=analysis-session-001

The Completed phase indicates claim completion. Inspect the allocated sandbox too. Claim cleanup and session termination are separate: ttlAfterCompleted, when configured, deletes the claim object rather than the delivered sandboxes. (openkruise.io)

4. Moving to an E2B integration

A working integration needs agent-runtime configured through spec.runtimes, a compatible image and internal connectivity. Injection adds helper containers, mounts and hooks. Validate the resulting Pod, not just the original template. Removing skipInitRuntime alone does not turn this laboratory into a production integration. (openkruise.io)

The run_code operation also requires an appropriate code-interpreter environment. Test domains, certificates, authentication and SDK compatibility explicitly. A healthy Pod does not prove that the application-level integration works. (openkruise.io)

Security: a sandbox resource is not a strong isolation boundary

Namespaces and ordinary containers do not automatically provide sufficient isolation for hostile code. Evaluate sandboxed runtimes, dedicated nodes or separate clusters against your threat model, including container escape, lateral access and host exposure. (kubernetes.io)

For a production baseline, I recommend:

• Restricted Role-Based Access Control (RBAC) for backends and operators.
• Unprivileged workload ServiceAccounts.
• Policies rejecting privileged containers, hostPath and host namespaces.
• Secrets kept out of images and Git, with rotation and encryption.
• Transport Layer Security (TLS) on access paths.
• Applying the same security review to injected containers as to the primary workload. (kubernetes.io)

On Amazon EKS, EKS Pod Identity associates AWS Identity and Access Management (IAM) permissions with ServiceAccounts. Credentials available to a process are not protected from code executing within that process's environment. Restrict node metadata access as well. For sensitive operations, prefer an authorized tool broker over broad credentials inside the sandbox. (docs.aws.amazon.com)

An initial network policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sandbox-default-deny
  namespace: agent-lab
spec:
  podSelector:
    matchLabels:
      app: python-tools
  policyTypes:
    - Ingress
    - Egress

This establishes default-deny behavior for selected Pods when the network plugin enforces NetworkPolicy. Other allow policies remain additive. It also blocks DNS and normal remote access, so add narrowly scoped rules for required paths. This is not a complete E2B integration policy. (kubernetes.io)

Enterprise example: per-session file analysis

Problem: a company wants users to analyze uploaded files without granting direct access to production PostgreSQL.

Proposed solution: an API authenticates the user, enforces a quota and requests a sandbox from an approved template. A queue bounds concurrency, while an authorized intermediary controls file access.

Proposed implementation: store session metadata in PostgreSQL, results in object storage, and keep execution environments separate from critical services. The backend records the user-to-claim-to-sandbox mapping, enforces a business timeout and confirms persistence before terminating the session.

Expected outcome: more predictable provisioning and clearer ownership. This is an architectural hypothesis to validate, not a benchmark. Compare time to the first useful operation with and without a pool, keeping the image, workload and infrastructure consistent.

Capacity planning, autoscaling and FinOps

PoolAutoscaler adjusts unused inventory through available-capacity policies or schedules. It does not replace node autoscaling or independently manage the lifecycle of allocated sessions. Avoid having multiple controllers write spec.replicas concurrently. (openkruise.io)

For an initial planning estimate, use:

free inventory ≈ peak arrival rate × replenishment time + burst allowance

As a hypothetical example, two new sessions per second and a 15-second replenishment time suggest starting tests with 30 free environments before adding a burst allowance. This calculation does not replace latency distributions, saturation testing or active-concurrency measurements.

Budget for active sessions, unused inventory, control components and failure headroom. Oversized requests reduce capacity efficiency; undersized requests make behavior under contention less predictable. (kubernetes.io)

For financial planning, include nodes, storage, snapshots, network traffic, load balancers and telemetry retention. Evaluate Spot capacity only where interruption and replay are acceptable, and preserve baseline capacity for critical sessions. Validate savings through cost per completed session, including failures and idle time, rather than assuming that pooling is always cheaper.

Availability and recovery

Service availability does not automatically replicate individual session state. A PodDisruptionBudget (PDB) constrains certain voluntary disruptions; it does not prevent node failures or replace data recovery. (kubernetes.io)

My production recommendation is to spread manager and gateway replicas across zones, validate controller leader election before increasing its replicas, and test node loss. Add overload admission controls: bounded queues, explicit rejection and retries with backoff rather than unrestricted session creation.

Pause, resume and checkpoints

Pause and resume are exposed through the API, but memory and filesystem preservation depend on the infrastructure. The reviewed documentation limits its stated memory-preservation support to Alibaba Cloud ACS; do not extrapolate that guarantee to EKS. Open connections also need to be re-established after resuming. (openkruise.io)

Define a Recovery Point Objective (RPO, acceptable data loss) and Recovery Time Objective (RTO, acceptable restoration time). Back up durable data, metadata and configuration, then test restoration. Evaluate execution checkpoints separately from business-data backups.

Observability from the user's perspective

The controller and manager expose Prometheus metrics. The controller endpoint requires authentication and authorization, so configure the scraper accordingly. The metrics reference follows development code; verify metric names and labels against the deployed version. (openkruise.io)

I recommend a dashboard covering:

• p50, p95 and p99 latency from request to first useful operation.
• Failed or timed-out claims and available inventory.
• Replenishment time, queue depth and active sessions.
• CPU, memory, restarts, temporary storage and node saturation.
• Authentication, gateway and resume failures.

Correlate logs and traces by session identifier without recording sensitive payloads. One possible architecture combines Prometheus and Grafana for metrics, Loki for logs, and OpenTelemetry instrumentation. These signals complement one another; none alone establishes user-perceived availability. (kubernetes.io)

Upgrades, GitOps and rollback

Separate operator upgrades, template changes and active-session migration. The project distinguishes rolling updates of unused pool instances from operations on allocated sandboxes, including those coordinated through SandboxUpdateOps. (openkruise.io)

Use the scale subresource for manual capacity changes and patches rather than full-object replacement for template modifications. (openkruise.io)

kubectl scale sandboxsets.agents.kruise.io python-tools -n agent-lab --replicas=4

Operationally, I recommend creating a separately named canary pool, gradually routing new sessions to it and leaving existing sessions on their original revision. Rollback redirects new requests to the previous pool; it does not reverse modified data or restore lost memory.

In GitOps, assign one owner to each dynamic field. Before CRD upgrades, record versions, export resources and validate backward compatibility. Reverting a chart is not a complete recovery strategy.

Troubleshooting

Investigate in this order: resource → Pod → node → network. The following commands collect evidence without assuming chart-specific Deployment names. (kubernetes.io)

kubectl get sandboxsets.agents.kruise.io -n agent-lab -o yaml
kubectl get sandboxclaims.agents.kruise.io -n agent-lab -o yaml
kubectl get pods -A
kubectl get deployments -n sandbox-system
kubectl get nodes
kubectl get events -n agent-lab --sort-by=.metadata.creationTimestamp
kubectl describe pod <pod> -n agent-lab
kubectl logs <pod> -n agent-lab -c sandbox
kubectl logs <pod> -n agent-lab -c sandbox --previous

Slow or failed claims

Symptom → allocation does not complete.

Possible cause → unavailable inventory, failed readiness or incompatible runtime initialization.

Investigation → inspect claim status, inventory and events.

Correction → fix the provisioning blocker. For this laboratory, check skipInitRuntime; do not use it to hide failures in integrations that require the runtime. (openkruise.io)

Pending or restarting Pods

Symptom → environments never become usable.

Possible cause → insufficient capacity, unavailable images or process failures.

Investigation → inspect Pod descriptions, events and previous logs.

Correction → address the evidenced cause before increasing timeouts or replica counts. (kubernetes.io)

Healthy Pod, inaccessible session

Symptom → session access times out.

Possible cause → a network policy blocks a required path.

Investigation → review selectors and ingress/egress permissions.

Correction → add the narrowest necessary allowance rather than removing isolation altogether. (kubernetes.io)

Development versus production

Do not promote the laboratory unchanged. Missing quotas, hardcoded credentials, unnecessary public exposure, unvalidated probes, absent observability and unbounded session lifetime should block production admission.

When to use it—and what to compare

Consider OpenKruise Agents when isolated sessions, pre-warmed inventory and declarative management justify operating a specialized platform. My recommendation is to require support capacity, isolation testing and clear lifecycle ownership before adoption.

For finite, mostly non-interactive processing, Kubernetes Jobs may be simpler. They model execution to completion rather than interactive session management. (kubernetes.io)

Kubernetes SIG Agent Sandbox also provides abstractions for isolated, stateful workloads. Compare its APIs, extensions, runtimes and operating model; do not treat it as a drop-in manifest replacement. (github.com)

Include managed services in the evaluation when reducing operational responsibility matters more than controlling the full infrastructure stack. Compare per-session cost, data residency, portability and contractual guarantees. For occasional tasks, maintaining an operator and idle capacity may not be justified.

Practical summary

• Distinguish sandbox instances, free inventory and allocated sessions.
• Validate capabilities against the installed release.
• Treat the walkthrough as a laboratory, not a hostile-code boundary.
• Size total infrastructure demand, not just pool replicas.
• Test persistence, resumption, termination and rollback before onboarding users.

Conclusion

I recommend introducing OpenKruise Agents as an internal Platform Engineering product: approved templates, explicit access contracts, capacity budgets and reliability criteria.

For DevOps, that means reproducible changes. For SRE, it means measurable objectives and tested recovery. For cloud architecture, it means deliberate decisions about where to share infrastructure and where to separate risk. The goal is not merely fast environment creation, but sessions whose behavior, cost and failure modes can be understood and controlled.