Designing ephemeral sandboxes for AI agents on Kubernetes

An agent that reviews repositories, runs tests, or transforms files needs tools with real effects: shells, interpreters, data access, and network calls. For this design, assume the code being executed may be hostile even when the original request is legitimate.

The proposed approach separates planning from execution. An agent requests a task; a trusted platform determines where it runs, which capabilities it receives, and when its environment must be destroyed. The agent does not receive permission to provision arbitrary infrastructure.

The goal is not simply fast Pod creation. It is an execution service with enforceable boundaries around identity, networking, resources, lifetime, and persistence.

1. What makes a sandbox ephemeral?

Here, a sandbox is a disposable environment created for one execution or session, with an explicit policy and temporary working storage. The language model can remain outside it: tool execution is the primary workload that needs containment.

A namespace groups resources and policies, but it is not, by itself, a strong security boundary against hostile code. Conventional containers share the node kernel, so runtime selection must follow the threat model. (kubernetes.io)

Use three properties as design requirements:

• Bounded lifetime: every execution has a deadline, an owner, and a destruction procedure.
• Limited authority: the executor receives only the data and operations needed for its task.
• Verifiable external state: accepted results, audit records, and operational state survive environment deletion.

2. Start with a threat model

Before writing YAML, identify assets, adversaries, and consequences. For a platform executing code from external repositories, I recommend the following contract:

Treat files, dependencies, tool responses, and agent output as untrusted content. Infrastructure policy must remain outside their control.

For mutually untrusted customers or stringent regulatory requirements, consider separate clusters and cloud accounts in addition to runtime isolation. Kubernetes multitenancy guidance distinguishes namespace-based separation from stronger isolation models. (kubernetes.io)

3. Reference architecture

Proposed execution flow:

Client → execution API → authorization and budget checks → queue → controller → namespace and policies → Job → isolated runtime → validated result → cleanup

Separate two planes:

• Trusted control plane: authentication, authorization, queuing, reconciliation, auditing, and issuance of short-lived capabilities.
• Untrusted execution plane: the agent or tool process, input files, and working directory.

The controller is a platform component, not an invented Kubernetes API. It can operate entirely on native resources. A CustomResourceDefinition, or CRD, is optional if a dedicated session API becomes useful.

Resources and responsibilities

• Job: tracks a finite task. It does not guarantee exactly-once execution; task code must tolerate duplicate execution. (kubernetes.io)
• RuntimeClass: selects a runtime configuration already installed on eligible nodes. (kubernetes.io)
• ServiceAccount: identifies a Kubernetes workload. Disabling automatic token mounting avoids exposing a token when one is unnecessary. (kubernetes.io)
• ResourceQuota: constrains aggregate resource usage and object counts within a namespace; it does not reserve physical capacity. (kubernetes.io)
• NetworkPolicy: controls selected traffic when the networking implementation supports enforcement. (kubernetes.io)

For interactive sessions, I recommend a Pod with an explicit session protocol and an externally enforced deadline. A Deployment that continually recreates processes is not automatically the right abstraction for a disposable session.

4. Choose isolation before optimizing startup time

gVisor implements a Linux-like interface through an application kernel running in userspace. Kata Containers uses lightweight virtual machines and hardware virtualization. These are different approaches whose compatibility requirements must be evaluated against the actual workload. (gvisor.dev)

Do not select a runtime using a generic benchmark alone. Test system calls, native libraries, filesystem behavior, networking, initialization, and any GPU requirements with representative tasks.

Example RuntimeClass

This manifest assumes the runsc handler is already installed and configured in the node runtime. The sandbox-runtime: gvisor label is a local convention managed by the platform team.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: agent-gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox-runtime: gvisor

This object selects a handler; it does not install gVisor. Adapt the name, handler, and node selection. In production, configure measured overhead values where applicable so runtime resource costs participate in capacity calculations. (kubernetes.io)

I recommend a dedicated node pool, platform-controlled scheduling policies, and restrictions preventing executors from changing these controls. Scheduling separation complements security isolation; it does not replace it.

5. Complete example: an offline, disposable execution

The following example is educational. It runs a harmless command without network access to validate the execution envelope. It does not implement input downloads, inference, or artifact uploads.

Prerequisites include a functioning runtime configuration, eligible nodes, and a network plugin that enforces NetworkPolicy. The platform must establish and verify the baseline before admitting untrusted code. API acceptance of a policy object does not, by itself, prove that the dataplane is already enforcing it.

apiVersion: v1
kind: Namespace
metadata:
  name: agent-run-7f3a
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sandbox
  namespace: agent-run-7f3a
automountServiceAccountToken: false
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: sandbox-budget
  namespace: agent-run-7f3a
spec:
  hard:
    pods: '2'
    count/jobs.batch: '2'
    requests.cpu: '1'
    limits.cpu: '2'
    requests.memory: 512Mi
    limits.memory: 1Gi
    requests.ephemeral-storage: 512Mi
    limits.ephemeral-storage: 1Gi
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: agent-run-7f3a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: batch/v1
kind: Job
metadata:
  name: agent-task
  namespace: agent-run-7f3a
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 600
  template:
    metadata:
      labels:
        app: agent-sandbox
    spec:
      runtimeClassName: agent-gvisor
      serviceAccountName: sandbox
      automountServiceAccountToken: false
      restartPolicy: Never
      terminationGracePeriodSeconds: 10
      securityContext:
        runAsNonRoot: true
        runAsUser: 10000
        runAsGroup: 10000
        fsGroup: 10000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: executor
          image: busybox:1.37.0
          command:
            - sh
            - -c
          args:
            - |
              set -eu
              echo sandbox-ready > /workspace/result.txt
              cat /workspace/result.txt
          workingDir: /workspace
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              cpu: 250m
              memory: 128Mi
              ephemeral-storage: 128Mi
            limits:
              cpu: '1'
              memory: 512Mi
              ephemeral-storage: 512Mi
          volumeMounts:
            - name: workspace
              mountPath: /workspace
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: workspace
          emptyDir:
            sizeLimit: 256Mi
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi

What to change, and why

The restricted profile enforces Pod security controls, not the entire proposed architecture. readOnlyRootFilesystem is an additional design choice. For production, pin the profile to the Kubernetes version you have validated rather than allowing latest to follow changes automatically. (kubernetes.io)

backoffLimit: 0 disables Job retries for failures. activeDeadlineSeconds bounds the Job's active duration, but it is neither the complete deadline from queue admission nor an instantaneous physical shutdown guarantee. (kubernetes.io)

The 600-second Time to Live, or TTL, begins after completion or failure. It enables cleanup of the Job and its dependents, but does not automatically remove the namespace and every independently created resource. (kubernetes.io)

An emptyDir follows the Pod lifecycle and is not durable storage. Its sizeLimit is not a disk reservation; monitor node storage pressure as well. (kubernetes.io)

Requests inform scheduling. CPU limits can cause throttling, while memory limits can result in out-of-memory termination. Size ephemeral storage for both working files and logs. (kubernetes.io)

The quota applies to one namespace. Platform-wide and per-tenant budgets remain platform responsibilities. Replace the fixed namespace with unique execution identifiers. In production, use an approved image pinned by digest, preinstalled dependencies, and image signature validation according to your policy.

6. Networking, identity, and mediated tools

The manifest allows neither DNS nor external access. NetworkPolicies are additive: another permissive policy can broaden access. The standard API does not provide HTTP-domain authorization, and node traffic and networking implementations have important exceptions. Explicitly test cloud metadata and internal destinations rather than treating the YAML as proof of isolation. (kubernetes.io) To size IPs, DNS, connections, and egress for that mediated traffic, see How to size Kubernetes networking for ephemeral AI agent sandboxes.

For connected tasks, I recommend allowing only an authenticated private gateway. It should authorize destinations and methods, cap bytes and duration, validate redirects, and reject inappropriate internal destinations. If DNS is required, use controlled resolution and consider DNS itself as an exfiltration path.

Do not place a broad model-provider, GitHub, or cloud credential in an untrusted process. Prefer short-lived, execution-scoped capabilities or calls mediated by trusted services.

Role-Based Access Control, or RBAC, must separate controller and executor permissions. Permission to create workloads can indirectly expose Secrets and identities; never accept arbitrary manifests from the agent. (kubernetes.io)

On Amazon Elastic Kubernetes Service, or Amazon EKS, EKS Pod Identity associates an Identity and Access Management, or IAM, role with a ServiceAccount. For this architecture, prefer assigning that identity to the trusted artifact service rather than the sandbox. (docs.aws.amazon.com)

Use Transport Layer Security, or TLS, between services and implement credential rotation and expiration. Kubernetes Secrets require access protection and encryption at rest: base64 encoding is not encryption. (kubernetes.io)

7. Lifecycle management and recovery

I recommend recording the states queued → provisioning → running → collecting → terminating → cleaned, with explicit failure and cancellation paths. These are application states, not native Pod phases.

The controller should reconcile desired and observed state using an execution-specific idempotency key. An independent reconciler should detect expired environments and orphaned resources.

Require durable result acknowledgement before reporting success. The artifact service should enforce size limits, validate paths, reject unsafe archive extraction, and treat generated files as untrusted. Do not depend on copying files after their Pod has already disappeared.

Kubernetes garbage collection uses ownership relationships with scope restrictions. A namespaced Job must not be used as the owner of a cluster-scoped namespace. Plan explicit deletion of namespaces and external resources. (kubernetes.io)

During a network partition, deleting an API object does not prove the process has stopped on its node. For sensitive operations, combine deadlines with expiring credentials, gateway revocation, and node fencing before declaring physical execution terminated. (kubernetes.io)

8. Enterprise example: validating a code change

Problem: a team wants agents to review pull requests without running external scripts inside privileged corporate runners.

Proposed solution: keep planning outside the sandbox and create an isolated execution for each validation attempt.

Implementation: a trusted service prepares a repository snapshot without credentials. The executor receives only required files and approved dependencies. Tests produce a report. A separate service validates the artifact and publishes the comment to the code-hosting platform.

Expected outcome: reduce the reach of a compromised execution while improving auditability and reproducibility. This is an architectural expectation, not a benchmark or a measured customer result.

Use synthetic fixtures for PostgreSQL or Redis when tests require those services. Avoid production databases and writable caches shared across customers. Keep human approval separate from execution when an action involves deployment, deletion, or another sensitive change.

9. Production availability, capacity, and cost

I recommend multiple API and gateway replicas, distribution across availability zones, and idempotent reconciliation. Controllers with concurrent writers need leader election or equivalent coordination. Protect the state database with backups and tested restoration procedures.

Apply readiness, startup, and liveness probes to persistent services according to their purpose. The short-lived Job in this example does not need an artificial HTTP endpoint: completion, deadlines, and externally observed progress are the meaningful signals. Probes have distinct semantics and should not be added mechanically. (kubernetes.io)

Use PodDisruptionBudget where appropriate, remembering that it limits certain voluntary disruptions rather than preventing involuntary failures. It is not a Job-completion guarantee. (kubernetes.io)

Capacity planning

As a hypothetical example, 60 tasks per minute with a two-minute average duration implies approximately 120 concurrent executions in steady state. At requests of 250m CPU and 128Mi memory, that represents 30 CPUs and 15Gi of requested memory before runtime overhead and platform services.

Also account for bursts, duration percentiles, images, disk, IP addresses, and API object counts. Enforce concurrency limits before creating thousands of pending Pods.

Node autoscaling considers scheduling needs and Pod constraints. It complements admission control and baseline capacity for startup delays; it does not replace them. (kubernetes.io)

For FinOps, track cost per accepted task and per completed task. Include idle capacity, egress, storage, logs, and retries. My suggested policy is to reserve stable capacity for critical services and evaluate Spot only for recoverable tasks whose deadlines accommodate interruptions.

Prefer shared gateways to a load balancer per sandbox. Use immutable, content-addressed caches; do not reuse previously occupied environments across tenants merely to reduce latency.

10. Observability and troubleshooting

OpenTelemetry provides signals including metrics, logs, and traces. Instrument the platform to connect admission, provisioning, execution, and cleanup. (opentelemetry.io)

Recommended signals include:

• queue latency and time to the first command;
• throughput, success rate, and categorized failures;
• CPU, memory, disk, throttling, and restarts;
• artifact collection duration and the age of expired environments;
• policy rejections, denied traffic, and gateway saturation.

Define separate service-level objectives, or SLOs, for API availability, execution startup, and cleanup. Put execution identifiers in logs and traces rather than creating a metric time series for every execution. Do not record full prompts or credentials by default.

The following commands inspect status, events, and logs. Run them using an authorized operational identity, not from inside the sandbox. (kubernetes.io)

kubectl get runtimeclass agent-gvisor
kubectl get nodes -l sandbox-runtime=gvisor
kubectl get jobs,pods -n agent-run-7f3a
kubectl describe job agent-task -n agent-run-7f3a
kubectl describe resourcequota sandbox-budget -n agent-run-7f3a
kubectl get networkpolicy -n agent-run-7f3a
kubectl logs job/agent-task -n agent-run-7f3a
kubectl get events -n agent-run-7f3a --sort-by=.metadata.creationTimestamp

Use this operational checklist, adapting remediation to the evidence:

11. Development, maintenance, and acceptance criteria

In development, one replica and simplified observability may be sufficient for synthetic files. That is not a justification for executing hostile content without isolation.

Production should require a redundant control plane, limits, alerts, audit trails, durable-state backups, and recovery tests. Do not indiscriminately back up disposable workspaces: retain only what the data policy requires.

Manage templates and policies through GitOps and continuous integration and delivery, or CI/CD. Let the execution controller manage ephemeral runs, without a GitOps reconciler recreating Jobs that have already been cleaned up.

Roll out upgrades to a canary node pool and test compatibility, isolation, latency, and recovery. For rollback, route new tasks to the previously validated combination of image, runtime, and policy. Do not try to turn a completed Job back into a previous execution.

Before release, I recommend tests demonstrating:

1. Denied access to the cluster, metadata endpoints, and unauthorized destinations.
2. Rejection of arbitrary ServiceAccounts, volumes, and privileges.
3. Termination and capability revocation after timeout or cancellation.
4. Recovery after controller failure and node loss.
5. No file carryover between executions and removal of orphaned resources.
6. Idempotent publication and other external side effects.

Avoid hardcoded credentials, Docker socket access, hostPath, broad privileges, unrestricted egress, and missing budgets. Do not confuse ttlSecondsAfterFinished with complete cleanup, or retries with exactly-once processing.

12. When to use this architecture—and when to simplify

This architecture is a good candidate when untrusted code execution is recurring, multiple users share the service, auditability matters, and a team can operate the platform.

For small, trusted automations, a restricted worker may be sufficient. For strict regulatory boundaries, a dedicated VM or separate environment may be more appropriate. A managed execution service also deserves evaluation, considering documented isolation, data location, limits, cost, and provider dependence.

The decision should balance risk against operational capability, rather than merely making use of an existing cluster.

Practical summary

• Separate trusted decisions from untrusted execution.
• Select isolation according to the threat model.
• Bound identity, network access, resources, lifetime, and concurrency.
• Persist accepted results before cleanup.
• Make failure, duplication, and expiration part of the service contract.
• Test dataplane enforcement, not just YAML validity.

Conclusion

Ephemeral sandboxes should be treated as a Platform Engineering product: a simple consumer interface backed by rigorous policies and operations.

DevOps provides reproducible delivery. Site Reliability Engineering, or SRE, turns availability, recovery, and cleanup into verifiable objectives. Cloud architecture establishes identity and capacity boundaries. Kubernetes coordinates these mechanisms, but security depends on the complete contract between authorization, runtime isolation, networking, and lifecycle management.