How to Evaluate AI Agent Isolation on Kubernetes

An agent that inspects a repository, runs tests and publishes a diagnosis needs tools. The important architectural question is not simply where those tools run. It is what damage remains possible if the executing process can no longer be trusted.

For this assessment, use a conservative assumption: the agent process may be compromised and may attempt to exercise every available permission. The goal is not to determine whether the model follows instructions. It is to establish whether infrastructure prevents actions outside its authorized contract.

This article develops an evaluation protocol: a threat model, positive and negative tests, independent evidence and release criteria. Connectivity implementation belongs in networking for AI agent sandboxes; creation and teardown are covered in ephemeral Kubernetes sandboxes.

1. Define isolation as a set of verifiable properties

Isolation separates trust domains. In Kubernetes, a namespace provides administrative scope, but it is not a complete boundary for hostile code. Node resources and cluster components remain part of the sharing model. (kubernetes.io)

Make the assessment concrete by asking six questions:

1. Execution: what boundary separates the agent process from the host?
2. Identity: which APIs and resources can its credentials authorize?
3. Network: which destinations and operations can it reach?
4. Data: what can it read, modify or leave behind for another execution?
5. Resources: how much can it consume before containment takes effect?
6. Lifecycle: what remains accessible after the execution ends?

“Runs in a sandbox” is not an adequate answer. Each dimension needs a testable hypothesis, a test and supporting evidence.

Scope and limitations

State whether the assessment covers process compromise alone or also malicious administrators, node compromise and mutually distrustful customers. A permissions test is not proof against unknown kernel vulnerabilities.

This protocol treats platform administrators and the external evidence collector as trusted. The agent, its subprocesses, dependencies and incoming files are untrusted.

2. A reference architecture for the assessment

Use the following as a proposed architecture rather than a product-specific requirement:

Request → orchestration API → authorization → queue → isolated executor → tool broker → authorized service

Alongside that path:

Kubernetes API + runtime + network + broker → external collector → evidence report

Separate three responsibilities:

• Orchestrator: accepts requests and creates executions from an approved template.
• Executor: processes untrusted content without administering the cluster.
• Tool broker: checks identity, tenant, action and resource before accessing corporate systems.

Do not give the executor administrative credentials so that it can conveniently provision its own environment. Kubernetes documents that workload-creation privileges can provide indirect access to Secrets and to the permissions of other ServiceAccounts in the namespace. (kubernetes.io)

Produce the primary report outside the assessed trust domain. Agent output can contribute to an investigation, but it should not be the only evidence that the agent remained isolated.

3. Compare boundaries rather than product labels

These approaches establish different boundaries; they are not universally interchangeable security tiers. gVisor and Kata describe distinct architectures, while Kubernetes multitenancy guidance explains the trade-offs of sharing cluster infrastructure. (gvisor.dev)

A RuntimeClass selects a runtime configuration that must already exist. Creating the object or setting runtimeClassName does not install that runtime on nodes. Verify both the requested class and actual execution using runtime or managed-service evidence. (kubernetes.io)

If the required class is unavailable, the proposed acceptance rule is to fail closed: do not run the task. Reject orchestrator behavior that silently falls back to a less isolated class.

4. Build a reproducible test campaign

Run assessments in an authorized environment with synthetic data and explicit limits. Saturation tests belong in a segregated environment. Do not run kernel exploits or destructive workloads on shared production nodes.

Record:

• Kubernetes, operating system, kernel, runtime and network-plugin versions;
• image digest, CPU architecture and executor template;
• RuntimeClass, admission policies and identity configuration;
• tenant, execution identifier and collection window;
• permitted destinations, forbidden destinations and expected outcomes;
• owner, expiration date and rationale for each exception.

Use four outcomes: pass, fail, inconclusive and not applicable with justification. A broken test tool produces an inconclusive result, not a pass.

Pair every negative test with a positive control. If a forbidden request fails but no permitted request succeeds, you have not yet shown that the intended policy caused the failure.

5. The minimum test matrix

The following matrix is a proposed assessment contract. Adapt expected outcomes to the agent's authorized execution profile.

Do not average these outcomes into a reassuring score. Cross-tenant data access cannot be offset by good latency or low cost.

6. Admission and identity: establish who can do what

Pod Security Admission can enforce the Restricted profile at namespace level. Its warn and audit modes are not equivalent to enforce. The profile also does not replace network controls, business authorization or a policy requiring a specific RuntimeClass. (kubernetes.io)

For admission testing, keep a valid baseline Pod in the test repository. Produce one copy per mutation: privileged: true, hostPID: true, a hostPath volume or removal of the required runtime class. Change one property at a time. Verify that the rejection message identifies the intended control rather than a syntax error or unrelated restriction.

For role-based access control, or RBAC, start with the executor's identity. This example assumes that namespace agent-eval and ServiceAccount agent-worker already exist:

NS=agent-eval
SA=agent-worker

kubectl auth can-i get secrets -n "$NS" \
  --as="system:serviceaccount:${NS}:${SA}" \
  --as-group=system:serviceaccounts \
  --as-group="system:serviceaccounts:${NS}" \
  --as-group=system:authenticated

kubectl auth can-i create pods -n "$NS" \
  --as="system:serviceaccount:${NS}:${SA}" \
  --as-group=system:serviceaccounts \
  --as-group="system:serviceaccounts:${NS}" \
  --as-group=system:authenticated

The operator needs permission to impersonate the user and groups. For the proposed profile, both queries should return no. These commands query Kubernetes authorization; they do not establish network blocking, credential unavailability or denial by cloud APIs. Complete the assessment with synthetic-resource tests using the effective identity. (kubernetes.io)

When the executor does not require the Kubernetes API, disable automatic token mounting. Still inspect projected volumes, environment variables and external identity mechanisms: automountServiceAccountToken: false is not a general credential-removal policy. (kubernetes.io)

On Amazon EKS, assess AWS Identity and Access Management, or IAM, separately. EKS Pod Identity associates a role with a ServiceAccount, but it does not make containers security boundaries. Also check whether access to the instance metadata service exposes node credentials. (docs.aws.amazon.com)

7. A practical check inside the executor

For an evaluation executor containing a POSIX shell, id, test and mktemp, run the following check. Replace the Pod and container names. This example's contract requires a non-root user, no default Kubernetes token and a writable /tmp directory:

NS=agent-eval
POD=replace-with-evaluation-pod
CONTAINER=worker

kubectl exec -i -n "$NS" "$POD" -c "$CONTAINER" -- sh -s <<'SH'
set -eu

if [ "$(id -u)" -eq 0 ]; then
  echo 'FAIL: root user'
  exit 1
fi

if [ -e /var/run/secrets/kubernetes.io/serviceaccount/token ]; then
  echo 'FAIL: default Kubernetes token is mounted'
  exit 1
fi

probe=$(mktemp /tmp/isolation-probe.XXXXXX)
rm -f "$probe"

echo 'PASS: non-root, default token absent, temporary directory writable'
SH

This script checks only three local conditions. It does not establish the absence of every token, a read-only root filesystem, kernel isolation or resistance to container escape. Compare runAsNonRoot and the Restricted controls with the admitted Pod, not just the original template. (kubernetes.io)

If the image has no shell, use a previously reviewed test binary. Do not modify the production image or introduce a privileged container simply to make the assessment pass.

8. Complete YAML: an evaluation namespace budget

The following manifest adds an admission budget to the campaign. It does not create a sandbox or enforce network isolation. Apply it only to an existing, dedicated evaluation namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: isolation-evaluation-budget
  namespace: agent-eval
spec:
  hard:
    pods: "8"
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    requests.ephemeral-storage: 8Gi
    limits.ephemeral-storage: 16Gi
    count/jobs.batch: "16"
    persistentvolumeclaims: "0"
    count/services: "0"

requests.* and limits.* constrain the declared totals subject to the quota. pods limits non-terminal Pods; count/jobs.batch limits Job objects, including completed Jobs that have not been removed. The zero values prevent new PersistentVolumeClaims, or PVCs, and Services in this namespace. They do not prevent network access to external storage. (kubernetes.io)

These values are illustrative, not production sizing recommendations. Adjust them for concurrency, sidecars and runtime overhead. Also validate cleanup of old Jobs so that the object-count limit does not block later assessments.

kubectl apply -f evaluation-quota.yaml
kubectl describe resourcequota isolation-evaluation-budget -n agent-eval
kubectl get events -n agent-eval --sort-by=.metadata.creationTimestamp

A quota does not directly bound every form of process consumption. Requests affect scheduling, while CPU and memory limits have different enforcement mechanisms. Evaluate actual consumption, throttling and out-of-memory termination separately. (kubernetes.io)

Test the quota by submitting harmless workloads with declared resources until the admission budget is reached. A destructive load is unnecessary to demonstrate an admission rule.

9. Network and data: distinguish blocking from unavailability

NetworkPolicy requires a compatible network-plugin implementation. Permissions from applicable policies combine, so an additional policy can broaden connectivity. The standard API does not provide HTTP authorization or business-identity validation. (kubernetes.io)

Without duplicating the networking article's manifests, include these assessment cases:

• an authorized broker request with valid TLS certificate verification;
• access to a forbidden destination by both hostname and IP address;
• direct destination access that bypasses the proxy;
• IPv4 and IPv6 coverage when both are enabled;
• a request for another tenant's resource through the permitted broker;
• behavior while the authorization component is unavailable.

Do not treat a single timeout as proof of isolation. Correlate source, destination, timestamp and the expected enforcement mechanism. Distinguish network blocking, resolution failure, connection refusal, TLS errors and authorization denial.

For persistence, write a unique synthetic marker during execution A and search for it during execution B. Include workspaces, caches, artifacts, checkpoints and external storage. Deleting a Pod does not prove deletion of a persistent volume: reclamation depends on the volume policy and provisioning arrangement. (kubernetes.io)

Also define tests for credential rotation, key scope and backup authorization. Storage encryption does not replace separation of read permissions between tenants.

10. A corporate example: pull request analysis

Problem: a platform receives external pull requests and runs tests to generate a diagnosis. Submitted code must not inherit corporate automation credentials.

Proposed solution: separate the executor from the broker that publishes comments, giving the broker access only to the authorized repository and operation.

Assessment implementation: create two synthetic tenants, one permitted repository and one forbidden repository. Run the matrix with the image and runtime configuration intended for production. Verify permitted publication, denial in the second repository, absence of administrative credentials and cleanup of temporary artifacts.

Expected outcome: release the execution profile only when evidence demonstrates its boundaries. This is a hypothetical scenario, not a benchmark or a customer case with measured improvements.

A cross-tenant test failure should block the profile's release even if all functional code-analysis tests succeed.

11. Production availability without weakening isolation

As an operating design, spread the API, broker and queue consumers across availability zones, with replicas, health checks and baseline capacity. An isolated execution does not necessarily need simultaneous replicas: platform availability and repeating a task are separate decisions.

Use readiness to control traffic reception, startup checks for slow initialization and liveness only when restarting is an appropriate recovery action. For finite tasks, consider Job deadlines and retry policy. Do not automatically repeat a non-idempotent external operation. (kubernetes.io)

My recovery recommendation is to persist only authorized checkpoints, use idempotency keys at the broker and test node interruption, duplicate delivery and resumption. Include restoration of queue state and platform metadata in the backup plan.

For upgrades, run the same suite before and after changing the kernel, runtime, image or network plugin. Start with a synthetic-traffic canary. Roll back only to a profile that remains approved; never remove sandboxing or open unrestricted egress to restore availability.

12. Observability and evidence

Metrics, logs and traces answer different questions. Kubernetes auditing records API operations according to its configured policy; it does not replace observation of activity inside processes. (kubernetes.io)

For this protocol, instrument:

• provisioning latency and queue age;
• tool-broker throughput, errors and latency;
• CPU, memory, ephemeral storage and restarts;
• admission rejections and denied authorization attempts;
• divergence between approved and effective runtimes;
• time from task termination to access invalidation;
• cost per completed task, including retries.

Prometheus and Grafana are possible choices for the metrics layer; select logging and tracing tools according to organizational standards. Keep detailed execution identifiers in logs and traces rather than creating a separate metric series for every execution.

Do not collect tokens, Secret contents or sensitive documents merely to produce evidence. Store reports outside the executor namespace, with access controls, retention rules and verifiable integrity.

13. Hypothesis-driven troubleshooting

Use the following commands for initial evidence collection, substituting actual environment names:

kubectl get pods -n agent-eval -o wide
kubectl describe pod "$POD" -n agent-eval
kubectl logs "$POD" -n agent-eval -c worker
kubectl logs "$POD" -n agent-eval -c worker --previous
kubectl get runtimeclass
kubectl get resourcequota -n agent-eval
kubectl get events -n agent-eval --sort-by=.metadata.creationTimestamp

--previous is useful only when an earlier container instance exists and its logs remain available.

Validate runtime, quota, network and identity hypotheses against their documented mechanisms rather than resolving failures through broad permission increases. (kubernetes.io)

14. Development, costs and adoption criteria

In development, lower redundancy and a reduced synthetic-data suite can be reasonable. Preserve the identity and admission contract, however: an environment using cluster-admin does not validate the restricted production profile.

For production, require the complete report, alerting, recovery capability, a rehearsed rollback and reassessment after relevant changes.

For FinOps, compare total cost per completed task: compute, initialization, idle capacity, networking, storage and observability. Measure each runtime's overhead using representative workloads rather than assuming equivalent performance. Include declared RuntimeClass overhead in capacity planning. (kubernetes.io)

Consider Spot capacity only for interruptible, recoverable tasks. Maintain stable capacity for critical components when availability objectives require it. Set concurrency using queue age and per-tenant budgets, not average CPU utilization alone.

Use the full assessment when agents execute external code, access sensitive information, serve multiple tenants or require compliance evidence. An agent with no code execution and a single tightly authorized tool may justify a smaller assessment; identity and data boundaries still need review.

If the team cannot safely operate additional runtimes, consider a managed isolated-execution service or separate environments. Assess cost, observability, portability and provider dependency before choosing.

15. Practical summary and conclusion

Before approving an execution profile, confirm:

• a documented threat model and explicit exclusions;
• agreement between requested and effective runtimes;
• positive and negative tests for identity, networking and data;
• explicit resource and concurrency budgets;
• post-termination credential and persistence checks;
• evidence collected outside the executor;
• release blocking for critical failures;
• rehearsed upgrades, rollback and recovery.

Avoid equating a namespace with a sandbox, warning mode with enforcement, a timeout with proof, or valid YAML with verified behavior.

For DevOps and Platform Engineering, the deliverable should be a versioned, reproducible execution profile. For SRE, its guarantees must survive failure, saturation and maintenance. Evaluating isolation turns an architectural promise into explicit properties that the team can test, observe and sustain.