Agent Substrate, gVisor, and OpenRL: a cloud-native agent architecture

Consider an enterprise platform where agents receive tasks, invoke tools, and produce results that help evaluate a new model version. Its engineering team must answer three separate questions: where each session runs, how untrusted code is contained, and how valid outcomes reach the training pipeline.

This article argues for three distinct layers: execution lifecycle, isolation, and post-training. Agent Substrate, gVisor, and OpenRL provide capabilities relevant to those layers. They should not, however, be presented as a single product or an automatically integrated stack. The composition below is an architectural proposal, not a claim of certified interoperability. (ate.dev)

This is a companion to the GKE Agent Sandbox reinforcement learning lab. Instead of repeating sandbox installation or network manifests, it focuses on the contracts connecting execution, evaluation, and learning.

First, which OpenRL?

Here, OpenRL refers to `gke-labs/open-rl`, a self-hosted post-training API compatible with the Tinker client. It is neither a managed service nor the owner of your learning algorithm. Its repository also states that it is not an officially supported Google product. (github.com)

There is a separate project called OpenRL-Lab/openrl: a reinforcement learning research framework. The similar names do not imply interchangeable software or functionality. Naming the repository is essential to a meaningful architecture discussion. (github.com)

The capabilities discussed reflect documentation consulted for this publication on November 5, 2026. For deployment, pin versions and verify the requirements of the selected combination. Examples on a development branch are not compatibility guarantees.

Three technologies with different jobs

Agent Substrate: execution lifecycle beyond the Pod

Agent Substrate manages actors: execution units that can be activated, suspended, and resumed. Its architecture separates actor lifetimes from worker Pod lifetimes. A worker fleet hosts multiple actors, while a specialized control plane coordinates assignment and state. This model is particularly relevant to intermittent sessions rather than exclusively long-running processes. (ate.dev)

Its main resource types have different roles:

• Actor: an execution instance.
• ActorTemplate: the definition of an actor version.
• Atespace: a logical scope for resources and authorization.
• WorkerPool: a Kubernetes resource managing worker capacity.
• SandboxConfig: a Kubernetes resource configuring the sandbox runtime.

In the documentation reviewed, Actor and ActorTemplate belong to the Substrate API, not the Kubernetes API. WorkerPool and SandboxConfig use Custom Resource Definitions, or CRDs. An invented Kubernetes ActorTemplate manifest would therefore be misleading. (ate.dev)

The atelet supervisor coordinates node-side execution. With the gVisor backend, ateworker-gvisor translates operations into runsc invocations. A worker Pod can host multiple actors in separate sandboxes. Operationally, this means a healthy Pod does not establish that every actor inside it is healthy. (learn.agentsubstrate.dev)

gVisor: an execution boundary

gVisor implements a userspace application kernel called the Sentry. It handles workload system calls while reducing direct exposure to the host kernel. It is more than a system call filter and is not a conventional hypervisor. Its runsc runtime executes these sandboxes. (gvisor.dev)

That boundary does not validate business decisions. An agent allowed to access an API can still use it incorrectly. Isolation also does not replace access controls over data already made available inside the sandbox. Containment and action authorization must therefore remain separate concerns. (gvisor.dev)

OpenRL: training infrastructure behind an API

With OpenRL, the client coordinates the learning loop while relying on training and sampling services. This separates accelerator infrastructure from experimental logic. Check the compatibility matrix before connecting a client: Tinker compatibility should not be read as unrestricted support for every method or behavior in the software development kit, or SDK. (github.com)

The architectural implication is straightforward: running an agent in a sandbox does not mean training a model, and training a model does not require putting the trainer inside that sandbox.

Proposed architecture: separate execution from learning

For an enterprise platform, I recommend two distinct paths:

Execution:
Client → platform API → orchestrator → Agent Substrate
       → worker → gVisor sandbox → authorized tool

Learning:
Execution evidence → trusted evaluator → approved dataset
       → training client → OpenRL → candidate model
       → independent evaluation → controlled promotion

The adapter connecting these paths is application-owned. This design does not assume a native Agent Substrate–OpenRL connector.

I recommend documenting five explicit contracts:

1. Identity: which tenant, user, and execution may access each resource.
2. Execution: image, environment version, budget, deadline, and tool policy.
3. Outcome: output, evidence, termination reason, and failure classification.
4. Evaluation: evaluator version, reward, and eligibility rules.
5. Promotion: candidate version, test suite, and production approval.

Under this proposal, a zero process exit code is insufficient for training eligibility. A trusted evaluator must verify evidence and distinguish infrastructure failures from undesirable agent behavior.

I also recommend treating session state, the durable record of external actions, and model checkpoints as separate persistence concerns. They are not interchangeable recovery artifacts. Before replaying an operation after session restoration, the application should reconcile external effects. That is part of the proposed recovery contract, not a promise of exactly-once execution.

Enterprise example: an agent that proposes code fixes

Problem

Suppose a team wants to evaluate agents that repair failing tests in an internal service. Each agent receives a sanitized repository, runs tools, and returns a patch. Some attempts fail because the patch is incorrect; others stop because their worker becomes unavailable.

Combining those outcomes would make the evaluation unreliable. The proposed design keeps the evaluator outside the environment controlled by the agent and preserves the reason each attempt ended.

Solution and implementation

In this hypothetical deployment:

• the orchestrator binds each task to an immutable environment version;
• the sandbox receives only the necessary code and tools;
• the agent cannot approve its own patch;
• a separate evaluator runs protected tests;
• approved outcomes undergo data review before entering the learning loop;
• model promotion depends on an independent evaluation set.

Use a stable episode key and track attempts separately. My recommendation is to retain genuine agent failures as evaluable examples while routing infrastructure interruptions to investigation or retry. Make that policy explicit so difficult cases are not silently removed from the dataset.

Expected outcome

The intended result is traceability: identifying the model, environment, attempt, and evaluator behind every decision. This example makes no claim of measured improvements in quality, latency, or cost.

Kubernetes YAML: validate the outcome contract

The following is a self-contained educational test, not an installation of Substrate or OpenRL. It runs trusted code against synthetic records to validate an application rule: infrastructure failures must not carry rewards, while completed episodes with zero reward remain eligible.

This Job does not execute generated code. It neither demonstrates gVisor isolation nor replaces a real evaluation environment.

apiVersion: v1
kind: Namespace
metadata:
  name: agent-evaluation
---
apiVersion: batch/v1
kind: Job
metadata:
  name: rollout-contract-check
  namespace: agent-evaluation
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 120
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: rollout-contract-check
    spec:
      automountServiceAccountToken: false
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: validate
          image: python:3.12-slim
          command: ["python", "-B", "-c"]
          args:
            - |
              import json
              import math

              records = [
                  {"episode_id": "e1", "outcome": "completed", "reward": 1.0},
                  {"episode_id": "e2", "outcome": "completed", "reward": 0.0},
                  {"episode_id": "e3", "outcome": "infra_error", "reward": None},
              ]
              seen = set()
              eligible = []
              for record in records:
                  key = record["episode_id"]
                  if key in seen:
                      raise ValueError("duplicate episode")
                  seen.add(key)
                  outcome = record["outcome"]
                  reward = record["reward"]
                  if outcome == "infra_error":
                      if reward is not None:
                          raise ValueError("infra_error must not carry reward")
                      continue
                  if outcome != "completed":
                      raise ValueError("unknown outcome")
                  if type(reward) not in (int, float) or not math.isfinite(reward):
                      raise ValueError("reward must be finite")
                  eligible.append(record)
              if not eligible:
                  raise ValueError("no eligible episodes")
              print(json.dumps({
                  "eligible": len(eligible),
                  "infra_errors": len(records) - len(eligible),
              }))
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

backoffLimit: 0 prevents automatic retries following failure; activeDeadlineSeconds bounds execution time; ttlSecondsAfterFinished enables cleanup after completion. These settings do not provide exactly-once processing. With real data, use durable deduplication and export evidence before cleanup. (kubernetes.io)

The security context selects a non-root user, drops capabilities, prevents privilege escalation, and makes the root filesystem read-only. These settings harden the container; they do not turn it into a gVisor sandbox. (kubernetes.io)

Requests and limits are illustrative. Size the real validator from measurements: requests affect scheduling, while CPU and memory limits use different enforcement mechanisms. (kubernetes.io)

Save the manifest as rollout-contract-check.yaml, then run:

kubectl apply -f rollout-contract-check.yaml
kubectl wait --for=condition=complete job/rollout-contract-check \
  -n agent-evaluation --timeout=150s
kubectl logs -n agent-evaluation job/rollout-contract-check

The expected output is {"eligible": 2, "infra_errors": 1}. These field names belong to this example's application contract, not an OpenRL API.

For production, I recommend a reviewed validator image pinned by digest, a versioned schema, and recorded model, environment, dataset, and evaluator revisions. Give each execution a unique Job name. This short-lived test does not need probes: completion is its success condition. Persistent services require a different health strategy. (kubernetes.io)

Designing for production operations

Availability and recovery

I recommend separate capacity for the control plane, execution workers, and training services. Spread replicable components across zones using topology constraints that match available capacity. Hard constraints can leave Pods pending when eligible topology domains are unavailable. (kubernetes.io)

Substrate's documentation recommends external PostgreSQL for production. Add high availability, tested backups, and explicit recovery objectives to the operating model. Calculate connection requirements across every replica and connection pool rather than sizing for a single API instance. (ate.dev)

My proposed recovery plan includes:

• injecting worker failure during a task;
• testing restoration of the snapshot types actually in use;
• reconciling external effects before replay;
• maintaining a queue of incomplete outcomes;
• blocking model promotion when evidence integrity is uncertain.

For persistent APIs, distinguish readiness, liveness, and startup probes. Traffic admission, deadlock detection, and startup tolerance are different concerns. Avoid restarting a healthy process merely because a remote dependency is temporarily unavailable. (kubernetes.io)

Security across layers

Substrate documents boundaries around actors, gateways, the control plane, and storage, including egress policies and gateway-side credential injection. Authenticating end users and deciding which actors they may access remain application responsibilities. (ate.dev)

For the proposed platform, I recommend:

• separate identities for orchestration, evaluation, training, and publication;
• role-based access control, or RBAC, that prevents execution users from modifying platform infrastructure;
• narrowly scoped cloud identity and access management, or IAM;
• no credentials in images, prompts, or result records;
• Transport Layer Security, or TLS, for authenticated communication;
• encryption, retention controls, and auditing for datasets and checkpoints;
• independent approval for destructive actions.

A Kubernetes Secret is not encryption by itself. Configure encryption at rest and least-privilege access. Components that only process outcomes should not receive broad permission to read Secrets. (kubernetes.io)

For deeper coverage, see evaluating agent isolation in Kubernetes and networking for agent sandboxes.

Capacity, performance, and FinOps

gVisor's overhead is workload-dependent. It adds memory and processing costs, and system-call-heavy, network-heavy, or filesystem-heavy workloads require representative measurements. Do not assume a universal performance penalty or improvement. (gvisor.dev)

For initial planning, I propose these two accounting relationships:

Average concurrency ≈ arrival rate × average execution duration

Cost per valid outcome =
  total experiment cost / number of valid outcomes

The first expression is a steady-state starting estimate, not a peak-capacity plan. Add headroom for bursts, recovery, and maintenance. Include CPU, accelerators, storage, cross-zone traffic, gateways, load balancers, and observability in the cost numerator.

I recommend bounding concurrency by the most constrained stage. More workers are not useful when evaluation or training cannot consume their output. Favor interruptible capacity for replayable work whose evidence is persisted, and more predictable capacity for critical components. Compare savings against lost work, not just machine rates.

Observability: track actors and outcome integrity

Substrate's monitoring guidance distinguishes API behavior, Kubernetes operations, actor execution, and networking. Pod availability alone therefore leaves important blind spots. (ate.dev)

For this architecture, I propose three groups of service-level indicators, or SLIs:

• Execution: queue delay, activation latency, completion within deadline, and infrastructure failures.
• Evaluation: invalid outcomes, duplicates, missing evidence, and evaluator disagreement.
• Learning: time to a candidate checkpoint, training failures, and independent evaluation approval.

Define separate service-level objectives, or SLOs, for those groups. A fast execution with unverifiable results should not count as business success.

My instrumentation proposal uses aggregated Prometheus metrics, Grafana dashboards, and OpenTelemetry trace correlation. Keep episode identifiers in logs and traces rather than creating a metric label for every execution. Map this design onto the signals actually exposed by the installed version instead of assuming metric names exist. (ate.dev)

Development, upgrades, and rollback

For development, I recommend a small installation, synthetic data, and local evaluation. Production requires explicit budgets, access reviews, recoverable backups, alerts, contingency capacity, and auditable promotion.

Version the runtime, environment image, evaluator, and model independently. Upgrade a canary pool before expanding traffic. Substrate documents specific version-skew and rollback rules; arbitrary version jumps and downgrades should not be assumed safe or supported. (ate.dev)

In the proposed delivery process, keep the previous model available and define rollback criteria before rollout. Reverting a model does not undo external actions already performed by agents. The application needs an action ledger and a compensation procedure where appropriate.

Troubleshooting by layer

Start with Kubernetes resources to identify scheduling, startup, and termination problems: (kubernetes.io)

kubectl get pods -A
kubectl get deployments -A
kubectl get nodes
kubectl describe job rollout-contract-check -n agent-evaluation
kubectl get pods -n agent-evaluation -l job-name=rollout-contract-check
kubectl logs -n agent-evaluation job/rollout-contract-check
kubectl get events -A --sort-by=.metadata.creationTimestamp

Use the following as a proposed investigation checklist:

For restarted containers, kubectl logs <pod> -n <namespace> --previous retrieves the previous container instance's logs. The example Job uses restartPolicy: Never, so that is not its normal diagnostic path. (kubernetes.io)

When to adopt this design—and when to simplify

I would consider this composition for large session fleets, untrusted execution, recovery requirements, and an established evaluation and post-training program. The team must be prepared to operate more than one control plane.

For a small number of trusted tasks, consider Jobs and a queue. For agents limited to approved tool calls, evaluate a conventional service with per-operation authorization. If the immediate goal is better behavior without training, begin with evaluation, tool design, and configuration. An agent does not automatically justify OpenRL.

If gVisor does not meet workload compatibility requirements or the threat model, evaluate virtual-machine-based isolation. Substrate documents gVisor and microVM classes with different resource and execution behavior. Select through testing rather than assuming one is universally superior. (ate.dev)

The broader trade-off is that specialized APIs provide a better session contract while increasing dependence on their state and recovery models. I recommend storing outcomes and evidence in independently versioned formats to make future migration more practical.

Practical summary and conclusion

The decision is not whether to install three technologies at once. It is how to establish verifiable boundaries between execution, containment, evaluation, and learning.

For incremental adoption, I recommend:

1. define outcome contracts and failure classifications;
2. test isolation and recovery with representative workloads;
3. measure cost per valid outcome;
4. introduce post-training only after establishing trustworthy data and evaluation criteria;
5. promote changes through canaries with tested rollback procedures.

For Platform Engineering, this creates capabilities with explicit operating limits. For SRE, it turns agent execution into measurable reliability objectives. For DevOps, it connects delivery and model improvement without confusing a completed process with a safe decision or a valid learning signal.