LangGraph and PostgreSQL: testing recovery and controlling checkpoint growth

Persisting workflow state does not prove that a workflow can recover correctly. The decisive test is a crash after a business operation commits but before the workflow records that the step completed. Meanwhile, retaining every checkpoint forever is not a capacity strategy.

Consider a proposed internal platform that prepares infrastructure changes, requests approval, and records their execution. We need to demonstrate two properties: recovery does not apply the change twice, and cleanup does not remove an approval that is still pending.

This guide uses the LangGraph Python library with PostgresSaver directly. It does not assume Agent Server thread-management services, where the server manages persistence. Checkpointers persist thread-scoped state; stores support data shared across threads. Those are separate responsibilities. (docs.langchain.com)

1. Know what you are retaining

Do not confuse workflow checkpoints with PostgreSQL's internal database checkpoints. Here, the retention target is application state stored in tables.

The essential concepts are:

• thread_id: identifies a continuing workflow or conversation.
• checkpoint_id: identifies a persisted point.
• checkpoint_ns: separates namespaces, including those used by subgraphs.
• Superstep: an execution step that may involve multiple nodes.

Make durability explicit: sync persists before advancing; async overlaps persistence with the next step; exit persists when execution exits, including interrupts, but cannot recover intermediate progress after an abrupt process crash. Use sync for this lab. (docs.langchain.com)

In the PostgreSQL implementation reviewed, checkpoints holds snapshots and metadata, checkpoint_blobs stores channel values, and checkpoint_writes contains task-related writes. checkpoint_migrations tracks migrations. Checkpoints reference channel versions and blobs; an individual row does not necessarily contain everything required to reconstruct its state. (github.com)

Resume and replay are different operations

A dynamic interrupt uses interrupt() and continues through Command(resume=...) with the same thread_id. The interrupted node restarts, so code preceding interrupt() can run again. Do not swallow its control-flow exception with a broad exception handler. (docs.langchain.com)

After a failure, pending work can continue through graph.invoke(None, config). That is different from submitting the initial input again. Selecting a historical checkpoint enables replay: subsequent nodes, including external calls, may run again. (docs.langchain.com)

2. Proposed architecture

Client → authenticated API → per-thread coordinator → LangGraph worker
                                                         ↓
                                                   PostgresSaver
                                                         ↓
                                                   HA PostgreSQL

Worker → idempotent operation → business system
Retention controller → thread catalog → authorized deletion

I recommend separating three responsibilities:

1. Execution: advance the graph and persist its state.
2. Coordination: prevent incompatible concurrent updates to a thread and block resumes during deletion.
3. Retention: decide when history is no longer needed.

Do not treat the checkpointer as business authorization. Before accepting a resume, the API should validate the tenant, approver identity, current workflow status, and whether the decision remains valid.

For the infrastructure-change scenario, the proposed sequence is prepare → approve → record/apply idempotently → complete. Correct recovery is an acceptance criterion to test, not a property to infer from using PostgreSQL.

3. Lab: interrupt, crash the process, and recover

Use a disposable database supplied through DATABASE_URL and a Python version supported by your selected dependencies. These installation commands bootstrap a lab; after validating the combination, pin dependencies and hashes in your build process.

python -m venv .venv
. .venv/bin/activate
python -m pip install langgraph langgraph-checkpoint-postgres 'psycopg[binary]'
python -m pip check
python -m pip freeze > requirements-tested.txt

The saver's setup() method prepares its schema. Run it as a controlled migration step, not on every request. When constructing connections manually, follow the documented requirements for autocommit=True and row_factory=dict_row. This example uses the from_conn_string() context manager. (pypi.org)

Save the following as recovery_demo.py:

import os
import sys
from typing import TypedDict

import psycopg
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import Command, interrupt

DB = os.environ['DATABASE_URL']
THREAD = os.environ['THREAD_ID']
CONFIG = {'configurable': {'thread_id': THREAD}}


class State(TypedDict):
    operation_id: str
    approved: bool
    status: str


def approval(state: State):
    answer = interrupt({'operation_id': state['operation_id']})
    if not isinstance(answer, bool):
        raise ValueError('Approval must be boolean')
    return {'approved': answer}


def apply_change(state: State):
    if not state['approved']:
        return {'status': 'rejected'}

    # The committed row IS the business effect in this laboratory.
    with psycopg.connect(DB) as conn:
        conn.execute(
            'INSERT INTO demo_effects (operation_id) VALUES (%s) '
            'ON CONFLICT (operation_id) DO NOTHING',
            (state['operation_id'],),
        )

    # Crash after the business commit, before the node returns.
    if os.environ.get('FAIL_AFTER_COMMIT') == '1':
        os._exit(17)
    return {'status': 'done'}


builder = StateGraph(State)
builder.add_node('approval', approval)
builder.add_node('apply_change', apply_change)
builder.add_edge(START, 'approval')
builder.add_edge('approval', 'apply_change')
builder.add_edge('apply_change', END)

mode = sys.argv[1]
with PostgresSaver.from_conn_string(DB) as saver:
    if mode == 'setup':
        saver.setup()
        with psycopg.connect(DB) as conn:
            conn.execute(
                'CREATE TABLE IF NOT EXISTS demo_effects ('
                'operation_id TEXT PRIMARY KEY)'
            )
    else:
        graph = builder.compile(checkpointer=saver)
        if mode == 'start':
            result = graph.invoke(
                {'operation_id': THREAD, 'approved': False,
                 'status': 'pending'},
                CONFIG, durability='sync',
            )
            assert result['__interrupt__']
            assert graph.get_state(CONFIG).next == ('approval',)
        elif mode == 'resume':
            graph.invoke(Command(resume=True), CONFIG, durability='sync')
        elif mode == 'recover':
            before = graph.get_state(CONFIG)
            assert before.values['approved'] is True
            assert before.next == ('apply_change',)
            result = graph.invoke(None, CONFIG, durability='sync')
            assert result['status'] == 'done'
            assert graph.get_state(CONFIG).next == ()
            with psycopg.connect(DB) as conn:
                count = conn.execute(
                    'SELECT count(*) FROM demo_effects '
                    'WHERE operation_id = %s', (THREAD,),
                ).fetchone()[0]
            assert count == 1
            print('PASS: recovered with exactly one stored effect')
        else:
            raise ValueError('Unknown mode')

Run each phase in a separate process:

set -eu
: "${DATABASE_URL:?Inject a disposable database connection string}"
export THREAD_ID="$(python -c 'import uuid; print(uuid.uuid4())')"

python recovery_demo.py setup
python recovery_demo.py start

rc=0
FAIL_AFTER_COMMIT=1 python recovery_demo.py resume || rc=$?
test "$rc" -eq 17

python recovery_demo.py recover

The test checks a persisted interrupt, approval recovered by another process, an abrupt crash after commit, and completion without duplicating the business row. It does not rely on memory from the previous process.

Important boundary: the database row is the entire business effect in this lab. Writing a local key and then calling an API does not make that API idempotent. For external operations, I recommend a stable key honored by the destination, outcome reconciliation, or an outbox pattern with an idempotent consumer. The key should identify the business operation, not an attempt or a newly generated checkpoint.

This example does not establish behavior during database failover, concurrent writers, or a network outage. Those require additional experiments.

Minimum acceptance matrix

Reusing completed-node work within a failed superstep depends on the pending writes retained by checkpointing. Include parallel-branch failures in the test suite rather than testing only linear graphs. (docs.langchain.com)

4. Measure growth before deleting data

Start with total physical relation size, including indexes and PostgreSQL's TOAST storage for large values. Adapt the schema name:

SELECT c.relname,
       pg_total_relation_size(c.oid) AS total_bytes,
       pg_size_pretty(pg_total_relation_size(c.oid)) AS total_pretty
FROM pg_class AS c
JOIN pg_namespace AS n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
  AND c.relname IN (
    'checkpoints', 'checkpoint_blobs', 'checkpoint_writes'
  )
ORDER BY total_bytes DESC;

pg_total_relation_size measures each table's total footprint rather than only its heap. Summing these results does not include every database storage component, such as WAL and backups. (postgresql.org)

Check maintenance activity as well:

SELECT relname, n_live_tup, n_dead_tup,
       last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
WHERE schemaname = 'public'
  AND relname IN (
    'checkpoints', 'checkpoint_blobs', 'checkpoint_writes'
  );

Tuple counts are estimates, not exact bloat measurements. Correlate them with growth, workload, and transaction duration. (postgresql.org)

For capacity planning, I recommend measuring incremental bytes per completed thread under representative load:

estimated footprint = threads/day × incremental bytes/thread × retention days
                      + active threads + indexes + operating headroom

Do not count indexes twice if your incremental measurement already includes them. Budget separately for Write-Ahead Log (WAL), replicas, backups, and maintenance headroom.

As a hypothetical example, 10,000 daily threads averaging 200 KiB of incremental storage represent approximately 1.9 GiB/day before components excluded from that measurement. This is not a LangGraph benchmark.

5. Reduce growth at the source

I recommend four design constraints:

1. Store immutable references to large documents, including a version and checksum.
2. Limit tool outputs and maximum input size.
3. Scope threads to units of work instead of reusing a global thread indefinitely.
4. Define an explicit policy for conversation history and audit evidence.

Reducing messages in current state does not automatically delete historical checkpoints. Treat state compaction and history expiration as separate operations. Message-management APIs change state; deleting checkpoints requires its own lifecycle. (docs.langchain.com)

Likewise, do not replace sync with exit merely to reduce storage without approving the resulting change to the recovery contract.

6. Safe retention: begin with whole threads

My starting recommendation is to delete completed, expired threads rather than write custom SQL that keeps an arbitrary last N rows.

Maintain an application-owned catalog with tenant_id, thread_id, status, last activity, expires_at, graph version, and a retention-hold flag. These are proposed application fields, not native saver columns.

Recommended deletion protocol

1. Select a small batch of completed, expired candidates.
2. Exclude retention-held threads and pending approvals.
3. Acquire the same coordination mechanism used by executors.
4. Mark the thread deleting and reject new admissions.
5. Drain existing executions and confirm that no authorized writers remain.
6. Delete through the saver.
7. Verify absence and record completion.
8. Retain a tombstone long enough to reject delayed deliveries.

The public delete_thread(thread_id) method removes the thread's checkpoints and writes; the PostgreSQL implementation also removes its blobs. It does not automatically delete an independent store or business-system records. (reference.langchain.com)

The deletion step below is appropriate only after enforcing the protocol above:

import os
from langgraph.checkpoint.postgres import PostgresSaver

thread_id = os.environ['EXPIRED_FENCED_THREAD_ID']
config = {'configurable': {'thread_id': thread_id}}

with PostgresSaver.from_conn_string(os.environ['DATABASE_URL']) as saver:
    saver.delete_thread(thread_id)
    assert saver.get_tuple(config) is None

The environment variable is an operational input, not a fencing mechanism. A lock observed only by the collector cannot protect against workers that ignore it. Design for safe retries when cleanup fails between deleting data and updating the catalog.

What about retaining only the latest checkpoint?

The reviewed main implementation includes prune(..., strategy='keep_latest'), but that does not establish availability in every installed release. It preserves required dependencies, applies namespace rules, and rejects threads using DeltaChannel. Confirm the pinned release's API and constraints before adopting it. Do not copy internal SQL into production. (github.com)

If historical replay is a requirement, keeping only the latest point changes product behavior, not merely storage consumption.

7. DELETE, VACUUM, and Kubernetes

After DELETE, disk space can remain allocated. Regular VACUUM makes space reusable but generally does not shrink files as an operator might expect. VACUUM FULL rewrites the table, requires an exclusive lock, and needs additional disk space. Do not schedule it automatically without operational planning. (postgresql.org)

The following manifest is a manually enabled maintenance template, not a retention collector. It supplements rather than replaces autovacuum:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: langgraph-checkpoint-vacuum
  namespace: platform
spec:
  schedule: "30 3 * * *"
  timeZone: "Etc/UTC"
  suspend: true
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 900
      template:
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: vacuum
              image: postgres:18
              command: ["psql"]
              args:
                - "-X"
                - "-v"
                - "ON_ERROR_STOP=1"
                - "-c"
                - "VACUUM (ANALYZE) public.checkpoints;"
                - "-c"
                - "VACUUM (ANALYZE) public.checkpoint_blobs;"
                - "-c"
                - "VACUUM (ANALYZE) public.checkpoint_writes;"
              envFrom:
                - secretRef:
                    name: langgraph-maintenance-db
              env:
                - name: PGSSLMODE
                  value: verify-full
                - name: PGSSLROOTCERT
                  value: /etc/postgres-ca/ca.crt
                - name: PGCONNECT_TIMEOUT
                  value: "10"
              resources:
                requests:
                  cpu: 100m
                  memory: 64Mi
                limits:
                  cpu: 500m
                  memory: 256Mi
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
              volumeMounts:
                - name: postgres-ca
                  mountPath: /etc/postgres-ca
                  readOnly: true
          volumes:
            - name: postgres-ca
              configMap:
                name: postgres-ca

Before deployment, create the namespace, a Secret containing PGHOST, PGPORT, PGDATABASE, PGUSER, and PGPASSWORD, and the ConfigMap containing ca.crt. Adjust the schema, schedule, resources, maintenance permissions, and pin the image to an approved digest. Container resource limits do not cap the VACUUM workload running inside PostgreSQL.

suspend: true requires deliberate activation. concurrencyPolicy: Forbid prevents overlap between this CronJob's Jobs; it does not coordinate workers or other maintenance tasks. CronJobs can experience duplicate or missed executions under some circumstances, so tasks should tolerate reruns. (kubernetes.io)

This finite Job uses a deadline and exit status. Probes are more useful for long-running services: readiness controls traffic admission, liveness detects process failure, and startup protects slow initialization. Do not configure liveness to restart every worker solely because PostgreSQL is temporarily unavailable. (kubernetes.io)

8. Production reliability, security, and cost

Availability and concurrency

For production, I recommend PostgreSQL high availability across zones and distributed workers with requests, limits, graceful shutdown, and a disruption budget. Scale against outstanding work and database capacity, not CPU alone.

Use a bounded connection pool. Psycopg exposes max_size and acquisition timeouts; establish a global budget that includes processes, replicas, migrations, and maintenance. (psycopg.org)

A proposed sizing rule is:

potential connections = pods × processes/pod × maximum pool size
                        + business connections + maintenance + reserve

I recommend serializing mutations per thread and enforcing a lease/fencing protocol that retention also observes. Additional replicas do not replace coordination.

Security

Separate migration, runtime, and maintenance credentials. Use Transport Layer Security (TLS) with identity verification, private database connectivity, role-based access control (RBAC), and NetworkPolicy rules supported by your cluster network.

Keep credentials out of Git. Kubernetes Secrets do not guarantee encryption at rest by default; configure storage protection and least-privilege access. Avoid broad Secret-reading permissions for operators and workloads. (kubernetes.io)

I also recommend tenant authorization independent of thread_id, credential rotation, personal-data minimization, and excluding full checkpoint payloads from routine logs.

Backup and rollback

Point-in-Time Recovery (PITR) combines a base backup with an archived WAL sequence to recover to a selected point. Replication does not replace this recovery strategy. (postgresql.org)

Define a Recovery Point Objective (RPO), the acceptable data-loss window, and a Recovery Time Objective (RTO), the recovery deadline. Test isolated restoration before enabling automatic deletion. Restoring checkpoints does not reverse external effects; disable real integrations while validating restored workflows.

For upgrades, I recommend fixtures containing older checkpoints, deserialization and resume tests, versioned state schemas, and an explicit backward-compatibility policy. Rolling back the container image does not prove that older code can read state written by the newer release.

FinOps and observability

Treat storage, input/output operations, WAL, backups, cross-zone traffic, and telemetry as separate cost components. Use Spot workers only where the recovery policy has been demonstrated; retain stable capacity for critical paths.

Instrument application metrics for persistence and resume latency, errors by phase, pool wait time, pending threads, expiration backlog, bytes per table, and cleanup duration. These are proposed measurements, not guaranteed built-in LangGraph metric names.

Define a Service Level Objective (SLO) for eligible resumes that complete correctly within the required time. Correlate attempts, graph versions, and checkpoints in traces; avoid using thread_id as a high-cardinality metric label.

9. Troubleshooting

For the Kubernetes maintenance Job:

kubectl get cronjobs,jobs,pods -n platform
kubectl describe cronjob langgraph-checkpoint-vacuum -n platform
kubectl logs -n platform job/<job-name>
kubectl describe pod <pod-name> -n platform
kubectl get events -n platform --sort-by=.metadata.creationTimestamp

Do not expose connection strings while gathering evidence. Distinguish graph, authentication, transport, pool, storage, and maintenance failures before changing configuration.

10. Development, alternatives, and adoption criteria

A single replica and disposable database may be sufficient during development. For production, I recommend automated retention with thread protection, monitoring, tested backups, distributed coordination, and a documented rollback procedure.

Use this approach when workflows must survive restarts, wait for approvals, or maintain thread continuity. For short, disposable tasks, the operating burden may not be justified.

Alternatives include in-memory persistence for ephemeral tests, SQLite for local execution, and an application-owned transactional state machine for simpler workflows. Checkpointer implementations target different operating contexts; a local option is not automatically a substitute for shared persistence. (docs.langchain.com)

My architectural recommendation is to choose PostgreSQL when the team can operate its availability, recovery, and capacity requirements. A managed service can reduce some operational work, but assess its costs, retention controls, and provider dependency before adopting it.

Practical checklist

• Test across separate processes and inject failure after the business effect.
• Distinguish recovery, new input, and historical replay.
• Use stable operation identities and verify idempotency at the destination.
• Measure tables, indexes, WAL, and backups separately.
• Delete expired threads only after preventing further writes.
• Separate state reduction, history retention, and physical space reclamation.
• Require restoration and compatibility tests before upgrades.

Conclusion

The goal is neither to accumulate checkpoints indefinitely nor to delete them as aggressively as possible. It is to maintain a verifiable contract between workflow execution, business effects, and recovery.

In a DevOps and SRE platform, that contract belongs in the test pipeline, capacity limits, SLOs, and operational procedures. Designing recovery and retention together turns PostgreSQL from a snapshot destination into an explicit part of the workflow's reliability strategy.