Cutting MTTR: building an AI agent for incident triage and automatic resolution

Introduction

In distributed environments, the problem is rarely just detecting that something failed. Prometheus, Grafana, CloudWatch, Datadog, Loki, OpenTelemetry and other tools can identify degradation within seconds.

The real challenge starts after the alert fires.

A typical incident may require an engineer to:

1. identify which application was affected;
2. figure out the namespace, cluster and environment;
3. check metrics;
4. read logs;
5. inspect Kubernetes events;
6. compare against recent deployments;
7. identify configuration changes;
8. validate dependencies such as the database, Redis or RabbitMQ;
9. decide on a corrective action;
10. execute the fix;
11. confirm the environment is healthy again.

Every one of these steps takes time and increases MTTR — Mean Time to Recovery (or Mean Time to Repair), the metric representing how long, on average, an organization takes to restore a service after a failure.

This is exactly the space where AI agents can be useful.

Instead of using AI only to summarize alerts, we can build an agent capable of automatically gathering evidence, correlating metrics, logs, traces and recent changes, executing pre-authorized runbooks, and validating whether the intervention actually solved the problem.

The goal should not be:

"Give an LLM admin access to the cluster so it can fix any incident."

A safe architecture should look like:

Alert
   ↓
Context collection
   ↓
Evidence correlation
   ↓
AI / decision engine
   ↓
Policy Engine
   ↓
Authorized runbook
   ↓
Controlled execution
   ↓
Validation
   ↓
Closure or escalation

This distinction matters a lot.

The AI agent should be treated as part of the operational automation platform, not as an unrestricted infrastructure administrator.

What is an AI agent for incidents

An AI agent for incidents is a system capable of receiving operational events, gathering context about the environment, analyzing possible causes and deciding which actions should run within a previously defined set of permissions.

It combines traditional observability and automation components with language models.

An implementation might integrate:

Prometheus
Grafana
Alertmanager
Loki
Tempo
OpenTelemetry
Kubernetes API
AWS APIs
Git
ArgoCD
CI/CD
CMDB
Runbooks
LLM

The LLM doesn't need to run commands directly against the infrastructure.

A safer architecture uses intermediate tools.

For example:

LLM
 ↓
restartDeployment(namespace, deployment)
 ↓
Policy Engine
 ↓
Kubernetes API

Instead of:

LLM
 ↓
shell("kubectl delete pod -A ...")

The first approach lets you restrict exactly which operations can be performed.

Key concepts

MTTA

Mean Time to Acknowledge represents the time needed for an incident to be acknowledged.

An automated triage system can shrink this to roughly the interval between the alert firing and the agent processing it.

MTTD

Mean Time to Detect represents the time needed to detect the failure.

This part is usually the responsibility of the observability stack.

MTTR

Mean Time to Recovery measures how long it takes to restore the service.

This is usually where automation and AI agents show the most potential.

Runbook

Runbooks are documented procedures for solving known situations.

Example:

Problem:
Deployment with no available pods.

Diagnosis:
- check replicas;
- check events;
- check logs;
- check cluster resources.

Remediation:
- restart the rollout, if allowed;
- validate pods;
- validate the endpoint.

An agent can turn static runbooks into executable workflows.

Tool Calling

In an agent-based architecture, the model is given tools it can request.

Conceptual example:

getPods(namespace)
getDeployment(name)
getPodLogs(pod)
getEvents(namespace)
getMetrics(workload)
restartDeployment(name)
rollbackDeployment(name)

This is significantly safer than allowing arbitrary shell execution.

Policy Engine

The Policy Engine determines whether the action suggested by the agent can actually be executed.

For example:

restartDeployment -> allowed

scaleDeployment 3 → 5 -> allowed

scaleDeployment 3 → 100 -> denied

deleteNamespace -> denied

deleteDatabase -> denied

Or:

Production:
restart -> automatic
rollback -> human approval
delete -> forbidden

Reference architecture

An architecture for Kubernetes can be structured like this:

                     ┌──────────────────┐
                     │   Prometheus     │
                     └────────┬─────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │   Alertmanager   │
                     └────────┬─────────┘
                              │ webhook
                              ▼
                  ┌─────────────────────────┐
                  │ Incident AI Controller  │
                  └────────────┬────────────┘
                               │
             ┌─────────────────┼─────────────────┐
             │                 │                 │
             ▼                 ▼                 ▼
          Kubernetes          Loki             Tempo
          API Server          Logs             Traces
             │
             ▼
       Prometheus API
             │
             ▼
          Git / CI/CD
             │
             ▼
       Context Builder
             │
             ▼
        AI Reasoning
             │
             ▼
        Policy Engine
             │
             ▼
      Remediation Executor
             │
             ▼
        Kubernetes / AWS

Alertmanager can forward notifications via webhook to external systems; rules are normally evaluated by Prometheus and then forwarded to Alertmanager.

How it flows

Imagine the following alert:

HighErrorRate

service=checkout-api
namespace=production
error_rate=18%

The agent receives the alert and starts collecting evidence.

1. Check the Deployment

kubectl get deployment checkout-api -n production

Result:

READY   UP-TO-DATE   AVAILABLE
2/5     5            2

2. Check the pods

kubectl get pods -n production -l app=checkout-api

Result:

checkout-api-abc12   Running
checkout-api-def34   CrashLoopBackOff
checkout-api-ghi56   CrashLoopBackOff
checkout-api-jkl78   Running
checkout-api-mno90   CrashLoopBackOff

3. Check events

kubectl get events -n production \
  --sort-by=.metadata.creationTimestamp

The agent finds:

Back-off restarting failed container

4. Check logs

kubectl logs checkout-api-def34 \
  -n production \
  --previous \
  --tail=200

It finds:

Connection pool exhausted
database timeout after 5000ms

5. Check metrics

Telemetry shows:

DB connections: 100/100
API latency p95: 4.8s
API error rate: 18%
CPU: 42%
Memory: 53%

The agent can already rule out CPU and memory as the main cause.

6. Correlate recent changes

The agent checks Git or ArgoCD:

Deployment shipped 7 minutes earlier.

Change:
DB_POOL_SIZE=50 → DB_POOL_SIZE=100

Five replicas with pools of up to 100 connections could potentially push the database limit much further than the previous configuration.

The AI then produces a hypothesis:

Likely cause:

The last deployment raised the max connection pool.
The database hit its max connection limit.
Pods started timing out and CrashLoopBackOff'ing.

Confidence: high.

But the AI still doesn't execute anything.

The decision moves to the Policy Engine.

Problem → Solution → Configuration → How it works

Before

The incident required an engineer to manually run:

kubectl get pods
kubectl describe pod
kubectl logs
kubectl get events

Then check Grafana, Loki, GitHub, ArgoCD and the database.

Average time:

Detection:         1 minute
Triage:            8 minutes
Investigation:    15 minutes
Fix:               5 minutes
Validation:        5 minutes

Approximate MTTR: 34 minutes

After

With the agent:

Detection:          1 minute
Collection:       20 seconds
Correlation:      10 seconds
Remediation:      30 seconds
Validation:        2 minutes

MTTR: approximately 4 minutes

The gain doesn't come from AI alone.

Most of the reduction comes from automating evidence gathering and runbook execution.

Real-world production example

Consider a company running about 40 microservices on Amazon EKS.

The architecture includes:

AWS
 └── EKS
     ├── APIs
     ├── Workers
     ├── RabbitMQ
     ├── Redis
     ├── PostgreSQL
     └── Observability
          ├── Prometheus
          ├── Grafana
          ├── Loki
          └── Tempo

The agent runs in a dedicated namespace:

incident-system

and has read access to the monitored namespaces.

Destructive operations are not allowed.

Allowed automatic actions could include:

restart a Deployment
restart a specific StatefulSet
scale a workload within defined limits
re-run a Job
remove an unhealthy Pod
trigger a previously defined rollback

Depending on risk, some of these actions may require human approval.

YAML example: ServiceAccount and RBAC

Kubernetes lets you use ServiceAccounts alongside RBAC to grant only the permissions a workload needs, following the principle of least privilege.

An initial configuration might look like:

apiVersion: v1
kind: Namespace
metadata:
  name: incident-system

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: incident-agent
  namespace: incident-system
automountServiceAccountToken: true

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: incident-agent
  namespace: production
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/log
      - events
    verbs:
      - get
      - list
      - watch

  - apiGroups:
      - apps
    resources:
      - deployments
      - replicasets
    verbs:
      - get
      - list
      - watch
      - patch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: incident-agent
  namespace: production
subjects:
  - kind: ServiceAccount
    name: incident-agent
    namespace: incident-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: incident-agent

What this configuration does

The ServiceAccount:

incident-agent

is used by the agent.

The Role allows querying:

pods
logs
events
deployments
replicasets

The verb:

patch

allows changing certain objects, needed for example by restart mechanisms based on updating the Deployment template.

Cautions

Avoid:

resources:
  - "*"

verbs:
  - "*"

especially in production.

Also avoid granting a ClusterRole if the agent only needs to act on specific namespaces.

Deploying the agent

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-agent
  namespace: incident-system
spec:
  replicas: 2

  selector:
    matchLabels:
      app: incident-agent

  template:
    metadata:
      labels:
        app: incident-agent

    spec:
      serviceAccountName: incident-agent

      containers:
        - name: agent
          image: registry.example.com/platform/incident-agent:1.4.2

          ports:
            - name: http
              containerPort: 8080

          env:
            - name: ENVIRONMENT
              value: production

            - name: PROMETHEUS_URL
              value: http://prometheus.monitoring.svc:9090

            - name: LOKI_URL
              value: http://loki.monitoring.svc:3100

            - name: AUTO_REMEDIATION
              value: "true"

            - name: MAX_SCALE_FACTOR
              value: "2"

          resources:
            requests:
              cpu: 250m
              memory: 512Mi

            limits:
              cpu: "1"
              memory: 1Gi

          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 15
            periodSeconds: 20

---
apiVersion: v1
kind: Service
metadata:
  name: incident-agent
  namespace: incident-system
spec:
  selector:
    app: incident-agent

  ports:
    - port: 80
      targetPort: http

This Deployment runs two replicas of the agent so the incident response tool itself doesn't depend on a single pod.

The CPU and memory values are just an initial reference. They should be tuned using real metrics.

It's also important to avoid storing API keys directly in variables declared on the Deployment.

Use Secrets or an external secret management mechanism.

Receiving alerts from Alertmanager

A simplified Alertmanager configuration could forward critical alerts:

route:
  receiver: default

  routes:
    - receiver: incident-agent
      matchers:
        - severity="critical"

receivers:
  - name: default

  - name: incident-agent
    webhook_configs:
      - url: http://incident-agent.incident-system.svc/incidents
        send_resolved: true

In this model:

Prometheus
   ↓
Alertmanager
   ↓
Webhook
   ↓
Incident Agent

The agent receives labels such as:

alertname
service
namespace
severity
cluster
environment

The better the alert labels, the better the automation.

An alert with just:

something-is-broken

isn't very useful.

An alert containing:

service=payment-api
namespace=production
cluster=eks-prod
team=payments
severity=critical
runbook=payment-error-rate

enables much more precise automation.

The decision engine

The agent shouldn't just send:

"There's an alert. What do I do?"

to the model.

It should first build structured context.

For example:

{
  "incident": {
    "alert": "HighErrorRate",
    "service": "checkout-api",
    "namespace": "production"
  },
  "metrics": {
    "error_rate": 18.4,
    "latency_p95_ms": 4800,
    "cpu_percent": 42,
    "memory_percent": 53
  },
  "kubernetes": {
    "desired_replicas": 5,
    "available_replicas": 2,
    "crashloop_pods": 3
  },
  "logs": ["connection pool exhausted", "database timeout"],
  "recent_changes": [
    {
      "type": "deployment",
      "minutes_ago": 7,
      "revision": "4fd92ab"
    }
  ]
}

The AI receives normalized data instead of thousands of lines of logs.

This reduces:

• token usage;
• latency;
• cost;
• noise;
• the chance of misinterpretation.

Separating diagnosis from execution

An important architectural decision is separating:

Reasoning Engine

from:

Execution Engine

The first can suggest:

{
  "diagnosis": "database_connection_exhaustion",
  "confidence": 0.94,
  "action": "rollback",
  "target_revision": "previous"
}

The executor validates:

Does the action exist?

Does the service allow automatic rollback?

Does the environment allow it?

Is the change within the allowed window?

Is there an incident in progress?

Was there already an attempt?

Is the rollback within the configured limit?

Only then does the operation run.

Autonomy by levels

Not every incident should get the same level of autonomy.

A mature architecture might adopt:

The safest path is to evolve gradually.

Don't start at the maximum level.

Security

Security is probably the most important aspect of this architecture.

RBAC

The agent should have only the resources it needs.

Kubernetes lets you restrict a ServiceAccount's permissions through RBAC.

Never use:

cluster-admin

as a shortcut to make implementation easier.

IAM

If the agent needs to query AWS:

CloudWatch
EC2
RDS
EKS
SQS
SNS

use an identity associated with the workload.

On EKS, EKS Pod Identity lets you associate an IAM Role with a ServiceAccount, avoiding the need to distribute static AWS credentials inside containers.

For example, the agent might have:

cloudwatch:GetMetricData
cloudwatch:GetMetricStatistics
logs:StartQuery
logs:GetQueryResults

but not:

ec2:TerminateInstances
rds:DeleteDBInstance
iam:*

NetworkPolicy

NetworkPolicies let you control Pod communication at the IP and port level, as long as the cluster's network plugin implements enforcement.

An agent doesn't necessarily need direct access to every workload.

Example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: incident-agent
  namespace: incident-system
spec:
  podSelector:
    matchLabels:
      app: incident-agent

  policyTypes:
    - Ingress
    - Egress

  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring

      ports:
        - protocol: TCP
          port: 8080

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring

    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system

This example should be adapted to the cluster's topology, DNS setup and required external endpoints.

NetworkPolicy shouldn't be copied directly without validating the CNI's behavior and the agent's actual dependencies.

Observability of the agent itself

Building an automated incident response system without observing the system itself would be a mistake.

The agent should produce metrics such as:

incident_agent_incidents_total

incident_agent_investigation_duration_seconds

incident_agent_actions_total

incident_agent_actions_failed_total

incident_agent_auto_remediation_total

incident_agent_escalations_total

incident_agent_llm_requests_total

incident_agent_llm_tokens_total

It's also worth tracking:

MTTA before/after

MTTR before/after

percentage of incidents diagnosed

percentage of automatic remediations

reverted-remediation rate

false-diagnosis rate

Logs

Every decision needs to leave an auditable trail.

For example:

{
  "incident_id": "inc-8472",
  "service": "checkout-api",
  "action": "restart_deployment",
  "reason": "pods_unhealthy_after_dependency_recovery",
  "approved": true,
  "automatic": true,
  "timestamp": "2026-09-26T15:10:32Z"
}

This structure can be sent to Loki.

Traces

OpenTelemetry can be used to track the full flow:

Alert received
      ↓
Collect Kubernetes data
      ↓
Collect Prometheus metrics
      ↓
Collect Loki logs
      ↓
Call LLM
      ↓
Policy validation
      ↓
Execute remediation
      ↓
Validate service

This is especially important for figuring out why the automation system itself was slow or failed during an incident.

Real-world example: a stuck worker

Imagine workers consuming messages from RabbitMQ:

RabbitMQ
   ↓
payment-worker
   ↓
PostgreSQL

Prometheus identifies:

queue_messages_ready > 100000

The agent checks:

current replicas: 4
CPU: 82%
consumer throughput: 400 msg/s
incoming rate: 900 msg/s

The runbook allows scaling up to 12 replicas.

The AI recommends:

4 → 8 replicas

The Policy Engine checks:

maxReplicas = 12

and authorizes it.

After five minutes:

incoming rate: 900 msg/s
consumer throughput: 1100 msg/s
queue depth: decreasing

The remediation is considered successful.

Afterward, the agent can scale the workload back down once the queue normalizes, or that responsibility can belong to mechanisms like HPA or KEDA.

In this case, the structural fix would probably be to use autoscaling driven by queue depth instead of permanently relying on the agent.

This distinction matters:

AI shouldn't replace deterministic automations that already solve the problem better.

Development vs. production

In development, it might be acceptable to allow:

automatic restart;
automatic scaling;
rollback tests;
runbook simulation;
higher log verbosity.

In production, additional controls are needed:

minimal RBAC;
minimal IAM;
audit trail;
rate limiting;
timeouts;
circuit breakers;
approval for critical actions;
execution limits;
loop protection;
concurrency control;
rollback of the runbook itself.

There should also be a kill switch.

For example:

AUTO_REMEDIATION=false

immediately turning the system read-only.

Troubleshooting

Agent can't list pods

Test:

kubectl auth can-i list pods \
  -n production \
  --as=system:serviceaccount:incident-system:incident-agent

If it returns:

no

check the Role and RoleBinding.

Checking the agent

kubectl get pods -n incident-system

Then:

kubectl describe pod \
  -n incident-system \
  -l app=incident-agent

Logs

kubectl logs \
  -n incident-system \
  -l app=incident-agent \
  --tail=200

Events

kubectl get events \
  -n incident-system \
  --sort-by=.metadata.creationTimestamp

Checking permissions

kubectl auth can-i patch deployments \
  -n production \
  --as=system:serviceaccount:incident-system:incident-agent

Webhook not arriving

Check the Service:

kubectl get svc -n incident-system

Then:

kubectl get endpoints \
  -n incident-system \
  incident-agent

Also check:

NetworkPolicy
DNS
Service selector
containerPort
targetPort
readinessProbe

Best practices

A mature implementation should follow a few principles.

1. Automate repetitive incidents first. Known CrashLoopBackOff, unhealthy pods, stuck workers, accumulated queues and known transient failures are better candidates than novel incidents.

1. Start with AI for investigation only. Compare the diagnoses produced by the AI against engineers' diagnoses before enabling remediation.

1. Use typed tools. Prefer restartDeployment() over executeShell().

1. Implement limits. An agent should never scale 5 pods to 500 simply because it misread a metric.

1. Control repetition. For example:

maximum of 1 restart per deployment every 15 minutes.

1. Implement idempotency. Receiving the same alert twice should not trigger two concurrent remediations.

1. Preserve evidence. Collect logs and relevant information before restarting a workload.

1. Measure results. A real reduction in MTTR matters more than the number of incidents handled by AI.

What to avoid

Giving the agent cluster-admin

It's the fastest way to build something that works, and one of the worst decisions for production.

Allowing arbitrary shell access

A tool like:

execute_command(command)

turns virtually any misinterpretation into a potentially dangerous action.

Prefer specific, previously validated operations.

Sending all logs to the LLM

Besides being expensive, this increases noise.

Do this first:

filtering
 ↓
aggregation
 ↓
extracting relevant events
 ↓
LLM

Executing actions based only on the model's "confidence"

Don't use something like:

confidence > 90%
→ execute any action

Model confidence doesn't replace deterministic controls.

Creating remediation loops

Imagine:

Pod fails
↓
agent restarts it
↓
Pod fails
↓
agent restarts it
↓
Pod fails
↓
agent restarts it

This can mask the root cause.

Set limits:

maxAttempts = 2
cooldown = 15m

After that:

escalateToHuman()

Costs and FinOps

An agent's cost includes several components:

CPU/memory for the workload itself
log storage
metric queries
log queries
traces
model calls
tokens
external APIs

The most common mistake is sending large volumes of telemetry directly to the model.

For example:

50,000 Loki lines
→ LLM

It's usually better to do:

50,000 lines
 ↓
specific query
 ↓
aggregation
 ↓
50 relevant events
 ↓
LLM

This can significantly reduce tokens, latency and cost.

AI doesn't replace autoscaling

There's an important distinction here.

If the problem is:

CPU > 80%

the solution is probably HPA.

If it's:

growing RabbitMQ queue

KEDA might be more appropriate.

If it's:

cluster out of capacity

Karpenter can solve it.

The agent should mainly step in when the problem requires correlating multiple pieces of evidence or choosing between different runbooks.

A mature platform tends to follow:

Deterministic automation
        ↓
Autoscaling
        ↓
Self-healing
        ↓
Runbooks
        ↓
AI agent
        ↓
Human

not:

Any problem
      ↓
LLM

When to use this

This approach makes sense especially in environments with:

many microservices;
Kubernetes;
high alert volume;
on-call rotation;
known runbooks;
recurring problems;
mature observability stack;
SRE or Platform Engineering;
high availability requirements.

It's particularly interesting when a large part of MTTR comes from manual information gathering and correlation.

When not to use this

An AI solution can be unnecessary when:

there are few services;
there are few incidents;
the environment is simple;
there are no runbooks;
observability is still weak;
failures can be fixed with simple scripts.

If a 30-line CronJob solves the problem with predictable behavior, adding an AI agent likely increases complexity without proportional value.

Recommended evolution

A safe rollout could look like:

Phase 1
Alerts
   ↓
AI generates a summary

Phase 2
Alerts
   ↓
AI collects metrics/logs
   ↓
AI suggests a diagnosis

Phase 3
AI suggests a runbook
   ↓
Human approves

Phase 4
AI automatically executes
low-risk runbooks

Phase 5
AI investigates
   ↓
executes
   ↓
validates
   ↓
reverts if needed

This progression builds operational trust based on real results.

Practical summary

The real value of an AI agent for incidents isn't letting a model run commands freely.

It's combining:

Observability
+
Context
+
Runbooks
+
Automation
+
AI
+
Policies
+
Audit trail

to turn a process like:

alert
→ engineer wakes up
→ opens Grafana
→ looks up the service
→ opens Kubernetes
→ checks pods
→ reads logs
→ checks Git
→ finds the cause
→ executes the fix
→ monitors recovery

into:

alert
↓
automatic collection
↓
correlation
↓
diagnosis
↓
policy
↓
runbook
↓
remediation
↓
validation

keeping the engineer in the loop only when necessary.

Conclusion

Cutting MTTR doesn't just mean responding to alerts faster.

It means reducing the number of manual decisions needed between detecting a failure and recovering the service.

AI agents can play an important role in this because they can correlate information from Kubernetes, Prometheus, Loki, OpenTelemetry, Git, CI/CD, databases and cloud providers within a single investigation flow.

However, the safest architecture isn't one where AI gets unrestricted access to the infrastructure.

The more robust model is:

Observability
      ↓
Incident Controller
      ↓
Context Builder
      ↓
AI Reasoning
      ↓
Policy Engine
      ↓
Runbook Executor
      ↓
Infrastructure
      ↓
Validation Engine

In this design, AI helps decide; deterministic policies define what's allowed; controlled executors make the changes; and observability confirms whether the intervention worked.

This approach also changes the role of SRE and Platform Engineering. Instead of continuously running the same manual procedures, teams start turning operational knowledge into automated runbooks, policies and recovery mechanisms.

The most interesting outcome isn't simply a "bot that fixes alerts."

It's creating a context-driven self-healing layer, capable of automating known incidents, speeding up investigation of unknown ones, and maintaining clear security and governance controls.

Combined with autoscaling, observability, Infrastructure as Code, GitOps, security policies and reliability engineering, this architecture can turn incident response from a mostly reactive process into a progressively automated system — cutting MTTR without turning autonomy into an absence of control.