Technical engineering assessment: how to map risks and bottlenecks before they become incidents

Introduction

Severe incidents rarely appear without earlier warning signs.

A database that starts showing a gradual increase in latency, a Kubernetes cluster with requests that don't match real consumption, applications without a readinessProbe, queues quietly growing, critical workloads running on a single node, or service accounts with excessive permissions are examples of situations that can stay invisible for months.

Until a traffic change, a deploy, or an infrastructure failure turns that fragility into an outage.

That's exactly where the technical engineering assessment comes in.

A technical assessment is a structured review of architecture, infrastructure, applications, security, observability, capacity, costs and operational practices. Its goal isn't just to find existing errors, but mainly to find conditions that could evolve into incidents.

In modern environments built on Kubernetes, AWS, microservices, distributed databases, CI/CD pipelines and dozens of integrations, checking only whether "the system is working" isn't enough.

The right question is:

Will the system keep working when there's a spike in load, a loss of infrastructure, a bad deploy, or a dependency failure?

A well-executed assessment answers that question before production answers it the worst possible way.

What a technical assessment is

A technical assessment is a systematic evaluation of a platform's or system's technical state.

It looks not just at individual components, but especially at how they relate to each other.

An assessment can cover:

• architecture;
• infrastructure;
• Kubernetes;
• Cloud;
• applications;
• databases;
• messaging;
• CI/CD;
• security;
• observability;
• scalability;
• reliability;
• costs;
• operational processes.

The output is usually a risk map containing:

Current state → Evidence → Risk → Impact → Priority → Recommendation

For example:

Workload:
payment-api

Current state:
2 replicas running on the same worker node

Risk:
a node failure takes down all replicas

Impact:
complete API outage

Recommendation:
topologySpreadConstraints + PodDisruptionBudget

Priority:
High

An assessment shouldn't be confused with a formal audit.

An audit usually checks compliance with requirements or standards.

An engineering assessment tries to answer questions like:

• Where are our Single Points of Failure?
• Can the cluster handle increased traffic?
• Does the database have enough capacity?
• How long does it take us to detect a failure?
• Can a deploy take the environment down?
• Are resources oversized?
• Do we have workloads without limits?
• Are there excessive permissions?
• Can we recover the environment after a failure?

Key concepts

Risk

Risk represents the combination of:

Probability of failure × Impact of failure

A simple way to classify risks is:

A low-probability but catastrophic-impact problem also deserves attention.

A classic example is a production database with no validated backup.

You may never actually need to restore it.

But when you do, discovering the backup doesn't work can be fatal.

Bottleneck

A bottleneck is a resource or component that limits the system's capacity.

It can exist at different layers:

Application
    ↓
CPU
    ↓
Memory
    ↓
Network
    ↓
Database
    ↓
Storage
    ↓
External dependency

A common mistake is assuming high latency means lack of CPU.

In practice, the cause could be:

• saturated database connections;
• slow SQL queries;
• DNS;
• garbage collection;
• HTTP pool;
• distributed lock;
• IOPS;
• accumulated queues;
• external calls.

The assessment should look at the whole system

An efficient technical assessment analyzes several dimensions at once.

Architecture

Important questions:

• Are there Single Points of Failure?
• Are dependencies synchronous or asynchronous?
• Is there isolation between services?
• Is there retry?
• Does retry have a limit?
• Is there a timeout?
• Is there a circuit breaker?
• Can a service failure cascade?

Consider:

API
 ↓
Service A
 ↓
Service B
 ↓
Service C
 ↓
Database

If every service uses 30-second timeouts and multiple retries, a database failure could block hundreds of threads simultaneously.

The result can be a retry storm.

A more resilient architecture might use:

API
 ↓
Service
 ↓
Queue
 ↓
Worker
 ↓
Database

The queue absorbs spikes and reduces temporal coupling between systems.

How an assessment is structured

An assessment can be organized in the following flow:

Inventory
    ↓
Architecture
    ↓
Infrastructure
    ↓
Applications
    ↓
Capacity
    ↓
Observability
    ↓
Security
    ↓
FinOps
    ↓
Risks
    ↓
Action plan

The first step is understanding the environment.

It makes no sense to evaluate Kubernetes without understanding:

• which applications are critical;
• what dependencies exist;
• which database is used;
• what traffic volume exists;
• which services could become unavailable.

After that, the analysis moves to technical evidence.

Examples:

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get deployments -A
kubectl get statefulsets -A
kubectl get hpa -A
kubectl get pdb -A
kubectl get networkpolicy -A

The goal is to build an accurate map of the environment.

A day-to-day example

Consider a SaaS company running dozens of microservices on Amazon EKS.

The architecture includes:

Internet
   ↓
ALB
   ↓
Ingress Controller
   ↓
Kubernetes
   ↓
APIs
   ↓
RabbitMQ
   ↓
Workers
   ↓
PostgreSQL

During an assessment, the following points are found:

Critical API:
2 replicas

Nodes:
3

PodDisruptionBudget:
not configured

Topology Spread:
not configured

CPU Request:
100m

Real CPU:
700m

HPA:
not configured

The application works normally.

But there's a problem.

The two replicas could get scheduled on the same node.

If that node fails:

Node
 ├── API Pod 1
 └── API Pod 2

the application becomes unavailable.

The assessment catches the risk before the incident.

The fix would be to spread the replicas out.

YAML example: distributing pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: production
spec:
  replicas: 3

  selector:
    matchLabels:
      app: payment-api

  template:
    metadata:
      labels:
        app: payment-api

    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: payment-api

      containers:
        - name: api
          image: company/payment-api:2.4.1

          ports:
            - containerPort: 8080

          resources:
            requests:
              cpu: 300m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 20

This configuration uses topologySpreadConstraints to try to spread pods across different nodes.

The parameter:

topologyKey: kubernetes.io/hostname

means the distribution should account for different hosts.

Meanwhile:

whenUnsatisfiable: DoNotSchedule

prevents the scheduler from placing additional pods when the rule can't be honored.

In production, this decision should be evaluated carefully.

In small clusters, an overly strict rule can leave pods stuck Pending.

PodDisruptionBudget

Another common risk found during assessments is a missing PodDisruptionBudget (PDB).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-api

This PDB requires that at least two replicas remain available during voluntary disruptions.

For example:

Node drain
Cluster upgrade
Karpenter consolidation
Nodegroup update

Without a PDB, multiple pods of a critical application can be removed at the same time.

However, the PDB doesn't protect against unexpected node failures.

It only controls voluntary disruptions.

Assessing Kubernetes resources

One of the most important points in Kubernetes clusters is comparing:

requests
limits
actual usage
node capacity

Consider:

resources:
  requests:
    cpu: 2
    memory: 4Gi

  limits:
    cpu: 4
    memory: 8Gi

But the application uses:

Average CPU: 180m
Average memory: 600Mi

The cluster will be reserving far more capacity than it actually needs.

The scheduler works mainly based on requests.

So:

oversized requests
        ↓
fewer pods per node
        ↓
more nodes
        ↓
more cost

This is one of the most common problems found during FinOps assessments.

Capacity assessment

A platform should be analyzed based on three numbers:

Average usage
Peak usage
Maximum capacity

Example:

Node Pool

CPU capacity:
64 vCPU

Requests:
52 vCPU

Actual usage:
22 vCPU

At first glance there's a lot of idle capacity.

However, the scheduler considers the 52 vCPU of requests.

That means new pods could get stuck Pending even if actual usage is much lower.

An assessment should catch these distortions.

Horizontal Pod Autoscaler

Another important point is checking whether applications can react to increased load.

Example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api

  minReplicas: 3
  maxReplicas: 20

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60

    scaleDown:
      stabilizationWindowSeconds: 300

This HPA scales the application when CPU utilization relative to requests goes up.

It's important to understand that relationship.

If the request is wrong:

Actual CPU: 600m
CPU request: 2000m

the HPA might read:

600 / 2000 = 30%

and not scale.

So the assessment should analyze the HPA and requests together.

Real-world production example

Consider a platform containing:

AWS
 └── EKS
      ├── APIs
      ├── Workers
      ├── Redis
      ├── RabbitMQ
      ├── PostgreSQL
      ├── Prometheus
      ├── Grafana
      └── Loki

The assessment finds:

1. Nodes spread across only one AZ
2. APIs without a PDB
3. Workers without autoscaling
4. RabbitMQ with no queue monitoring
5. Database nearing 90% of storage
6. No NetworkPolicy in place
7. Shared ServiceAccounts
8. Logs with no defined retention
9. Oversized requests
10. Alerts based only on CPU

None of these points necessarily causes an incident right now.

But all of them represent accumulated risk.

That's exactly where the assessment's biggest value lies:

catching problems while they're still cheap to fix

Database evaluation

An assessment should also analyze the database.

In PostgreSQL, some important points include:

SELECT
    datname,
    numbackends
FROM pg_stat_database;

To identify slow queries, when pg_stat_statements is available:

SELECT
    query,
    calls,
    mean_exec_time,
    total_exec_time
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

Other points to assess:

• number of connections;
• locks;
• indexes;
• slow queries;
• table growth;
• WAL;
• replication;
• backup;
• restore;
• IOPS;
• storage.

An environment can be healthy on CPU and still suffer from a database bottleneck.

Messaging

Asynchronous systems need specific metrics.

For RabbitMQ, for example:

messages_ready
messages_unacknowledged
consumer count
publish rate
ack rate
queue depth

Consider:

Publish rate:
1,000 messages/s

Consume rate:
800 messages/s

The system is accumulating:

200 messages/s

After one hour:

720,000 messages

Even with no visible error, there's a structural bottleneck.

The assessment should catch this gap before the queue grows enough to threaten memory, disk or SLA.

Observability

A technical assessment can't just look at infrastructure.

It should also check whether the team can actually see problems.

A modern architecture usually has:

Metrics → Prometheus
Logs → Loki
Traces → OpenTelemetry
Dashboards → Grafana
Alerts → Alertmanager

The four fundamental pillars are:

Metrics

Examples:

CPU
Memory
Latency
Throughput
Error rate
Queue depth
Database connections

Logs

Should let you answer:

What happened?
When did it happen?
Which service caused it?
Which user/request was affected?

Traces

Distributed tracing lets you follow:

API Gateway
   ↓
Service A
   ↓
Service B
   ↓
Database

Without tracing, investigating latency between microservices can be extremely hard.

Alerts

Alerts should represent impact.

Avoid:

CPU > 70%

Prefer service-related signals:

p95 latency > 800ms
error rate > 5%
queue backlog > threshold
pods unavailable > threshold

Golden Signals

An efficient strategy during assessments is checking the Golden Signals used in SRE:

Latency
Traffic
Errors
Saturation

For example:

Latency:
p95 HTTP

Traffic:
requests/s

Errors:
5xx rate

Saturation:
CPU, memory, connections, queues

These signals help identify whether the system is close to its limits.

Security

An infrastructure assessment also needs to evaluate security.

RBAC

Check for excessive permissions.

kubectl get clusterrolebindings
kubectl get rolebindings -A

Look for permissions like:

cluster-admin

granted unnecessarily.

ServiceAccounts

Every workload should use its own identity.

Avoid:

default ServiceAccount

for production applications.

On AWS EKS, prefer workload identity mechanisms like EKS Pod Identity or IRSA, depending on the adopted architecture.

Secrets

Never store credentials directly in versioned manifests.

Avoid:

env:
  - name: DB_PASSWORD
    value: super-secret-password

Prefer integrating with services like:

AWS Secrets Manager
HashiCorp Vault
External Secrets Operator

NetworkPolicy

Without a NetworkPolicy, generally any pod can talk to any other pod inside the cluster, depending on the CNI in use.

A basic policy could be:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database

  policyTypes:
    - Ingress

  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: payment-api

      ports:
        - protocol: TCP
          port: 5432

This rule lets payment-api pods reach pods selected as database on port 5432.

In production, teams usually take a:

default deny
+
explicit allow

approach. This model reduces lateral movement in case of a compromise.

CI/CD during an assessment

Pipelines can also introduce risk.

Some points to check:

Automatic deploy to production?
Is there an approval step?
Is there a rollback path?
Is there image scanning?
Is there dependency scanning?
Do secrets show up in the pipeline?
Is there image versioning?

Avoid:

image: company/api:latest

Prefer:

image: company/api:2.4.1

or even better, images referenced by digest when immutability is a requirement.

Troubleshooting during the assessment

Some commands help identify problems quickly.

Pods

kubectl get pods -A -o wide

Look for:

CrashLoopBackOff
Pending
OOMKilled
Evicted

Pod description

kubectl describe pod <pod> -n <namespace>

Look for:

FailedScheduling
FailedMount
OOMKilled
Unhealthy
BackOff

Events

kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

This helps identify:

scheduling failures
volume issues
image issues
probe failures

Resources

kubectl top nodes
kubectl top pods -A

This data should be compared against:

kubectl describe node <node>

to analyze aggregated requests and limits.

Common risk signals

During an assessment, some patterns show up repeatedly.

Critical pods with a single replica

replicas: 1

May be acceptable in development.

In production it can be a Single Point of Failure.

Missing readinessProbe

Without readiness, Kubernetes can send traffic to applications that aren't ready yet.

Missing limits

An application with a memory leak can consume the entire node's memory.

Limits set too low

Can cause:

OOMKilled
CPU throttling

Requests set too high

Waste capacity.

HPA without node capacity

Scaling pods without infrastructure capacity solves little.

That's why:

HPA
+
Node Autoscaling

need to be evaluated together.

Development vs. production

Configurations acceptable in development can be dangerous in production.

Development

Might be acceptable:

1 replica
no PDB
autoscaling disabled
small cluster
low log retention

The goal might be to cut costs.

Production

There should usually be:

multiple replicas
distribution across nodes/AZs
PDB
autoscaling
monitoring
backup
security policies
resource requests
capacity planning

Production needs to treat failure as an expected event.

Costs and FinOps

A modern assessment should also evaluate financial efficiency.

There are three common categories of waste:

idle resources
oversized resources
forgotten resources

In Kubernetes:

Total CPU Request:
100 vCPU

Average CPU used:
32 vCPU

This could indicate a rightsizing opportunity.

But reducing requests without analysis can cause problems.

Ideally you should analyze:

p50
p95
p99
historical peaks

before changing configuration.

Spot versus On-Demand

Interruption-tolerant workloads can use Spot Instances.

For example:

CI/CD runners
asynchronous workers
jobs
batch processing

More sensitive workloads may require a combination of:

On-Demand
+
Spot

An assessment should analyze interruption tolerance before recommending Spot.

What to avoid

An assessment loses value when it becomes just a generic list.

Avoid recommendations like:

"improve observability"

Prefer:

Problem:
no backlog alert on RabbitMQ

Risk:
the queue can grow silently

Recommendation:
alert when messages_ready exceeds a defined threshold for 10 minutes

Another mistake is applying a checklist without understanding context.

For example:

"every service must have 3 replicas"

isn't necessarily true.

A batch service that runs once a day may not need multiple replicas.

The assessment needs to account for criticality and architecture.

How to prioritize risks

A practical way is to use:

Severity = Probability × Impact

Example:

It's also useful to consider:

effort to fix

Some high-impact changes may require little effort.

Example assessment matrix

Category: Kubernetes

Item:
Topology Spread

State:
Not configured

Risk:
Critical replicas can run on the same node

Impact:
Outage during a node failure

Priority:
High

Recommendation:
Add topologySpreadConstraints

Another example:

Category: Database

Item:
Storage

State:
85% used

Growth:
4% per month

Risk:
Storage exhaustion

Estimate:
about 4 months

Priority:
Critical

Recommendation:
expand storage and review retention

This format turns technical observations into concrete actions.

When to run an assessment

Assessments are especially useful in situations like:

• fast platform growth;
• a recurring increase in incidents;
• migrating to Kubernetes;
• a Cloud migration;
• an architecture change;
• preparing for Black Friday or major events;
• cost reduction;
• audit preparation;
• company acquisition or integration;
• undocumented legacy environments;
• a new Platform Engineering team coming in;
• preparing for SRE.

It's also worth running assessments periodically.

For example:

quarterly
twice a year
before major events
after relevant architectural changes

When not to use one

Not every problem needs a full assessment.

If the architecture is small:

1 application
1 database
1 server

a simple technical review may be enough.

It also doesn't make sense to start a dozen-dimension assessment when there's a known, urgent problem.

If the database is at 99% disk:

stabilize first
then run the assessment

An assessment doesn't replace incident response.

An efficient assessment should produce actions

The final document should answer:

What's wrong?
Why does it matter?
What's the risk?
What evidence proves it?
How do we fix it?
What's the priority?

An example:

Risk:
critical pods can run on the same node

Evidence:
3 pods found on worker ip-10-0-4-21

Impact:
node failure causes a full outage

Fix:
topologySpreadConstraints

Priority:
high

Effort:
low

This lets you turn a technical evaluation into an engineering backlog.

Practical summary

An efficient technical assessment follows a flow similar to:

Inventory
   ↓
Architecture
   ↓
Infrastructure
   ↓
Capacity
   ↓
Applications
   ↓
Database
   ↓
Messaging
   ↓
Observability
   ↓
Security
   ↓
FinOps
   ↓
Risk matrix
   ↓
Action plan

The main points to evaluate are:

• Single Points of Failure;
• workload distribution;
• CPU and memory capacity;
• requests and limits;
• scalability;
• queues and asynchronous processing;
• database;
• backups;
• observability;
• security;
• identity;
• network;
• CI/CD;
• disaster recovery;
• cost efficiency.

The goal isn't to find as many problems as possible.

It's to identify the problems that could have the biggest impact on the business.

Conclusion

A technical assessment shouldn't be treated as a static snapshot of the infrastructure.

It's a reliability engineering tool.

An architecture can be running normally and still hold dozens of hidden risks.

The assessment's job is to turn those risks into technical evidence before they're discovered during an incident.

In modern environments with Kubernetes, Cloud, microservices and dynamic infrastructure, complexity grows fast. The more components there are, the harder it becomes to understand every possible interaction between applications, infrastructure and dependencies.

That's why mature SRE, DevOps and Platform Engineering teams don't just react to incidents.

They constantly look for signs of fragility.

The natural evolution is moving from:

incident
   ↓
investigation
   ↓
fix

to:

assessment
   ↓
risk identification
   ↓
prioritization
   ↓
preventive fix
   ↓
fewer incidents

This shift is one of the key differences between a purely reactive operation and a reliability-driven platform engineering practice.

The best incident is still the one that never happened.