Getting started with OpenTelemetry eBPF Instrumentation on Linux

An API is slow, but the available dashboards only show CPU, memory, and process availability. The team knows something is wrong, yet cannot separate HTTP handling time from database calls and downstream delays. Adding instrumentation libraries to every application may require code changes and coordination across several teams.

OpenTelemetry eBPF Instrumentation, or OBI, provides an entry point for collecting traces and metrics from supported Linux applications without modifying their source code. It does not remove the need for business instrumentation; it establishes an initial view of requests and dependencies. (opentelemetry.io)

This guide moves from a Linux lab to a Kubernetes canary. Its focus is application instrumentation, not eBPF-based threat detection. For that separate use case, see runtime security with eBPF on GKE.

Reference version and stability boundaries

The project released OBI v1.0.0 on October 30, 2026. That is the version pinned in this guide. Its stability contract covers explicitly documented surfaces, including Config v2, deployment interfaces, and telemetry classified as stable. It does not extend uniformly to every feature or internal API; the Collector receiver follows a separate release line. (github.com)

The examples use documented environment variables. Avoid combining older Config v1 snippets with Config v2: the latter places OBI-specific settings under extensions.obi, while resources and exporters follow the OpenTelemetry declarative configuration structure. (opentelemetry.io)

What OBI observes

eBPF, originally extended Berkeley Packet Filter, enables verified programs to run at Linux instrumentation points. OBI uses this capability to observe executables and network activity and generate telemetry for supported protocols and runtimes. It is not simply a host utilization collector. (opentelemetry.io)

The main concepts are:

• Process selection: determines which applications are instrumented, for example through a listening port or executable path.
• Span: describes an observed operation, including its duration and attributes.
• Trace: connects spans through identifiers and distributed context.
• RED metrics: request rate, errors, and duration, used to assess service behavior.
• OTLP: the OpenTelemetry Protocol for transporting telemetry.
• Collector: receives, processes, and forwards signals to configured destinations. (opentelemetry.io)

The proposed architecture is:

Linux application → OBI instrumentation → OTLP → OpenTelemetry Collector → observability backend

OBI captures data; the Collector centralizes processing and delivery. Attribute transformation is a complementary topic covered in OTTL for traces, metrics, and logs.

Set realistic expectations

Capturing spans does not guarantee complete distributed traces. Context propagation depends on protocols, runtimes, and topology; Layer 7 proxies can interrupt transport-level propagation mechanisms. Observing TLS traffic also does not imply that context can be injected into every encrypted connection. (opentelemetry.io)

Do not expect OBI to infer business steps such as credit approval or discount calculation. Keep manual instrumentation or language agents for domain-specific semantics. Some features also integrate with application runtimes, so zero-code should not be interpreted as universally passive observation. (opentelemetry.io)

1. Check the host before installing

The reference release requires Linux on amd64 or arm64, BTF information—BPF Type Format—and sufficient permissions. The general kernel baseline is 5.8 or later. A documented exception covers RHEL-family 4.18 kernels with the required backports. A kernel version string alone does not establish compatibility. (github.com)

Run these checks on the Linux host:

uname -r
uname -m
test -r /sys/kernel/btf/vmlinux && echo 'BTF available'
findmnt -t tracefs
sudo ss -lntp

Treat the output as an initial inventory, not a certification. Review host security policies, tracefs availability, and the capabilities required by the enabled features. OBI may need to read /proc, load BPF programs, and attach probes; a Kubernetes ServiceAccount does not authorize those operations by itself. (opentelemetry.io)

For the first experiment, I recommend a disposable virtual machine. Avoid starting with every process on a shared production node.

2. Install a pinned OBI release

The project publishes architecture-specific Linux archives and a SHA256SUMS file. The following procedure pins the version and verifies archive integrity before installation. (opentelemetry.io)

set -euo pipefail
VERSION=1.0.0
case "$(uname -m)" in
  x86_64) ARCH=amd64 ;;
  aarch64|arm64) ARCH=arm64 ;;
  *) echo 'Unsupported architecture' >&2; exit 1 ;;
esac

WORKDIR=$(mktemp -d)
cd "$WORKDIR"
BASE="https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v${VERSION}"
curl -fLO "${BASE}/obi-v${VERSION}-linux-${ARCH}.tar.gz"
curl -fLO "${BASE}/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
tar -xzf "obi-v${VERSION}-linux-${ARCH}.tar.gz"
sudo install -m 0755 obi /usr/local/bin/obi
obi --version

For production, add source approval, signature or attestation verification following the project procedure, and review of the software bill of materials, or SBOM. A checksum is an integrity check, not a replacement for the entire software supply-chain trust policy. (github.com)

3. Prepare a local Collector

Prerequisite: an approved, pinned otelcol-contrib binary is already installed. Save the following as collector-lab.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  batch:
    timeout: 2s

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

This lab pipeline accepts OTLP/HTTP on loopback, processes traces and metrics, and prints the results. memory_limiter controls memory pressure, while batch groups telemetry for export. The values are illustrative rather than production sizing guidance. The debug exporter does not provide durable storage, and detailed output can expose sensitive attributes. (opentelemetry.io)

Start it in a dedicated terminal:

otelcol-contrib --config=collector-lab.yaml

Do not reuse this configuration unchanged for a Kubernetes gateway: a receiver listening on 127.0.0.1 cannot accept connections from other Pods.

4. Instrument a simple HTTP service

In another terminal, create a temporary directory containing no real application data and start a test server:

LAB_DIR=$(mktemp -d)
printf 'obi lab\n' > "${LAB_DIR}/index.html"
python3 -m http.server 8080 --bind 127.0.0.1 --directory "$LAB_DIR"

Run OBI in a third terminal:

sudo env \
  OTEL_EBPF_OPEN_PORT=8080 \
  OTEL_SERVICE_NAME=orders-lab \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
  /usr/local/bin/obi

The port selects the target process, and the endpoint identifies the Collector. The transport is explicit to avoid confusing OTLP/HTTP with OTLP/gRPC. When using the shared endpoint, the exporter appends signal-specific paths such as /v1/traces and /v1/metrics. (opentelemetry.io)

After checking the startup logs, generate traffic from another terminal:

for i in $(seq 1 20); do
  curl -fsS http://127.0.0.1:8080/ >/dev/null
  sleep 0.2
done
curl -sS -o /dev/null -w '%{http_code}\n' \
  http://127.0.0.1:8080/not-found

Lab acceptance criteria

Find orders-lab in the Collector output and inspect spans, durations, and HTTP attributes. Allow time for metric export as well: the default interval is 60 seconds. The 404 response helps validate captured status information; it does not imply that every 4xx response should be counted as a server error. (opentelemetry.io)

My validation sequence is: known traffic → selected process → received data → correct identity → reviewed attributes. A running OBI process alone is not proof of successful instrumentation.

5. Move to a Kubernetes canary

The following is a lab deployment, not a production-ready configuration. It assumes:

• An existing service whose executable is /usr/local/bin/orders-api.
• That workload is running on the selected canary node.
• An otel-collector Service in the observability namespace, with an accessible OTLP/HTTP receiver on port 4318 and enabled trace and metric pipelines.
• Approval to run a privileged agent in the dedicated namespace.

A DaemonSet maintains an agent on each eligible node. In this design, hostPID provides visibility into node processes, while the executable path narrows operational selection. That selector is not a security boundary between tenants. (kubernetes.io)

Save this as obi-canary.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: obi-canary
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: obi
  namespace: obi-canary
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: obi
  namespace: obi-canary
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: obi
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: obi
    spec:
      serviceAccountName: obi
      automountServiceAccountToken: false
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
        observability-canary: "true"
      containers:
        - name: obi
          image: otel/ebpf-instrument:v1.0.0
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 0
            privileged: true
          env:
            - name: OTEL_EBPF_AUTO_TARGET_EXE
              value: /usr/local/bin/orders-api
            - name: OTEL_SERVICE_NAME
              value: orders-api
            - name: OTEL_EBPF_KUBE_METADATA_ENABLE
              value: "false"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector.observability.svc.cluster.local:4318
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: http/protobuf
            - name: OTEL_EBPF_INTERNAL_METRICS_EXPORTER
              value: prometheus
            - name: OTEL_EBPF_INTERNAL_METRICS_PROMETHEUS_PORT
              value: "6060"
          ports:
            - name: internal
              containerPort: 6060
          startupProbe:
            httpGet:
              path: /internal/metrics
              port: internal
            periodSeconds: 5
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /internal/metrics
              port: internal
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          volumeMounts:
            - name: tracefs
              mountPath: /sys/kernel/tracing
      volumes:
        - name: tracefs
          hostPath:
            path: /sys/kernel/tracing
            type: Directory

The manifest adapts the documented DaemonSet pattern. The tracefs mount supports attachment and propagation requirements documented for particular kernel and OBI combinations. /internal/metrics is the actual endpoint exposed by the internal reporter. (opentelemetry.io)

These probes only check endpoint availability. They do not establish that capture or backend delivery is working. Omitting a livenessProbe is intentional for this canary: define and test a local failure condition before automating restarts. A downstream Collector outage should not cause continuous agent restarts.

Resource values are hypothetical starting points. Adjust the executable, destination, and capacity. Kubernetes metadata enrichment is disabled, so no API token or ClusterRole is included. If enrichment is required, use the documented configuration and list/watch permissions rather than granting cluster-admin. (opentelemetry.io)

The namespace permits privileged workloads and should have tightly controlled administrative access. Additional cluster admission policies may still reject the Pod; do not disable cluster-wide protections to bypass that decision. (kubernetes.io)

Apply the configuration after replacing the node name:

NODE=worker-canary-01
kubectl label node "$NODE" observability-canary=true --overwrite
kubectl apply -f obi-canary.yaml
kubectl -n obi-canary rollout status daemonset/obi
kubectl -n obi-canary get pods -o wide
kubectl -n obi-canary logs daemonset/obi --tail=100

Operational example: an orders API

Consider this hypothetical scenario:

Problem → an orders API uses PostgreSQL and Redis, but the team lacks sufficient traces to guide an investigation.

Solution → evaluate OBI on one node, targeting only that API, while retaining a comparable control group without the agent.

Implementation → compare equivalent traffic, inspect available spans, and verify coverage for the actual clients and protocols. PostgreSQL and Redis appear in the support matrix, but library details, existing connections, and request volume still matter. (github.com)

Expected outcome → identify dependencies worth investigating further. This is a hypothesis about operational usefulness, not a benchmark or a promised latency reduction. The next step might involve queries, connection pools, or concurrency; installing observability does not fix those issues automatically.

From demonstration to production

I recommend approving rollout only after answering the following questions.

Security and data handling

• What is the minimum capability set for the selected features? Replace privileged: true once the kernel, runtime, and feature combination has been validated.
• Who can change the DaemonSet, image, and process selection?
• Which attributes could contain personal information, sensitive URLs, or queries?
• Does remote transport use Transport Layer Security, or TLS, with certificate verification?
• Are credentials stored in Secrets or an organizational secrets system, with rotation?

Linux capabilities and Kubernetes access controls address different boundaries. A minimally privileged ServiceAccount does not neutralize a privileged container. I also recommend limiting egress to the Collector and required dependencies, then testing enforcement with the actual networking plugin. (opentelemetry.io)

Availability and recovery

For node agents, coverage means a healthy agent on every selected node—not two arbitrary replicas competing to instrument the same workload. For the Collector gateway, I propose multiple replicas spread across zones, probes, reserved capacity, and a PodDisruptionBudget aligned with the availability target. (kubernetes.io)

Configure exporter queues and retries in production. If losing buffered telemetry during restarts is unacceptable, evaluate persistent queues using file_storage and suitable storage. Queues remain bounded, so prolonged outages can still cause drops. Collector persistence cannot recover events that never reached it. (opentelemetry.io)

Version configurations, dashboards, and policies. Define backend storage backup and recovery around retention and business requirements rather than treating the agent as a database.

Performance, capacity, and cost

Measure comparable traffic before and after rollout, including application latency, CPU, memory, restarts, export failures, and ingested volume. Avoid assuming a universal overhead percentage. Capture settings and buffers are part of capacity planning. (opentelemetry.io)

For FinOps, account for agents per node, Collector processing, cross-zone traffic, ingestion, and retention. Estimate daily volume from observed spans per second and average payload size. Remove unnecessary dimensions before simply adding resources: cardinality multiplies time series and consumption. (opentelemetry.io)

My recommendation is to keep user and order identifiers out of metric labels. For Spot nodes, include churn and temporary coverage gaps in testing. Choose interruptible or non-interruptible gateway capacity according to loss tolerance, rather than assuming that the lowest compute price delivers the lowest total cost.

If tail sampling is enabled, scaling requires preserving the grouping of spans belonging to the same trace at the component making the sampling decision. Do not treat that processing as entirely stateless. (opentelemetry.io)

Observe the instrumentation itself

Separate three assessments:

1. Application: throughput, duration, and service-relevant failures.
2. Agent: node coverage, resource usage, restarts, and internal metrics.
3. Pipeline: received data, refusals, queues, retries, and export failures.

OBI provides optional internal metrics, and the Collector exposes signals for capacity planning and pressure detection. I additionally recommend a periodic synthetic check: issue a known request and verify its arrival at the destination. An accessible metrics endpoint is not a substitute for that end-to-end test. (opentelemetry.io)

Keep log ingestion an explicit architectural decision. Trace-log correlation should not be mistaken for a guarantee that every application log is being collected. (opentelemetry.io)

Troubleshooting

Start with inspection commands:

kubectl get nodes
kubectl -n obi-canary get daemonsets,pods -o wide
kubectl -n obi-canary describe daemonset obi
kubectl -n obi-canary describe pod <pod>
kubectl -n obi-canary logs <pod> --tail=200
kubectl -n obi-canary logs <pod> --previous
kubectl get events -A --sort-by=.metadata.creationTimestamp

Replace <pod> with the actual Pod name. Use the following diagnostic sequence:

Check probe and propagation limitations before attributing every missing signal to the backend. (opentelemetry.io)

Upgrades, rollback, and environment differences

A development environment may only need one instance, local unencrypted export, and detailed console output. For production, I recommend protected transport, measured capacity, alerts, gateway redundancy, and a rehearsed recovery procedure.

Promote versions through canaries and record the tested image, configuration, and kernel. Review telemetry changes before updating dashboards. The v1 compatibility guarantee applies to surfaces classified as stable, not every experimental metric. (github.com)

To undo a DaemonSet update or stop the canary:

kubectl -n obi-canary rollout undo daemonset/obi
kubectl -n obi-canary rollout status daemonset/obi

# Stop instrumentation on the selected node
kubectl label node "$NODE" observability-canary-

Undo requires an earlier revision. With GitOps, also revert the source commit or reconciliation may reapply the change. After stopping instrumentation, confirm application recovery and check for errors. Unchanged source code is not proof of a risk-free rollback.

When to use OBI—and when to choose something else

OBI is worth evaluating when Linux services lack initial application visibility and the platform team controls the host. Language agents and SDKs remain relevant for framework detail and business semantics. Environments missing kernel or privilege requirements do not become compatible merely because they run containers. (opentelemetry.io)

My practical selection framework is:

Avoid adding a second automatic instrumentation layer unless it closes a demonstrated gap. Establish ownership for each signal and check for duplication before expanding coverage.

Practical summary

1. Inventory the kernel, BTF, architecture, and permissions.
2. Pin the release and target one service.
3. Verify capture and delivery using known traffic.
4. Run a canary with explicit stop conditions.
5. Review sensitive attributes, capacity, and ingestion cost.
6. Keep SDK instrumentation where business semantics require it.
7. Expand only after testing recovery and rollback.

Conclusion

OBI's platform value is not automatic collection of everything. It is the opportunity to reduce the initial effort required to obtain useful operational evidence. Adoption should therefore be managed as an infrastructure change with potential application impact: bounded scope, validation, security controls, and reversibility.

For DevOps, SRE, and Platform Engineering teams, the most valuable deliverable is a repeatable path: a selected service, understandable telemetry, known cost, and clear operational ownership. That is a more sustainable outcome than simply increasing the number of stored spans.