OTTL v1 in practice: transforming traces, metrics, and logs in the Collector

A platform receives spans with inconsistent operation names, metrics with incompatible dimensions, and logs containing information that should never reach the backend. Fixing instrumentation is the preferred long-term solution, but application release cycles may not match an urgent migration or operational requirement. The Collector provides a transformation point before export, supporting data quality, governance, and data control. (opentelemetry.io)

This guide develops a practical design for a checkout API: normalize span names, enrich a custom metric, and reshape structured logs. The scenario is illustrative, not a report of measured business results.

What OTTL v1 actually guarantees

The October 27, 2026 announcement establishes OTTL as stable for traces, metrics, and logs starting with Collector Contrib v0.164.0. Its 1.x compatibility commitment covers the language, standard functions, and stable Go API. Experimental xottl features, lambda functions, and profiles are outside that commitment. (opentelemetry.io)

Language stability does not make every Collector component stable: the transform processor remains listed as beta for these signals in the documentation for this release. This guide pins image 0.164.0 as a reproducible baseline rather than using a floating tag. (github.com)

OTTL stands for OpenTelemetry Transformation Language, a domain-specific language for processing telemetry structures. It is not a backend query language or a historical analytics engine; components use it while handling incoming data. (github.com)

The concepts behind a reliable rule

A statement combines a function, arguments, and an optional where condition:

set(span.attributes["platform.team"], "payments") where resource.attributes["service.name"] == "checkout-api"

Paths identify fields, and brackets access map entries. A missing map key returns nil. Context determines which paths are available. Before writing a rule, establish where the value lives and which type it has instead of assuming that everything belongs under attributes. (github.com)

The example uses these concepts:

• `resource`: source identity, accessible from signal contexts.
• `scope`: instrumentation scope information.
• `span`: a span's name, attributes, identifiers, and related fields. (github.com)
• `metric`: metadata such as name, description, and unit. (github.com)
• `datapoint`: attributes and values associated with an individual metric point. (github.com)
• `log`: the body, attributes, severity, and correlation fields of a log record. log.cache holds intermediate transformation results. (github.com)

Functions such as set, delete_key, and keep_keys edit data. ParseJSON converts a JSON string, while IsString provides a type check. A string is not necessarily valid JSON, so parsing can still fail after that check. (github.com)

Architecture: place transformations deliberately

The proposed flow is:

API and workers
  → OTLP/HTTP
  → Collector gateway
  → memory_limiter
  → transform/platform
  → batch
  → exporter
  → observability backend

OTLP is the OpenTelemetry Protocol. Receivers ingest data, processors apply intermediate operations, and exporters deliver the result. Defining a component does not activate it: it must be referenced in service.pipelines, where processor ordering controls the flow. (opentelemetry.io)

This design places memory limiting before transformation. The limiter can refuse data under pressure; it does not replace the container memory limit or guarantee that the process cannot be killed for excessive memory use. Sender retry behavior must also be considered. (github.com)

Complete Collector configuration

Save the following as collector.yaml.

Scope: local testing and staging with synthetic data. The receiver does not use TLS, and the debug exporter prints telemetry. Do not expose this endpoint publicly or send real sensitive data into this lab.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 384
    spike_limit_mib: 96

  transform/platform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - 'set(span.name, span.attributes["http.route"]) where span.kind == SPAN_KIND_SERVER and IsString(span.attributes["http.route"])'
          - 'delete_key(span.attributes, "user.email")'
          - 'delete_key(span.attributes, "http.request.header.authorization")'

    metric_statements:
      - context: metric
        statements:
          - 'set(metric.description, "Number of completed checkout orders") where metric.name == "checkout.orders.count"'
      - context: datapoint
        statements:
          - 'set(datapoint.attributes["platform.pipeline"], "normalized") where metric.name == "checkout.orders.count"'

    log_statements:
      - context: log
        statements:
          - 'merge_maps(log.cache, ParseJSON(log.body), "upsert") where IsString(log.body)'
          - 'set(log.attributes["event.name"], "checkout.failed") where log.cache["event"] == "checkout.failed"'
          - 'set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.cache["level"] == "ERROR"'
          - 'set(log.severity_text, "ERROR") where log.cache["level"] == "ERROR"'
          - 'keep_keys(log.attributes, ["event.name"])'
          - 'set(log.body, "[body omitted by policy]")'

  batch:
    timeout: 1s
    send_batch_size: 512

exporters:
  debug:
    verbosity: detailed

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/platform, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, transform/platform, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/platform, batch]
      exporters: [debug]

Traces: normalize names without changing correlation

Under this example's contract, a server span receives the parameterized route supplied by instrumentation, such as /orders/{id}, as its name. A missing route does not trigger a rename. The rule leaves trace_id, span_id, and parent relationships untouched; these fields are accessible, but name normalization does not require rewriting them. (github.com)

Deleting two attribute keys is intentionally narrow. It is not a complete protection policy: span names, events, resource attributes, and other fields also require a sensitive-data review. I recommend preventing inappropriate collection at the source and treating the gateway as an additional safeguard.

Metrics: metadata is not a data point dimension

The metric block edits the description. The datapoint block adds a controlled dimension. checkout.orders.count and platform.pipeline are custom names for this example, not official semantic conventions. The numeric measurement remains unchanged. (github.com)

Do not indiscriminately delete dimensions such as user.id or pod to reduce cardinality. Previously distinct points can acquire the same series identity. Changing unit also does not convert values, histogram buckets, or temporality. Identity and aggregation semantics must remain consistent. (opentelemetry.io)

My preferred approach is to remove inappropriate dimensions in instrumentation. When that is not immediately possible, handle the transformation as a migration: specify aggregation behavior, historical continuity, affected queries, and acceptance tests.

Logs: extract selectively and address the original body

Parsing happens in the cache rather than directly in exported attributes. The example promotes one recognized event and the ERROR level, restricts attributes, and replaces the entire body. Avoid copying every key from an uncontrolled JSON document. The selected conversion and editing functions support this staged approach. (github.com)

This policy is intentionally destructive and applies to every log in this pipeline. Even if parsing fails, subsequent statements should replace the body. That trades away the original message and stack trace. Establish whether the application's logging contract can tolerate that loss before adopting it.

The example also leaves resource attributes unchanged and does not validate an existing event.name value. For production, I recommend an explicit allowed-value policy and tests covering every exported field, not just the body.

Error handling: availability is not a security guarantee

ignore logs an error and continues; silent continues without logging it; propagate returns the error to the pipeline and can cause the payload to be dropped. Statements run in configured order. Syntax errors still require configuration fixes. (github.com)

Choose the policy explicitly. For sensitive data, do not treat ignore as a blocking guarantee. If a required transformation fails, a tested safe outcome must prevent the prohibited content from being exported.

Validation and integration testing

Validate with the exact image used for deployment. The Collector provides a validate subcommand, but successful configuration validation does not prove correct behavior for every payload shape. (opentelemetry.io)

docker run --rm \
  -v "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.164.0 \
  validate --config=/etc/otelcol-contrib/config.yaml

docker run --rm --name ottl-lab \
  -p 127.0.0.1:4318:4318 \
  -v "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.164.0 \
  --config=/etc/otelcol-contrib/config.yaml

From another terminal, submit a synthetic record using the OTLP/HTTP JSON envelope and /v1/logs endpoint. The command generates the timestamp when it runs. (opentelemetry.io)

NOW_NS="$(date +%s)000000000"
curl --fail-with-body \
  -H 'Content-Type: application/json' \
  http://127.0.0.1:4318/v1/logs \
  --data-binary @- <<EOF
{
  "resourceLogs": [{
    "resource": {
      "attributes": [{
        "key": "service.name",
        "value": {"stringValue": "checkout-api"}
      }]
    },
    "scopeLogs": [{
      "scope": {"name": "ottl-lab"},
      "logRecords": [{
        "timeUnixNano": "$NOW_NS",
        "body": {
          "stringValue": "{\"event\":\"checkout.failed\",\"level\":\"ERROR\",\"email\":\"synthetic@example.invalid\"}"
        }
      }]
    }]
  }]
}
EOF

Expected output contains the replacement body, event.name=checkout.failed, and ERROR severity. Inspect the processed record through the debug exporter. This is a reproducible procedure, not a claim that these commands have been executed in your environment. (opentelemetry.io)

I recommend including these fixtures in continuous integration:

Compare normalized OTLP structures while excluding variable fields such as timestamps. Include multiple records sharing a resource and explicit negative tests for sensitive-data leakage.

Kubernetes: a complete staging deployment

Create the namespace and ConfigMap from the configuration above:

kubectl create namespace observability
kubectl create configmap otel-config-v1 \
  --from-file=collector.yaml \
  -n observability

Save this manifest as collector-k8s.yaml. It references the ConfigMap created above and requires neither an Operator nor a Custom Resource Definition, or CRD.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-gateway
  namespace: observability
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-gateway
  namespace: observability
spec:
  replicas: 2
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: otel-gateway
  template:
    metadata:
      labels:
        app: otel-gateway
    spec:
      serviceAccountName: otel-gateway
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: otel-gateway
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.164.0
          args:
            - --config=/etc/otelcol-contrib/collector.yaml
          ports:
            - name: otlp-http
              containerPort: 4318
            - name: health
              containerPort: 13133
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          startupProbe:
            httpGet:
              path: /
              port: health
            periodSeconds: 5
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /
              port: health
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: health
            periodSeconds: 10
            failureThreshold: 3
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol-contrib
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: otel-config-v1
---
apiVersion: v1
kind: Service
metadata:
  name: otel-gateway
  namespace: observability
spec:
  type: ClusterIP
  selector:
    app: otel-gateway
  ports:
    - name: otlp-http
      port: 4318
      targetPort: otlp-http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: otel-gateway
  namespace: observability
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: otel-gateway

The Deployment defines a rolling update and initial test resource allocations, not universal capacity recommendations. The versioned ConfigMap is part of the deployment contract. (kubernetes.io)

Probes use the health_check extension. Startup handles initialization, readiness controls traffic eligibility, and liveness supports restart-based recovery. This endpoint does not prove successful backend delivery. (github.com)

Hostname spreading is a preference because the manifest uses ScheduleAnyway; it does not guarantee separation. A PodDisruptionBudget constrains voluntary disruption, not every possible node or zone failure. (kubernetes.io)

kubectl apply -f collector-k8s.yaml
kubectl rollout status deployment/otel-gateway -n observability
kubectl get pods,svc,pdb -n observability
kubectl logs -n observability -l app=otel-gateway \
  --all-containers=true --tail=100

This is still not production-ready: it has two replicas but no authentication, TLS, NetworkPolicy, durable storage, or backend exporter. An internal port is not a sufficient trust boundary.

Production design decisions and trade-offs

Security and isolation

I recommend enabling the pipeline only after implementing TLS, sender-to-gateway authentication, restricted egress, and NetworkPolicy supported by the cluster's Container Network Interface, or CNI. Credentials should come from Secrets or an external secrets manager, never the repository. Official guidance emphasizes endpoint protection and least privilege. (opentelemetry.io)

This design does not query the Kubernetes API, so it does not grant Role-Based Access Control, or RBAC, permissions to the ServiceAccount. If Kubernetes enrichment is added, review its permissions separately.

Remove debug and test the actual exporter with certificates, credential rotation, and authentication failures. Detailed OTTL logging can expose transformation context before the rules run, including the very information the rules are intended to remove. (github.com)

Availability and recovery

For a critical service, I propose three replicas spread across zones, sufficient capacity to lose one replica, and an explicit autoscaling policy. Inspect actual connection distribution: extra replicas do not automatically fix every imbalance. Tail sampling and other stateful processing also require routing decisions that account for state. (opentelemetry.io)

Configure exporter queues and retries. An in-memory queue does not survive process loss; persistent queues require storage and recovery planning. Queue exhaustion, retry expiration, and storage failures remain possible sources of data loss. (opentelemetry.io)

Specify gateway availability, maximum delivery delay, and acceptable loss independently. Backing up configuration cannot recover telemetry that has already been discarded.

Observe the Collector itself

Monitor accepted and exported throughput, failures, CPU, memory, restarts, queue occupancy, and delivery delay. Useful metrics include otelcol_exporter_queue_size and otelcol_exporter_queue_capacity. Check the metric names and suffixes actually exposed by your version and exporter. (opentelemetry.io)

I recommend per-signal dashboards and alerts for sustained queue growth, unexpected absence of exports, and transformation failures. Add an end-to-end synthetic check: a healthy process alone does not prove that intact data reached its destination.

Performance and FinOps

Benchmark representative payload sizes and shapes. Advanced transformations can increase processing cost, so records per second alone are not an adequate capacity estimate. (opentelemetry.io)

Include CPU, memory, standby capacity, cross-zone traffic, and backend ingestion in the cost model. Compare total cost before and after the rules rather than assuming that fewer exported bytes justify any amount of additional processing. For critical workloads, consider a non-interruptible capacity baseline before relying on Spot capacity.

Development versus production

Do not promote the lab configuration without reviewing these requirements. Missing requests, probes, monitoring, or rollback should not become permanent shortcuts.

Troubleshooting

Use symptom → hypothesis → investigation → correction. Official troubleshooting tools include internal logs, component inspection, and controlled local export. (opentelemetry.io)

kubectl get pods -n observability
kubectl describe deployment otel-gateway -n observability
kubectl get events -n observability \
  --sort-by=.metadata.creationTimestamp
kubectl logs -n observability <pod> --previous
kubectl describe pod -n observability <pod>

Replace <pod> with the actual name. When sensitive data is involved, reproduce the issue with a sanitized fixture before increasing log verbosity.

CI/CD, upgrades, and rollback

Version the image, configuration, fixtures, and expected outputs together. In continuous integration and delivery, validate configuration, exercise all three signals, and compare resource usage before promoting a canary.

For each change, create a new ConfigMap, such as otel-config-v2, and update the Pod template reference. Retain the previous object. rollout undo restores a Deployment revision, not the historical contents of a ConfigMap that was overwritten. (kubernetes.io)

kubectl rollout history deployment/otel-gateway -n observability
kubectl rollout undo deployment/otel-gateway -n observability
kubectl rollout status deployment/otel-gateway -n observability

With GitOps, also revert the desired-state commit so reconciliation does not restore the faulty configuration. Suggested rollback triggers include detected leakage, unexpected semantic changes, loss beyond the agreed budget, and sustained saturation.

OTTL v1 does not eliminate the need to review Collector and component changelogs. Release 0.164.0, for example, includes feature-gate removals and changes beyond language stabilization. (github.com)

When to use OTTL—and when to choose something simpler

I recommend OTTL when conditional policies must be shared across teams and the organization can maintain their contracts and tests. I would not use it to hide faulty instrumentation indefinitely or to implement business logic.

For straightforward attribute insertion or deletion, attributes or resource may be easier to maintain. For dropping telemetry, consider filter, whose purpose differs from transformation. metricstransform provides metric-specific operations. (opentelemetry.io)

My decision order is: fix the source where practical, choose the simplest suitable processor, and use OTTL for genuine conditional requirements. Centralization can make governance easier, but it also concentrates operational responsibility in the telemetry pipeline.

Practical takeaways

• Pin a known distribution and version.
• Establish context and types before writing conditions.
• Maintain separate contracts for traces, metrics, and logs.
• Do not confuse attribute deletion with correct aggregation.
• Test malformed inputs, missing fields, and data leakage.
• Design monitoring, capacity, security, and rollback alongside the rules.

Conclusion

A mature OTTL implementation is more than a statement that works. It needs an owner, an input contract, a verifiable output, and a recovery procedure.

For DevOps, that means versioned and automated transformation. For SRE, it means protecting pipeline reliability without corrupting signal meaning. For Platform Engineering, it means providing reusable standards while keeping instrumentation quality an application responsibility. The goal is a predictable observability platform—not simply a Collector with more rules.