🌱 Cultivating Unique service.instance.id on NGINX Ingress with OpenTelemetry

A task to set a unique service.instance.id on NGINX Ingress Controller using the OpenTelemetry should be simple, right? But as it turned out, the ingress controller doesn't expose all of NGINX’s OTel knobs out of the box, so I had to roll my own tweak garden.

Why should I care?

Without unique instance IDs, your tracing data looks like a tangled tangleweed—hard to trace, difficult to debug, and completely undermines what we’re trying to do with observability.

The solution

Part 1 - Nginx Ingress Controller - Helm Chart

To get POD_UID injected into each trace, here’s the minimal yet powerful config snippet I landed on:

Set the POD_UID environment variable, once NGINX Ingress Controller set by default only the POD_NAME and POD_NAMESPACE.

extraEnvs:
  - name: POD_UID
    valueFrom:
      fieldRef:
         fieldPath: metadata.uid

Set the POD_UID as span-attribute for instance.

controller:
  config:
    main-snippet: |
      env POD_UID;
    server-snippet: |
      set $pod_uid "unknown";
      access_by_lua_block {
        ngx.var.pod_uid = os.getenv("POD_UID") or "unknown"
      }
      opentelemetry_attribute service.instance.id $pod_uid;

Part 2 - OpenTelemetry Collector Config

Configure the transform processor to set the resource-attribute.

    processors:
      transform:
        error_mode: ignore
        trace_statements:
          - set(resource.attributes["service.instance.id"], span.attributes["service.instance.id"])
          - delete_key(span.attributes, "service.instance.id")

How it works:

Explicitly exposes the POD_UID environment variables so NGINX can see them.
Initializes a default $pod_uid—my fail-safe in case the env var goes missing.
Uses Lua to pull in the real POD_UID.
Finally, sets the OpenTelemetry attribute service.instance.id to match the actual UID.
Through OpenTelemetry Collector capture the service.instance.id from span-attributes and set it on resource-attributes.

Benefits to your telemetry:

Without unique `service.instance.id`	With unique `service.instance.id`
Pods share the same instance ID, causing confusion in telemetry.	Each pod is individually identifiable.
Difficult to isolate errors and debug effectively.	Allows precise tracing to individual pods.
Limited visibility into pod-specific issues.	Enables accurate root-cause analysis.

Final result

By implementing this solution, each NGINX ingress pod is clearly distinguishable in tracing data. This improves observability significantly by providing accurate, pod-specific telemetry, facilitating precise troubleshooting and diagnostics.