Our Top 10 Observability Talks from KubeCon + CloudNativeCon North America 2026
- Events
- Community
- OpenTelemetry
We analyzed more than 60 recorded talks from Observability Day and KubeCon + CloudNativeCon North America 2026 in Salt Lake City and picked ten.

Salt Lake City gave the observability community two programs in one week: Observability Day, the CNCF co-located event on November 9, and the Observability track of KubeCon® + CloudNativeCon® North America 2026, which ran from November 9 to 12. Counting the OpenTelemetry® sessions from other KubeCon + CloudNativeCon tracks, that is more than 60 recorded talks. We analyzed all of them and picked ten.
1. OpenTelemetry Far and Wide
Tyler Yahn (Splunk) & Nikola Grcevski (Grafana Labs) | Observability Day | Watch on YouTube
Tyler and Nikola maintain the two projects this talk is about: OpenTelemetry eBPF Instrumentation (OBI) and the OpenTelemetry Injector. Their argument is that OpenTelemetry instrumentation is no longer only APIs and SDKs that developers code against. It is also something a platform team installs. They make the case with a live demo instead of slides: a checkout that succeeds, but makes the customer wait two seconds or more.
The demo application has 18 uninstrumented services plus PostgreSQL and Redis. One command installs OBI, which uses eBPF probes on the Linux nodes, and Grafana shows a connected trace across all 18 services. That trace locates the slow span in the checkout service, but OBI only sees the process from the network boundary. To go deeper, they add the OpenTelemetry Java agent to the checkout deployment through an init container and JAVA_TOOL_OPTIONS. The agent reports a java.net.SocketTimeoutException and shows that the service still calls a legacy pricing URL the operators had already retired. Removing that environment variable brings the request from about two seconds to 165 milliseconds in the metrics, without rebuilding the application.
The second half addresses the part that does not scale: repeating this for services that other teams own. The OpenTelemetry Operator can inject agents, but the platform team still has to know each image's details, such as whether a Python image uses musl, which Python version it runs, and which native packages match. The OpenTelemetry Injector moves that knowledge into a shared library that loads the matching language agent when the process starts. After one declarative change to the namespace, 14 services run language agents and the four Go services stay covered by OBI, all in the same trace. The speakers estimate 247 engineering hours and about $24,000 saved over two years for these services, and invite the audience to plug in their own numbers.
Why this talk stood out: Every claim is demonstrated live, the problem is one most platform teams have, and the people on stage maintain the tools they recommend.
2. Scaling OpenTelemetry in a Financial Enterprise: Lessons from CapitalOne's Journey
Sateesh Mamidala & Chandra Siva (Capital One) | Observability Day | Watch on YouTube
Sateesh and Chandra describe five years of taking their OpenTelemetry deployment from a proof of concept to an enterprise platform. Collectors now run on more than 85,000 compute instances, thousands of containers on Fargate and Kubernetes, and AWS Lambda functions. Their constraint is regulatory: the pipeline is evidence for federal examiners, an account number in a span attribute is a compliance incident, and a backend that silently drops data breaks an auditable record.
On Lambda, they made exports asynchronous and moved from gRPC to HTTP, which cut cold starts by 180 to 350 milliseconds depending on the runtime. For backend outages, they developed a failover connector that redirects telemetry to a secondary destination and contributed it upstream. To contain the blast radius of a bad release, a new Collector version reaches an EC2 instance only when the instance is rebuilt from a new image, releases move through rings from canary to production, and each agent has a centrally managed throughput cap.
The redaction section is the most practical part. Account and Social Security numbers must never leave the trusted environment, so redaction runs in three layers. Go's regular expression engine has no lookahead or lookbehind, which keeps execution linear and rules out regular expression denial of service, but a 12-digit AWS account number looks like the 12-digit driver's license numbers some US states issue. The team partitioned its patterns by attribute key so infrastructure IDs survive while personal data is scrubbed, and tuning the pattern sets cut redaction CPU by about 50%. They close with tail sampling that keeps every error trace, a load-balancing tier that routes each trace ID to one sampling node, and their own control plane built on the Open Agent Management Protocol (OpAMP). Their advice: build compliance in from the start, extend OpenTelemetry components instead of forking them, and run your own distribution.
Why this talk stood out: It shows an OpenTelemetry deployment running under bank regulation on 85,000 instances, and nearly every decision in it, from the HTTP export switch to the partitioned redaction patterns, transfers to other enterprises.
3. Two-Stage OTel Aggregation: Taming Istio Cardinality at 10k-Pod Scale
Evelyn (Chungyu) Chen (Yahoo) | KubeCon + CloudNativeCon, Observability track | Watch on YouTube
Evelyn's production cluster has more than 800 nodes, 24,000 pods, and 400,000 requests per second, and its Istio metrics had Prometheus one bad query away from an out-of-memory kill. Using only upstream Collector components, with the configuration kept in Git, her team cut active series from over 2 million to about 280,000, an 86% reduction worth about $140,000 a year on that cluster alone.
Yahoo runs agent Collectors as a DaemonSet in front of a gateway tier. Evelyn explains why a single PromQL sum by (app) turns into three processors whose order matters: the transform processor deletes pod name, pod ID, and container ID at the resource level, the groupbyattrs processor promotes app from a data point attribute to a resource attribute, with a batch processor before it so data from different pods meets in the same batch, and the metricstransform processor performs the sum. The gateway repeats the pattern to remove node identity.
She also shows what broke. With a valid configuration and a green pipeline, rate panels showed spikes up to 600 times the real traffic, because summing cumulative counters from pods with different baselines corrupts the result whenever a pod restarts. The fix was the cumulativetodelta processor at the start of the agent pipeline, followed by rewriting every query that used rate(), which returns wrong values on delta data without failing. The team validated the result by running raw and aggregated data side by side on the same dashboards, and that comparison exposed a remaining problem: when three gateway pods send the same series in the same second, a backend that stores one point per series per second keeps only the last one and drops the rest silently. She presented it as unsolved.
Why this talk stood out: It is a complete recipe built from upstream components, and it includes the failure that a green pipeline hid as well as the problem that is still open.
The rest of the top 10
Edging Closer to AI Assisted Troubleshooting - AI Assisted Dashboard Analysis, Vijay Samuel & Jing Hu (eBay), KubeCon + CloudNativeCon. eBay's dashboard analyzer is an agent built around one rule: the large language model (LLM) never computes anything. Code reduces the data with statistics, the LLM never authors trace IDs or service names, plans and loops are bounded, and every finding links to the exact panel and time range that supports it. By Vijay's rough math, sending the raw data behind a 40-panel dashboard to a model would take about 3.6 million tokens.
Not All Memory Is Equal: How To Properly Observe Memory Usage, Braydon Kains (Google) & Roger Coll (Grafana Labs), KubeCon + CloudNativeCon. Two maintainers of the system semantic conventions and the Collector's host metrics receiver explain why memory metrics disagree, down to the kernel files each one reads. A live memory-pressure lab shows why the contrib Collector, with 12,000 init functions, uses more memory than core, and soft reclaim brought a Collector's usage from 300 MB to 80 MB without hurting throughput.
Tracing at Scale: Running a 16 Billion Spans/Hour Pipeline at Nubank, Felipe Araujo & Kaynan Coelho (Nubank), KubeCon + CloudNativeCon. Five engineers run a tracing pipeline that now receives more than 23 billion spans per hour. The Collector writes micro-batches to Amazon S3 and a queue carries a reference to each batch, a claim check pattern that is cheaper than streaming, and a move from fixed-rate to dynamic sampling surfaced services and errors they had never seen in their traces.
Filter, Transform, Route: Mastering OTTL in the OpenTelemetry Collector, Evan Bradley (Dynatrace), Edmo Vamerlatti Costa (Elastic), Jamie Danielson (honeycomb.io) & Tyler Helmuth (Grafana Labs), KubeCon + CloudNativeCon tutorial. Collector maintainers announce that the OpenTelemetry Transformation Language (OTTL) reached 1.0, shipped in Collector v0.164.0. The 70-minute tutorial covers debugging statements, common redaction mistakes, custom functions in a distribution built with the OpenTelemetry Collector Builder (OCB), and exercises in the ottl.run playground.
The Race to Standardize AI Observability, Liudmila Molkova (Google) & Trask Stalnaker (Microsoft), KubeCon + CloudNativeCon. A small AI application instrumented by three libraries sends OpenTelemetry Protocol (OTLP) data in three different schemas, and its token dashboard disagrees with the provider bill. The maintainers walk through the generative AI (GenAI) semantic conventions, conformance checks built on OpenTelemetry Weaver, and the plan to stabilize the conventions.
The Promises We Keep at Scale: A Contract-Driven Guide to Platform SLOs, Vasudev Bongale & Swati Shridhar Nair (LinkedIn), KubeCon + CloudNativeCon. LinkedIn's Kubernetes platform runs more than 5 million pods, and its service level objectives (SLOs) start from promises to users instead of internal metrics. The speakers show how each failure is attributed to the platform or the application, and walk through real breaches, including a containerd bug that left pods stuck in terminating.
Designing an Internal SLO Platform, Julio Leal (Nubank), Observability Day. Nubank's metrics platform ingests 58 million samples per second, yet nobody could say whether the credit card product was healthy. Julio covers the SLO platform his team built on VictoriaMetrics and slo-generator, the burn-rate alert math, why multi-region SLOs must sum good requests instead of averaging percentages, and what he would do differently next time.
What the top three have in common
The top three talks describe the same shift from different angles. In each one, a platform team takes ownership of telemetry centrally instead of asking every application team to get it right: Splunk and Grafana Labs maintainers install instrumentation across a namespace, Capital One redacts, samples, and controls agents through its own distribution, and Yahoo reshapes metrics in Collector configuration that lives in Git. None of them relied on documentation or mandates to change what developers emit.
All ten talks are worth watching, and the recordings are on the CNCF YouTube channel. If you want to start with one, start with Far and Wide.
Our sessions in Salt Lake City
Our KubeCon + CloudNativeCon North America 2026 page has more on our week in Salt Lake City.
Observability Day welcome and opening remarks
Juraci Paixão Kröhling (OllyGarden), Austin Parker (honeycomb.io) & Eduardo Silva (Palo Alto Networks) | Observability Day | Watch on YouTube
Juraci framed the day around one shift: agents are becoming the largest consumers of telemetry, and they are also new producers of it, as more of the code we run is generated. The primitives for observing agents already exist, from the instrumentation libraries and signals to most of the semantic conventions. What does not carry over is interpretation: a backend trace and an agentic workflow trace are both spans with names, attributes, and timings, but they do not mean the same thing, and instincts built on microservices do not apply one to one.
Agents Are Not Microservices: Operating Autonomous AI on Kubernetes
Henrique Santana (AWS) & Lanay Marques (OllyGarden) | Cloud Native AI + Inference Day | Watch on YouTube
Henrique and Lanay walk through five microservice assumptions that agents break: probes, scaling, disruption, communication, and observability. It was Lanay's first international talk. Henrique covers what happens inside the pod: readiness that reflects how many tasks an agent can take, scaling the downstream bottleneck instead of the agent's CPU, and checkpoints that survive an out-of-memory kill. Lanay covers what happens between pods. A retried model call can choose a different action, so the agent code, not the model, must generate idempotency keys and keep them in a shared store. And because a trace with HTTP 200 and no errors can still carry a wrong answer, an evaluation step records its verdict as an OpenTelemetry GenAI evaluation event on the same trace. All five labs run on a laptop with kind, with no GPU, no model, and no cloud account.
Agents on the Platform: Four Perspectives on the Conversation We're Already Having in Private
Kasper Borg Nissen (Dash0, moderator), Juraci Paixão Kröhling (OllyGarden), Alexa Griffith (Red Hat), Jed Salazar (Edera) & Joseph Sandoval (Resolve) | KubeCon + CloudNativeCon, Platform Engineering track | Watch on YouTube
The panel asks what platform teams must guarantee once agents run in production: identity, access, cost, and audit. Jed argues that agent workloads should be treated as adversarial, Alexa explains why inference routing has to reach the GPU that holds the right cached context, and Joseph urges end users to give upstream feedback instead of building homegrown agent platforms. Juraci's point is that an HTTP 200 seen from outside no longer tells you whether an agent did its job. The business instrumentation that teams skipped for microservices now has to come from inside the agent code, and tracing, which teams were slow to adopt for microservices, is the default signal for agents.






