Your Agents Trust Your Telemetry. Should They?

A joint post by OllyGarden and Causely.

Zero errors, healthy service

An agent is asked whether a payment service had problems in the last hour. It queries the logs for records at ERROR severity, gets an empty result, and reports that the service is healthy. The query was correct. The logs were not. The service wrote lines like level=ERROR msg="executor failed" err="unexpected EOF", but the level only lived in the text of the message. The severity field on every one of those records was empty, so the filter matched nothing.

A person scrolling through the raw logs would probably have spotted the word ERROR. An agent that does what it is told, and does it fast, has no reason to look. It reads an empty result as an answer, and in this case the answer was wrong.

That is the first of two questions this post sets out to answer: can your agents trust the telemetry they read? The second is harder, and it takes up the second half: even when the data is trustworthy, can an agent tell what it means?

Part one: Can the data be trusted?

By OllyGarden

Bad telemetry versus untrustworthy telemetry

Some bad telemetry announces itself. A log storm that writes the same line ten thousand times a minute is obviously wrong, and so is a password sitting in a log attribute. Anyone who looks at that data, human or agent, knows something needs fixing. Those are real problems, with real costs and real risks, but they are visible.

Untrustworthy telemetry is a different kind of problem. It is well-formed, it fits the schema, and it returns plausible results. It just does not say what happened. An agent queries it and gets an empty result, a smaller count, or a healthy-looking service, and nothing in the response suggests the answer is wrong. That is the telemetry this post is about.

We audit OpenTelemetry® data for a living. Between July and October 2026 we sampled telemetry from multiple organizations and looked for the problems that change what a query returns. The sample size isn't representative of all organizations out there, and all of them are teams that chose to work with us on telemetry quality, so treat the numbers below as what we see in practice, not as an industry survey. We group what we found into three ways telemetry loses trust.

Silent absence: the data is not there

The most common problem is data that should exist and does not. An agent cannot reason about what it cannot see, and nothing in the result tells it that something is missing.

Error spans are the clearest case. In every organization where we checked trace data in depth, between 38% and 96% of spans with an error status had no status message and no exception event, and most were above 90%. The agent learns that something failed but not why, and has to guess the cause from the span name and duration.

Ownership goes missing too. Nearly every organization sent telemetry without a usable service.name, either with no value at all or with the SDK default, such as unknown_service:java. For an agent, that telemetry belongs to nobody. It cannot route the problem to a team, correlate it with a deployment, or compare it against the service's normal behavior.

Then there are traces that stop at the first hop. Every organization in the sample had services where most traces consisted of a single span, ranging from 59% to 100% of their traces, with one service at 99.9% of 1.4 million traces in a window of 5 minutes. Some single-span traces are legitimate, such as a worker processing one job. At these rates, though, they usually point to context propagation breaking at a gateway, a queue, or a thread pool. An agent following a slow request across services reaches that boundary and stops, and the hop it ends up blaming is the last one it can see, not the one at fault.

Contradiction: the fields disagree

The second way is telemetry that describes the same event in two incompatible ways. A human weighs both and gets suspicious. An agent usually trusts whichever field its query touches.

Log severity is where we see this most. In most organizations we checked, between roughly a quarter and three quarters of log records whose body reads like an error were recorded at INFO, at DEBUG, or with no severity at all. One record we found carried its own contradiction: {"error":"context canceled","level":"INFO","msg":"Unable to proxy request"}. Another service logged Error starting ApplicationContext at INFO during a failed startup.

Spans do this too. One span we saw reported HTTP status 200 for a batch export, while the gRPC status underneath was 8, resource exhausted, with a message saying the batch exceeded the maximum message size. An agent checking the HTTP status concludes that the batch was delivered. It was rejected.

Split vocabulary: the query sees part of the data

The third way is the hardest to spot, because every query looks correct. The telemetry uses more than one name for the same thing, and the query only knows one of them.

Every organization we checked had more than one generation of attribute names in flight at the same time. In one of them, 125 services reported http.method while 16 reported http.request.method. An agent that writes its query against the current conventions sees about an eighth of the services and draws conclusions about traffic and error rates from that slice. Severity has the same problem: one organization used 30 different spellings for its log levels, including ERR, error, SEVERE, and a JSON fragment that leaked into the field. Most organizations still used deployment.environment alongside, or instead of, deployment.environment.name, so a filter for production returns some of production.

The fix is a deliberate minimum

None of these problems are solved by collecting more data. More auto-instrumentation adds more spans with the same missing reasons, more logs with the same missing severity, and more volume for an agent to read. What helps is a small set of telemetry that is correct on purpose.

We call this Minimum Viable Instrumentation (MVI). It instruments the boundaries of each service: inbound requests, outbound calls to databases, queues, and other services, and the context propagation that ties them together. It records errors with a reason. It settles on one set of attribute names across the organization. And it is verified against a real Collector before it is merged, so a broken service.name or a lost trace context is caught in review, not during an incident. We wrote about the approach in Introducing Minimum Viable Instrumentation.

When to skip this: if a service is about to be retired, or nobody, human or agent, will ever query its telemetry, fixing its instrumentation is not a good use of time. Start with the services your agents already read. The Pareto principle applies here as well: prioritize the top 20% of critical services to achieve the maximum impact.

When good data still misleads

Making telemetry trustworthy removes one class of wrong answers, but not all of them. Some telemetry is well-formed, consistent, and correctly instrumented, and still easy to misread.

A refund worker reports a p99 latency of 20 seconds, which looks like a severe slowdown until you notice that 20 seconds is the queue's polling interval, so the "latency" measures how long the worker waited for work. A trace that appears to start at a ledger consumer has a root span that still references a parent, because the producer that created it never exported its spans.

Neither of these is untrustworthy data. The values are accurate, and no quality check would flag them. Reading them correctly requires knowing how the system fits together, and that is where Causely picks up.

Part two: Can an agent tell what the data means?

By Causely

Those two cases are the shape of the second problem. The p99 that measures waiting and the trace whose producer never reported are both correct, and both invite a confident wrong answer. Fix every field in the logs from the opening, and an agent can still miss the payment service entirely, even when every signal it reads is accurate.

An agent opens a PR

Say you run a background agent that watches for reliability issues and opens a pull request when it finds one. Here is how that can go wrong, using the checkout flow in the OpenTelemetry Astronomy Shop demo app, where checkout calls payment to charge each order, and an illustrative fault.

Checkout's error alert fires. The agent picks it up, pulls checkout's recent errors, logs, and traces, and makes two observations: requests are failing, and checkout's calls to product-catalog hit connection-refused errors from a product-catalog pod that restarted earlier that afternoon. Every trace is intact, and both observations are real.

The agent comes to a reasonable conclusion: checkout does not handle a flaky dependency well. It opens a PR that adds retries with backoff to checkout's product-catalog client, with a clear description and evidence linking the restart to the errors. It is the kind of PR a busy reviewer approves.

The actual problem is a code change in payment that rejects every transaction. Payment never alerted, and there were no restarts, OOMKills, or resource problems on its pods. The errors in checkout, accounting, and shipping all trace back to it.

After the merge, checkout is still failing. The team has spent a review cycle on a change nobody needed, and because a fix shipped, the problem can look handled while transactions keep failing.

Why the agent got it wrong

The agent did not make anything up. Every piece of evidence it used was in the telemetry, and its reasoning from that evidence was sensible.

What it lacked was a way to separate cause from effect. Starting from the alert, it searched outward until something explained the errors, and a restart on a dependency explained them well enough. Payment raised no alert and showed no infrastructure symptoms, so nothing pointed the agent there first. A better model or a longer prompt does not fix this, because the misleading evidence is real.

It is also the expensive path. Every extra log query or probe costs tokens and time, and gives the agent another chance to find something that fits. In many cases the gap is wider than one application. An infrastructure issue can surface as application errors, and a slow external dependency can set off alerts in components that did nothing wrong.

The same alert, with causal context

Causely is a causal reasoning system that continuously maintains a live map of your environment: what depends on what, how each kind of fault propagates to the parts of the stack around it, and which active symptoms point back to which cause (more on the semantics behind this here). Through Causely's MCP server, the agent can check that model before it starts searching.

Here, it can see that the errors in checkout, accounting, and shipping point back to payment, and that the product-catalog restart is unrelated and already over. The PR it opens targets the payment change, and the description says why: which services were affected, which cause explains them, and what it ruled out. The reviewer checks an argument, including the causal graph, rather than redoing the investigation. The agent doesn't have to wait for checkout's alert at all. Causely can trigger it directly, with payment already identified as the cause, so the investigation starts from the answer.

The numbers behind it

We ran this fault in a public benchmark on the OpenTelemetry Astronomy Shop, with four agent configurations (two coding agents and two SRE agents), each with and without Causely's MCP server. The samples are small, so read the size of the gap rather than the exact values:

	Without causal context	With Causely
Cause questions answered correctly	75%	100%
Tool calls per investigation	19	4
Tokens per run	308K	123K
Time to diagnosis	73 seconds	25 seconds
False incidents reported on a healthy cluster	4 of 12 runs	1 of 12 runs

We saw the same pattern in a background-agent setup. With Claude Managed Agents on a 36-service app, both agents found the right fix, but the one without causal context made 3.6x to 7x more tool calls. Full results are on the benchmark page and in the paper.

Where the two halves meet

Causely's reasoning draws on your telemetry, so the problems in the first half affect it too. A service without a usable service.name can end up with no clear identity in the dependency graph, and broken context propagation can hide connections between services. Causely closes some of these gaps itself: its eBPF auto-instrumentation fills in missing coverage, and because it reasons from how faults propagate, it can identify causes that were never directly instrumented. When inputs are poor, the diagnosis gets less specific rather than wrong. Clean telemetry makes that diagnosis sharper, and causal reasoning is what lets an agent use clean telemetry well.

Checklist: can your agents trust what they read?

Nine things to check. The first five are queries you can run against your telemetry this week. The last four are questions to ask of your agent's recent output.

On the data, from OllyGarden

1. What share of your error spans carry a reason, either a status message or an exception event?
2. Which services report unknown_service or no service.name at all?
3. What share of traces are a single span, broken down by service?
4. How many attribute names are in use for HTTP methods, log severity, and deployment environment?
5. Do log severity and log body agree? Search error-like bodies and count how many are not at ERROR or WARN.

On the reasoning, from Causely

6. Of your agent's last ten reliability PRs, how many changed the service that alerted rather than the one that caused the alert?
7. When you ask your agent what it ruled out, can it name alternatives and say why it rejected them?
8. Does the evidence in each PR line up in time with the problem it claims to fix? A restart that ended before the errors started does not explain them.
9. After your agent's fixes merge, how often does the original alert fail to clear?

Try it yourself

Every telemetry example in the first half of this post comes from real data, and we turned twenty of them into a short game: each round shows an anonymized snippet and a claim, and you decide whether to trust it. Play it at ollygarden.com/resources/trust-your-telemetry. If you get fooled, your agents probably would be too.

The benchmark queries, ground-truth answers, and setup from the second half are in Causely's repo, with full results on the benchmark page and on arXiv. If you run it and get different results, open an issue.

Both teams will be at KubeCon North America in Salt Lake City: OllyGarden at booth 356 and Causely at booth 354. Come try the full set of telemetry cases, or bring us the one that fooled your agent.