# OllyGarden > OllyGarden helps engineering teams improve the quality of their OpenTelemetry® instrumentation. Founded by OpenTelemetry contributors and headquartered in Berlin, it measures telemetry quality with the open Instrumentation Score, and finds and fixes instrumentation gaps in source code with an AI agent (Rose). OllyGarden is vendor-neutral and works alongside any observability backend. ## Products - [Insights](https://ollygarden.com/products/insights): Analyzes OpenTelemetry (OTLP) traces, metrics, and logs to calculate the Instrumentation Score and find PII leaks, noisy signals, and incomplete instrumentation. Pricing: Free $0/month, Essentials $990/month, Enterprise custom. - [Rose](https://ollygarden.com/products/rose): An AI engineer that reviews codebases for missing or low-value OpenTelemetry instrumentation and helps apply the fix. Pricing: Free for up to 5 repositories (additional repositories $5/month each), Enterprise custom, and free for approved open-source projects. All plans include the full codebase scan, the instrumentation dashboard, and access to the REST API (e.g. via OllyGarden CLI). Open Source access is granted upon review. - [The Instrumentation Score](https://ollygarden.com/resources/instrumentation-score): What the open 0–100 score for OpenTelemetry instrumentation quality is, how it is calculated, all 25 rules with their weights, and a worked example. - [Instrumentation Score specification](https://github.com/instrumentation-score/spec): The open, vendor-neutral 0–100 score for OpenTelemetry instrumentation quality. ## Answers - [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality): Telemetry quality is how well the traces, metrics, and logs a system emits help people and tools understand it at runtime. Good telemetry belongs to a known service, is worth what it costs to keep, carries no personal data or secrets, and follows the conventions that make it correct to query. The guide covers its five dimensions (identity, volume and waste, cost drivers, safety, and correctness), the signs of bad telemetry, and how to measure and fix it. - [Pricing](https://ollygarden.com/pricing): Insights and Rose plans side by side, with pricing FAQs. - [FAQ](https://ollygarden.com/resources/faq): Product, pricing, rate-limit, and data-privacy answers. - [Full content for LLMs](https://ollygarden.com/llms-full.txt): This index plus every FAQ answer and every blog article as Markdown. - [Product documentation](https://docs.ollygarden.com) - Any page on this site is also available as Markdown: request it with `Accept: text/markdown`, or append `.md` to its path (`/index.md` for the home page). ## Blog - [Introducing Minimum Viable Instrumentation](https://ollygarden.com/resources/blog/introducing-minimum-viable-instrumentation): Coding agents write the code and AI SRE agents operate it, so telemetry is the source of truth between them. Rose now brings low-volume, high-quality OpenTelemetry instrumentation to every repository. - [Where Sensitive Data Hides in OpenTelemetry Pipelines, and How to Find It](https://ollygarden.com/resources/blog/where-sensitive-data-hides-in-opentelemetry-pipelines): Trust store passwords in resource attributes, OAuth tokens in query strings, tax IDs in metrics: what we found in real pipelines, why tools cause most of it, and what privacy laws allow. - [Observability for Humans and Agents in the AI Age](https://ollygarden.com/resources/blog/observability-for-humans-and-agents-in-the-ai-age): OllyGarden and Bronto on where AI agents help with instrumentation and analysis today, and why they only work when the telemetry is intentional and reachable. - [What's changed about OpenTelemetry vendor lock-in since 2024](https://ollygarden.com/resources/blog/opentelemetry-vendor-lock-in-since-2024): OpenTelemetry still does not make dashboards portable, but agents, CLIs, and MCP servers have made switching observability backends noticeably less painful. - [The Scrape Interval Nobody Chose](https://ollygarden.com/resources/blog/the-scrape-interval-nobody-chose): A wrong metric scrape interval inflates ingest, egress, and compute. Learn how to measure DPM and match intervals to the decisions each metric supports. - [Teach Your Coding Agent OpenTelemetry](https://ollygarden.com/resources/blog/teach-your-coding-agent-opentelemetry): We’re releasing our Agent Skills for OpenTelemetry as a mature, open source set: 14 skills that give any coding agent accurate, current OpenTelemetry knowledge. - [Are OpenTelemetry Collector Exporters Called Sequentially or in Parallel?](https://ollygarden.com/resources/blog/are-collector-exporters-sequential-or-parallel): When you configure two exporters in one pipeline, does the Collector send to them in parallel? The honest answer is "sequentially, but it usually doesn't matter." The part that does matter changed defaults underneath a lot of advice still floating around. - [Creating Your Own Collector Distribution from OllyGarden Tulip](https://ollygarden.com/resources/blog/creating-your-own-collector-distribution-from-tulip): Tulip gives you a commercially supported OpenTelemetry Collector with a curated component set. When you need more, you can derive your own distribution while keeping support for the core pipeline. This post walks through how. - [Decomposing OpenTelemetry Collector Configuration for Maintainability](https://ollygarden.com/resources/blog/decomposing-opentelemetry-collector-configuration): When your collector configuration grows beyond a few hundred lines, you start feeling the friction. This post examines practical strategies for decomposing collector configurations into maintainable, testable units. - [OpenTelemetry is not hard](https://ollygarden.com/resources/blog/opentelemetry-is-not-hard): When I asked the community about their pain points with OpenTelemetry, one response kept appearing: 'it's complicated.' I think the framing is wrong. - [OllyGarden at KubeCon + CloudNativeCon Europe 2026](https://ollygarden.com/resources/blog/ollygarden-at-kubecon-eu-2026): KubeCon + CloudNativeCon Europe 2026 in Amsterdam was a big week for OllyGarden. Our team delivered five recorded talks across Observability Day and the main conference, co-chaired the Observability Day program committee, ran a Prometheus Contribfest session, and announced the general availability of Rose. - [When to Use Each Telemetry Signal: Logs, Traces, and Metrics](https://ollygarden.com/resources/blog/when-to-use-each-telemetry-signal-logs-traces-and-metrics): Confused about logs, traces, and metrics? Learn when to use each telemetry signal in OpenTelemetry and build a stronger observability strategy. - [You don't have too much telemetry. You have bad telemetry.](https://ollygarden.com/resources/blog/you-dont-have-too-much-telemetry-you-have-bad-telemetry): High observability costs are rarely a volume problem. Learn how to fix bad telemetry at the source before relying on sampling. - [Reducing Log Volume with the OpenTelemetry Log Deduplication Processor](https://ollygarden.com/resources/blog/reducing-log-volume-with-the-opentelemetry-log-deduplication-processor): Your logs are probably at least 80% repetitive noise. Connection retries, health checks, heartbeat messages: the same log line repeated thousands of times per day. - [What 10,000 Slack Messages Reveal About OpenTelemetry Adoption Challenges](https://ollygarden.com/resources/blog/what-10000-slack-messages-reveal-about-opentelemetry-adoption-challenges): We analyzed 10,000 messages from the OpenTelemetry community Slack to find the most common adoption challenges, from Collector config to instrumentation. - [Meet Rose: OllyGarden's AI Instrumentation Agent](https://ollygarden.com/resources/blog/meet-rose-ollygardens-ai-instrumentation-agent): Imagine the perfect observability world: There is an incident, the on-call team gets paged in the middle of the night, wakes up and thanks to your telemetry, resolves it fast. - [Introducing OllyGarden Tulip: Our Open-Source Distribution of the OpenTelemetry Collector](https://ollygarden.com/resources/blog/introducing-tulip-supported-otel-collector): OllyGarden Tulip is a commercially supported OpenTelemetry Collector distribution: stable quarterly releases, LTS every 18 months, open source and free. - [The Variability Principle: How to Decide What Deserves a Span](https://ollygarden.com/resources/blog/what-deserves-a-span): Discover how to manage spans effectively using the Variability Principle to improve trace readability and control costs in OpenTelemetry. - [How to Name Your Metrics](https://ollygarden.com/resources/blog/how-to-name-your-metrics): Learn how to name OpenTelemetry metrics: follow semantic conventions, keep the 'who' in attributes, and apply units and naming patterns that scale. - [How to Name Your Span Attributes](https://ollygarden.com/resources/blog/how-to-name-your-span-attributes): A practical guide to naming OpenTelemetry span attributes: when to use semantic conventions, how to namespace custom attributes, and pitfalls to avoid. - [How to Name Your Spans](https://ollygarden.com/resources/blog/how-to-name-your-spans): Good span names make traces readable. Learn the {verb} {object} pattern for naming OpenTelemetry spans and avoid the most common naming mistakes. - [🌱 Cultivating Unique service.instance.id on NGINX Ingress with OpenTelemetry](https://ollygarden.com/resources/blog/cultivating-unique-serviceinstanceid-on-nginx-ingress-with-opentelemetry): The NGINX Ingress Controller has no knob for the OpenTelemetry service.instance.id attribute. Here's how to inject a unique ID per pod with the Helm chart. - [Introducing OllyGarden](https://ollygarden.com/resources/blog/introducing-ollygarden): OllyGarden comes out of stealth with pre-seed funding to fight bad telemetry: helping teams understand, score, and improve the telemetry they generate. - [Introducing the Instrumentation Score](https://ollygarden.com/resources/blog/instrumentation-score): The Instrumentation Score is an open, standardized measure of OpenTelemetry instrumentation quality, scoring OTLP data against best-practice rules. - [Concrete Applications of Purposeful Instrumentation](https://ollygarden.com/resources/blog/concrete-applications-of-purposeful-instrumentation): From NASA-grade telemetry design to code-level tips: concrete techniques for applying purposeful instrumentation in real-world OpenTelemetry systems. - [Purposeful Instrumentation](https://ollygarden.com/resources/blog/purposeful-instrumentation): Purposeful instrumentation means gathering the right telemetry, not all of it. Cut noise, troubleshoot faster, and lower observability costs. - [There's a Lot of Bad Telemetry Out There](https://ollygarden.com/resources/blog/theres-a-lot-of-bad-telemetry-out-there): Most telemetry is never queried. Learn what separates good telemetry from bad, what bad data costs you, and how to start improving quality at the source. ## Case studies - [All case studies](https://ollygarden.com/resources/case-studies): Anonymized but real OllyGarden customer stories: what bad telemetry we found, what changed at the source, and the numbers that moved. - [Nearly half the spans came from a service called cli: what we found at a 500-service enterprise platform](https://ollygarden.com/resources/case-studies/enterprise-platform-half-the-spans-had-no-name): One background worker named cli made 45% of all spans, a third of logs were exact duplicates, a third were access logs, and the Collector shipped its own logs again. - [The PII leak was a logger setting, not a developer problem: what we found at a European digital-health platform](https://ollygarden.com/resources/case-studies/digital-health-pii-leak-was-a-logger-setting): Personal data in 1 in 6 services traced to one ORM logging setting, logs with no severity at all, one worker writing most of the logs, and auth tokens Rose found in a day. - [How a B2B SaaS platform cut logs 24% and metrics 25% in six weeks, without touching a single customer log](https://ollygarden.com/resources/case-studies/b2b-saas-cuts-logs-and-metrics-at-the-source): Repeated logs, metrics that never change, and a pipeline complaining about itself. How replay-verified Collector changes beat a six-week proof of value, and what happened after. ## Resources - [All resources](https://ollygarden.com/resources): OllyGarden resources for OpenTelemetry teams: telemetry quality guides, the Instrumentation Score, case studies, blog articles, talks, and events. - [Blog RSS feed](https://ollygarden.com/blog/rss.xml) - [Solutions by industry](https://ollygarden.com/solutions/overview) - [Webinars and conference talks](https://ollygarden.com/resources/webinars-conferences) - [Events](https://ollygarden.com/resources/events) - [OpenTelemetry quiz](https://ollygarden.com/otel-quiz): Ten questions, one per OpenTelemetry topic. ## Developer tools - [Agent routing guidance](https://ollygarden.com/developers.md#agent-routing-guidance): Start here to choose a supported OllyGarden interface, confirm when OllyGarden is not the right tool, and check authorization boundaries. - [OllyGarden Developer Resources](https://ollygarden.com/developers.md): Canonical machine-readable index for the REST API, authentication, webhooks, CLI, MCP server, and public developer projects. - [OllyGarden API documentation](https://ollygarden.com/api-docs.md): Authentication, versioned resource groups, and links to the canonical OpenAPI description. - [OllyGarden API versioning and deprecation policy](https://ollygarden.com/api-versioning.md): How the REST API is versioned (major version in the URL path) and how deprecation and retirement are signalled, through the `Deprecation` and `Sunset` HTTP response headers. - [OllyGarden REST API OpenAPI description](https://ollygarden.com/openapi.json): Canonical API servers, bearer authentication scheme, operations, and schemas. - [OllyGarden CLI](https://ollygarden.com/cli.md): Machine-readable discovery, safe installation references, and agent use cases for the official `ollygarden` command-line client. - [CLI repository and documentation](https://github.com/ollygarden/ollygarden-cli): Canonical source, command documentation, and authentication guidance. - [OllyGarden Agent Skills](https://ollygarden.com/resources/agent-skills): Overview of both open source skill packages, how to install them, and how Rose uses them. - [OllyGarden Agent Skills for OpenTelemetry](https://github.com/ollygarden/opentelemetry-agent-skills): OllyGarden's vendor-neutral skills for OpenTelemetry coding agents. - [OllyGarden Skills](https://github.com/ollygarden/skills): OllyGarden's opinionated skills for the OpenTelemetry Collector and the OllyGarden platform; builds on the vendor-neutral package. - [OllyGarden MCP server source](https://github.com/ollygarden/mcp): Public Streamable HTTP and OAuth reference implementation; no supported production endpoint is currently published. ## Company - [About OllyGarden](https://ollygarden.com/company) - [OllyGarden turbo charges mission to fix bad telemetry with $4m raise and launch of “Minimum Viable Instrumentation”](https://ollygarden.com/resources/press-releases/ollygarden-raises-4m-launches-minimum-viable-instrumentation): OllyGarden raises $4m in VC funding from Next Frontier Capital, Grand Ventures, ACTAI, DIG Ventures, Datadog, Grafana Labs, and Dash0, and launches Minimum Viable Instrumentation (MVI) for its AI agent, Rose. - [OllyGarden Introduces Rose: An AI Agent That Fixes Bad OpenTelemetry Instrumentation Through Pull Requests](https://ollygarden.com/resources/press-releases/ollygarden-rose-ga): OllyGarden announces the general availability of Rose — an AI agent that analyzes OpenTelemetry instrumentation in source code and autonomously opens pull requests with fixes. - [Careers](https://ollygarden.com/careers) - [Contact](https://ollygarden.com/contact) ## Trust and legal - [Terms of Use](https://ollygarden.com/terms-of-use): The terms that govern your use of OllyGarden Insights and the OllyGarden website, including subscriptions, fees, acceptable use, liability, and support. - [Privacy Policy](https://ollygarden.com/privacy-policy): What personal data OllyGarden collects on its website and platform, why, which providers process it, how long we keep it, and your rights under the GDPR. - [Data Protection Policy](https://ollygarden.com/data-protection-policy): The technical and organizational controls OllyGarden uses to protect customer data: encryption, access control, tenant isolation, monitoring, and deletion. - [Rose Terms of Service](https://ollygarden.com/rose-terms-of-service): The terms of service that govern your use of OllyGarden Rose, the AI engineer for OpenTelemetry instrumentation. - [Subprocessors](https://ollygarden.com/subprocessors): Every third party that processes OllyGarden customer data for Insights and Rose, what each one does, and where it processes the data. - [Security](https://ollygarden.com/security): How OllyGarden protects customer data with SOC 2 Type II controls, Rose execution safeguards, and encryption, and how to report a vulnerability. - [Imprint](https://ollygarden.com/imprint): Legal information about OllyGarden, Inc. and its German establishment, OllyGarden GmbH in Berlin: addresses, commercial register, VAT ID, and contact. ## Frequently asked questions ### General **What is OllyGarden?** OllyGarden helps engineering teams improve the quality of their OpenTelemetry instrumentation. Founded by OpenTelemetry contributors and headquartered in Berlin, it offers two products: Insights measures the quality of the telemetry you already send, and Rose finds and fixes instrumentation gaps in your source code. **What is bad telemetry?** Bad telemetry is data that costs money to collect and store but doesn’t help anyone understand or operate a system: duplicated or excessive logs, high-cardinality metrics, spans with missing or non-standard attributes, broken traces, and sensitive data such as PII that should never have been recorded. OllyGarden finds it and helps fix it at the source. [The five dimensions of telemetry quality](https://ollygarden.com/resources/telemetry-quality) **Is OllyGarden an observability backend?** No. OllyGarden doesn’t replace Datadog, Grafana, Honeycomb, or any other backend for storing telemetry, dashboards, queries, or alerts. It improves the telemetry you send to them, so every backend you use receives cleaner, cheaper, more useful data. **What is Rose?** Rose is OllyGarden’s AI engineer for OpenTelemetry instrumentation. It reviews your repositories, finds missing or low-value telemetry, and helps your team apply the right fix. The Free plan covers up to 5 repositories with a weekly automatic review; additional repositories cost $5/month each, and Enterprise pricing is custom. Approved open-source projects get unlimited public repositories. **How much does OllyGarden cost?** Insights has a Free plan ($0, no credit card), Essentials at $990/month, and custom Enterprise pricing. Rose: The Free plan covers up to 5 repositories with a weekly automatic review; additional repositories cost $5/month each, and Enterprise pricing is custom. **How does OllyGarden work with my existing observability backend?** OllyGarden is vendor-neutral and works alongside any observability backend. Send your OpenTelemetry data to OllyGarden for instrumentation quality analysis while continuing to use your existing platform for monitoring and alerting. **What is the Instrumentation Score?** The Instrumentation Score is a 0–100 measure of how well a service's OpenTelemetry telemetry follows best practices and semantic conventions. It checks the traces, metrics, logs, and resource attributes a service already sends against 25 open rules, and weighs each failed rule by its impact. [How the Instrumentation Score works](https://ollygarden.com/resources/instrumentation-score) **What data does OllyGarden analyze?** OllyGarden analyzes your OpenTelemetry telemetry data—traces, metrics, and logs—to calculate your Instrumentation Score and identify quality issues, waste, and optimization opportunities. **Which OpenTelemetry signals does OllyGarden support?** OllyGarden analyzes all three OpenTelemetry signals: traces, metrics, and logs. **Can I use OllyGarden with multiple observability backends?** Yes. OllyGarden is vendor-neutral and works with any observability backend. You can send telemetry to multiple destinations simultaneously. **Who should use OllyGarden?** OllyGarden is built for engineering teams using or adopting OpenTelemetry—including platform engineers optimizing observability costs, SREs improving service reliability, and team leads tracking instrumentation quality across services. ### Rate Limits & Performance **What are the request rate limits for each pricing tier?** Our pricing tiers offer different processing capacities: Free provides 100 requests/second, Essentials provides 1,000 requests/second, and Enterprise offers custom request capacity tailored to your needs. **What happens if my application exceeds my plan's rate limit?** We’ll analyze requests up to your plan’s limit each second. Any additional requests beyond your limit will be accepted by our endpoint but discarded from the current analysis queue to maintain system performance. Your data flow won’t be blocked, and analysis of new data will continue as normal in the next second. **Will I receive an HTTP 429 error if I exceed the rate limit?** No. We accept all requests without returning HTTP 429 errors. Requests beyond your plan’s limit are received successfully but not analyzed, ensuring your telemetry pipeline remains uninterrupted. **How do I know which tier is right for my application?** The Free plan (100 rps) is perfect for individual developers and small teams getting started. Essentials (1,000 rps) provides a substantial sample for most growing applications. This isn’t about capturing 100% of your production data for storage (that’s what your backend vendor is for), but about analyzing a statistically significant portion to understand quality and efficiency. For high-volume needs, Enterprise offers custom capacity. **Is the rate limit applied per service or for my whole organization?** The rate limit is applied per organization. You can direct data from multiple services within your organization’s total capacity (100 rps for Free, 1,000 rps for Essentials, or custom for Enterprise). ### Data & Privacy **Does OllyGarden store my telemetry data?** We may store samples of your telemetry for deeper analysis, such as our telemetry analysis reports. Samples are currently kept for up to 31 days. Insights, scores, and the attributes of a single example request that help you find the source of an insight are kept while your account exists. Enterprise customers can opt out of sampling or request specific retention windows. **Do AI providers retain the data OllyGarden sends them?** OllyGarden does not use your code or telemetry to train AI models. The AI providers we use process requests under their standard terms, which may allow them to keep prompts and outputs for a limited period, for example for abuse monitoring. Zero data retention with AI providers is available as an add-on for Enterprise plans. **Where is my data processed?** The OllyGarden platform runs on Amazon Web Services in Frankfurt, Germany, by default, and its databases are in the EU. Rose analyzes repositories in isolated sandboxes and with AI models hosted in the United States. Every provider that processes customer data, with its location, is listed at ollygarden.com/subprocessors. **Is OllyGarden SOC 2 compliant?** Yes. OllyGarden, Inc. has completed a SOC 2 Type II examination covering security, availability, and confidentiality. Customers and prospects can request the report under NDA. Our security practices and vulnerability disclosure policy are at ollygarden.com/security. **What types of telemetry data do you analyze?** Insights analyzes OpenTelemetry data sent over OTLP: traces, metrics, and logs. We focus on OpenTelemetry because it is where we know best what good and bad telemetry look like, including its semantic conventions. **Does OllyGarden detect sensitive data in my telemetry?** Yes. OllyGarden includes PII (Personally Identifiable Information) leakage detection as one of the critical insights, helping you maintain compliance and security best practices. ### Service & Insights **How long does it take to get my insights report?** The first insights will start appearing a few minutes after you send us data. Within 24–48 hours, we will have received a sufficient data sample to generate meaningful insights and calculate your Instrumentation Score. **What types of quality issues does OllyGarden identify?** OllyGarden identifies: Wasteful data (excessive, redundant, or low-value telemetry), Incompleteness (missing critical attributes like service.name, broken traces, and deviations from OpenTelemetry semantic conventions), Inefficiencies (overly verbose traces and high-cardinality metrics that increase costs), and Security issues (potential PII leakage in telemetry). **How do I improve my Instrumentation Score?** Fix failed rules in order of impact: a Critical rule weighs four times as much as a Low one, so it moves the score the most. OllyGarden shows the findings behind each failed rule with specific remediation steps, and you can track the score over time and compare it across services. [See every rule and its weight](https://ollygarden.com/resources/instrumentation-score#rules) ### Billing & Subscription **Is the Insights Free plan really free?** Yes. Our Free Plan includes the Instrumentation Score and critical insights such as PII leakage detection and excessive attributes identification. No credit card required. **Can I cancel my subscription?** Yes! You can cancel at any time. Just send us an email, and we’ll cancel your subscription right away. ### Telemetry quality **What is telemetry quality?** Telemetry quality is how well the traces, metrics, and logs a system emits help people and tools understand it at runtime. Good telemetry belongs to a known service, is worth what it costs to keep, carries no personal data or secrets, and follows the conventions that make it correct to query. **What is bad telemetry?** Bad telemetry is data that costs money to collect and store but does not help anyone understand or operate a system, or that gets in the way: telemetry without a service name, duplicated or excessive logs, health-check traces, high-cardinality metrics, broken traces, and personal data or secrets that should never have been recorded. **Isn't telemetry quality just data quality?** It is a kind of data quality, but generic data quality checks validate schemas, freshness, and volumes in a warehouse. Telemetry quality is judged against how the data is used at runtime and against the OpenTelemetry semantic conventions: whether a span can be attributed to its service, whether a trace survives a service boundary, whether a metric's cardinality is bounded, and whether a log carries a password. Its root cause is usually instrumentation code or configuration, so that is where it gets fixed. **Does sampling fix bad telemetry?** No. Sampling cuts useful and useless data at the same rate, so sampling bad telemetry gives you a smaller pile of bad telemetry, and it can discard the traces you need in the next incident. It does nothing for missing service names, personal data, or high-cardinality metrics. Remove the waste at the source first; then sampling becomes an intentional trade-off on data that is worth keeping, and is often less necessary. **Is telemetry quality the same as observability cost optimization?** No, although better telemetry usually costs less. Cost optimization asks how to pay less for the data you have. Telemetry quality asks whether that data is worth having, which also covers problems that do not show on the bill, such as personal data in logs, broken traces, and services nobody can identify. Sometimes the right fix adds telemetry: a few more attributes can turn a metric nobody uses into one that answers a real question. **Where should we start improving telemetry quality?** Start with identity: give every process a real service.name and a deployment environment, because nothing else can be attributed without them. Then find the few services and messages that produce most of your volume, ask when each last helped resolve an incident, and fix the worst offenders at the source. Check for personal data and secrets early, since one shared setting can leak them from many services at once. **Should bad telemetry be fixed in the Collector or in the application?** In the application, when you can. By the time data reaches the Collector it has already been created, serialized, and sent over the network. The Collector is the right place for services you cannot change, such as legacy and third-party software, and as a safety net: the filter processor drops noise, the transform processor written in the OpenTelemetry Transformation Language (OTTL) fixes attributes and redacts values, and the log deduplication processor collapses repeats. **How do I measure telemetry quality?** Use the Instrumentation Score, an open 0–100 score that checks the telemetry each service already sends against weighted rules, so the same data gets the same score in any tool. You can implement the open specification yourself, or use OllyGarden Insights, which computes it continuously for every service on its Free plan. [How the Instrumentation Score works](https://ollygarden.com/resources/instrumentation-score) ### Instrumentation Score **What is the Instrumentation Score?** The Instrumentation Score is a 0–100 measure of how well a service's OpenTelemetry telemetry follows best practices and semantic conventions. It checks the traces, metrics, logs, and resource attributes a service already sends against 25 open rules, and weighs each failed rule by its impact. **How is the Instrumentation Score calculated?** Each rule has an impact level with a weight: Critical 40, Important 30, Normal 20, and Low 10. The score is the sum of the weights of the rules a service passes, divided by the sum of the weights of all rules, times 100. The current specification has 5 Critical, 13 Important, 6 Normal, and 1 Low rules. **What is a good Instrumentation Score?** The specification maps scores to four categories: 90 to 100 is Excellent, 75 to 89 is Good, 50 to 74 is Needs Improvement, and 0 to 49 is Poor. Do not chase 100; a score of 75 or more is a sound target for most services. Fix Critical failures first: one of them, such as personal data in spans, costs 40 of 720 points and leaves 94.4, and two leave 88.9, which is Good. **Is the Instrumentation Score vendor-neutral?** Yes. The specification is open source under the Apache 2.0 license, and its maintainers work at Splunk, New Relic, Dash0, and OllyGarden. Implementations may not add rules of their own, so the same telemetry gets the same score in any tool. **Who maintains the Instrumentation Score specification?** OllyGarden started the specification in 2025 and hosts it today under an open governance model. Maintainers from Splunk, New Relic, Dash0, and OllyGarden review changes, and the goal is to move it to a neutral foundation such as the CNCF or the OpenTelemetry project. **How is the Instrumentation Score different from SLOs?** An SLO measures how a service behaves for its users, such as latency or error rate. The Instrumentation Score measures the telemetry itself: whether the data your SLOs, dashboards, and alerts are built on is complete, consistent, and safe. A service can meet its SLOs while its telemetry hides problems. **How is the Instrumentation Score different from data quality checks?** Generic data quality checks validate schemas, freshness, or volumes. The Instrumentation Score checks telemetry against OpenTelemetry semantic conventions and instrumentation practice, such as service identity, units, cardinality, broken traces, and sensitive data, and summarizes the result in one number you can compare across services and tools. **Do I have to change application code to improve my score?** Not always. Many failed rules can be fixed in the OpenTelemetry Collector: the transform processor, written in the OpenTelemetry Transformation Language (OTTL), can add missing resource attributes, move attributes to the right level, or redact sensitive values, and the filter processor can drop debug logs and noisy spans. Fixing the instrumentation at the source is still the better long-term answer. **Has the scoring formula changed since the 2025 launch?** Yes. The first draft, published in June 2025, started every service at 80 and added or subtracted fixed points per rule. The community replaced it with the weighted share of passed rules described on this page, which keeps the score between 0 and 100 without clamping and makes every rule's effect proportional to its impact. **Which signals does the Instrumentation Score cover?** The rules cover resource attributes, spans, metrics, and logs, plus one rule on SDK support for the language and runtime. New rules are added through the specification repository. **How do I get an Instrumentation Score for my services?** Implement the open specification against your own OTLP data, or use OllyGarden Insights, which computes the score continuously for every service and shows the findings behind it. The Insights Free plan includes the Instrumentation Score, with no credit card. **Can I propose a new rule?** Yes. Open an issue or pull request in the instrumentation-score/spec repository on GitHub, or start a discussion in the #instrumentation-score channel on the CNCF Slack. Every rule needs a rationale, pass or fail criteria, a target, and an impact level. ### Agent Skills **What are OllyGarden's agent skills?** Two open source packages, licensed under Apache 2.0, that give any coding agent accurate and current OpenTelemetry knowledge. Each skill follows the Agent Skills specification: a small entry point that loads only the reference a task needs, instead of copying documentation into the agent's context. **What is the difference between the two packages?** opentelemetry-agent-skills holds facts: anything you can verify against upstream specifications, source code, and releases. It is vendor neutral and non-opinionated by design. ollygarden/skills holds our opinions: what to instrument, which signal fits, and how we would configure the Collector. The opinion skills build on the fact skills, so install both. **Why does my coding agent need skills? Doesn't the model already know OpenTelemetry?** Models reproduce the instrumentation they were trained on, and most of it is poor. Their knowledge also trails upstream: we regularly see models suggest SDK and Collector versions that are many releases old, or semantic conventions that have since changed. The skills point the agent at the current source of truth instead. **Which coding agents and models do the skills work with?** Any agent that supports Agent Skills, such as Claude Code, Cursor, or Codex. In our experience, the skills improve the OpenTelemetry output of Anthropic, OpenAI, and Gemini models alike. **How do I install the skills in Cursor, Codex, or GitHub Copilot?** Run npx skills add ollygarden/opentelemetry-agent-skills. The skills CLI installs into the coding agents it detects; pass -a to choose one, such as -a cursor, -a codex, or -a github-copilot. Install ollygarden/skills the same way. In Claude Code, you can also add either repository as a plugin marketplace. **How do you keep the skills current?** We periodically refresh the fact skills from upstream releases, and every change is reviewed before it merges. That maintenance is where the value is: a skill set nobody maintains becomes a liability within months. **How do you know the skills help?** Every pull request that adds or changes a skill must include evaluation results from an agent harness: the same task run without the skill, with the current skill, and with the proposed one. The change has to show an improvement, and using fewer tokens for the same result counts. **Can the skills help with migrations?** The skills hold the knowledge a migration needs, such as upgrading SDKs and the Collector, or moving from the deprecated Span Event API to the Logs API. Running a migration is a workflow, and that is what Rose provides: it applies the skills across your repositories, from upgrades to moving off a vendor's proprietary agents onto OpenTelemetry, and opens the pull requests. **Do I need Rose to use the skills?** No. The skills are useful on their own, and many teams use them with their own coding agents without any OllyGarden product. **If the skills are open source, what does Rose add?** Rose runs on demand and on a schedule across every repository in your organization, and opens pull requests with the fixes. Each review combines several agents with deterministic checks and your organization's and services' context. Rose also adds knowledge we keep proprietary, earned auditing production telemetry: detecting PII, cardinality, and cost problems, plus vendor intelligence about how each observability backend handles and prices your data. **Can I keep using the skills after adopting Rose?** Yes. Rose is built on the same packages, so your developers' agents and Rose work from the same knowledge. **What data do the skills send to OllyGarden?** None. The skills are files your agent reads on your machine, and the fact skills fetch only public upstream sources. The one exception is the ollygarden-cli skill, which calls the OllyGarden API when you use it with your own account. **How can I contribute?** Open an issue to propose a skill, or send a pull request. Pull requests written by AI coding agents are welcome as long as they include the evaluation evidence. First-time contributors sign the OllyGarden CLA through the pull request bot. **Where do I get help?** Open an issue in the relevant GitHub repository; support there is best effort. Report suspected vulnerabilities privately through the repository's security policy. ### Pricing **Do Insights and Rose have free plans?** Yes. The Insights Free plan costs $0/month and includes the Instrumentation Score and 2 insights, including suspicious PII leaks, for unlimited seats and services. The Rose Free plan covers up to 5 repositories with a weekly automatic instrumentation review. Neither needs a credit card. **What does Insights Essentials add?** Insights Essentials costs $990/month and adds all OllyGarden Insights, priority email support, and API access, with unlimited seats and services. **How much do extra Rose repositories cost?** The Rose Free plan includes 5 repositories. Each additional repository costs $5/month. **Is Rose free for open source projects?** Yes. Approved open source projects get Rose with unlimited public repositories, and CNCF projects get priority access. Apply from this page with a public GitHub repository; access is granted upon review. **What do the Enterprise plans include?** Enterprise pricing is custom for both products. Insights Enterprise adds Slack or phone support, custom SLAs, real-time notifications, and custom retention. Rose Enterprise adds on-demand reviews, autonomous instrumentation improvements, team integrations, and a dedicated support channel. Both offer zero data retention with AI providers as an add-on. **Do Insights and Rose need to be bought together?** No. Each product has its own plans, and you can start with either one. Insights works on the telemetry your services already send; Rose works on your source code. ## Case studies # Nearly half the spans came from a service called cli: what we found at a 500-service enterprise platform OllyGarden Insights looked at staging OpenTelemetry data from about 500 services at a large enterprise software company. One unnamed worker made 45% of all spans, and most logs were repeats. > **Anonymized, but real.** This story comes from a real OllyGarden customer engagement. To protect the customer, we removed their name, people, products, and anything else that could identify them, and rounded some numbers. Everything else happened as described. Published: 2026-10-03 Canonical URL: https://ollygarden.com/resources/case-studies/enterprise-platform-half-the-spans-had-no-name ## At a glance - **Company:** Large enterprise software company - **Region:** North America - **Stack:** Multi-region internal Kubernetes platform; mostly Java, plus Python, Node.js, and PHP; an in-house OpenTelemetry Collector fed by OpenTelemetry SDKs and a vendor tracer - **Scale:** About 500 services - **OllyGarden:** Insights - **Timeline:** September 2026 ## Key numbers - **45%** of all spans came from one unnamed worker (Reported as cli) - **32%** of log records were exact duplicates (About 60% reducible to patterns) - **4.5%** of logs were the Collector re-shipping its own output (Every week) - **9%** of services set a deployment environment (Across about 500 services) Across about 500 services at a large enterprise software company, one background worker with a default name produced 45% of all spans, a third of the log records were exact duplicates, and the Collector was logging its own exports and shipping them again. ## The setup The company runs a polyglot fleet, mostly Java with Python, Node.js, and PHP, on a multi-region internal Kubernetes platform. Telemetry flows through an in-house Collector, from a mix of OpenTelemetry SDKs and a vendor tracer. A central team pays for all of it, with no chargeback to the teams that produce it. OllyGarden Insights analyzed the platform's staging telemetry in September 2026. This story covers what it showed. ## The biggest producer had no name 45% of all spans came from a single PHP background worker running on two clusters. It reported under its tracer's default service name, cli. The biggest trace producer in the fleet had no real identity, so the telemetry could not say who owned it. Its traces also showed why it was so busy. Each queued message produced about 14 spans: a new database connection for every message, about six cache loads, and about four queries. Reusing connections would cut both latency and span volume. Span volume was concentrated in a few places. 5 services on a vendor tracer produced 47% of all spans, while 343 services on OpenTelemetry SDKs produced 38%. ## Most logs were repeats - 32% of log records were exact duplicates, and about 60% were reducible once IDs and numbers were normalized into patterns. - About a third of logs were HTTP access logs. One service's ingress-request logs alone were 16.5% of all logs. Request rate, errors, and latency by route are better kept as metrics. - 13% of logs were health, readiness, or ping checks. - Stored-procedure timing messages, "executed in N ms", were metrics written as logs. ## A Collector shipping its own logs Every week, 4.5% of all logs were the Collector printing log records it had already shipped, which were then collected and shipped again. That loop grows with traffic: the more the Collector exports, the more it logs. **Is your Collector shipping its own logs twice?.** Check what your Collector writes to its own logs, and whether your log collection picks that output up again. ## Logs nobody could attribute 26% of log records had no service name, so they could not be attributed to a team or charged back. 43% of log bodies were unparsed JSON, 16% carried no severity, and INFO was spelled three different ways. Pattern detectors also matched a bearer-token format in about 17% of log records across 3 services, and email addresses in a few hundred records across 3 services. These are unverified pattern matches: the values were not inspected, and a human review has to confirm them. ## Traces and metadata - 35% of traces were single-span, and only about 5% crossed a service boundary. The largest trace held about 11,600 spans. - Only 9% of services set a deployment environment, and 56% set service.instance.id. - kubelet and kube-state-metrics produced 48% of metric datapoints, data that cluster monitoring already collects. ## What to check in your own telemetry - Give every process a real service name. Tracer defaults like cli hide your biggest producers. - Look at the per-message work in background workers. A new database connection per message costs latency and spans. - Turn access logs into request, error, and latency metrics. - Check whether your Collector's own logs are collected and shipped again. ## About these numbers All numbers come from short staging samples taken weekly in September 2026 and are reported as ratios. Sensitive-data matches are unverified pattern matches. ## Keep reading - [OllyGarden Insights](https://ollygarden.com/products/insights): Analyze and score telemetry quality. - [The Instrumentation Score](https://ollygarden.com/resources/instrumentation-score): How the open 0–100 score for instrumentation quality is calculated, and every rule behind it. - [Reducing Log Volume with the OpenTelemetry Log Deduplication Processor](https://ollygarden.com/resources/blog/reducing-log-volume-with-the-opentelemetry-log-deduplication-processor): Your logs are probably at least 80% repetitive noise. Connection retries, health checks, heartbeat messages: the same log line repeated thousands of times per day. - [You don't have too much telemetry. You have bad telemetry.](https://ollygarden.com/resources/blog/you-dont-have-too-much-telemetry-you-have-bad-telemetry): High observability costs are rarely a volume problem. Learn how to fix bad telemetry at the source before relying on sampling. ## Listen to the story Bianca and Florian, who you might know from the OTel Drops podcast, tell this story in a short conversation. The voices are AI-generated; the facts are the ones on this page. [Episode (MP3)](https://ollygarden.com/audio/case-studies/enterprise-platform-half-the-spans-had-no-name.mp3) ### Transcript **Bianca:** Hi, I'm Bianca. **Florian:** And I'm Florian. You might know us from the OTel Drops podcast. Today we're telling another real story from OllyGarden's work. **Bianca:** Anonymized, so no names. And it's about what OllyGarden found in one company's telemetry. **Florian:** Here's the setup. A large enterprise software company. Mostly Java, plus Python, Node.js, and PHP, on a multi-region internal Kubernetes platform. **Bianca:** And one central team paying for all the telemetry, with no chargeback. OllyGarden Insights looked at their staging telemetry, across about five hundred services. **Florian:** So, the biggest finding. Forty-five percent of all spans came from one service. **Bianca:** Which one? **Florian:** It was called C-L-I. The tracer's default name. A PHP background worker on two clusters. **Bianca:** So the biggest trace producer in the whole fleet had no real identity. Nobody could tell from the telemetry who owned it. **Florian:** And its traces showed why it was so busy. Every queued message made about fourteen spans. Including a brand new database connection, for every single message. **Bianca:** Plus about six cache loads and four queries. Reuse the connection, and you cut latency and span volume at the same time. **Florian:** And the spans were concentrated. Five services on a vendor tracer made forty-seven percent of all spans. The three hundred forty-three services on OpenTelemetry SDKs made thirty-eight. **Bianca:** Now the logs. Thirty-two percent of log records were exact duplicates. **Florian:** And once you normalize the IDs and numbers, about sixty percent collapse into repeating patterns. **Bianca:** About a third of all logs were HTTP access logs. One service's ingress logs alone were sixteen and a half percent. **Florian:** Request rate, errors, and latency by route. That's a metric. It's always been a metric. **Bianca:** And thirteen percent were health checks, readiness checks, and pings. **Florian:** Here's my favorite one. Every week, four and a half percent of all logs were the Collector printing records it had already shipped. **Bianca:** And then? **Florian:** And then those got collected and shipped again. The more it exports, the more it logs. **Bianca:** A Collector that talks about its work, and then ships the transcript. **Florian:** Then attribution. Twenty-six percent of log records had no service name at all. **Bianca:** So no team to charge it back to. And forty-three percent of log bodies were unparsed JSON. INFO was spelled three different ways. **Florian:** Only nine percent of services set a deployment environment. **Bianca:** And thirty-five percent of traces were single spans. Only about five percent crossed a service boundary. **Florian:** There were also pattern matches for bearer tokens and email addresses in a few services. Those still need a human review, so treat them as suspects for now. **Bianca:** So, what would you check in your own telemetry after hearing this? **Florian:** First, give every process a real service name. Defaults hide your biggest producers. **Bianca:** Second, look at what your background workers do per message. **Florian:** Third, turn access logs into metrics. And fourth, check whether your Collector is shipping its own logs twice. **Bianca:** The full written story is on the OllyGarden website. Thanks for listening. **Florian:** Until next time. --- # The PII leak was a logger setting, not a developer problem: what we found at a European digital-health platform OllyGarden Insights and Rose looked at staging OpenTelemetry data and code from a European digital-health platform. Most of the personal data in its logs came from one shared ORM setting. > **Anonymized, but real.** This story comes from a real OllyGarden customer engagement. To protect the customer, we removed their name, people, products, and anything else that could identify them, and rounded some numbers. Everything else happened as described. Published: 2026-10-03 Canonical URL: https://ollygarden.com/resources/case-studies/digital-health-pii-leak-was-a-logger-setting ## At a glance - **Company:** Small-to-mid-size digital-health platform with a small engineering team - **Region:** Europe - **Stack:** Node.js and TypeScript microservices with shared OpenTelemetry auto-instrumentation, a self-hosted open-source observability stack - **Scale:** 50 to 120 services; in staging, about 15 million log records and 28 million metric datapoints a day - **OllyGarden:** Insights, with weekly written assessments, and Rose reviewing the code - **Timeline:** April to June 2026 ## Key numbers - **17%** of services flagged for personal data in week one (12 of about 70) - **100k** log lines a week with personal data, from one service (One ORM logging setting) - **100%** of logs arrived with no severity (In every sample) - **1 day** for Rose to find auth tokens in logs (Confirmed by the customer) A European digital-health platform knew personal data might be in its logs, but not where it came from. Five weekly staging samples and a code review later, the answer was mostly a single shared setting: an ORM configured to log every SQL query with its values. On one service, that came to about 100,000 log lines a week carrying clinicians' personal data. ## The setup The platform runs a fleet of Node.js and TypeScript microservices, between 50 and 120 depending on the sample, instrumented through a shared internal auto-instrumentation package. Telemetry goes to a self-hosted open-source observability stack. The engineering team is small, and nobody owned observability full time. With security and privacy audits ahead, personal data in logs was the engineering lead's top priority. > However careful a team is, it never feels sure it isn't leaking secrets or personal data. > > — CTO (paraphrased) OllyGarden Insights analyzed short staging samples and sent five written assessments between April and June 2026, and Rose reviewed the code. This story covers what they found. ## Personal data in 1 in 6 services In the first weekly sample, Insights flagged personal data in 12 of about 70 services, or 17% of the fleet. As more services started sending traces, the count grew to 18 and then 25 over the next two weeks. The source was not careless developers. One shared ORM configuration logged every SQL query with its bound values at INFO level. On a service that handles clinician records, about 60% of the query logs in a 30-minute window touched personal-data columns: email, phone, full name, and address. Over a full week, that is roughly 170,000 query logs from one service, about 100,000 of them carrying personal data. Two more services used the same configuration. **Fix the setting, not the people.** When one shared setting causes most of a leak, the fix belongs in that setting, not in a reminder to developers to be more careful. Later samples found more: a bundle of customer personal data (email, name, phone, street, and postal code) in one service, authorization headers in three services, a date of birth on a trace attribute, and OAuth tokens captured verbatim in spans. ## Tokens in the headers Within about a day of being connected to the repositories, Rose found authentication tokens being written to logs inside HTTP request headers. The customer's DevOps lead checked the finding against their own logs, confirmed it, and called it critical. ## The hits we ruled out A scanner that cries wolf gets ignored. Several hits were checked and ruled out as false positives: git SSH URLs, millisecond timestamps that looked like phone numbers, and cloud service-account addresses. In the first week, traces were clean, so only the log pipeline needed work. One trace-side issue did show up: medical-image and signature file fetches were traced as one span per file, which created high-cardinality spans and exposed storage bucket paths. ## Logs with no severity Every log record in every sample, about 50,000 to 60,000 per sample, arrived with no severity. The bridge that converted the logs to OTLP was dropping the level. That one cause accounted for about 70% of all active insights in the first week, 90 of 128, and a single collector transform would close almost all of them. ## Where the volume came from - 16% of logs were exact duplicates in the first week, mostly from a secrets-cache loop that logged on every check. By June, the staging application logs were 84% to 99% reducible by deduplication. - One background worker produced about 60% to 65% of all log volume, three weeks in a row. - Once metrics arrived, the monitoring stack's own self-monitoring was about 40% of all metric volume. Scrape intervals were already efficient, so the lever was dropping series, not slowing scrapes. ## Gaps in the picture Tracing barely existed. In the first week, about 3 of 72 services sent traces, and every trace was a single span, so there was no distributed tracing at all. There were no metrics for the first three weeks, and deployment.environment.name was set on 2 of 72 services, so staging and production could not be told apart. Some checks came back clean: no metric attribute exceeded 1,000 distinct values, and no attribute values were oversized. ## What to check in your own telemetry - Before looking for careless code, look at shared settings. ORM and HTTP logging that records query values or headers can put personal data in every service that uses it. - Check that your log pipeline keeps severity. A bridge that drops it makes every log look the same. - Find the one service or worker that writes most of your logs. - Review PII hits before acting on them. Timestamps and service accounts can look like personal data. ## About these numbers All numbers come from short staging samples, about five minutes each, analyzed between April and June 2026, plus Rose's review of the code. Figures are rounded, and the weekly figures for the clinician-records service are extrapolated from a 30-minute window. Production was not analyzed. Several PII hits were checked and ruled out as false positives before reporting. ## Keep reading - [OllyGarden Insights](https://ollygarden.com/products/insights): Analyze and score telemetry quality. - [OllyGarden Rose](https://ollygarden.com/products/rose): Fix instrumentation at the source. - [The Instrumentation Score](https://ollygarden.com/resources/instrumentation-score): How the open 0–100 score for instrumentation quality is calculated, and every rule behind it. - [Where Sensitive Data Hides in OpenTelemetry Pipelines, and How to Find It](https://ollygarden.com/resources/blog/where-sensitive-data-hides-in-opentelemetry-pipelines): Trust store passwords in resource attributes, OAuth tokens in query strings, tax IDs in metrics: what we found in real pipelines, why tools cause most of it, and what privacy laws allow. - [Teach Your Coding Agent OpenTelemetry](https://ollygarden.com/resources/blog/teach-your-coding-agent-opentelemetry): We’re releasing our Agent Skills for OpenTelemetry as a mature, open source set: 14 skills that give any coding agent accurate, current OpenTelemetry knowledge. ## Listen to the story Bianca and Florian, who you might know from the OTel Drops podcast, tell this story in a short conversation. The voices are AI-generated; the facts are the ones on this page. [Episode (MP3)](https://ollygarden.com/audio/case-studies/digital-health-pii-leak-was-a-logger-setting.mp3) ### Transcript **Bianca:** Hi, I'm Bianca. **Florian:** And I'm Florian. You might know us from the OTel Drops podcast. Today we're telling another real story from OllyGarden's work. **Bianca:** Anonymized, so no names. And this one is about what OllyGarden found, start to finish. **Florian:** Here's the setup. A European digital-health platform. A small engineering team, a fleet of Node.js and TypeScript microservices, and nobody owning observability full time. **Bianca:** And security and privacy audits coming up. So personal data in logs was their top priority. **Florian:** Their CTO put it well, and I'm paraphrasing: however careful a team is, it never feels sure it isn't leaking secrets or personal data. **Bianca:** That's honest. And it's true for almost every team I know. **Florian:** So OllyGarden Insights looked at their staging samples every week. And in the very first one? **Bianca:** Personal data in twelve of about seventy services. Roughly one in six. **Florian:** That sounds like a lot of careless code. **Bianca:** That's the interesting part. It wasn't. Most of it came from one shared ORM setting. It logged every SQL query, with its values, at INFO level. **Florian:** So every query, values included, went straight into the logs. **Bianca:** On one service that handles clinician records, about sixty percent of the query logs touched personal data columns. Email, phone, full name, address. **Florian:** And over a week? **Bianca:** Roughly a hundred thousand log lines carrying personal data. From one service. And two more services used the same setting. **Florian:** So the fix isn't a training session for developers. It's one configuration. **Bianca:** Exactly. Fix the setting, not the people. **Florian:** Then Rose got connected to the repositories. **Bianca:** And within about a day, it found authentication tokens being written to logs, inside HTTP request headers. **Florian:** The customer's DevOps lead checked it against their own logs, confirmed it, and called it critical. **Bianca:** I also like what OllyGarden didn't report. Several hits turned out to be false positives. Git SSH URLs, service-account addresses, and millisecond timestamps that looked like phone numbers. **Florian:** A timestamp with a phone number's haircut. **Bianca:** Right. But a scanner that cries wolf gets ignored. So those were ruled out before they went into a report. **Florian:** Then the less dramatic, but very expensive, findings. Every single log record, in every sample, arrived with no severity. **Bianca:** None at all? **Florian:** None. The bridge converting their logs to OTLP was dropping the level. That one cause was about seventy percent of all their open insights in the first week. **Bianca:** And one collector transform would close almost all of them. **Florian:** Then volume. One background worker wrote about sixty percent of all the logs, three weeks in a row. **Bianca:** And a secrets-cache loop logged on every single check. That was most of their exact duplicates. **Florian:** When metrics showed up, about forty percent of them were the monitoring stack watching itself. **Bianca:** And tracing barely existed. A handful of services, and every trace was a single span. **Florian:** So, what would you check in your own telemetry after hearing this? **Bianca:** First, look at shared settings before you blame anyone's code. ORM and HTTP logging can put personal data in every service at once. **Florian:** Second, check that your log pipeline keeps severity. And third, find the one worker that writes most of your logs. **Bianca:** And fourth, review your PII hits before acting on them. Not everything that looks like a phone number is one. **Florian:** The full written story is on the OllyGarden website. **Bianca:** Thanks for listening. **Florian:** Until next time. --- # How a B2B SaaS platform cut logs 24% and metrics 25% in six weeks, without touching a single customer log A Latin American SaaS platform paid its observability backend per record. OllyGarden found where the volume came from and fixed it in their own OpenTelemetry Collector pipelines, with customer data off limits from day one. > **Anonymized, but real.** This story comes from a real OllyGarden customer engagement. To protect the customer, we removed their name, people, products, and anything else that could identify them, and rounded some numbers. Everything else happened as described. Published: 2026-10-03 Canonical URL: https://ollygarden.com/resources/case-studies/b2b-saas-cuts-logs-and-metrics-at-the-source ## At a glance - **Company:** Mid-size, multi-tenant B2B SaaS platform that runs its customers' workloads - **Region:** Latin America - **Stack:** Kubernetes on two public clouds, upstream OpenTelemetry Collector, a backend built for OpenTelemetry, billed per record - **OllyGarden:** Insights, telemetry assessments, and a six-week proof of value - **Timeline:** November 2025 to September 2026 ## Key numbers - **24%** fewer log records (Target was 20%) - **25%** fewer metric datapoints (Target was 10%) - **61%** less volume on dedup-eligible logs (Counts preserved) - **Zero** customer workload logs touched (Passed through one to one) The bill grew with every pod. A Latin American B2B SaaS platform paid its observability backend for every log record and every metric datapoint, and nobody on the team could see, at the source, what was driving the volume. Six weeks later, log records reaching the backend were down 24% and metric datapoints about 25%, and not one customer workload log had been touched. ## A bill that grew with every pod The company runs a multi-tenant platform that executes its customers' workloads on Kubernetes, across several production clusters on two public clouds. After moving away from a legacy APM vendor, it chose an observability SaaS built for OpenTelemetry that bills per ingested record. That model is fair, but unforgiving: every duplicate log line and every metric series nobody reads shows up on the invoice. The platform team suspected waste. Nobody could say where the volume came from, and more than half of their metric data arrived without a service name, so they could not even say which team was paying for what. > Once a log reaches the backend, the backend can't make it better. We want to fix telemetry where it's born. > > — Platform lead (paraphrased) That conviction shaped the whole engagement. Rather than add more filtering in the backend, the team wanted control at the edge, in their own Collector pipelines, where they decide what leaves the cluster at all. ## One rule that could not bend Most of the platform's log volume is customer workload output: the content of jobs that its customers run. That content may contain personal data, and for regulatory reasons it had to stay out of any reduction policy and out of any analysis. Whatever we changed, customer content had to pass through exactly as it came in. The security review was just as strict. Work started in staging and pre-production, and production samples were shared only after the contract and the security review had cleared. Then the team set the bar for a paid six-week proof of value, with pass/fail targets: at least 20% fewer log records and 10% fewer metric datapoints reaching the backend. ## What the first samples showed OllyGarden Insights analyzed recurring telemetry samples, first from staging and then from production. The staging assessment, based on about 1.8 million records sampled over a week, was blunt: - Metrics made up about 97% of all records, and more than half of the metric data had no service.name. Cost could not be attributed. - About 89% of logs were duplicates. Deduplication alone could remove about 70% of log volume. - About 29% of traces were single-span noise from cache health pings and liveness probes. - No signal set deployment.environment.name, and SDK versions were fragmented across services. Earlier in the pilot, something more urgent had already surfaced: passwords, usernames, API keys, and Authorization headers were leaking into logs from about five internal services. Insights flagged them, and the team fixed them at the source. **Metrics that carry no signal.** About 57% of the Kubernetes volume-metric series came from static mounts: config maps, secrets, projected volumes, and the downward API. Those volumes never change size, and the same metric family was scraped three times more often than needed. High-volume access logs from the API gateway and an HTTP trigger service, about 14% of log records, could become request-rate, error-rate, and latency metrics without losing anything an on-call engineer would look for. ## Six weeks, in waves We delivered the fixes as numbered waves of drop-in Collector configurations. Every wave was replayed against real production samples before it went anywhere near production. - Wave 1 cut the volume-metric scrape from three datapoints per minute to one. - Wave 2 converted API gateway access logs into metrics. - Wave 3 dropped the static-mount series and converted trigger access logs into metrics. - Wave 5 added a log pipeline with template mining, attribute stripping, and count-preserving deduplication, which keeps how many times an event happened while dropping the copies. - Wave 7 filtered pipeline self-noise on a newly onboarded cluster. Customer workload content was routed around every one of these pipelines and passed through one to one. ## The day the replay paid for itself Turning access logs into metrics is a well-known win, but it has a trap. One per-user path segment in the gateway routes was not templated, so a new log-to-metric rule copied real user email addresses into a metric route label. That was a privacy leak and an unbounded cardinality problem at the same time. The next sample caught it. A fix shipped right away, and the following day's sample confirmed zero personal data in the metric. Without the replay-and-verify loop, those addresses would have sat in a backend that nobody was auditing for them. ## Noise that hid an outage When a new cluster joined in June, about 78% of its log volume turned out to be the telemetry pipeline complaining about itself. An in-app agent could not reach its export endpoint and logged a stack trace on every pod, nonstop. Another 24% of the cluster's metrics was the logging stack observing itself. The noise was hiding something worse: the cluster's application traces and metrics were not reaching the backend at all. Wave 7 removed about 70% of the cluster's log records and about 24% of its metrics, touched zero business logs, and made the real export failure impossible to miss. ## The results In production, Wave 5 alone removed about 24% of log records overall and about 61% on the deduplication-eligible branch, consistently across two samples taken at midday and at midnight. Measured service by service, deduplication cut eligible log volume by 63% to 69%. The proof of value closed at 24% fewer log records and a 24.6% average reduction in metric datapoints, against targets of 20% and 10%. The customer accepted it and kept OllyGarden on for continuous optimization. An engineering lead on their side called it a success for both teams. ## What happened next, honestly A proof of value is a snapshot, and production keeps moving. After the six weeks, the production clusters moved off the vendor's operator and onto a self-managed upstream Collector, with OllyGarden checking each cutover. Those checks found the hidden tax of a Collector migration. Service-name resolution had lived in the operator's configuration and was never carried over: after the cutover, 99.5% of logs and 77% of metrics reported service.name as unknown, which breaks dashboards and cost attribution alike. Then, in August, the monthly backend bill still jumped by about a third, driven by incident-related metrics and deployments with hundreds to thousands of replicas. The analysis that followed showed that 10 metric groups made up about 48% of all metric volume, that self-monitoring was about 13% of metric datapoints, and that the median series sent about one datapoint per minute. The bill was driven by series count, not by scrape frequency. So the relationship changed shape. Wave 8 moved 20 slow Kubernetes metric families to one datapoint per minute, added a selective log deduplication policy, and delivered a narrowly scoped email-redaction rule for cloud load-balancer logs. The team now brings OllyGarden in early whenever the bill starts to move, and the backlog is real: restoring service and cluster names, investigating single-span traces, and a filter that would remove about 43% of the in-scope logs. ## What other teams can take from this - Decide what is off limits before you optimize. Routing customer content around every reduction made the rest of the work safe to do. - Replay every change against real samples. That is how an email address in a metric label was caught within a day. - Treat a Collector migration as a telemetry change. Logic that lived in a vendor operator, like service-name resolution, does not move by itself. - Count series, not scrapes. Cardinality drives a per-datapoint bill far more than scrape intervals do. ## About these numbers The proof-of-value results were measured on the pilot cluster with count-preserving pipeline measurements, not as a bill-to-bill comparison. Sample-based counts are subsampled, so this story reports ratios rather than absolute volumes. We do not publish dollar savings here: none have been confirmed, and the August bill increase shows why a single savings number would mislead. ## Keep reading - [OllyGarden Insights](https://ollygarden.com/products/insights): Analyze and score telemetry quality. - [The Instrumentation Score](https://ollygarden.com/resources/instrumentation-score): How the open 0–100 score for instrumentation quality is calculated, and every rule behind it. - [Reducing Log Volume with the OpenTelemetry Log Deduplication Processor](https://ollygarden.com/resources/blog/reducing-log-volume-with-the-opentelemetry-log-deduplication-processor): Your logs are probably at least 80% repetitive noise. Connection retries, health checks, heartbeat messages: the same log line repeated thousands of times per day. - [The Scrape Interval Nobody Chose](https://ollygarden.com/resources/blog/the-scrape-interval-nobody-chose): A wrong metric scrape interval inflates ingest, egress, and compute. Learn how to measure DPM and match intervals to the decisions each metric supports. ## Listen to the story Bianca and Florian, who you might know from the OTel Drops podcast, tell this story in a short conversation. The voices are AI-generated; the facts are the ones on this page. [Episode (MP3)](https://ollygarden.com/audio/case-studies/b2b-saas-cuts-logs-and-metrics-at-the-source.mp3) ### Transcript **Bianca:** Hi, I'm Bianca. **Florian:** And I'm Florian. You might know us from the OTel Drops podcast, but today we're here to tell you a story. **Bianca:** A real one. OllyGarden shared it with us, anonymized. So no company names and no people, but every number is real. **Florian:** Here's the setup. A mid-size B2B SaaS platform in Latin America. Multi-tenant, running customer workloads on Kubernetes, across two public clouds. **Bianca:** And an observability backend that bills per record. Every log line, every datapoint. So the bill grew with volume, and nobody could say what was driving it, because nobody had a view at the source. **Florian:** What I like about this team is their platform lead's instinct. They didn't want more filtering in the backend. Their argument was simple: once a log reaches the backend, the backend can't make it better anymore. Fix it where it's born. **Bianca:** That's the right instinct. But there was a hard constraint. Most of their log volume is customer workload content, the output of their customers' jobs. It can contain personal data. **Florian:** So that content was off limits. Not reduced, not analyzed, not touched. Full stop. **Bianca:** The security review was strict, too. They started in staging, and production data only flowed once the contract and the security approval were in place. **Florian:** Then they set a bar. A six-week proof of value, with pass or fail targets. Twenty percent fewer log records, and ten percent fewer metric datapoints reaching the backend. **Bianca:** So OllyGarden Insights started looking at samples. And the staging assessment was... a lot. **Florian:** Metrics were about ninety-seven percent of all records. And more than half of the metric data had no service name. Just "unknown". **Bianca:** Which means you can't attribute cost. You're paying for data, and you can't even say whose it is. **Florian:** It gets better. About eighty-nine percent of the logs were duplicates. **Bianca:** Eighty-nine?! **Florian:** And almost a third of the traces were single spans. Cache health pings, liveness probes. Noise. **Bianca:** And production? **Florian:** Production had a Kubernetes detail I love. More than half of the volume metric series came from static mounts. Config maps, secrets, projected volumes. They never change size. **Bianca:** So those series carry no signal at all. **Florian:** None. And they were scraped three times more often than needed. **Bianca:** So how do you fix all of that without breaking anything? OllyGarden shipped the work in numbered waves of drop-in OpenTelemetry Collector configurations. **Florian:** And here's the part the operator in me appreciates. Every wave was replayed against real samples before it went anywhere near production. **Bianca:** One wave cut those volume metrics from three datapoints a minute to one. Another turned high-volume access logs into request rate, error rate, and latency metrics. **Florian:** And then the replays earned their keep. One of the new log-to-metric rules put real user email addresses into a metric label. **Bianca:** Oof. How? **Florian:** One per-user path segment wasn't templated. So every email became a label value. A privacy problem and a cardinality explosion, in one line of config. **Bianca:** And the next sample caught it. A fix shipped, and the day after, zero personal data. **Florian:** Then a new cluster came online. About seventy-eight percent of its log volume was the pipeline complaining about itself. **Bianca:** Complaining about what? **Florian:** An in-app agent couldn't reach its export endpoint, so it logged a stack trace on every pod, nonstop. And that noise was hiding a real outage. That cluster's traces and metrics weren't reaching the backend at all. **Bianca:** That's the thing about noise. It doesn't just cost money. It hides the signal you actually need. **Bianca:** So, the six weeks. Did they hit the targets? **Florian:** They did. Twenty-four percent fewer log records, against a target of twenty. And about twenty-five percent fewer metric datapoints, against a target of ten. **Bianca:** And the log deduplication is my favorite part. It's count-preserving, so you still know how many times something happened. On the eligible logs, it collapsed volume by about sixty-one percent. **Florian:** And the customer workload logs? **Bianca:** Passed through one to one. Untouched. Exactly as promised. **Florian:** The customer accepted the proof of value, and the work kept going. **Bianca:** Now, this is the part I respect most about this story. It doesn't end with a perfect bill. **Florian:** No. After the proof of value, the clusters moved from the vendor's operator to a self-managed Collector. And service name resolution silently disappeared. **Bianca:** How bad? **Florian:** Ninety-nine and a half percent of logs, and seventy-seven percent of metrics, showed service name unknown. That logic lived in the operator's config, and nobody carried it over. **Bianca:** The hidden tax of a Collector migration. **Florian:** And in August, the monthly bill still jumped by about a third. Incident-driven metrics, and deployments with hundreds to thousands of replicas. **Bianca:** Which is the real lesson. The cost driver was series count. Cardinality. Not scrape frequency. **Florian:** So it stopped being a one-time project. The team now brings OllyGarden in early, whenever the bill starts to move. **Bianca:** So, what would I take from this? First, look for telemetry that never changes. Static series and repeated logs are the cheapest wins you'll find. **Florian:** Second, replay every change against real data before rollout. That's how you catch an email address in a metric label. **Bianca:** Third, protect your customers' data by design. Decide what's off limits first, then optimize everything else. **Florian:** And fourth, telemetry quality is a habit, not a project. **Bianca:** The full written story, with all the numbers, is on the OllyGarden website. Thanks for listening. **Florian:** Until next time. ## Blog articles # Introducing Minimum Viable Instrumentation Coding agents write the code and AI SRE agents operate it, so telemetry is the source of truth between them. Rose now brings low-volume, high-quality OpenTelemetry instrumentation to every repository. Published: 2026-10-09 Canonical URL: https://ollygarden.com/resources/blog/introducing-minimum-viable-instrumentation In 2026, companies generate code with coding agents and operate their systems with AI SRE agents. There are agents on both sides of the equation now: one writes the code, the other runs it. The engineers who direct those coding agents read less of the code than ever. When something breaks in production, the first responder is increasingly an agent connected to the observability backend, investigating before anyone is paged. What connects the agent that writes the code with the agent that runs it? Telemetry. People no longer read every line of code, but they, and their agents, can look at the telemetry. It has become the single source of truth about what is actually happening in production. Telemetry used to be something we looked at. Now it is the source of truth that AI acts on, so it has to be top-notch. That is why today we are launching **Minimum Viable Instrumentation (MVI)** in [OllyGarden Rose](https://ollygarden.com/products/rose), together with the news that OllyGarden has raised $4m from Next Frontier Capital, Grand Ventures, ACTAI, DIG Ventures, Datadog, Grafana Labs, and Dash0. You can read the details in our [press release](https://ollygarden.com/resources/press-releases/ollygarden-raises-4m-launches-minimum-viable-instrumentation). This post is about the product, and about why the old way of getting telemetry does not work anymore. ## High volume, low quality no longer works For years, the fastest path to telemetry was the wide net. Attach the Java agent, enable eBPF-based instrumentation, add instrumentation libraries, and data shows up for every request, query, and DNS lookup. When all you have is darkness, a floodlight is welcome. But these tools cannot know what is important to your business, so they capture everything. That is high-volume, low-quality telemetry. Here is what that looks like in practice. At one company we work with, a single batch job that ran for 59.7 seconds produced 633,718 spans through the Java agent. More than 90% of them came from the database layer. About 19% were orphaned from their trace. Not one span described the business operation the job performed. Our plan for the same job, built around its real boundaries, needs six spans. This was already expensive. Observability is the second largest infrastructure expense after compute, and in our audits we keep finding the same waste: 82% log duplication, 63% trace waste, and credentials or personal data leaking into telemetry at five out of six companies. In 2026, the problem compounds. Coding agents write most of the new instrumentation that ships to production, at machine speed, and they were trained on the bad patterns of the past. If agents help us write ten times more code, a wide-net approach gives us ten times more telemetry to store, query, and pay for. That volume of data was already hard to justify yesterday. It is worse tomorrow. The cost does not stop at the bill. When one agent writes bad telemetry and another agent reads it, nobody really knows what is happening in production. AI SRE agents take every signal at face value. Noisy, duplicated, or misleading telemetry makes them spend more time and more tokens to reach an answer, and sometimes they confidently fix the wrong problem. High-volume, low-quality telemetry costs more money and more resources in 2026, not less. ## Low volume, high quality, at 2026 speed The alternative has always been manual instrumentation: deciding in the code which operations deserve a span, which attributes they carry, and how context flows between services. It produces low-volume, high-quality telemetry, including the business context no library can guess, such as the installation ID that lets you pull every trace for the one customer whose install failed. Its problem was never quality. It was speed and scale. It takes OpenTelemetry expertise, and most organizations have only a handful of people with it, who cannot review every repository. Plain coding agents do not solve this on their own. When we tested them, they suggested outdated SDK versions and APIs, struggled with the SDK setup, and broke context propagation between services. But agents guided by real expertise change the equation. Today we can instrument source code with the same speed and effort a wide-net agent needed a few years ago, at a quality that is far better. If teams are going to write ten times more code, they cannot afford ten times more telemetry. They need low volume and high quality, delivered at the speed and scale of 2026. ## What Minimum Viable Instrumentation means Minimum Viable Instrumentation is the smallest set of telemetry that gives a team, and its agents, a foundation it can operate on. By default, that means the boundaries of a service are instrumented: the requests it serves, the outbound calls it makes to databases, queues, and other services, and the context that connects them. On top of that come the few business attributes that turn a generic trace into one you can search for during an incident. Rose goes deeper only where you ask it to. The word "minimum" is deliberate. More instrumentation is not better instrumentation. The goal is the telemetry you need, and nothing you don't. ## How Rose establishes the baseline Until now, Rose reviewed the instrumentation you already had and suggested how to improve it. With MVI, Rose also finds the parts of your code that production cannot see, and gives you a path to an instrumentation baseline. Rose does not start by adding an SDK. It starts by reading the code base and writing an instrumentation plan. The plan identifies the service boundaries and the external systems the code connects to, and decides which of those boundaries should be instrumented. Knowing which boundaries are instrumented is the core of MVI. Rose then sets up the OpenTelemetry SDK using declarative configuration, so the setup lives in a configuration file instead of language-specific code spread across the application. It implements the plan with targeted manual instrumentation, and tunes instrumentation libraries where they remain the better starting point, for example by turning off duplicate database spans or DNS resolution spans that nobody will ever query. Along the way, Rose adds business context. In one of our demo applications, a NestJS service with two OAuth sign-in flows, one for Apple and one for Google, the default instrumentation would not have separated them. Rose added a `provider` attribute to each flow, so a person or an agent chasing a sign-in issue can filter by provider and see whether only one of them is failing. Before it opens a pull request, Rose verifies its work. When it can start the application, it runs an OpenTelemetry Collector locally and sends sample telemetry to it, never to a remote backend, to confirm the instrumentation actually produces data. The result arrives as a pull request with the plan, the configuration, and the code changes, all plain OpenTelemetry in your own repository. Your engineers review it and decide what merges, as with every other change Rose proposes. **Find the services production cannot see.** OllyGarden Rose reviews the instrumentation in your repositories, and with Minimum Viable Instrumentation it also plans, implements, and verifies a baseline where none exists, as a pull request your developers review before anything merges. [Meet Rose](https://ollygarden.com/products/rose) ## An OpenTelemetry expert in every repository MVI completes the loop Rose started. For services with instrumentation, Rose reviews and improves what is there, and catches the regressions coding agents introduce. For services without it, Rose now establishes the baseline. Together, that puts an OpenTelemetry expert in every team, every repository, and every pull request, watching around the clock, even in organizations with only a handful of observability engineers. This works on top of the observability backend you already run. Better telemetry makes every backend more valuable, and it gives your AI SRE agents a source of truth they can actually trust. If you want to see the approach before you try it, our [Minimum Viable Instrumentation with AI Agents](https://www.youtube.com/watch?v=GqXic9ohr8o) session walks through Rose instrumenting an uninstrumented application, step by step. If you prefer to work with your own coding agent, much of the knowledge behind Rose is public: the [OpenTelemetry Agent Skills](https://github.com/ollygarden/opentelemetry-agent-skills) give any agent current, token-efficient OpenTelemetry references, and the [OllyGarden skills](https://github.com/ollygarden/skills) carry our opinions on instrumentation planning, declarative configuration, and manual instrumentation. See our [Agent Skills page](https://ollygarden.com/resources/agent-skills) for how to install them. ## Availability Minimum Viable Instrumentation is available today to all existing OllyGarden enterprise customers, with a free trial available to new customers. [Talk to us](https://ollygarden.com/contact) to start, or [connect Rose to your repositories](https://ollygarden.com/products/rose) and see what it finds in the instrumentation you already have. To our investors, our customers, and the OpenTelemetry community: thank you. Coding agents write the code, and AI SRE agents operate it. Our job is to make sure the telemetry between them is something both can trust. --- # Where Sensitive Data Hides in OpenTelemetry Pipelines, and How to Find It Trust store passwords in resource attributes, OAuth tokens in query strings, tax IDs in metrics: what we found in real pipelines, why tools cause most of it, and what privacy laws allow. Published: 2026-10-03 Canonical URL: https://ollygarden.com/resources/blog/where-sensitive-data-hides-in-opentelemetry-pipelines The evening we first ran our personal-data detector against a customer's telemetry, I sent their platform lead a short message: "I have your trust store password. So does SiloLogic." Nobody at that bank had typed the password into a log statement. A Java process had started with the password as a command-line flag, the OpenTelemetry Java agent had copied the full command line into a resource attribute, and from there it rode along on every span the service produced, all the way into SiloLogic, the bank's observability vendor. The story is real, but we anonymized it: SiloLogic is a fictional name, and we left out every detail that could identify the bank. Sensitive data in OpenTelemetry pipelines is more frequent than you might think, and more diverse in shape than you can imagine. Sensitive data in telemetry means any personal data, credential, or secret that ends up in a span, metric, or log record: a customer's email address, a national tax ID, an OAuth access token, a database password. Since then, we have found it at banks, fintechs, health platforms, and IoT companies, in traces, metrics, and logs alike. This post walks through what we found, why most of it is caused by misconfigured tools rather than careless developers, what the main data privacy frameworks allow and prohibit, and where to look in your own telemetry. ## What we found in real OpenTelemetry pipelines The table below lists findings from organizations we work with. We anonymized every one of them, and most came from staging or development environments, which matters for reasons we cover later. | What leaked | Where it showed up | What caused it | | --- | --- | --- | | Java trust store passwords | process.command_line resource attribute on every span | Password passed as a -D flag, captured by the Java agent's process resource detector | | OAuth authorization codes, access tokens, and JSON Web Tokens (JWTs) | url.query and url.full span attributes, sometimes URL-encoded twice | HTTP auto-instrumentation records the query string by default | | Authorization headers with bearer tokens | Log records and span attributes | HTTP header capture enabled for all headers | | Brazilian tax IDs: individual (CPF) and company (CNPJ) numbers | Log messages, url.full, and metric attributes | Developer log statements and identifiers in REST paths | | Email addresses | Metric attributes, span attributes, and logs | Identifiers chosen as dimensions, logged objects | | Names, emails, phone numbers, and street addresses of clinicians | Log records | An object-relational mapper (ORM) configured to log every SQL statement with its bound values | | Date of birth | A span attribute | Request object recorded as attributes | | Customer phone numbers | Prometheus metric labels | The phone number doubled as a money-transfer key | | Wi-Fi passwords | Span attributes | A decorator that recorded every method parameter | | Database connection strings with embedded credentials | A span attribute on hundreds of thousands of spans | Client library instrumentation recording the connection string | | Signed storage URLs | Span attributes | The signature in the URL granted 15 minutes of access to signed contracts | | Japanese tax numbers | Logs | Regional identifiers no existing redaction rule covered | Two things stand out. The first is how often the leak sits in a field nobody thinks about. Teams that review their log statements carefully never look at `process.command_line`, because nobody wrote it. The second is shape. A Brazilian CPF looks nothing like a JWT, which looks nothing like a street address, which looks nothing like a Wi-Fi password. A single regular expression, or a single list of forbidden attribute names, catches a fraction of it. ### A password nobody logged The trust store password is the story we tell most often, because we have seen it at more than one company. Java applications that talk to services over mutual Transport Layer Security (TLS) need a trust store, and the quickest way to configure one is `-Djavax.net.ssl.trustStorePassword=...` on the command line. The OpenTelemetry Java agent, like other process resource detectors, can record the full command line as `process.command_line` or `process.command_args`. Resource attributes are attached to every span, metric, and log the process emits, so one flag turns into millions of copies of the password in the backend. Someone might argue that anyone who can read a process's command line already owns the host. That is not true once the command line leaves the host. Telemetry backends have far broader access than production machines: developers, support engineers, vendors, and now AI agents can query them. ### Tokens in the query string Our most common finding is an OAuth token or authorization code in `url.query`. Every OAuth and OpenID Connect flow ends with a redirect back to the application, and that redirect often carries a `code`, an `access_token`, or an `id_token` in the query string. HTTP auto-instrumentation records the query string by default. The semantic conventions do ask instrumentations to redact query parameters by default, but the list covers signed cloud storage URLs: `AWSAccessKeyId`, `Signature`, `sig`, and `X-Goog-Signature`. An OAuth `code` is not on it. The sneakiest variant we found was a token inside a `redirect` parameter, URL-encoded inside another URL-encoded value. To read it, you have to decode the attribute twice. Most users we show this to do not remember that their login flow involves an OAuth exchange at all. ## Most leaks come from tools, not careless developers When teams find personal data in their telemetry, the first reaction is often to remind developers to be careful. In our experience, that is aimed at the wrong target. Most of what we find is a tool doing what it was configured to do. The Java agent records the command line because a resource detector is enabled. HTTP instrumentation records the query string because that is the default. At a European digital-health platform, one shared ORM setting logged every SQL query with its values at the INFO level, which produced about 100,000 log lines a week carrying clinicians' personal data from a single service; we wrote about it in [a case study](https://ollygarden.com/resources/case-studies/digital-health-pii-leak-was-a-logger-setting). Header capture configured with a wildcard records `Authorization`. A log bridge that serializes whole request objects records whatever those objects contain. None of these required a developer to write the word "password" anywhere. Developers do make mistakes as well. We found a CPF in a log message that announced a successful "CPF validation", Wi-Fi passwords recorded by a decorator that logged every method parameter, a customer email three levels deep inside a data transfer object whose `toString` method was logged, and a redaction rule that masked the field the team was worried about while an API key in the same log line went through untouched. These are real, but they are the minority. Fixing the shared setting removes far more personal data than asking each developer to review each log line. ## Why your SIEM did not catch it Some of these findings came from organizations that pay good money for specialized security information and event management (SIEM) tools, with teams whose job is finding this kind of leak. The tools were not broken. They were looking in places where OpenTelemetry data does not look like what they expect. SIEM tools are built around text logs. OpenTelemetry data travels as OpenTelemetry Protocol (OTLP) messages, usually Protocol Buffers, and it is structured: a value can sit in a resource attribute, a span attribute, an event, a metric data point attribute, or a log body. A SIEM rule that greps for `password=` never sees a resource attribute that was never written to a log file, and it does not decode a token that has been URL-encoded twice inside `url.full`. It also does not know that `url.query` is where OAuth redirects end up, or that `process.command_args` exists. We found these leaks because we know OpenTelemetry deeply. We know which instrumentation libraries record which attributes by default, which semantic conventions carry user input, and which resource detectors copy host details into every signal. Knowing where to look matters more than the sophistication of the pattern matching. That knowledge is what the Suspicious PII Leakage insight in [OllyGarden Insights](https://ollygarden.com/products/insights) encodes: it checks attribute names and values across traces and logs, URL query strings included. ## Where to look in your own telemetry If you want to start a review today, these are the places that produced the most findings for us: - `process.command_line` and `process.command_args` resource attributes, especially on Java services - `url.query` and `url.full` span attributes, decoded at least twice - `http.request.header.*` and `http.response.header.*` attributes, especially `authorization`, `cookie`, and `set-cookie` - `db.query.text` (formerly `db.statement`) and ORM or driver logs that include bound parameter values - `exception.message` and `exception.stacktrace`, which often embed the input that caused the error - Log bodies from frameworks that serialize whole objects, such as request bodies, DTOs, and HTTP client debug logs - Metric attributes, where an email address or tax ID also causes a cardinality problem - Messaging payloads and generative AI prompts recorded as span attributes or events Attribute names help, but values matter more. A key called `customer.reference` can hold an email address, and a key called `id` can hold a tax ID. ## What data privacy frameworks allow and prohibit Privacy law does not ban personal data in telemetry. It requires that every piece of personal data you process has a reason, and that you process no more than that reason needs. The frameworks below are the ones our customers ask about most. This is a practitioner's summary, not legal advice; your privacy officer or counsel makes the call for your organization. | Framework | Where it applies | What it says about data like this | | --- | --- | --- | | General Data Protection Regulation (GDPR) | European Union and European Economic Area | Personal data needs a lawful basis (Article 6), must be limited to what is necessary (data minimisation, Article 5), kept no longer than needed, and secured. Health data is a special category (Article 9). Breaches must usually be reported within 72 hours (Article 33). | | UK GDPR and Data Protection Act 2018 | United Kingdom | Keeps the EU GDPR's principles after Brexit: lawful basis, data minimisation, storage limitation, and security. Breaches go to the Information Commissioner's Office (ICO) within 72 hours. Fines reach £17.5 million or 4% of worldwide turnover, whichever is higher. | | Privacy Act 1988 and the Australian Privacy Principles (APPs) | Australia | Collect personal information only when reasonably necessary for your functions (APP 3), protect it and destroy or de-identify it when no longer needed (APP 11). Tax file numbers have their own rule that limits their use to tax purposes. The Notifiable Data Breaches scheme requires an assessment within 30 days. Penalties for serious interferences reach the greater of 50 million Australian dollars, three times the benefit gained, or 30% of adjusted turnover. | | Lei Geral de Proteção de Dados (LGPD) | Brazil | Modeled on the GDPR, with similar legal bases and necessity principle. A CPF is personal data. Fines reach 2% of revenue in Brazil, up to 50 million reais per infraction. | | California Consumer Privacy Act (CCPA), as amended by the CPRA | California | IP addresses and email addresses are personal information. Social Security numbers and account logins with passwords are "sensitive personal information". Collection must be reasonably necessary and proportionate to its purpose. | | Health Insurance Portability and Accountability Act (HIPAA) | United States health care | Protected health information includes names, dates of birth, email addresses, and IP addresses linked to health data. The minimum necessary standard applies, and a vendor that stores it needs a business associate agreement. | | Payment Card Industry Data Security Standard (PCI DSS) 4.0 | Anyone handling payment cards | Card verification codes and other sensitive authentication data must never be stored after authorization. Card numbers must be unreadable wherever they are stored, and a telemetry backend that holds them is in scope for the audit. | Credentials such as passwords, tokens, and connection strings are a separate category. They are rarely personal data, but every security standard and audit, from ISO 27001 to SOC 2, expects you to keep them out of logs. They are also the leak an attacker can use immediately. ### Client IP addresses are fine, until you can tie them to a user IP addresses are the most common debate we have with customers. Under the GDPR, and the UK GDPR that mirrors it, an IP address is usually personal data: the Court of Justice of the European Union ruled in 2016 (Breyer, C-582/14) that even a dynamic IP address is personal data when the site operator has legal means to identify the person behind it. The same regulation also recognizes network and information security as a legitimate interest (Recital 49). Keeping `client.address` on a span to investigate abuse, rate limiting, or an attack is usually easy to justify, especially with a short retention period. The picture changes when the same span, or the same trace, also carries `enduser.id`, an email address, or an account number. At that point the telemetry is a record of which known person did what, from where, and when. That record needs its own justification, and it is a much harder one to write. ### Every piece of personal data needs a justification Frameworks allow some personal data when it is necessary for the business operation. A payment service may need an account identifier to trace a failed transfer. A support team may need a customer ID to find a user's request. What they do not allow is personal data because it was convenient, or because nobody turned the default off. In practice, this means each personal attribute in your telemetry should have an answer to three questions: why it is there, who can see it, and how long it stays. If nobody can answer them, the attribute should go. There is also a timing aspect. A leak you do not know about is a problem. A leak you do know about, and leave in place, is a decision, and auditors treat it as one. Finding sensitive data creates an obligation to act on it. ## We rarely see real sensitive data, and we do not need to There is an irony in this work. We have found sensitive data at almost every organization we have worked with, but we have seen very little real sensitive data ourselves. We are happy to analyze staging data. We do not need production telemetry to find a production leak. If it looks like sensitive data, it might be sensitive data. A span attribute named `customer.name` with the value "Jane Doe" is clearly test data. The application that records it, though, will record real customer names the moment it runs in production. The same code path, the same instrumentation, and the same defaults produce the same attribute. A trust store password from a staging environment usually means the production deployment passes its own password the same way. This also means detection has to tolerate false positives. In one assessment, millisecond timestamps looked like phone numbers, and cloud service-account addresses looked like personal email addresses. Every hit needs a human or an agent to confirm it before anyone acts on it. **Ready to discover which sensitive data you are leaking?.** OllyGarden Insights inspects the traces and logs you already send, including resource attributes and URL query strings, and flags suspicious PII leaks per service. Staging data is enough, and the free plan includes PII leakage detection. [Explore Insights](https://ollygarden.com/products/insights) ## Summary Sensitive data in OpenTelemetry pipelines is common and comes in many shapes: trust store passwords in resource attributes, OAuth tokens in query strings, tax IDs and email addresses in metric attributes, and personal data in SQL logs. Most of it comes from tools doing what they were configured to do, such as auto-instrumentation defaults, ORM logging, and header capture, and a smaller share from developer mistakes. SIEM tools tend to miss it because they read text logs, not structured OTLP data. Privacy frameworks such as the GDPR, the UK GDPR, the Australian Privacy Act, LGPD, CCPA, HIPAA, and PCI DSS allow personal data only when it is necessary and justified, and an IP address that can be tied to a user ID needs a stronger justification than one that cannot. Staging data is enough to find these leaks, because the same code records the same fields in production. --- # Observability for Humans and Agents in the AI Age OllyGarden and Bronto on where AI agents help with instrumentation and analysis today, and why they only work when the telemetry is intentional and reachable. Published: 2026-09-28 Canonical URL: https://ollygarden.com/resources/blog/observability-for-humans-and-agents-in-the-ai-age Look around and it can feel like everyone is using LLMs for everything. In software, that impression is close to true. Teams are not only building applications with agents, they instrument them with agents, and when the application goes live another set of agents handles the incidents. You might already live in that reality, with agents across your whole software development lifecycle. You might be at the other end of the spectrum, doing all of this without AI. Not by hand, since you already have a high degree of automation in software, just without a model in the loop. Or you sit somewhere in between and ask the obvious question: what is the next thing worth handing to an agent, and which parts are a hype trap? We want to answer that in the one area where we can claim to be subject matter experts: observability. Two pieces of it in particular. - **Instrumentation.** The code you add to your software so it emits telemetry, so that someone can later ask questions about it. - **Analysis.** The work of using that telemetry to answer those questions: to fix issues, to prevent them, and to make the system better. Our companies sell products in both halves, and we obviously think they are good. This post is written so you can act on it either way, whether or not you ever become a customer of OllyGarden or [Bronto](https://bronto.io). Everything here works on plain OpenTelemetry. We believe AI agents can help with both instrumentation and analysis, but they only work well when the telemetry they rely on is intentional, trustworthy, and easy to access. So: what are the levels at which you can put LLMs to work on your instrumentation and your analysis? ## How We Use Agents for Instrumentation at OllyGarden The short version of what works for us (and probably for you, too): - Use coding agents during local development to facilitate the manual instrumentation process Be careful with the business context: deciding what matters is still (mostly) on you - Use agent skills to improve coding agents Enrich them with your custom conventions and guidelines - Make instrumentation part of the SDLC Agentic PR reviews with a focused look at instrumentation, so blind spots and over-instrumentation ideally don't reach main - Set up a continuous background agent focused on instrumentation across the whole codebase If you want to take it to the next level: let it create the fixes autonomously ### Auto-Instrumentation Most codebases that use OpenTelemetry get it through auto-instrumentation: eBPF ([OBI](https://opentelemetry.io/docs/zero-code/obi/)), monkey patching ([the Python agent](https://opentelemetry.io/docs/zero-code/python/)), or bytecode manipulation ([the Java agent](https://opentelemetry.io/docs/zero-code/java/agent/)). This works, and it is a quick way to get baseline telemetry about a running application. But auto-instrumentation comes with an inherent drawback: it captures generic data to get you started, and it cannot capture business context. The result is high volume and low signal: you can see that an HTTP call happened, but not which customer, which plan, or which feature flag it ran under. ### Manual Instrumentation As the name suggests, manual instrumentation is the opposite: it puts engineers in control of deciding which telemetry is useful and worth collecting. Could this information be relevant during a 3 a.m. outage? If yes, instrument it proactively so troubleshooting is easier when it matters. Could an attribute expose sensitive data to the backend? If there is any doubt, redact or sanitize it at the source before it is emitted. While the theory sounds promising, the reality is that manual instrumentation is not trivial. Done right, it requires consistency across application boundaries, correct context propagation, OpenTelemetry domain knowledge, and, most importantly, engineers with the time and knowledge to maintain it. ### Instrumentation with AI Agents But code is cheap now, and to some extent, so is manual instrumentation. For an AI agent, there is little difference between adding a new feature to a codebase and instrumenting an application with OpenTelemetry. At the end of the day, it is "just code." So if code, and therefore in some sense manual instrumentation, has become cheap, does that mean instrumentation is solved? Unfortunately, no. At least not yet. [A recent study](https://arxiv.org/pdf/2607.05785) found that LLMs still struggle to decide what to instrument and how to instrument it effectively. They have learned that exceptions and network calls matter. But they are not reliable at deciding which business context matters. Fortunately, there are a few things you can do to improve an agent's ability to instrument your codebase. An obvious, yet impactful, improvement is to provide agents with dedicated agent skills. One example is the [OllyGarden OpenTelemetry agent skills](https://github.com/ollygarden/opentelemetry-agent-skills). These skills provide curated knowledge, guidance, and pointers that steer agents in the right direction, and they make sessions cheaper and faster along the way. The *otel-semantic-conventions* skill, for example, helps the agent navigate the fairly large semantic convention registry without pulling the whole thing into context, so it finds the released attribute names and span naming rules it needs at a fraction of the tokens. The language skills such as *otel-go* point it to the right contrib instrumentation libraries like *otelhttp* and *otelgrpc* instead of hand-rolled spans. You can load them directly into your agents, or use them as a baseline and extend them with organization-specific context, conventions, and instrumentation guidelines. This custom knowledge does not exist in the LLM's training data, so the model cannot know it unless it eventually picks it up on its own, or better: you provide it. Based on our experience, supplying this custom domain context is one of the highest-leverage ways to improve the instrumentation capabilities of coding agents. At OllyGarden, we like to incorporate agents into different phases of the SDLC (Software Development Lifecycle). The most obvious one is, of course, during local development, but there is more. As mentioned earlier, manual instrumentation is business-relevant code and should be treated accordingly. Just as your agent reviews code changes before merging them into main, it should also take a focused look at the existing, or missing, instrumentation in a PR. If a new feature could leave you flying blind after it is merged, you, or in this case your agent, can catch that before it happens. The same applies to over-instrumentation. There are many good AI code review tools out there, and most of them already catch basic instrumentation issues. We noticed that custom instructions can push them further. Unfortunately, some things inevitably slip through, because a PR only shows a small part of the picture: the instrumentation surface of a change sometimes spans (pun intended) the whole codebase, and a diff alone is not enough context to judge it. That is why we believe instrumentation additionally benefits from an agent (or multiple ones) that runs continuously and autonomously in the background and watches how your instrumentation evolves alongside the codebase. It surfaces gaps and over-instrumentation before they hurt you. Looking further ahead, the natural next step is closing the loop: the same process raising PRs on its own to optimize the instrumentation. A human should still review and manually approve those PRs since agents love to over-instrument or apply inconsistent conventions. ![Auto-instrumentation and manual instrumentation feed four agent steps: local development with skills, PR review, a background agent watching the whole codebase, and autonomous PRs, with a human review before every merge](https://ollygarden.com/blog-images/observability-for-humans-and-agents-in-the-ai-age/image-1.png) Overall, introducing manual instrumentation to your codebase has already gotten a lot easier thanks to AI agents. However, applying *purposeful* and *consistent* manual instrumentation in particular still requires oversight, guidance, and time. **Put an instrumentation agent on your repos.** OllyGarden Rose connects to your GitHub repositories, reviews code for missing or low-value telemetry, explains each finding, and prepares source-level fixes for your developers to review before anything merges. [Meet Rose](https://ollygarden.com/products/rose) ## How We Use Agents for Analysis at Bronto The primary use case for observability data has always been incident response. Something behaves in a way you did not want, someone gets notified, and that someone starts digging: suspicious logs, slow traces, a metric bending the wrong way. It is tedious work, and it gets slower as the environment gets bigger. The gap between "the issue appeared" and "we know why" grows with the size of your system. When LLMs arrived, the obvious first move was to put a chat box in the observability product. You could ask it to explain what you were looking at, to find the evidence you needed, or to produce a set of hypotheses about the root cause. If you have never tried this workflow, start there. If your observability solution has no built-in chat, connect a general-purpose model to it over MCP or a CLI and ask the questions you would have clicked your way to: what caused the spike at 3pm, why is the authentication service slow, and summarize every log from the last thirty minutes. Do not stop at incidents. The same interface answers questions about bottlenecks and optimization: which service consumes the most time on the critical path of a payment transaction? And if your backend keeps data long enough to make the question meaningful, which services were used less over the last three months? You can also have the model build dashboards for you, or vibe code an entire custom interface on top of your data. All of it by asking questions, and all of it faster than doing it by hand. Run that way for a while and you notice the repetition. The same questions, the same context, the same instructions and restrictions typed again every session. That is what skills are for: markdown files you write once and hand to the model whenever it does a particular task. Write your own, or start from the examples the community publishes. At this level analysis starts to feel like programming. You tell the model up front whether this session is incident response, incident prevention, or optimization, and it comes with the right assumptions. It is still reactive, though. A human has to walk over to the machine and ask. Once you know where the model helps and where it does not, you can drop the reactive part. Enrich your alerts with LLM-powered runbooks: for a defined set of well-understood alerts, add a trigger that hands the alert to an agent with instructions to investigate. Give it skills, a few CLI tools, and the MCP servers it needs, and it comes back with hypotheses and candidate fixes on its own. Your job becomes reviewing the output, accepting what holds up, and tuning the agent for next time. This is what the first generation of "AI SRE" products did, at its core: a domain-specific agent with defined inputs, tools, and outputs. Some observability platforms ship it as a feature. You can also build your own quickly with a workflow builder like n8n or an agent framework such as AWS Strands Agents. ![Four maturity steps for analysis, from manual investigation to a chat interface, skills, and an agentic workflow, all depending on alert quality and on tools and data, which together decide investigation quality](https://ollygarden.com/blog-images/observability-for-humans-and-agents-in-the-ai-age/image-2.png) Follow that path and you have gone from manual, to chat, to skills, to an agentic workflow. From here the remaining gains are almost entirely about inputs, tools, and the quality of what you feed the thing. Those choices matter far more than which model or which harness you picked: - **Inputs.** Alerts were a hard problem when humans were the only consumers, and they do not get easier. They get more expensive. An alert storm that fans out into hundreds of agent investigations will drain a token budget in an afternoon. You can gate it with suppression, or put a triage agent in front, but the real fix is the same as it always was: alert on a clear signal. - **Tools and data.** Your agent can only find what is in your data, and this is where the two halves of this post meet. Instrumentation decides whether the answer exists. Your backend decides whether the agent can reach it, how fast, and at what fidelity. An agent that has to wait on a rehydration job, or that gets handed a one-percent sample and asked to explain a specific customer's failed request, will do what agents do when the evidence is thin: it will produce a confident, well-written hypothesis that happens to be wrong. ## From "Human Only" to "Mostly Agents" We hope this helps on the way from doing observability by hand to having most of it done by agents. If you are an SRE or in ITOps, this is not the part where your job disappears. It changes shape. Instead of working through the tasks yourself, you orchestrate the agents that do, and your real work becomes improving that workflow by upgrading the building blocks underneath it, one at a time, as you find the weak one. You can build some of those blocks yourself. You can also get help. OllyGarden's [Rose](https://ollygarden.com/products/rose) works on the instrumentation side. It reviews your OpenTelemetry, adds what is missing, and fixes what is off, running continuously in the background, so the telemetry your agents read was written on purpose. [Bronto](https://bronto.io) works on the other side: full fidelity with no sampling, twelve months of hot retention by default, sub-second search across petabytes, and MCP so your logs, metrics, and traces are a first-class tool call for whatever agent you point at them. Site reliability engineering has always been about making systems reliable and scalable through better automation. Using agents for it is just what that looks like in 2026. We shared our perspectives as experts on instrumentation (OllyGarden) and analysis (Bronto), but we are curious: what is your perspective? What works for you, and what doesn't? Do you agree, or did we miss something important? --- # What's changed about OpenTelemetry vendor lock-in since 2024 OpenTelemetry still does not make dashboards portable, but agents, CLIs, and MCP servers have made switching observability backends noticeably less painful. Published: 2026-08-21 Canonical URL: https://ollygarden.com/resources/blog/opentelemetry-vendor-lock-in-since-2024 In late 2024, David Allen and I co-authored a post on the Grafana blog about [OpenTelemetry and vendor neutrality](https://grafana.com/blog/2024/09/12/opentelemetry-and-vendor-neutrality-how-to-build-an-observability-strategy-with-maximum-flexibility/). The argument was straightforward: OpenTelemetry removes vendor lock-in at the instrumentation and collection layers, but the storage, visualization, and alerting layers are a different story. Most of that argument still holds. What has changed, and changed quickly, is the practical experience of moving from one observability vendor to another. In 2025 and 2026, that experience became noticeably less painful, and the reason has less to do with OpenTelemetry itself than with how engineers now interact with their telemetry. It is worth being precise about what was ever promised. Nobody in the OpenTelemetry community claimed that switching vendors would be easy or effortless. OpenTelemetry came with a narrower promise: the lock-in that lived in the instrumentation and collection layers, where proprietary agents tied data collection to a specific backend, would be removed. Storage, visualization, alerting, dashboarding, and everything that sits on top of the raw telemetry are deliberately outside the project's scope. They always were. Treating OpenTelemetry as if it guaranteed zero lock-in across the entire stack is reading a promise that was never made. ## Data portability, not dashboard portability The part of that post that has aged best is the observation that an engineer's experience with a tool is not portable. OpenTelemetry makes your data portable. It does not make your familiarity portable. If you have spent two years learning where to look in a particular backend, which queries answer which questions, and how to read a specific waterfall view under pressure, that knowledge does not transfer when you switch. It has to be rebuilt, and rebuilding it takes real time. The same is true for the artifacts a team accumulates around a backend: dashboards, saved queries, alert definitions, runbooks that reference specific views. None of that moves with the data. The sunk cost of all that work is a genuine form of lock-in, and it is the form OpenTelemetry was never going to solve, because it lives above the layer OpenTelemetry standardizes. ## What changed What shifted is the interface between engineers and their telemetry. For most of observability's history, that interface was a web UI: you logged into a backend and looked at dashboards someone had built. The dashboards were the product, and rebuilding them somewhere else was the cost of leaving. Engineers living a little ahead of the curve now interact with telemetry differently. At OllyGarden, we increasingly reach for coding agents wired into Model Context Protocol (MCP) servers and command-line interfaces (CLIs) rather than opening a dashboard first. Tools like `gcx` or `pup` let us pull telemetry, connect it to other systems, and reshape it on the fly. MCP servers and CLIs make moving data between systems, and connecting systems together, easier than it has ever been. When the interface is a CLI and an agent rather than a vendor's bespoke UI, the question of which backend stores the data starts to matter less. A concrete example: I have a daily digest skill that, among other things, gives me the state of our critical services every morning. It tells me which customers deserve attention based on the traffic they are sending and, just as importantly, the shape of that traffic. Are they sending more logs today than they ever have? Did their metrics drop off? Instead of opening a static dashboard and scanning for anomalies myself, I get a pre-triaged "what's important right now" view, assembled by an agent against the raw telemetry. The shift goes further than telemetry, though, and that is the part that matters most for how a workday actually flows. Because the operational picture now arrives through the same interface as everything else, it gets triaged alongside the rest of my work rather than in a separate tab. Is there an urgent Linear issue waiting for me, or is a system on the verge of breaking? Should I be reading an email thread instead, or does a Slack message need my attention right now? Those used to be questions I answered by checking several different places, each with its own UI and its own mental context switch. Now there is one interface to all of them, and the agent helps me weigh a service trending toward trouble against a customer email and an open issue, in one place, with one ranking. ## The caveat This is not a story about agents replacing observability backends. The agents are not perfect. They require constant supervision and double-checking, and they make mistakes that a human reading a dashboard would (hopefully) not. Having access to the raw telemetry does not replace the real engineering that goes into a full observability solution: the curation, the correlation, the years of accumulated judgment about what matters. We are nowhere near the point where raw data access makes a mature backend irrelevant. What we are seeing is that access to the raw telemetry is more valuable, and more usable, than it used to be. The case in point is recent and direct: we experimented with moving from one observability vendor to another. The result was not perfect. But it was good enough that we did not feel the same level of lock-in we would have felt a couple of years ago. The data came across, the agents and CLIs let us keep working in familiar ways, and the parts that were hard were the parts the 2024 post predicted would be hard: relearning a new tool's way of surfacing information. The boundary I would draw today is between consuming telemetry and operating the infrastructure behind it. Teams can already build observability tools around their own workflows and preferences. For bounded failures, especially in low-criticality services, a capable model can investigate the telemetry, identify a likely cause, and sometimes arrive at a pull request. We have done this ourselves. Manual, dashboard-first triage is no longer the only practical default. What has not become easy is maintaining a scalable and reliable telemetry store and query engine. Agents still need durable data, fast queries, and trustworthy evidence against which their conclusions can be checked. Proactive alerts also remain necessary; the agent-generated morning brief complements them rather than replacing them. Agents are changing the consumption and triage layers, not eliminating the backend that makes those workflows possible. **Check the telemetry your agents will read.** OllyGarden Insights analyzes the traces, metrics, and logs you already send and flags per-service problems such as noisy signals, high cardinality, and incomplete instrumentation before agents start reasoning over that data. [Explore Insights](https://ollygarden.com/products/insights) ## Where this goes next Eighteen months ago, nobody could have predicted where we would be today, and it is just as impossible to predict where we will be eighteen months from now. One thing seems safe to bet on, though. OpenTelemetry's work on standardizing instrumentation and collection, combined with the increasingly important standardization of the shape of telemetry through semantic conventions, means that good telemetry will be even easier for agents to consume in the future. Consistent attribute names and predictable structures are what an agent needs to reason about data it has never seen before. As a result, telemetry is becoming accessible to more than dedicated observability specialists. When agents can reach it through application programming interfaces (APIs), CLIs, and MCP servers, the same data can inform software development, capacity planning, customer support, and routine operational decisions. Portability therefore matters not only because it lowers switching costs, but because it allows telemetry to participate in more workflows. We are starting to see artificial intelligence (AI) agents for site reliability engineering (SRE) become useful for triage and remediation. The next step is to use telemetry for one of observability's original promises: improving running systems, rather than only helping people explain why they failed. Diagnosis produces an answer. A closed loop uses that answer to make the system better. Experiments are already moving in that direction. Autonomous agents continuously query telemetry, form hypotheses, make changes, run tests, open pull requests, merge and deploy within defined guardrails, observe the results, and roll back when a new version does not improve on the one before it. Then the loop starts again. This may sound too futuristic to some teams, but it already describes experiments happening today. The lock-in conversation has been stuck on the cost of leaving a backend. The more interesting shift is that standardized, portable telemetry is becoming a substrate for systems that do more than explain failures. They can continuously improve the software producing that telemetry. --- # The Scrape Interval Nobody Chose A wrong metric scrape interval inflates ingest, egress, and compute. Learn how to measure DPM and match intervals to the decisions each metric supports. Published: 2026-07-31 Canonical URL: https://ollygarden.com/resources/blog/the-scrape-interval-nobody-chose Ask an engineering team why their metrics arrive every 10 seconds and the answer is usually silence. Nobody decided that. A Helm chart shipped with a default, an agent came preconfigured, and nobody ever touched the receiver's `collection_interval`. That unexamined number now multiplies across every metric series, every pod, and every month of the observability bill. The unit that matters here is datapoints per minute (DPM). A series scraped every 10 seconds produces 6 datapoints per minute, or 8,640 per day. That is the quantity an organization buys, whether its platform prices it directly or hides it inside ingest volume. Across the organizations sending telemetry through our pipelines, wrong scrape intervals, and the closely related habit of emitting metrics nobody can act on, are the single most common source of wasted metric spend. The waste is not subtle. In the measurements below, roughly two-thirds of one organization's metric stream repeated unchanged values throughout the capture. ## The visible cost: the bill Most metric platforms charge in one of two ways: per ingested datapoint, often per million datapoints, or per active time series. A wrong interval inflates the first directly. Scraping at 10 seconds instead of 60 costs six times as much for a metric whose value changes once an hour. The second model is inflated by the sibling problem: emitting series that should not exist at all, such as state metrics that expand one enum into a dozen always-zero series per container. Both models respond to the same fix, applied at the source. Stop emitting the series that carry no signal, and slow down the ones that change rarely. This is a configuration change, not an engineering project. Receivers expose `collection_interval`, SDKs expose export intervals, and the OpenTelemetry Collector's `intervalprocessor` can thin what cannot be changed at the origin. ## The hidden costs: egress and compute The bill is the visible part. Even on platforms that do not charge per datapoint, wrong intervals cost money in two places that rarely appear in an observability business case. The first is network egress. Metrics leave the cluster, cross a network boundary, and land at a backend. Every cloud provider meters that traffic. In the experiment below, one organization's metric stream produced 12.4 GB of compressed egress per day. After source-side fixes, it produced 5.2 GB. That difference is paid whether or not the vendor charges for ingest. The second is compute at the origin. Every datapoint is produced, serialized, compressed, and shipped by CPUs inside the organization's pods, on nodes it pays for. Collectors that forward six times more datapoints than necessary need more memory and CPU, and they compete for those resources with the workloads they observe. This cost responds to cleanup immediately. At a large bank we work with, dropping health-check noise at the source cut the inbound volume of its central collector deployment by 70 to 75%. The deployment itself went from 25 replicas at 7 GB of memory each to 10 replicas at 3 to 5 GB. The observability pipeline had been consuming a mid-sized cluster's worth of compute to move data nobody wanted. ## The other ditch: intervals too long to alert on Interval choice has two failure modes, and the second one is worse than a large bill. If a metric feeds an alert that evaluates every minute, but the metric arrives every five minutes, the alert fires late or flaps on stale data. We see this less often than over-scraping, but it is the reason the answer to "what interval should I use" is not "as slow as possible." The interval is a decision per metric family: fast enough for the alerts and dashboards that consume it, and no faster. A request-latency histogram powering a critical service-level objective may deserve a short interval. A filesystem-capacity gauge may not. A rule of thumb makes this concrete: aim for five samples within the alerting period, enough to establish a trend instead of reacting to a blip. If the objective is reacting within one minute of a service becoming slow, the metric behind that alert needs five datapoints per minute, which means a 12-second interval. For a highly critical endpoint, that cadence is justified, and the organization pays for it knowingly. For a slow-moving target like disk usage, one datapoint per minute limits the sampling-related detection delay to roughly one minute. Whether that is safe depends on the worst-case fill rate: an alert at 80% is early enough only if the disk cannot plausibly consume the remaining 20% before the team can respond. ## Fixing at the source: two field cases An integration platform company we work with had a metric stream dominated by synthetic state metrics: Kubernetes status enums expanded into one series per possible state per container, `*_info` metrics that exist only to carry labels, and network-error counters flat at zero. The synthetic group alone was 12.5 to 14% of all metric datapoints. Series whose value never changed during our measurement window accounted for 65 to 75% of the stream. A flat sample alone does not prove that a metric is safe to remove, because it may matter during an incident outside the capture window. Before disabling a metric at the receiver or dropping it in the collector, review the alerts and dashboards that use it and confirm its purpose with the owning team. If nobody can establish whether it will matter later, reduce its scrape frequency instead of deleting it. This organization had already replaced the component responsible once, but the replacement reintroduced the same pattern under different metric names. Volume problems recur unless the review does too. At a large bank, the leverage was concentration. A single Prometheus federation endpoint accounted for 28.8% of the organization's entire metric volume, including 74,000 series of one HAProxy counter family. One `scrape_interval` line in one configuration file governed almost a third of what the organization paid for metrics. The same sample held a 929-series metric family scraped every 30 seconds whose value changed on 0.2% of observed scrapes. When the biggest offenders are this concentrated, stretching a handful of intervals moves the whole bill. ## What one real payload shows Claims like these deserve a measurement, so we ran one. We took a capture of a financial services company's metric stream from a non-production environment, roughly 90 minutes and 66 million datapoints, and analyzed every series for scrape cadence and information content. Two findings stood out. First, 0.9% of the series, emitted every 2 to 5 seconds by an internal framework's stats sender, carried 20.6% of the entire volume. The dominant family was a counter that never advanced, emitted 20 times per minute. Second, 65% of all datapoints belonged to series whose value did not change once across the full capture. We then replayed a five-minute steady-state window of that capture through a live OpenTelemetry Collector into a byte-counting sink, twice. The first run was the stream as captured. For the second run, we modeled a source-side cleanup: we dropped the series that never changed across the full capture, and thinned the fast-scraping, rarely changing series to a 60-second interval. The collector configuration was identical in both runs, so the difference on the wire is attributable entirely to the modeled source-side fix. This experiment measures the volume opportunity; it does not establish that every flat series is safe to remove from production without the alert, dashboard, and ownership review described above. | Measure (5-minute window) | Before | After | Reduction | | --- | --- | --- | --- | | Datapoints | 2,944,901 | 985,860 | 66% | | Datapoints per minute | 588,980 | 197,172 | 66% | | Bytes on the wire, compressed | 42.9 MB | 17.9 MB | 58% | | Projected daily egress | 12.4 GB | 5.2 GB | 58% | On a platform that prices per datapoint, this stream drops from 25.4 billion to 8.5 billion datapoints per month, and the egress savings apply regardless of vendor. Bytes fall less than datapoints because the surviving data still carries its resource metadata and compression already exploits repetition. The wire is the conservative measure, and it still halved. Worth repeating: this is a non-production environment. This organization pays for two-thirds of its metric stream to repeat unchanged values throughout the capture, before a single customer transaction is involved. **Find the metric series worth thinning.** OllyGarden Insights analyzes the metrics your services already send, flags noisy and high-cardinality series per service, and gives each service an Instrumentation Score so you know where to start. [Explore Insights](https://ollygarden.com/products/insights) ## When not to touch the interval Not every high-DPM metric is waste. Histograms almost always change between scrapes, so the thinning above excluded them. Counters that advance steadily carry signal at any interval. The question for them is whether anyone consumes the resolution. Any metric feeding a fast-evaluating alert needs an interval comfortably shorter than the alert window. The test is not only "does this change often?" It is also "does anything downstream need it this often?" Apply fixes series by series, with the consumers in view. ## Make the interval a decision Scrape intervals are one of the few budget lines an engineering organization can cut by two-thirds without losing capability, once it confirms which resolution its alerts and dashboards actually need. The work is measurable in advance. A few queries over a sample of the metric stream will show what fraction of the datapoints never changes and which series dominate the volume. Measure before the next renewal, review the consumers and owners, fix at the source, and make the interval something the organization chose instead of something it inherited. --- # Teach Your Coding Agent OpenTelemetry We’re releasing our Agent Skills for OpenTelemetry as a mature, open source set: 14 skills that give any coding agent accurate, current OpenTelemetry knowledge. Published: 2026-07-14 Canonical URL: https://ollygarden.com/resources/blog/teach-your-coding-agent-opentelemetry More of the code running in production today was written by a coding agent than a year ago, and next year that share will be larger still. We see no sign of this trend slowing down; if anything, it is accelerating. This changes something fundamental about telemetry. When agents write the code and agents help operate it, telemetry stops being something humans occasionally glance at on a dashboard. It becomes the primary source of truth that agents use to understand what is actually happening in a running system. That world breaks down quickly when telemetry is high volume and low quality. The observability backend bill becomes prohibitive: ingestion and storage costs grow with every span nobody needed, and query costs compound on top, because agents query telemetry far more often and far more broadly than humans ever did. Agents reasoning over noisy signals reach wrong conclusions with full confidence. And incidents take longer to resolve, for humans and agents alike, because the signal that matters is buried under data nobody asked for. ## Instrumentation moves up the stack There is a second consequence of agents writing most of the new code, and it is a positive one. For years, the pragmatic answer to "how do we get telemetry out of this service" was to instrument at lower levels: bytecode agents, sidecars, kernel probes. These approaches produce telemetry in bulk, but they cannot know what a request means to the business, so the result skews toward high volume and low value. An agent that is writing the code anyway, and that actually knows OpenTelemetry, can do better. It can add high value, low volume instrumentation directly in the source code, at the moment the code is written: a span where a business decision happens, an attribute that captures why a request took the slow path, a metric that tracks the thing the on-call engineer will actually ask about. Lower-level instrumentation still has its place for baseline visibility, but the center of gravity shifts to purposeful instrumentation in the code itself. The catch is the qualifier: an agent that actually knows OpenTelemetry. Base models know a lot about OpenTelemetry, but their knowledge is frozen at training time, in an ecosystem that ships breaking changes, new semantic conventions, and new components every week. That gap is what we are addressing today. ## Announcing OllyGarden Agent Skills for OpenTelemetry At KubeCon® + CloudNativeCon® Europe 2026 in Amsterdam earlier this year, our colleague Nicolas Wörner, together with our friend Pavol Loffay from Red Hat, presented "Let Me Be Your OpenTelemetry Champ" and released the initial set of our [OpenTelemetry Agent Skills](https://github.com/ollygarden/opentelemetry-agent-skills). Since then, we have used these skills every day: internally, in our own engineering work, and as a foundational layer of [Rose](https://ollygarden.com/products/rose), our AI instrumentation agent, where they run alongside our more specialized skills. Today we are ready to say that this set has matured into something we recommend for any base model. In our experience, the skills improve the OpenTelemetry output of Anthropic, OpenAI, and Gemini models alike. They form what we call the "OTel Facts" layer of our knowledge fabric: facts gathered from the OpenTelemetry repositories themselves, with a strong preference for the most current source of truth there is, the source code. The skills are open source under the Apache 2.0 license, and we believe they can bring value to everyone working with OpenTelemetry, whether or not they ever use an OllyGarden product. If they lower the barrier to entry for OpenTelemetry adoption, that alone makes them worth releasing. ## What is inside The repository launched with 14 skills, and they cluster into three groups. The [current catalog](https://ollygarden.com/resources/agent-skills) lists every skill in both of our skill packages. The first group covers instrumenting services in every major ecosystem: Go, Java, Node.js, Python, .NET, and the browser. Each language skill bundles references for software development kit (SDK) setup, the instrumentation API surface, available instrumentation libraries, performance tuning, and breaking changes. The browser skill covers real user monitoring, Core Web Vitals, and connecting frontend telemetry to backend traces. The second group covers operating the telemetry pipeline: the OpenTelemetry Collector skill knows the configuration surface of individual components across the core and contrib repositories, the OpenTelemetry Transformation Language (OTTL) skill writes and debugs telemetry transformations, and the telemetrygen skill generates synthetic telemetry for testing pipelines. The third group is about staying correct over time: a skill that looks up the latest compatible SDK versions per language, a skill for semantic convention naming and compliance, a skill for OpenTelemetry Weaver, a skill for declarative configuration, which sets up the SDK through YAML instead of scattered environment variables, and a skill that migrates span events to the Logs API following the accepted OTEP 4430 plan. Those last two illustrate the whole point: declarative configuration only recently became a community recommendation, and the span events migration plan did not exist when current base models were trained. The skills bring knowledge the models could not have picked up in training. The skills follow the [agentskills.io specification](https://agentskills.io/specification) and are built around progressive disclosure: each skill's entry point is deliberately small, and the agent pulls in only the reference it needs for the task at hand. Instead of copying documentation into context, the skills use compact lookup tables and scripts that point at the upstream sources of truth. The agent retrieves what it missed during training, and the answers stay current as the OpenTelemetry project evolves. The skills are also non-opinionated by design; there are many valid ways to use OpenTelemetry, and prescribing conventions is not their job. **Add organizational context with Rose.** Rose builds on these skills, reviews the instrumentation in your GitHub repositories for missing or low-value telemetry, and prepares source-level fixes as pull requests or context for your coding agent. [Meet Rose](https://ollygarden.com/products/rose) ## Day one is easy Here is what we learned building these: shipping a set of skills on day one is the easy part. Anyone with a weekend and a scraper can do it. The hard part, and the part where the value actually lives, is everything that comes after. OpenTelemetry ships new SDK releases, renames Collector components, and evolves semantic conventions continuously. Base models get replaced, and each new model knows different things and fails in different ways. A skill set that was accurate in March quietly becomes a liability by September if nobody maintains it. That maintenance is the commitment we are making, with this repository and with our products in general. It also means tracking two moving targets, not one: the OpenTelemetry ecosystem itself, and the fast-moving practice of how agent skills are built. Techniques like progressive disclosure were not common vocabulary a year ago; we expect the state of the practice to keep changing, and we will keep adapting the skills to match it. This is also why these skills sit at the foundation of Rose. Rose layers organizational context, our specialized detection and remediation skills, and cross-service analysis on top of the OTel Facts layer. The better the foundation, the better everything built on it. ## Try them, break them, tell us about it Installing the skills takes one command: ``` npx skills add https://github.com/ollygarden/opentelemetry-agent-skills ``` Claude Code users can also add the repository as a plugin marketplace and install individual skills from there; the [README](https://github.com/ollygarden/opentelemetry-agent-skills) has the details. We built these skills with instrumentation in mind, but we keep finding new uses for them: reviewing Collector configurations, planning migrations, generating test telemetry, checking semantic convention compliance. We would love to hear where you take them, where they shined, and just as importantly, where they failed. Open an issue, and if you want to go further, contributions are welcome. Our hope is that this repository gains enough momentum that we can eventually donate it to the OpenTelemetry project, where a community-maintained knowledge layer for agents rightfully belongs. And if you want to see what these skills enable when combined with organizational context and specialized instrumentation knowledge, [try Rose](https://ollygarden.com/products/rose). It reviews the OpenTelemetry instrumentation in your pull requests and helps your team produce the high quality, high value telemetry that this new era demands. --- # Are OpenTelemetry Collector Exporters Called Sequentially or in Parallel? When you configure two exporters in one pipeline, does the Collector send to them in parallel? The honest answer is "sequentially, but it usually doesn't matter." The part that does matter changed defaults underneath a lot of advice still floating around. Published: 2026-06-29 Canonical URL: https://ollygarden.com/resources/blog/are-collector-exporters-sequential-or-parallel You have configured multiple exporters in a single OpenTelemetry Collector pipeline: one for your production observability backend, and perhaps another for a local debugging tool or a data archive. It is tempting to assume the Collector broadcasts telemetry to all of them simultaneously. The reality is more nuanced. The Collector calls exporters *sequentially*, in the order they appear in the pipeline. But the sending queue normally hides that from you, so in practice it behaves *as if* it were parallel. The interesting question is what happens when that illusion is put under pressure, and the answer changed at some point, which is why a lot of the advice you will find online is now wrong. ## The fan-out mechanism The component responsible for delivering data to multiple exporters within a single pipeline is the fan-out consumer. It iterates over the configured exporters in the order they appear in your configuration. Consider this pipeline: ``` receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: otlp/prod: endpoint: 'api.vendor.com:4317' otlp/archive: endpoint: 'archive.internal:4317' service: pipelines: traces: receivers: [otlp] exporters: [otlp/prod, otlp/archive] ``` When the Collector processes a batch of traces, the fan-out consumer performs a simple loop: it calls `ConsumeTraces` on `otlp/prod`, waits for that call to return, then calls `ConsumeTraces` on `otlp/archive`. The hand-off is inherently sequential. There is one detail worth knowing for correctness: the fan-out consumer accumulates errors rather than aborting on the first one. If an exporter returns an error, the loop records it and still proceeds to the next exporter. A hard failure in `otlp/prod` does not, by itself, prevent `otlp/archive` from receiving its copy. ## The role of queues If the execution is sequential, does `otlp/archive` have to wait for `otlp/prod` to complete a network request across the internet? No. The OTLP exporter (like most core exporters) is built on the `exporterhelper` package, which enables a **sending queue by default**. That queue changes what `ConsumeTraces` actually does. With the queue enabled, `ConsumeTraces` does not send anything over the network. It enqueues the data into an in-memory buffer and returns almost immediately. Each exporter has its own pool of background consumers, ten by default, that drain its queue and handle network transmission independently. So the real flow is: - The fan-out consumer enqueues the batch on `otlp/prod`'s queue and returns immediately. - It enqueues the batch on `otlp/archive`'s queue and returns immediately. - Background consumers for each exporter pick up the data and transmit it over the network in parallel. On the happy path, the system behaves *as if* it were parallel, even though the initial hand-off is strictly sequential. Two exporters with full queues drain concurrently; one slow backend does not make the other wait for its network round-trip. ## What happens when a queue fills up The decoupling holds only while there is room in the queue. The important question, and the one that trips people up, is what the Collector does when a queue is *full*. This is governed by a single setting, `sending_queue.block_on_overflow`, and **its default is `false`**: ``` exporters: otlp: sending_queue: enabled: true queue_size: 1000 # default; measured in requests block_on_overflow: false # default ``` With `block_on_overflow: false`, enqueuing onto a full queue **returns a retryable error immediately** instead of waiting. It does not block the caller. So if `otlp/prod` is unhealthy and its queue is saturated, here is what happens on the next batch: - The push to `otlp/prod` fails fast with a "queue is full" error. The fan-out consumer records it. - The loop moves on and enqueues the batch on `otlp/archive`, which is healthy and has room. - `otlp/archive` keeps receiving data normally. In other words, **by default, latency is isolated**. A slow or stalled backend cannot hold the fan-out loop hostage, because the hand-off to its queue never blocks; it just fails fast for the data it cannot accept. There is a cost, of course: the data that could not be enqueued for `otlp/prod` is rejected back up the pipeline. Note that this is *not* the same as `retry_on_failure`, which only applies to network send failures *after* a batch has been dequeued. An enqueue failure is returned to whoever called the pipeline. For the OTLP receiver, that means a gRPC error goes back to the upstream client, which can decide to retry. The default trades durability for isolation: when a destination is overwhelmed, you shed its load rather than letting it stall everything else. ## Opting into backpressure If you would rather not drop data, you can flip the behavior: ``` exporters: otlp: sending_queue: enabled: true block_on_overflow: true ``` With `block_on_overflow: true`, enqueuing onto a full queue **blocks until space frees up**. This is genuine backpressure, and it is exactly the head-of-line-blocking scenario people warn about, except now it is something you opted into: - The push to the full `otlp/prod` queue blocks, waiting for space. - The fan-out consumer is stuck on the first exporter. - The call to `otlp/archive` is delayed for as long as `otlp/prod` stays saturated. Worse, that backpressure propagates upstream. A blocked enqueue means the receiver cannot return, so it stops reading from the network, which stalls *every* exporter in the pipeline and every producer feeding it. With this setting, a single slow vendor really can degrade delivery to your local archive. This is the right choice when losing data is unacceptable and you would rather slow everything down than drop anything. It is the wrong choice when you have a critical, healthy destination that must not be held hostage by a struggling one. The point is that it is now a deliberate decision, not a hidden default. ## A note on why the old advice says the opposite If you have read that "the Collector applies backpressure by default" or that "a full queue blocks the pipeline," that advice was not wrong when it was written. Earlier Collector releases used a bounded in-memory queue that **blocked the caller when full**: blocking was the default behavior, with no knob to turn it off short of disabling the queue entirely. The queue implementation was later reworked (the `queuebatch` refactor in `exporterhelper`), and the default flipped to non-blocking, with `block_on_overflow` added as the explicit opt-in. The version number on the binary did not announce this in a way most readers noticed, so a lot of still-circulating guidance describes behavior that is no longer the default. When you read advice about Collector queue behavior, check which era it was written for, and better yet, check the behavior of the exact build you ship. **Give your agent current Collector knowledge.** Our free, open source OpenTelemetry Agent Skills point Claude Code, Cursor, and other coding agents at upstream sources of truth, so their Collector advice reflects current behavior, not outdated defaults. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Recommendations **1. Keep sending queues enabled.** They are on by default in the OTLP exporter, and they are what gives you the as-if-parallel behavior. Disabling the queue forces the fan-out consumer to wait for the actual network request, which guarantees that your slowest exporter dictates the throughput of the entire pipeline. **2. Decide your overflow policy on purpose.** The default (`block_on_overflow: false`) isolates latency by shedding load from an overwhelmed destination. Setting `block_on_overflow: true` protects against data loss but couples the fate of every exporter (and the receiver) to your slowest backend. Pick the one that matches what you are optimizing for; do not inherit it by accident. **3. Monitor your queues.** Watch `otelcol_exporter_queue_size` against `otelcol_exporter_queue_capacity`. A queue that constantly hovers near capacity is a bottleneck. Pair it with `otelcol_exporter_enqueue_failed_spans` (and the metric/log equivalents): under the default policy, that is where shed data shows up, silently, unless you are looking. **4. For true isolation, use separate Collector instances.** Splitting destinations into separate pipelines within one Collector helps, but remember the receiver caveat: if you have set `block_on_overflow: true` on any pipeline, a stalled enqueue there can stop the shared receiver from reading the network, which affects every pipeline. Hard isolation between a critical destination and a flaky one is most reliably achieved with separate Collector processes. ## Summary The OpenTelemetry Collector calls exporters sequentially, and the `sending_queue` is what turns that sequential hand-off into effectively parallel delivery. By default, a full queue fails fast (`block_on_overflow: false`), so one slow backend isolates its own latency at the cost of shedding its own data. Backpressure and head-of-line blocking are real, but they are the opt-in behavior (`block_on_overflow: true`), not the default they once were. Keep your queues enabled, choose your overflow policy deliberately, and monitor queue size and enqueue failures so you know which trade-off you are actually living with. --- # Creating Your Own Collector Distribution from OllyGarden Tulip Tulip gives you a commercially supported OpenTelemetry Collector with a curated component set. When you need more, you can derive your own distribution while keeping support for the core pipeline. This post walks through how. Published: 2026-05-27 Canonical URL: https://ollygarden.com/resources/blog/creating-your-own-collector-distribution-from-tulip One of the most common complaints I hear from platform teams adopting OpenTelemetry: "We love the vendor neutrality, but we can't run a critical piece of infrastructure without support." The OpenTelemetry Collector is powerful. It lets you decouple your instrumentation from your backend, so switching observability platforms doesn't mean reinstrumenting your entire stack. But for many organizations, running the upstream Collector in production without a support contract is a non-starter. [OllyGarden Tulip](https://github.com/ollygarden/tulip) solves this. It's a commercially supported distribution of the OpenTelemetry Collector that gives you ready-to-use binaries and container images with a curated set of components — plus a manifest system that lets you derive your own custom builds, mixing supported components with community modules. You get the vendor neutrality that OpenTelemetry promises, backed by the support your organization requires. And when you need components beyond the curated set, you can extend Tulip while keeping commercial support for the core pipeline. This post walks through creating your own distribution derived from Tulip, using the **v26.05.1** release — the first Long-Term Support (LTS) release — as the baseline. ## What's in the Tulip Manifest? Before extending, understand what you're starting with. Tulip v26.05.1 includes: **Extensions:** zpages, pprof, basicauth, bearertokenauth, oauth2client, oidc, filestorage **Receivers:** otlp, nop, hostmetrics, filelog **Processors:** attributes, resource, span, probabilisticsampler, filter, transform, redaction **Exporters:** otlp, otlphttp, file, debug, nop **Connectors:** forward This is a deliberately focused set covering the most common telemetry pipeline patterns. If you need components beyond this list — say, the Kafka receiver, Prometheus exporter, or tail sampling processor — you'll add them to your fork. One thing to keep in mind: commercial support covers only the components in the Tulip manifest. Anything you add yourself is yours to maintain. We cover exactly where that line sits in the "Understanding Support Boundaries" section below. **No batch processor?** That's intentional. The `batchprocessor` was removed in the LTS May 2026 release: it has been deprecated upstream and carries a known data-loss bug. Instead, batching now happens at the exporter level via `sending_queue` and `batch` settings, which is more robust under backpressure. You'll see this reflected in the configuration examples below. ## Step 1: Clone the Tulip Repository Structure Start by creating your distribution directory. Tulip uses a multi-distribution repository structure: ``` mkdir -p my-collector/distributions/my-distro cd my-collector ``` Create the essential files: ``` distributions/my-distro/ ├── manifest.yaml # Component manifest (the important one) ├── config.yaml # Default runtime configuration ├── Dockerfile # Container image ├── my-distro.service # systemd service file ├── my-distro.conf # systemd environment file └── my-distro-test.yaml # Test configuration ``` ## Step 2: Create Your Manifest The manifest is the heart of your distribution. It declares which components to include and their versions. Here's a manifest that extends Tulip with the Kafka receiver and tail sampling processor: ``` # distributions/my-distro/manifest.yaml dist: module: github.com/my-org/my-collector/my-distro name: my-distro description: Custom collector with Kafka and tail sampling output_path: ./_build version: 0.151.0 build_tags: "grpcnotrace" extensions: # Keep Tulip's extensions - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/oauth2clientauthextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/oidcauthextension v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.151.0 receivers: - gomod: go.opentelemetry.io/collector/receiver/nopreceiver v0.151.0 - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.151.0 # ADD: Kafka receiver - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.151.0 processors: - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor v0.151.0 # ADD: Tail sampling processor - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.151.0 exporters: - gomod: go.opentelemetry.io/collector/exporter/nopexporter v0.151.0 - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.151.0 - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.151.0 - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.151.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.151.0 connectors: - gomod: go.opentelemetry.io/collector/connector/forwardconnector v0.151.0 providers: - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.57.0 - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.57.0 - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.57.0 ``` ### Version Alignment Matters Notice that all `opentelemetry-collector-contrib` components use the same version (`v0.151.0`). Keeping them aligned matters because mixing incompatible core and contrib versions leads to dependency conflicts and runtime panics. For the components you inherit from Tulip, use the versions in the Tulip manifest — that's the exact combination OllyGarden tests and supports. For components you add yourself, we strongly recommend matching that same version so the whole build stays consistent, but it isn't enforced: anything beyond the manifest is yours to keep compatible. The version mapping for Tulip v26.05.1: - **Collector core / contrib components:** `v0.151.0` - **Confmap providers:** `v1.57.0` (these follow their own semver line) When upgrading, update all core and contrib versions together, and bump the confmap providers to their matching release. ## Step 3: Build Your Distribution Install the OpenTelemetry Collector Builder (ocb). For the v0.151.0 release, download the released binary rather than using `go install` — `go install go.opentelemetry.io/collector/cmd/builder@v0.151.0` resolves to the parent `go.opentelemetry.io/collector` module (which no longer carries the builder) and fails for this specific tag. Grab the prebuilt `ocb` instead, adjusting the asset name for your OS and architecture: ``` curl --proto '=https' --tlsv1.2 -fL -o ocb \ https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.151.0/ocb_0.151.0_linux_amd64 chmod +x ocb sudo mv ocb /usr/local/bin/ocb ``` Generate and build: ``` cd distributions/my-distro # Generate Go source code from manifest ocb --config manifest.yaml # Build a static binary (CGO_ENABLED=0) so it runs on the scratch-based image cd _build CGO_ENABLED=0 go build -o ../bin/my-distro . ``` The builder creates a `_build/` directory with generated Go code. Don't edit these files directly — regenerate from the manifest instead. ## Step 4: Create a Default Configuration Your distribution needs a sensible default configuration. Note that batching is configured on the exporter (`sending_queue` + `batch`), not as a pipeline processor: ``` # distributions/my-distro/config.yaml extensions: pprof: {} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 kafka: brokers: - ${env:KAFKA_BROKERS} traces: topics: [otlp_spans] encoding: otlp_proto processors: tail_sampling: decision_wait: 10s policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: slow-traces type: latency latency: threshold_ms: 1000 - name: probabilistic type: probabilistic probabilistic: sampling_percentage: 10 exporters: otlp_grpc: endpoint: ${env:OTLP_ENDPOINT} tls: insecure: ${env:OTLP_INSECURE:-false} sending_queue: enabled: true queue_size: 1000 batch: flush_timeout: 200ms min_size: 8192 retry_on_failure: enabled: true initial_interval: 5s max_interval: 30s max_elapsed_time: 300s service: extensions: [pprof] pipelines: traces: receivers: [otlp, kafka] processors: [tail_sampling] exporters: [otlp_grpc] ``` **`otlp_grpc`, not `otlp`?** As of otelcol 0.151.0 the OTLP exporter is registered as `otlp_grpc` (and the HTTP variant as `otlp_http`). The old `otlp`/`otlphttp` names still resolve as deprecated aliases, but new configurations should use the explicit names. Note the OTLP *receiver* is unaffected — it's still `otlp`. ## Step 5: Containerize Create a Dockerfile for your distribution: ``` # distributions/my-distro/Dockerfile FROM alpine:3.21 AS certs RUN apk add --no-cache ca-certificates FROM scratch COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --chmod=755 bin/my-distro /my-distro COPY config.yaml /etc/my-distro/config.yaml USER 10001:10001 EXPOSE 4317 4318 ENTRYPOINT ["/my-distro"] CMD ["--config=/etc/my-distro/config.yaml"] ``` Build and run: ``` docker build -t my-distro:latest distributions/my-distro/ docker run -p 4317:4317 -p 4318:4318 my-distro:latest ``` The build context is `distributions/my-distro/`, so the `COPY bin/my-distro` and `COPY config.yaml` paths in the Dockerfile resolve relative to that directory — exactly where Step 3 wrote the binary. ## Step 6: Test Your Distribution Create a test configuration that exercises your added components: ``` # distributions/my-distro/my-distro-test.yaml receivers: otlp: protocols: grpc: endpoint: localhost:4317 processors: tail_sampling: decision_wait: 1s policies: - name: test-policy type: always_sample exporters: debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [tail_sampling] exporters: [debug] ``` Run a smoke test: ``` # Start the collector ./bin/my-distro --config my-distro-test.yaml & # Send test traces telemetrygen traces --otlp-insecure --traces 5 # Verify output shows the trace # Then stop the collector kill %1 ``` **Help your coding agent write Collector config.** OllyGarden's free, open source OpenTelemetry Agent Skills give Claude Code, Cursor, and other coding agents current Collector configuration knowledge, so the manifests and pipelines they draft draw on upstream documentation. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Understanding Support Boundaries When you derive from Tulip, support scope follows a clear hierarchy: | Component | Support Level | | --- | --- | | Tulip manifest components | Full commercial support | | Contrib components you add | Best-effort guidance | | Custom/private components | Your responsibility | This means: - If `transformprocessor` has a bug, OllyGarden will triage and fix it - If `kafkareceiver` (which you added) has issues, OllyGarden provides integration tips but won't ship patches - If you add a custom receiver from your own codebase, that's entirely on you This model lets you extend without losing the value of commercial support for the core pipeline. ## Best Practices for Maintaining Your Fork ### Pin to Tulip Release Versions Don't chase upstream head. Align your distribution versions with Tulip releases: ``` # Good: Match Tulip v26.05.1 version: 0.151.0 # Bad: A random version that doesn't match a Tulip release version: 0.153.0 ``` This ensures your base components match a tested, supported configuration. ### Document Your Additions Keep a CHANGELOG noting which components you've added beyond Tulip's manifest: ``` ## Custom Components ### Receivers - `kafkareceiver` - Consumes traces from Kafka topics ### Processors - `tailsamplingprocessor` - Intelligent trace sampling ### Rationale Kafka integration required for our event-driven architecture. Tail sampling reduces storage costs while preserving error traces. ``` ### Separate Configuration from Distribution Your distribution binary should be configuration-agnostic. Ship a minimal default config, but let operators provide their own via: ``` ./my-distro --config file:/etc/my-distro/config.yaml ``` ## When to Consider a Custom Distribution Deriving from Tulip makes sense when: - You need specific receivers/exporters not in Tulip's manifest - You want commercial support for the core pipeline - Your organization has compliance requirements around supported software - You're standardizing on a single collector binary across teams Consider sticking with vanilla Tulip if: - The included components meet your needs - You prefer zero maintenance overhead - You're still evaluating your telemetry architecture ## Summary Creating a custom distribution from Tulip gives you the best of both worlds: a production-tested, commercially supported foundation with the flexibility to add components your organization needs. The process: - Clone Tulip's manifest structure - Add your required components (keeping versions aligned) - Build with ocb - Containerize and deploy - Maintain version alignment with Tulip releases Your core pipeline remains supported. Your extensions remain flexible. And when issues arise, you know exactly which components fall under commercial support and which are your team's responsibility. *Ready to get started? Clone the [Tulip repository](https://github.com/ollygarden/tulip) and check out the [OpenTelemetry Collector Contrib repository](https://github.com/open-telemetry/opentelemetry-collector-contrib) for the full component catalog.* --- # Decomposing OpenTelemetry Collector Configuration for Maintainability When your collector configuration grows beyond a few hundred lines, you start feeling the friction. This post examines practical strategies for decomposing collector configurations into maintainable, testable units. Published: 2026-05-11 Canonical URL: https://ollygarden.com/resources/blog/decomposing-opentelemetry-collector-configuration When your collector configuration grows beyond a few hundred lines, you start feeling the friction. Pull request reviews become exercises in scrolling. Testing a single processor change means deploying an entire pipeline. Environment-specific variations creep in through copy-paste, and suddenly you have three nearly-identical YAML files that drift apart over time. The monolithic collector configuration that worked well during initial deployment becomes a liability at scale. The OpenTelemetry Collector provides configuration providers that enable modular configurations, but the documentation tends to focus on individual features rather than composition patterns. This post examines practical strategies for decomposing collector configurations into maintainable, testable units. ## The monolith problem Consider a typical production collector configuration. It starts innocently enough: OTLP receiver, batch processor, OTLP exporter. Then you add Kubernetes metadata enrichment, tail sampling for traces, filtering for noisy health check spans, resource detection for cloud provider attributes, and suddenly you have a 500-line YAML file that handles traces, metrics, and logs across multiple pipelines. The problems compound in predictable ways. When someone submits a pull request to modify the tail sampling policy, the reviewer must mentally parse the entire configuration to understand context. When a team wants to test a new transform processor statement, they cannot easily isolate that piece from the rest of the pipeline. When you deploy to staging versus production, environment-specific values get mixed with structural configuration, making it difficult to identify what actually differs between deployments. The collector's configuration merging behavior and provider system offer a path forward, but the patterns for using them effectively are not immediately obvious from the documentation. ## Configuration providers and merging The collector supports multiple configuration sources through providers. The most commonly used are the file provider (`file:`), environment provider (`env:`), HTTP provider (`http://`, `https://`), and YAML provider (`yaml:`). Each provider resolves a URI to configuration content, and the collector merges configurations from multiple sources in the order specified. ``` otelcol --config file:base.yaml --config file:overrides.yaml ``` When the collector receives multiple configuration sources, it performs a deep merge. Keys from later sources override keys from earlier sources at each level of the hierarchy. This merge behavior is the foundation for decomposition: you can split configuration by concern and let the merge operation assemble the final result. The environment provider substitutes environment variable values within configuration files using `${env:VAR_NAME}` or the shorthand `${VAR_NAME}` syntax. The file provider supports recursive inclusion through the `${file:path}` syntax within configuration files. This enables configuration fragments to reference other fragments, building up complex configurations from smaller pieces. ## Decomposition strategies Three primary patterns emerge for organizing collector configurations: splitting by component type, splitting by signal pipeline, and layering environment-specific overlays. Each serves different organizational needs, and they can be combined. ### Splitting by component type The first pattern separates receivers, processors, and exporters into distinct files. This works well when teams own different parts of the telemetry pipeline. The platform team might own receiver configurations, while the observability team owns processor logic, and the SRE team manages exporter destinations. ``` collector/ base.yaml # service section, extensions receivers.yaml # all receiver definitions processors.yaml # all processor definitions exporters.yaml # all exporter definitions ``` The base configuration defines the service section and references components by name: ``` # base.yaml extensions: health_check: endpoint: 0.0.0.0:13133 service: extensions: [health_check] pipelines: traces: receivers: [otlp] processors: [memory_limiter, k8sattributes, batch] exporters: [otlp] metrics: receivers: [otlp, prometheus] processors: [memory_limiter, batch] exporters: [otlp] ``` Component files define the actual configurations: ``` # receivers.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus: config: scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod ``` ``` # processors.yaml processors: memory_limiter: check_interval: 1s limit_mib: ${env:MEMORY_LIMIT_MIB:-512} spike_limit_mib: ${env:SPIKE_LIMIT_MIB:-128} k8sattributes: auth_type: serviceAccount extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.deployment.name batch: timeout: 1s send_batch_size: 1024 ``` The collector assembles these with multiple `--config` flags: ``` otelcol --config file:base.yaml \ --config file:receivers.yaml \ --config file:processors.yaml \ --config file:exporters.yaml ``` This pattern makes pull requests smaller and more focused. A change to the batch processor configuration only touches `processors.yaml`, and reviewers can evaluate it in isolation. ### Splitting by signal pipeline When different teams own different telemetry signals, splitting by pipeline makes more sense. The traces team iterates on sampling policies while the metrics team focuses on aggregation rules. Each signal gets its own configuration file containing receivers, processors, and exporters relevant to that signal. ``` collector/ common.yaml # shared extensions, telemetry settings traces.yaml # trace pipeline: receivers, processors, exporters, service.pipelines.traces metrics.yaml # metrics pipeline: receivers, processors, exporters, service.pipelines.metrics logs.yaml # logs pipeline: receivers, processors, exporters, service.pipelines.logs ``` The common file contains shared infrastructure: ``` # common.yaml extensions: health_check: endpoint: 0.0.0.0:13133 pprof: endpoint: localhost:1777 processors: memory_limiter: check_interval: 1s limit_mib: ${env:MEMORY_LIMIT_MIB:-512} service: extensions: [health_check, pprof] telemetry: logs: level: ${env:LOG_LEVEL:-info} encoding: json metrics: level: detailed readers: - pull: exporter: prometheus: host: 0.0.0.0 port: 8888 ``` Each signal file is self-contained for its domain: ``` # traces.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: tail_sampling: decision_wait: 30s num_traces: 50000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: slow-requests type: latency latency: threshold_ms: 2000 - name: baseline type: probabilistic probabilistic: sampling_percentage: 5 batch: timeout: 1s send_batch_size: 512 exporters: otlp: endpoint: ${env:TRACES_BACKEND_ENDPOINT} tls: insecure: false service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ``` The merge operation combines the `service.pipelines` sections from each file, resulting in a complete configuration with all three signal pipelines. ### Environment-specific overlays Production, staging, and development environments differ in endpoints, resource limits, and sometimes pipeline structure. The overlay pattern uses a shared base with environment-specific files that override particular values. ``` collector/ base.yaml env/ production.yaml staging.yaml development.yaml ``` The base file defines the complete structure with placeholders or defaults: ``` # base.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: memory_limiter: check_interval: 1s limit_mib: 512 spike_limit_mib: 128 batch: timeout: 1s send_batch_size: 1024 exporters: otlp: endpoint: localhost:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp] ``` Environment files override specific values: ``` # env/production.yaml processors: memory_limiter: limit_mib: 2048 spike_limit_mib: 512 exporters: otlp: endpoint: ${env:BACKEND_ENDPOINT} tls: insecure: false ca_file: /etc/ssl/certs/ca-bundle.crt retry_on_failure: enabled: true max_elapsed_time: 300s sending_queue: enabled: true num_consumers: 10 queue_size: 5000 service: telemetry: logs: level: info encoding: json ``` ``` # env/development.yaml processors: memory_limiter: limit_mib: 256 exporters: otlp: endpoint: localhost:4317 tls: insecure: true debug: verbosity: detailed service: pipelines: traces: exporters: [otlp, debug] telemetry: logs: level: debug ``` Deployment selects the appropriate overlay: ``` # Production otelcol --config file:base.yaml --config file:env/production.yaml # Development otelcol --config file:base.yaml --config file:env/development.yaml ``` ## Nested file inclusion For deeply modular configurations, the file provider supports nested inclusion. This is particularly useful for complex processor configurations like tail sampling policies, where individual policies might be maintained by different teams. ``` # processors/tail_sampling.yaml processors: tail_sampling: decision_wait: 30s num_traces: 50000 policies: - ${file:policies/errors.yaml} - ${file:policies/slo-violations.yaml} - ${file:policies/baseline.yaml} ``` Each policy file contains a single policy definition: ``` # policies/errors.yaml name: errors type: status_code status_code: status_codes: [ERROR] ``` ``` # policies/slo-violations.yaml name: slo-violations type: and and: and_sub_policy: - name: latency-threshold type: latency latency: threshold_ms: 2000 - name: high-priority-services type: string_attribute string_attribute: key: service.tier values: [critical, high] ``` This granularity enables teams to own individual policies, submit focused pull requests, and test policies in isolation before integration. ## Testing decomposed configurations The collector's `validate` command accepts the same configuration sources as runtime, enabling validation of decomposed configurations: ``` # Validate merged configuration otelcol validate --config file:base.yaml --config file:env/production.yaml # Validate with environment variables set BACKEND_ENDPOINT=backend:4317 otelcol validate --config file:base.yaml --config file:env/production.yaml ``` For more complex validation, the `print-config` command outputs the fully resolved configuration after merging and environment variable substitution: ``` otelcol print-config --config file:base.yaml --config file:env/production.yaml ``` This output shows exactly what the collector would receive, useful for debugging merge issues or unexpected environment variable values. Individual component files can be validated in isolation by wrapping them in minimal configurations. For a processor file to validate independently, it needs at least one receiver, exporter, and pipeline that uses the processor: ``` # test-harness.yaml receivers: otlp: protocols: grpc: endpoint: localhost:4317 exporters: debug: verbosity: basic service: pipelines: traces: receivers: [otlp] processors: [tail_sampling] # processor under test exporters: [debug] ``` ``` otelcol validate --config file:test-harness.yaml --config file:processors/tail_sampling.yaml ``` **Bring Collector expertise to your coding agent.** Our free, open source OpenTelemetry Agent Skills give coding agents like Claude Code and Cursor current Collector configuration knowledge from upstream sources of truth, so they can help you split and validate configurations. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Practical considerations The merge operation has limitations worth understanding. Arrays are not merged; the later source completely replaces the earlier source's array. This affects pipeline definitions: if `base.yaml` defines `processors: [a, b, c]` and an overlay defines `processors: [a, b]`, the result is `[a, b]`, not a combination. Plan your decomposition accordingly, keeping arrays that need to vary together in the same file. File paths in nested inclusions are relative to the working directory, not the file containing the inclusion. A `${file:policies/errors.yaml}` reference resolves relative to where the collector process runs, regardless of which configuration file contains the reference. This behavior can surprise when organizing configurations in subdirectories; consider using absolute paths or ensuring the working directory is set appropriately. Environment variable defaults (`${env:VAR:-default}`) only apply when the variable is unset. An empty string is not the same as unset; if `VAR=""` is exported, the default is not used. For required variables without reasonable defaults, validate externally before starting the collector: ``` : ${BACKEND_ENDPOINT:?BACKEND_ENDPOINT must be set} otelcol --config file:config.yaml ``` Custom collector distributions built with `ocb` need appropriate providers included in the manifest. The file, env, http, https, and yaml providers are not included by default. Missing providers cause cryptic errors when the configuration tries to use unsupported URI schemes: ``` # builder.yaml providers: - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.57.0 - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.57.0 - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.57.0 ``` ## When not to decompose Decomposition adds indirection. A single-file configuration that fits on a screen is easier to understand than multiple files that must be mentally merged. Small teams with straightforward pipelines may find decomposition overhead exceeds its benefits. The patterns described here target configurations that have grown painful to maintain. If your configuration is not yet painful, keeping it simple might be the right choice. The collector's configuration system supports decomposition when you need it; you are not required to use it from the start. ## Summary The OpenTelemetry Collector's configuration merging and provider system enable modular configurations that scale with organizational complexity. Splitting by component type aligns with team ownership of pipeline stages. Splitting by signal pipeline aligns with team ownership of telemetry domains. Environment overlays separate deployment concerns from structural configuration. The key insight is that the collector's deep merge behavior lets you compose configurations from independent pieces. Each piece can be reviewed, tested, and modified in isolation. When combined with validation in CI pipelines, decomposed configurations become easier to maintain than their monolithic alternatives. Start with the decomposition pattern that matches your organizational boundaries. If platform and observability teams have clear ownership, split by component type. If traces, metrics, and logs teams operate independently, split by signal. Layer environment overlays on either approach. The collector's configuration system is flexible enough to support the structure that works for your team. --- # OpenTelemetry is not hard When I asked the community about their pain points with OpenTelemetry, one response kept appearing: 'it's complicated.' I think the framing is wrong. Published: 2026-05-04 Canonical URL: https://ollygarden.com/resources/blog/opentelemetry-is-not-hard When I asked the community about their pain points with OpenTelemetry, one response kept appearing: "it's complicated." The project has too many components, too many specifications, too much to learn. I understand why people feel this way. But I think the framing is wrong. OpenTelemetry is not hard. OpenTelemetry is deep. The difference matters. Hard means you struggle to get started. Deep means there is more to discover as you go further. Getting started with OpenTelemetry takes two commands. Mastering every corner of the project takes years. Most people only need something in between. ## Two commands Here is a Java application getting full observability. First, download the Java Agent: ``` curl -L -o opentelemetry-javaagent.jar \ https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar ``` Then run your application with the agent attached: ``` java -javaagent:opentelemetry-javaagent.jar \ -Dotel.service.name=spring-petclinic \ -jar build/libs/spring-petclinic-4.0.0-SNAPSHOT.jar ``` The application now emits traces, metrics, and logs to `localhost:4317` using the OpenTelemetry Protocol (OTLP). Point it at any OTLP-compatible backend and you have observability. For local testing, a single container gives you everything: ``` docker run --rm -p 3000:3000 -p 4317:4317 grafana/otel-lgtm:latest ``` Browse your application at `localhost:8080`, then check `localhost:3000`. You will find traces showing HTTP requests flowing through your code, database queries with their durations, and even an N+1 query pattern if your application has one. Metrics and logs are there too. No code changes. No configuration files. Two commands. This is not a toy example. The Java Agent instruments dozens of libraries automatically. For many applications, this level of observability is enough to debug production issues, understand performance characteristics, and identify bottlenecks. To be fair, Java is one of the most mature implementations in the OpenTelemetry ecosystem. The Java Agent has been battle-tested in production for years, and the auto-instrumentation covers a wide range of frameworks and libraries out of the box. This makes getting started with Java particularly straightforward. The experience can be quite different for other languages. Some SDKs require more manual setup, have fewer auto-instrumentation options, or lack the same level of polish in their tooling. If you are working with Ruby, Rust, or even Python, you might find yourself writing more instrumentation code by hand, or discovering that certain libraries are not yet covered. This is not a criticism of those implementations; OpenTelemetry is a massive undertaking, and not all languages have the same level of community investment. It is simply a reality worth acknowledging. ## The map is not the territory If OpenTelemetry is this simple, why do people think it is complicated? Because they see the map before they walk the territory. The OpenTelemetry project includes the Collector, the Operator, semantic conventions, the API, the SDK, Weaver, OTLP, eleven language implementations, and dozens of Special Interest Groups (SIGs) working on specific domains. Looking at that list, anyone would feel overwhelmed. But that list represents everything the project offers, not everything you need to use. A menu with two hundred items does not mean you have to order two hundred dishes. You pick what you need. ## Problems create components Every OpenTelemetry component exists because someone had a specific problem. Understanding which problem each component solves tells you whether you need it. The Java Agent we used above sends telemetry directly to a backend. This works fine until you have hundreds of services and want to change your backend. Updating hundreds of deployments to point at a new endpoint is painful. The OpenTelemetry Collector solves this: it acts as a gateway between your applications and your backends. Applications send to the Collector; the Collector sends to the backend. Change the backend, update one place. If you have five services and a stable backend, you do not need a Collector. The Collector requires configuration and deployment. On Kubernetes, the OpenTelemetry Operator simplifies this by managing Collectors through Custom Resource Definitions (CRDs). It can also inject the Java Agent into your workloads automatically. If you are not on Kubernetes, or you have twenty services and a working deployment script, you do not need the Operator. Auto-instrumentation captures common operations, but sometimes you need to record business-specific data. The OpenTelemetry API lets you create custom spans and metrics. If auto-instrumentation covers your needs, you do not need to learn the API. Different languages should use consistent attribute names so you can query across services. Semantic conventions define those names. Auto-instrumentation already follows them. You only care about semantic conventions when you add custom instrumentation or query data across polyglot services. Large organizations find that different teams instrument the same concepts differently. "team" versus "squad", "org" versus "organization", "cluster.name" versus "cluster_name". Weaver lets you define and enforce your own conventions. If you have three services and five engineers, you do not need Weaver. Each component is an answer to a question. If you are not asking the question, you do not need the answer. **Let your coding agent pick the right pieces.** OllyGarden's free, open source OpenTelemetry Agent Skills give coding agents current knowledge of SDK setup per language, Collector configuration, and semantic conventions, pointing them at upstream sources of truth. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Why OpenTelemetry cannot be smaller Some specialized tools are simpler because they make decisions OpenTelemetry cannot make. The Prometheus client library, for example, has excellent performance for metrics. But using it commits you to Prometheus-compatible backends at instrumentation time. You also need a separate solution for traces. OpenTelemetry optimizes for flexibility. Some backends expect cumulative metrics, others expect deltas. Some thrive on high-cardinality data, others choke on it. OpenTelemetry works with all of them. The cost of that flexibility is that OpenTelemetry cannot take shortcuts that assume a specific backend. This is a trade-off, not a flaw. If you know your backend will never change and you need maximum performance for one signal type, a specialized library might serve you better. If you want one instrumentation approach that works everywhere and gives you options, OpenTelemetry is the answer. ## A framework that grows with you Most teams follow a similar progression. Auto-instrumentation comes first, providing immediate visibility without code changes. The exporter points directly at a backend, and telemetry starts flowing. As the number of services grows, the Collector enters the picture. It decouples applications from backend decisions and centralizes concerns like sampling, enrichment, and routing. Teams running on Kubernetes often adopt the Operator next, automating agent injection and Collector management across the cluster. Custom instrumentation arrives when auto-instrumentation cannot capture business-specific operations. The API provides the primitives; semantic conventions provide the vocabulary. At enterprise scale, tools like Weaver enforce consistency across teams that would otherwise drift apart. The framework meets you where you are and offers more when you need it. ## Summary OpenTelemetry has many components because observability has many problems. Not everyone has all the problems, and not everyone has the same problems. The project gives you tools to solve the problems you have and leaves the rest on the shelf. Getting started is two commands. The Java Agent, or equivalent auto-instrumentation for other languages, provides meaningful observability with no code changes. From there, you add components as needs arise: a Collector for backend flexibility, the API for custom instrumentation, the Operator for Kubernetes scale, Weaver for organizational consistency. The project is deep, not hard. You do not need to learn everything before you start. You need to start, and learn what you need as you go. --- # OllyGarden at KubeCon + CloudNativeCon Europe 2026 KubeCon + CloudNativeCon Europe 2026 in Amsterdam was a big week for OllyGarden. Our team delivered five recorded talks across Observability Day and the main conference, co-chaired the Observability Day program committee, ran a Prometheus Contribfest session, and announced the general availability of Rose. Published: 2026-04-15 Canonical URL: https://ollygarden.com/resources/blog/ollygarden-at-kubecon-eu-2026 KubeCon® + CloudNativeCon® Europe 2026 in Amsterdam was a big week for OllyGarden. Our team delivered five recorded talks across Observability Day and the main conference, co-chaired the Observability Day program committee, ran a Prometheus Contribfest session, and announced the general availability of Rose from booth 1343. Here is what we presented and why it matters. ## Observability Day Observability Day kicked off the week on Monday, March 23, as a co-located event before the main conference. Juraci Paixão Kröhling served as program committee co-chair alongside Austin Parker and Eduardo Silva, helping shape the day's program. In the [opening remarks](https://www.youtube.com/watch?v=rRJFrWzlruE), Juraci framed the day around a simple observation: the systems we build today are beyond the ability of any single individual to fully understand. Microservices, service meshes, serverless functions, GPU clusters running inference workloads. A request comes in, touches multiple services belonging to multiple teams on multiple clusters across multiple clouds. Observability exists not as a buzzword or product category, but as a fundamental engineering practice, because if we cannot understand our systems, we cannot fix them, improve them, or trust them. But adoption does not mean the problems are solved. The economics of observability are still broken. Some companies spend six to nine digits on observability tooling alone and still have to turn off tracing or reduce log retention to stay within budget. The choice between visibility and affordability should not be normal. And perhaps the biggest challenge is the human one: teams of five platform engineers supporting organizations with thousands of developers, most of whom start from zero when it comes to good telemetry practices. No single tool solves that. These are structural problems, and the day's program was built around people facing them in production. Two OllyGarden talks ran in parallel tracks that afternoon. ### OpenTelemetry gateways: enforce, transform, route **Juraci Paixão Kröhling (OllyGarden) & Natalie Ujuk (IG Group)** | [Watch on YouTube](https://www.youtube.com/watch?v=S6z2gd666qg) This talk makes the case for the OpenTelemetry Collector gateway as a centralized control point in the telemetry pipeline, using IG Group's production deployment as the running example. IG Group is a financial services company with over 800 engineers across four offices, a mix of on-premises and multi-cloud infrastructure, and thousands of services including legacy systems that cannot be modified. They operate over 10,000 edge collectors that forward telemetry through Kafka into a gateway layer. The core argument is that scaling observability by teaching every team to instrument correctly does not work. Developers are busy delivering features, teams are reluctant to touch legacy code, and documentation does not get read. A centralized gateway solves this by enforcing standards, transforming data, and routing telemetry without requiring changes to application code. The talk organizes gateway capabilities into three categories. Enforcement covers mandatory resource attributes (service.name, service.namespace, deployment.environment), service identity validation against a Configuration Management Database (CMDB), and Personally Identifiable Information (PII) protection using OpenTelemetry Transformation Language (OTTL) regex patterns. Transformation covers filling in missing attributes, normalizing inconsistent naming conventions, and enriching telemetry with external data sources like GeoIP databases. Routing covers directing PII-containing services through resource-intensive redaction pipelines while others take a fast path, aggregating Java Virtual Machine (JVM) metrics to reduce time series counts by 97.5%, and distributing traces across sampling instances using trace ID for load balancing. The speakers also address the trade-offs honestly. The gateway becomes critical infrastructure that requires high availability planning. It is tempting to decompose the pipeline into too many layers, so teams should start simple and add complexity only as needed. During Q&A, an audience member raised the challenge of opacity when gateways drop or transform telemetry. The response: redact attribute values rather than removing entire attributes, tag telemetry to indicate when attributes have been dropped, and use OTel Weaver to shift enforcement left to CI time. ### Let me be your OpenTelemetry champ **Pavol Loffay (Red Hat) & Nicolas Worner (OllyGarden)** | [Watch on YouTube](https://www.youtube.com/watch?v=pdKemf65KGc) Nicolas and Pavol tested how well AI coding agents handle OpenTelemetry instrumentation and Collector configuration. The setup is a practical Friday-afternoon scenario: a developer and an SRE need to instrument a multi-service demo application (browser, Go backend, Python service calling a Large Language Model) and deploy a Collector in Kubernetes before the weekend. The first attempt, using a coding agent with a simple prompt, produced a working pull request in five minutes. But closer inspection revealed problems. The agent used an outdated Python SDK version from 2024. Context propagation failed, leaving each service with disconnected traces instead of a single correlated trace. Span names used raw function names instead of the semantic convention pattern of method and route. The talk then introduces "agentic skills" as a mechanism for improving agent performance. An SDK version skill maintains a table of the latest version for each language SDK, updated automatically on new releases. A semantic conventions skill provides a script that fetches only the relevant subset of the conventions registry. With these skills loaded, the agent produced correct span names, proper semantic convention attributes, working context propagation, and functioning log correlation with traces. On the Collector side, the agent consistently struggled. It hallucinated deprecated configuration fields, produced invalid OTTL statements, suggested extensions that exist only in the repository and not in any standard distribution, and generated different configurations each time the same prompt ran. These findings led to concrete proposals: a collection of agentic skills for common OpenTelemetry use cases (Kubernetes monitoring, PII removal, cost reduction) and a Model Context Protocol (MCP) server for the Collector that would teach agents exactly which components exist in a given distribution, what their configuration looks like for a specific version, and how they map to the deployment. **Give your coding agent current OpenTelemetry knowledge.** OllyGarden's free, open source OpenTelemetry Agent Skills give coding agents current guidance on SDK setup, semantic conventions, Collector configuration, and OTTL by pointing them at upstream sources of truth. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Main conference ### OpenTelemetry project update and ask the experts **Pablo Baeyens (Datadog), Juraci Paixão Kröhling (OllyGarden), Marylia Gutierrez (Grafana Labs) & Severin Neumann (Causely)** | [Watch on YouTube](https://www.youtube.com/watch?v=mvSGusgcVuw) The four OpenTelemetry Governance Committee members presented progress since the previous project update in Atlanta. Three deep-dive topics anchored the session. Declarative configuration reached stable status. This feature replaces the fragmented environment variable approach with a single YAML file that works across all SDKs (Java, JavaScript, Python, Go, and others) as well as the Collector. Users can configure resources, propagators, tracer providers, meter providers, and logger providers in one place. Juraci presented OTel Weaver, a tool for defining, validating, and evolving telemetry conventions within an organization. Teams create a registry of YAML files that define metrics, attributes, instrument types, stability levels, and required fields. Weaver resolves these into a schema and can generate code from it. Custom Rego policies enforce naming conventions and attribute patterns. A live check mode acts as an OpenTelemetry Protocol (OTLP) server that validates incoming telemetry against the registry in real time and reports policy violations as JSON output. Severin Neumann introduced the OpenTelemetry Injector, which uses LD_PRELOAD to automatically inject the correct auto-instrumentation into applications without code changes. It supports Java, Node.js, .NET, and Python, detecting the application runtime and setting the appropriate environment variables before the process starts. Rapid updates covered 4x to 30x performance improvements in the Go SDK metrics path, the deprecation of the span event API in favor of log-based events, profiling reaching alpha in OTLP version 1.11, the extended Berkeley Packet Filter (eBPF) instrumentation project targeting a 1.0 release candidate, Kubernetes semantic conventions reaching release candidate status, and new Special Interest Groups (SIGs) for Collector MCP, Browser, Zig, PHP, Kotlin, and the Ecosystem Explorer. ### How manual OTel instrumentation saves more than just money **Juliano Costa (Datadog) & Yuri Oliveira (OllyGarden)** | [Watch on YouTube](https://www.youtube.com/watch?v=13Lyvy48Y6A) Many companies that have reached a mature observability practice continue to rely on auto-instrumentation without realizing it is quietly inflating their budgets. Yuri and Juliano show how manual instrumentation can deliver leaner and more business-oriented telemetry. By comparing traces produced through automatic and manual instrumentation, the talk demonstrates how controlling spans and their attributes can cut data volume by up to 60%, reducing resource overhead and storage costs while improving signal-to-noise ratio. The talk covers the do's and don'ts of span and attribute management, how to identify unnecessary telemetry metadata, and practical steps to achieve higher observability efficiency. The key insight is that the problem may lie not in how much you collect, but in what you collect. At scale, the difference between thoughtful manual instrumentation and default auto-instrumentation translates directly into cost savings, cleaner traces, and faster debugging. ### Day-2 reality check: taming wasteful telemetry **Juraci Paixão Kröhling (OllyGarden) & Elena Kovalenko (Delivery Hero)** | [Watch on YouTube](https://www.youtube.com/watch?v=cpB5NTtUdwQ) This talk examines the sources of telemetry waste that accumulate after the initial instrumentation push. The speakers identify three root causes: the convenience trap of auto-instrumentation at day zero, lack of telemetry governance in large organizations (especially those formed through acquisitions, like Delivery Hero), and telemetry hoarding where teams keep data "just in case." The real-world examples are striking. Elena describes a security incident where over 700 customer phone numbers leaked into a telemetry backend because neither the application nor the ingestion pipeline masked PII. She also shows "traces of doom," where auto-instrumentation on a pub/sub system produced a single trace running for 7 days and 12 hours with no useful information in its spans. A slide of actual service names found in production includes database calls, broadcast IP addresses, "net http," AWS S3 endpoints, and "Java SDK," none of which are valid service names. For immediate relief, the talk demonstrates several OpenTelemetry Collector processors: the transform processor for masking PII and deleting obsolete attributes, and the log deduplication processor for aggregating repeated log entries. For long-term improvement, the speakers advocate source-level governance through OTel Weaver for schema enforcement, SDK wrappers that provide opinionated defaults, and agentic tooling that reviews pull requests for cardinality issues and suggests instrumentation improvements. ## Beyond the stage ### Contribfest: Prometheus new contributor introduction **Ben Kochie (Reddit), Bryan Boreham (Grafana Labs), Saswata Mukherjee (Red Hat) & Arianna Vespri (OllyGarden)** Arianna co-led a hands-on Contribfest session guiding newcomers through their first contributions to the Prometheus project. Contribfest sessions are not recorded, but they are one of the most effective ways to grow the contributor base for open source projects. ### Rose general availability OllyGarden announced the general availability of Rose at KubeCon + CloudNativeCon, an AI agent that fixes bad OpenTelemetry instrumentation. The announcement came from booth 1343, where the team spent the week talking with platform teams and observability engineers about the challenges covered in the talks above: telemetry waste, governance at scale, and the gap between auto-instrumentation defaults and production-quality observability. ## A common thread A theme runs through all five talks: the hard problems in observability are no longer about getting started with instrumentation. They are about what happens after. Telemetry waste accumulates silently. Governance does not scale through documentation and mandates. Auto-instrumentation provides a starting point, not an ending point. AI coding agents can accelerate instrumentation work, but they need structured knowledge to produce correct results. And centralized gateways offer a practical path to enforcing standards without requiring every team to become an observability expert. These are the problems OllyGarden works on every day, and KubeCon + CloudNativeCon Europe 2026 was an opportunity to share what we have learned with the community. --- # When to Use Each Telemetry Signal: Logs, Traces, and Metrics Confused about logs, traces, and metrics? Learn when to use each telemetry signal in OpenTelemetry and build a stronger observability strategy. Published: 2026-02-17 Canonical URL: https://ollygarden.com/resources/blog/when-to-use-each-telemetry-signal-logs-traces-and-metrics Understanding when to use logs, traces, or metrics is fundamental to building effective observability. Each signal serves a distinct purpose, and choosing the right one for a given situation directly impacts your ability to debug, monitor, and understand your systems. The challenge is that these signals overlap in capability, leading teams to either over-instrument with redundant data or miss critical insights by using the wrong signal for the job. ## Logs: The system lifecycle narrator Logs are the grandfather of all telemetry signals. Most of us learned to read logs and error messages early in our computing journey, first as users trying to understand why something failed, and eventually as developers writing log statements to communicate system state. Because of this history, logs remain the most universal and accessible signal. Most legacy systems rely exclusively on logs. Before modern observability practices, aggregating log records was the standard approach to understanding request rates, queue depths, or user journeys across systems. Many teams implemented correlation IDs to tie logs for a specific request across multiple services, essentially building a primitive form of distributed tracing before dedicated tracing systems existed. Today, logs serve a more focused purpose: understanding the lifecycle of an application. They excel at recording when a polled connection to a database was established, when a ring was rebalanced, when a circuit breaker opened or closed, when a critical resource stopped working, or when a service recovered from degraded mode. These are events about the application itself, not about individual business transactions. The strength of logs lies in their flexibility and low barrier to entry. Any developer can add a log statement. The weakness is that this flexibility often leads to inconsistent structure, making analysis difficult at scale. ## Traces: The transaction investigator Traces capture telemetry for business transactions, typically in the context of an end-user request that might touch dozens or hundreds of services. Unlike logs, which describe system state, traces describe what happened during a specific operation and how long each step took. Think of spans, the building blocks of traces, as super-logs. A span is essentially a log entry with a timestamp, a duration, causality relationships through parent span references, and built-in correlation IDs through the trace ID. The critical addition is context propagation: a standardized mechanism to pass trace and span IDs to downstream services, ensuring that all participants in a transaction can contribute to the same trace. This is what those hand-rolled correlation ID solutions were trying to achieve, but traces provide it as a first-class capability with standard protocols and automatic propagation. The power of traces becomes evident when debugging errors. A trace shows not just that an error occurred, but the exact path the request took, which services were involved, and where the failure originated. When aggregated across many transactions, traces reveal user behavior patterns, system bottlenecks, and optimization opportunities. One pattern that traces expose with unusual clarity is N+1 queries. While this anti-pattern is difficult to spot with other signals, a trace immediately reveals when a single request triggers dozens of sequential database or network calls. The visual representation of span timing makes the problem obvious in a way that logs or metrics cannot match. Spans carry highly detailed attributes: which feature flags were active, the user's IP address, whether this is a VIP customer, which authentication mechanism was used, which payment method was selected, and which specific service instances processed the request. This level of detail makes traces the backbone of observability. When you have questions or theories about service behavior, traces often provide the answers. This power comes with trade-offs. The detail that makes traces valuable also makes them expensive. The sheer volume of spans in a complex system creates significant storage and processing costs. Traces are also the most difficult signal to learn and implement correctly, which drives teams toward auto-instrumentation. While convenient, auto-instrumentation often increases volume further without adding proportional value. ## Metrics: The pre-calculated answer engine Metrics are aggregations of events or numeric representations of system state. They answer questions like: what is the current queue depth? How many users visited this page? What is the p99 latency for a specific endpoint? As aggregations, metrics require choosing dimensions upfront. You might aggregate by endpoint path, service location, or page visited. You typically do not store the IP address for each individual user visit unless you specifically need to count visits per IP. Time-series databases, the systems specialized for storing metrics, are optimized for aggregated data rather than high-cardinality dimensions. Metrics excel at pre-calculating answers to questions you know you will ask. RED metrics (requests, errors, duration) for HTTP services are the classic example. If you know you will want to track request rates, error percentages, and latency distributions for every endpoint, metrics provide this efficiently and at low query cost. The limitation appears during ad-hoc exploration. While metrics can answer many questions, there will inevitably be investigations where you need a dimension you did not anticipate. Am I seeing high latency for all users, or only those in Europe? Only Germany? Only Berlin? If you did not include geographic dimensions in your metrics, you cannot answer these questions without re-instrumenting. Metrics are the classic signal in monitoring. When operating a database, experienced operators know which metrics to watch: connection pool utilization, query latency distributions, replication lag. These indicators quickly reveal the health of the system without requiring investigation into individual transactions. ## Choosing the right signal The decision framework is straightforward once you understand each signal's purpose. Use traces to record events related to business transactions. When an HTTP request arrives, when a user places an order, when a payment is processed, these are trace-worthy operations. The value is in understanding the complete path and timing of individual transactions. Use metrics to pre-calculate answers to questions you know you will ask. If you need to monitor request rates, error percentages, or latency distributions, define those metrics upfront. The value is in fast, cheap access to known indicators. Use logs to understand lifecycle events of your services. When dependencies change state, when configuration reloads, when the application starts or stops gracefully, these belong in logs. The value is in understanding the application as a running system, not the transactions it processes. **Check signal choices where they are made.** Rose reviews the instrumentation in your GitHub repositories for missing or low-value logs, spans, and metrics, explains each finding, and prepares source-level fixes your developers can review before merging. [Meet Rose](https://ollygarden.com/products/rose) ## When signals overlap Real systems often require multiple signals for the same event. A database connection failure might warrant a log (lifecycle event: dependency unavailable), affect a metric (connection error count), and appear in traces (failed span for database operations). This overlap is expected and appropriate. The mistake is using one signal where another would be more effective. Aggregating log records to compute request rates works, but metrics do this more efficiently. Searching traces to understand when a service entered degraded mode works, but logs make this pattern explicit. Understanding why a specific request failed from metrics alone is nearly impossible, while a trace makes the answer visible. Match the signal to the question. System health and known indicators call for metrics. Transaction debugging and behavior analysis call for traces. Application lifecycle and operational events call for logs. ## Summary Each telemetry signal has a distinct purpose that reflects its design and history. Logs narrate system lifecycle events: startups, configuration changes, dependency state transitions. Traces capture business transaction details: request paths, timing, errors, and the attributes that explain behavior. Metrics pre-calculate answers to monitoring questions: rates, distributions, and aggregate states. Effective observability uses all three signals appropriately. The goal is not coverage through redundancy, but precision through choosing the right tool for each question you need to answer. --- # You don't have too much telemetry. You have bad telemetry. High observability costs are rarely a volume problem. Learn how to fix bad telemetry at the source before relying on sampling. Published: 2026-02-04 Canonical URL: https://ollygarden.com/resources/blog/you-dont-have-too-much-telemetry-you-have-bad-telemetry The quarterly budget review arrives, and the observability line item has doubled again. The reflexive response is familiar: "We need to sample more aggressively" or "Let's only observe critical services." These tactics will reduce costs. They will also destroy your ability to debug production incidents, trading a financial problem for an operational one. The uncomfortable truth is that most organizations do not have a volume problem. They have a governance problem. Before reaching for the sampling dial, engineering leaders should ask a more fundamental question: do we actually know what we are collecting, and is it worth keeping? ## The governance gap Most organizations cannot answer three basic questions about their telemetry: What are we collecting? Who owns it? Is it valuable? Telemetry tends to grow organically. A developer enables debug logging during an incident and forgets to disable it. Auto-instrumentation captures every internal function call by default. Library internals generate spans that no one ever examines. Over months and years, this accumulation becomes the baseline that everyone assumes is necessary. The result is a telemetry estate where no one understands the data, no one owns the data, and no one has consciously decided the data is worth the cost of keeping it. When the bill arrives, the only lever that seems available is sampling, which treats all telemetry as equally valuable and cuts it indiscriminately. ## Patterns of bad telemetry Before implementing sampling, engineering leaders should understand the common patterns of telemetry that provides minimal debugging value while consuming significant resources. Our guide [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality) groups them into five dimensions, with real numbers for each. Health check floods represent one of the most common offenders. Kubernetes probes, load balancer checks, and monitoring systems generate millions of traces daily. These traces confirm that services are responding, but they reveal nothing about application behavior, user experience, or system bottlenecks. They crowd out useful signal and consume pipeline capacity. Debug logs abandoned in production create similar waste. During incident response, engineers often increase logging verbosity to understand system behavior. Once the incident resolves, these verbose settings remain in place, generating enormous log volumes that no one examines until the next billing cycle. High-cardinality metric attributes cause a different kind of problem. Adding user identifiers or transaction IDs to metric labels seems useful until the metrics backend collapses under millions of unique time series. The cost grows multiplicatively with each additional high-cardinality attribute. Internal span proliferation occurs when auto-instrumentation, especially via eBPF, captures every method call within a service. A single user request might generate fifteen spans, ten of which complete in under a millisecond and represent internal implementation details rather than meaningful system boundaries. These spans add noise to traces without aiding debugging. Orphaned spans result from broken context propagation between services. These spans cannot be assembled into coherent traces, rendering them useless for understanding request flow. They consume storage and processing resources while providing zero debugging value. **Find the bad telemetry in your own data.** Insights analyzes the OpenTelemetry data you already send and flags noisy signals, broken traces, and high-cardinality attributes per service, so you know what to fix before you reach for sampling. [Explore Insights](https://ollygarden.com/products/insights) ## Fix at source, not at pipe Many organizations attempt to address telemetry waste by adding filters in their collection pipeline. This approach misses the fundamental inefficiency. By the time data reaches the collector, the application has already generated, serialized, and transmitted it across the network. Filtering at the collector reduces storage costs, but the computational and network costs have already been incurred. Source-level fixes eliminate waste entirely. Configuring instrumentation agents to exclude health check endpoints prevents those traces from being created. Establishing log level policies in deployment configurations ensures debug logging stays in development environments. Code review practices can catch high-cardinality metric attributes before they reach production. The collector should serve as a safety net for edge cases, not the primary mechanism for data governance. Filter processors handle scenarios that cannot be addressed at the source, such as legacy applications or third-party services. For everything else, the most cost-effective solution is preventing waste from being generated. ## The volume question remains Even after addressing bad telemetry, some organizations will still face legitimate volume challenges. High-traffic systems generate substantial telemetry even when every span and log provides genuine value. The difference is what happens next. Sampling garbage gives you a smaller pile of garbage. When telemetry is a mix of useful signal and noise, sampling cuts both indiscriminately. You reduce costs, but you also reduce your ability to debug the specific incidents that sampling happened to discard. Sampling after cleanup is a strategic decision about valuable data. When you have eliminated the noise, every piece of remaining telemetry serves a purpose. Sampling decisions become intentional trade-offs between cost and observability coverage rather than desperate cuts to an unmanaged data stream. Tail-based sampling can preserve error traces while reducing successful request volume. Rate limiting can cap burst traffic while maintaining baseline visibility. The key insight is that cleanup dramatically reduces the volume that needs sampling in the first place. Organizations often discover that addressing bad telemetry alone brings costs within acceptable ranges, eliminating the need for aggressive sampling entirely. ## A practical approach for engineering leaders Addressing telemetry governance requires visibility before action. Start by inventorying what you collect, identifying the top contributors to volume across traces, metrics, and logs. Most organizations find that a small number of sources account for the majority of data. Categorize that volume by type. Health checks, internal spans, debug logs, and high-cardinality metrics each require different remediation strategies. Understanding the composition of your telemetry guides where to focus effort. Assess value honestly by asking when each category of telemetry last contributed to resolving an incident. If no one can recall using health check traces for debugging, they are candidates for elimination or aggressive filtering. Implement fixes at the source where possible. Agent configuration changes, log level policies, and instrumentation code reviews address the root cause rather than treating symptoms. Reserve collector-level filtering for cases where source changes are impractical. Finally, if volume remains a concern after cleanup, implement sampling with intention. Document what is being sampled and why. Ensure that sampling policies preserve the traces most likely to matter during incidents, such as errors, high-latency requests, and specific customer traffic. The path from reactive cost cutting to intentional data governance requires effort, but the reward is an observability system that costs less and works better. The next time the budget conversation surfaces, the answer should not be "sample more." It should be "we know exactly what we collect, and it is worth keeping." --- # Reducing Log Volume with the OpenTelemetry Log Deduplication Processor Your logs are probably at least 80% repetitive noise. Connection retries, health checks, heartbeat messages: the same log line repeated thousands of times per day. Published: 2026-01-19 Canonical URL: https://ollygarden.com/resources/blog/reducing-log-volume-with-the-opentelemetry-log-deduplication-processor Your logs are probably at least 80% repetitive noise. Connection retries, health checks, heartbeat messages: the same log line repeated thousands of times per minute. You pay storage costs for each one while the signal drowns in noise. The OpenTelemetry Collector's log deduplication processor offers an elegant solution to this problem. ## The repetitive log problem Modern distributed systems generate enormous volumes of logs, but much of that volume provides diminishing returns. Consider a typical microservice that logs connection errors when a downstream dependency is unavailable. If the service retries every 100 milliseconds for 30 seconds, that's 300 nearly identical log entries for a single incident. Each entry consumes storage, network bandwidth, and processing capacity in your logging backend. Health check endpoints compound the problem. Kubernetes probes, load balancer checks, and monitoring systems all generate log entries at regular intervals. A single service might log thousands of health check responses per hour, none of which provide meaningful insight beyond "the service was running." The logdedupprocessor in the OpenTelemetry Collector solves this by aggregating identical logs over a configurable time window. Instead of forwarding every duplicate entry, it emits a single log with a count of how many times that message appeared. ## How log deduplication works The core concept is straightforward. Logs are considered identical when they share the same resource attributes, scope, body, attributes, and severity. The processor computes a hash of these fields and tracks occurrences within a configurable interval. When the interval expires, the processor emits a single log entry with three additional attributes: `log_count` (the number of duplicates), `first_observed_timestamp`, and `last_observed_timestamp`. You keep full visibility into frequency patterns without storing every identical entry. This approach differs from sampling in an important way. Sampling discards data permanently. Deduplication preserves the information that matters (what happened, how often, and when) while eliminating redundant storage. **Find your noisiest log streams first.** OllyGarden Insights analyzes the logs your services already send and flags noisy, repetitive signals per service, so you know where deduplication is worth configuring. [Explore Insights](https://ollygarden.com/products/insights) ## Practical configuration Here is a configuration that deduplicates connection errors while preserving audit logs: ``` processors: logdedup: interval: 1s conditions: - severity_number >= SEVERITY_NUMBER_ERROR - attributes["log.type"] == "connection" exclude_fields: - attributes.request_id - attributes.timestamp ``` The `conditions` field uses OpenTelemetry Transformation Language (OTTL) expressions to filter which logs get deduplicated. Logs that do not match pass through unchanged. In this example, only ERROR-level logs with the `log.type=connection` attribute are candidates for deduplication. The `exclude_fields` option removes high-cardinality fields from the comparison. Fields like request IDs and timestamps differ between entries even when the log message is semantically identical. By excluding them, logs that differ only in these volatile fields are treated as duplicates. ## A complete pipeline example To use the log deduplication processor, include it in your collector pipeline: ``` receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: logdedup: interval: 1s conditions: - severity_number >= SEVERITY_NUMBER_ERROR - attributes["log.type"] == "connection" exclude_fields: - attributes.request_id - attributes.timestamp exporters: debug: service: pipelines: logs: receivers: [otlp] processors: [logdedup] exporters: [debug] ``` ## Testing with telemetrygen To test this configuration locally, use telemetrygen to generate connection error logs: ``` telemetrygen logs \ --otlp-insecure \ --logs 100 \ --rate 10 \ --severity-text ERROR \ --severity-number 17 \ --body "Connection refused: failed to connect to database at 10.0.0.5:5432" \ --telemetry-attributes 'log.type="connection"' \ --telemetry-attributes 'service.name="order-service"' \ --telemetry-attributes 'db.system="postgresql"' ``` This generates 100 logs at 10 per second, all with ERROR severity and the `log.type=connection` attribute that triggers deduplication. After a few seconds, you should see a few log entries with `log_count: N` in your backend instead of 100 separate entries. ## Tradeoffs and considerations The log deduplication processor introduces latency equal to your interval setting. Logs are held until the interval expires before being forwarded. For most use cases, a 1-second delay is acceptable, but real-time alerting systems may need adjustment. For compliance-critical logs where every occurrence must be preserved with its original timestamp, skip deduplication entirely. Audit logs, security events, and regulatory records often require complete fidelity. The tradeoff is straightforward: reduced storage and clearer signal at the cost of slight delay and losing individual timestamps. For high-volume repetitive logs, that tradeoff is usually worth it. ## Conclusion The log deduplication processor provides a practical solution to the noise problem in modern logging pipelines. By aggregating identical entries while preserving frequency information, you can dramatically reduce storage costs and improve signal clarity without sacrificing observability. Combined with other OpenTelemetry Collector processors like filtering and sampling, log deduplication gives you fine-grained control over your telemetry pipeline. The result is a logging system that captures what matters while discarding the noise. --- # What 10,000 Slack Messages Reveal About OpenTelemetry Adoption Challenges We analyzed 10,000 messages from the OpenTelemetry community Slack to find the most common adoption challenges, from Collector config to instrumentation. Published: 2026-01-06 Canonical URL: https://ollygarden.com/resources/blog/what-10000-slack-messages-reveal-about-opentelemetry-adoption-challenges The OpenTelemetry community has grown tremendously over the past few years, and with that growth comes valuable insights hidden in our community conversations. We analyzed nearly 10,000 messages from the `#otel-collector` and `#opentelemetry` Slack channels spanning from May 2019 to December 2025 to understand what challenges users face most often, which components generate the most discussion, and where the community might need additional documentation or tooling improvements. ## The Dataset Our analysis covered 9,966 messages across two of the most active OpenTelemetry Slack channels: - **#otel-collector**: 5,570 messages (56%) - **#opentelemetry**: 4,396 messages (44%) These messages break down into several categories: | Category | Percentage | | --- | --- | | Questions | 46.7% | | Error Reports | 25.9% | | Discussions | 23.3% | | Configuration | 3.0% | | Help Responses | 1.0% | The high proportion of questions and error reports (over 72% combined) tells us that these channels serve as critical support resources for the community, and the topics that appear most frequently represent real adoption challenges. We applied topic modeling using BERTopic to cluster similar messages, then analyzed sentiment and frustration indicators to identify which topics cause the most difficulty. Messages containing error reports, repeated requests for help, or expressions of confusion scored higher on our frustration metric. ## Most Discussed Collector Components Topic modeling revealed clear patterns in which Collector components generate the most community discussion. Here are the top components by message volume: ### 1. Prometheus Receiver and Exporter (498 messages, 5.0%) Prometheus integration dominates community discussions. Users frequently ask about: - Configuring the Prometheus receiver to scrape metrics - Setting up the Prometheus remote write exporter - Understanding metric type and metadata preservation across the pipeline - Integrating with existing Prometheus infrastructure This makes sense given Prometheus's widespread adoption. Many organizations start their OpenTelemetry journey by wanting to integrate with or migrate from existing Prometheus setups. The remote write exporter in particular sees heavy use, as it allows teams to continue using Prometheus as a storage backend while adopting OpenTelemetry for collection and processing. ### 2. k8sattributes Processor (258 messages, 2.6%) Kubernetes metadata enrichment is the second most discussed topic. Common challenges include: - Pod association and metadata extraction in DaemonSet deployments - RBAC permissions for accessing the Kubernetes API - Performance implications in large clusters - Interaction with the kubeletstats receiver The complexity of Kubernetes environments and the desire for rich metadata context makes this processor essential but sometimes tricky to configure correctly. Users often discover that running the Collector as a DaemonSet requires different pod association rules than running it as a gateway, leading to troubleshooting cycles that could be avoided with clearer guidance. ### 3. Tail Sampling Processor (167 messages, 1.7%) Tail-based sampling generates significant discussion, often with a higher frustration level than other topics. Users struggle with: - Policy configuration and interaction between multiple policies - Stateful sampling across distributed services - Head sampling vs. tail sampling trade-offs - Debugging why traces are or aren't being sampled - Understanding the decision wait period and its impact on latency The stateful nature of tail sampling, which requires collecting all spans of a trace before making a decision, adds operational complexity that head sampling avoids. Many teams end up running both approaches, using head sampling at the SDK level for baseline reduction and tail sampling in the Collector for intelligent retention of interesting traces. ### 4. Kafka Receiver and Exporter (131 messages, 1.3%) Kafka integration appears frequently, particularly around: - Connection and authentication issues with managed Kafka services (AWS MSK) - Topic configuration and consumer group management - Message format and serialization - High-availability deployment patterns ### 5. Memory Limiter Processor (125 messages, 1.3%) Resource management is a consistent concern: - Proper memory limit configuration relative to container limits - GOMEMLIMIT interaction with the memory limiter - Debugging memory spikes and OOM situations - CPU usage profiling with pprof Understanding the relationship between Go's memory management, container limits, and the memory limiter processor requires knowledge that spans multiple domains. The recent addition of `GOMEMLIMIT` support has helped, but users still need guidance on proper configuration for their specific deployment scenarios. ## Top 10 Problem Areas and Pain Points Beyond component-specific discussions, our frustration analysis identified the topics that cause the most difficulty for users. These represent areas where improved documentation, better error messages, or tooling enhancements could have the highest impact. ### 1. Connection and Export Failures The most frustrating experiences relate to OTLP export failures, particularly: - `DEADLINE_EXCEEDED` errors when exporting to backends - TLS configuration issues with load balancers - gRPC vs. HTTP protocol confusion - Connectivity issues behind proxies or in cloud environments ### 2. Custom Collector Distributions Building custom distributions with `ocb` (OpenTelemetry Collector Builder) generates significant frustration: - Version conflicts between components - Build failures on specific platforms (Windows MSI notably painful) - Dependency resolution issues - Understanding which components to include ### 3. Configuration Syntax and Validation Many users struggle with basic configuration: - YAML syntax errors that produce cryptic error messages - Understanding the relationship between receivers, processors, and exporters - Pipeline configuration and data flow - Environment variable substitution syntax ### 4. Context Propagation Distributed tracing fundamentals cause confusion: - B3 vs. W3C trace context formats - Baggage propagation across service boundaries - Extract and inject operations in SDKs - Cross-language propagation issues ### 5. Attribute and Resource Management Understanding the data model proves challenging: - When to use resource attributes vs. span/metric/log attributes - Moving attributes between resource and signal levels - Semantic conventions compliance - Attribute cardinality and its impact ### 6. OTTL (OpenTelemetry Transformation Language) While powerful, OTTL generates confusion: - Function syntax and available operations - Context-specific paths and accessors - Debugging transformation failures - Performance implications of complex transforms ### 7. Kubernetes Operator and Auto-Instrumentation The Operator simplifies deployment but introduces its own challenges: - Instrumentation injection not working as expected - Multiple collector deployment modes (DaemonSet vs. Sidecar vs. Deployment) - CRD configuration options - Troubleshooting injected agents ### 8. Backend Integration Connecting to observability backends requires effort: - Jaeger configuration and migration from legacy setups - Vendor-specific exporter configuration - Authentication and authorization with managed services - Multi-backend routing ### 9. Docker and Container Deployment Container-related issues appear regularly: - Image selection (contrib vs. core) - Version availability on Docker Hub - Custom image building - Resource limits and performance tuning ### 10. Queue and Retry Behavior Understanding the exporter helper's behavior: - Persistent queue configuration and storage - Retry policies and backoff behavior - Data loss scenarios and prevention - Queue sizing for high-volume deployments **Give your coding agent current Collector knowledge.** OllyGarden's free, open source OpenTelemetry Agent Skills point your coding agent at upstream sources for Collector configuration, OTTL, SDK setup, and semantic conventions, so it works from current documentation instead of stale training data. [Get the skills](https://ollygarden.com/resources/agent-skills) ## What This Tells Us Several themes emerge from this analysis: **The Prometheus ecosystem remains central.** Organizations aren't abandoning Prometheus; they're integrating it with OpenTelemetry. Documentation and tooling that bridges these ecosystems will continue to be valuable. **Kubernetes complexity compounds OTel complexity.** The k8sattributes processor and Operator discussions show that Kubernetes environments introduce additional layers of configuration and troubleshooting. Simplified deployment patterns and better defaults could help. **Sampling is conceptually difficult.** Tail sampling, despite being well documented, generates ongoing confusion. Interactive tools or visualization of sampling decisions might help users understand and debug their configurations. **Error messages need improvement.** Many frustration-heavy discussions start with a cryptic error message. Investing in actionable error messages with suggested fixes would significantly improve the user experience. **The gap between "getting started" and "production ready" is real.** Basic tutorials work, but scaling to production with proper memory limits, persistent queues, and multi-backend routing requires significant learning. ## Moving Forward We hope this analysis helps maintainers and SIGs identify areas where documentation improvements would have the highest impact. The data clearly shows that certain topics, particularly around configuration patterns, sampling strategies, and multi-backend deployments, generate recurring questions that better guides could address. On my end, I have lined up a series of articles that tackle some of these pain points directly, covering topics like decomposing Collector configuration files into manageable pieces, routing telemetry to multiple backends based on tenant or environment, and building effective tail sampling strategies. ## Acknowledgments Thank you to everyone who participates in the OpenTelemetry Slack community. Your questions, error reports, and discussions not only help each other but also provide valuable signal for where the project can improve. A special thanks to the community members who take time to answer questions and share their experiences - the 1% of help responses in our data represent countless hours of volunteer effort that makes this community welcoming for newcomers. *This analysis used topic modeling and sentiment analysis on publicly available Slack messages. Individual messages were aggregated into topics; no personally identifiable information was used in this report.* --- # Meet Rose: OllyGarden's AI Instrumentation Agent Imagine the perfect observability world: There is an incident, the on-call team gets paged in the middle of the night, wakes up and thanks to your telemetry, resolves it fast. Published: 2025-10-29 Canonical URL: https://ollygarden.com/resources/blog/meet-rose-ollygardens-ai-instrumentation-agent ![Rose analyzing OpenTelemetry instrumentation and suggesting fixes via pull requests](https://ollygarden.com/blog-images/meet-rose-ollygardens-ai-instrumentation-agent/image-1.png) Imagine the perfect observability world: There is an incident, the on-call team gets paged in the middle of the night, wakes up and thanks to your telemetry, the root-cause is identified within just a few minutes. Telemetry is produced without sensitive data or passwords and you only pay for what you *actually* need. Dashboards aren't broken or inconsistent and you have confidence in your data. Unfortunately this perfect observability world is often an utopia and the harsh reality looks different. But why is that and what can we do to bring today's reality closer to the perfect observability world? Modern tools and instrumentation approaches have made it easier than ever to collect telemetry data of applications. You press the "easy button" and thanks to powerful tools like eBPF or auto-instrumentation agents, a lot of data will magically appear in the observability backend. Within almost no time, the whole system is suddenly instrumented. Sounds great, doesn't it? Where's the catch? Such approaches are great to get you started quickly and have valid use cases for baseline visibility. However, as your applications scale, the produced amount of telemetry data grows significantly as well. The telemetry data is often low-quality and at a certain point it becomes challenging (and expensive) to maintain and make sense out of the data. To have more control about telemetry, it's possible to manually instrument applications. That gives the power to capture what actually is needed. Even better: Telemetry quality can be enhanced thanks to custom application/business-specific attributes while guaranteeing that no sensitive data is produced. You only pay for what you need and thanks to the reduced amount of low-signal data, relevant issues can be identified faster. While the theory sounds promising, the reality is that manual instrumentation isn't trivial. When done right, it requires consistency across application boundaries, correct context propagation, domain knowledge specific to OpenTelemetry and most importantly engineers who have time and the knowledge to maintain the instrumentation. ## Introducing Rose Today we are announcing the research preview of [**OllyGarden Rose**](https://ollygarden.com/products/rose), our AI instrumentation agent. Rose integrates seamlessly into your development workflow as a **GitHub** app that analyzes OpenTelemetry instrumentation in pull requests, identifies pitfalls and suggests improvements to ensure consistent, high-quality telemetry practices. It’s designed to facilitate the manual instrumentation process by reducing engineering time, ensuring consistency, and providing clear guidance that builds confidence in your telemetry. At a later stage, OllyGarden Rose will be able to do assessments of the instrumentation quality of an entire code repository, or even install and perform an initial instrumentation on its own, guided by our knowledge about what’s good telemetry, as well as other external sources. **Research preview launches October 29, 2025.** [**Learn more about Rose**](https://ollygarden.com/products/rose). ### Key Features #### Context-Aware Analysis Rose understands your entire codebase, not just the diff. It knows your organization's telemetry patterns, recognizes which semantic conventions apply, and understands whether you're instrumenting an HTTP client, database call, or message queue. It provides guidance specific to your exact situation. #### OllyGarden Knowledge Base While general-purpose coding assistants understand OpenTelemetry SDK syntax, they often lack the depth to guide instrumentation across application boundaries and understand the why and what. Built on OllyGarden's expertise from years of contributing to OpenTelemetry, best-practices and industry standards are encoded into actionable rules and patterns that Rose applies automatically. #### OpenTelemetry Education Every comment Rose makes includes an explanation of why something matters, optionally a concrete code suggestion showing how to fix it, and links to relevant OpenTelemetry documentation. The goal isn’t to just fix issues, but to teach and share observability best practices with every pull request. **Measure the telemetry you already send.** OllyGarden Insights analyzes the traces, metrics, and logs your services already emit, flagging noisy signals, PII leaks, and incomplete instrumentation for each service before you decide what to instrument by hand. [Explore Insights](https://ollygarden.com/products/insights) ## Join the Research Preview Our mission at OllyGarden is to bring the reality closer to the perfect observability world, where it's easy to achieve and maintain high-quality telemetry data. In addition to OllyGarden Insights, Rose is another step towards that goal. We'll provide Rose free of charge to selected participants during the research preview period. In return, we expect participants to provide feedback and access to the target source code repository (the one to be instrumented), so that we can analyze what worked and what didn't. NDAs are available for organizations with security requirements. Our goal is to learn from real-world code out there, with any level of instrumentation. We're especially interested in teams already using or planning to adopt manual instrumentation practices. **Ready to participate?** [Contact us](https://ollygarden.com/products/rose) to join the research preview for free. --- # Introducing OllyGarden Tulip: Our Open-Source Distribution of the OpenTelemetry Collector OllyGarden Tulip is a commercially supported OpenTelemetry Collector distribution: stable quarterly releases, LTS every 18 months, open source and free. Published: 2025-10-16 Canonical URL: https://ollygarden.com/resources/blog/introducing-tulip-supported-otel-collector **TL;DR:** We're launching [OllyGarden Tulip](https://github.com/ollygarden/tulip), a commercially supported OpenTelemetry Collector distribution with stable releases, predictable upgrade paths, and professional support from the people who helped build the Collector. It's open source and free to use, with optional commercial support for production deployments. Quarterly releases start with v25.11, with LTS releases every 18 months. Back in 2019, I encountered something that would shape the next six years of my career: the OpenCensus Service. This unassuming piece of infrastructure could receive telemetry data in one format (OpenCensus, Zipkin) and export it in another, like Jaeger. Simple, elegant, powerful. When OpenCensus merged with OpenTracing, the service became OpenTelemetry Service. Coming from the Jaeger world, I immediately saw a naming problem: having an "OpenTelemetry Service" that functioned like Jaeger Collector would confuse everyone. In my first SIG calls, I suggested renaming it to OpenTelemetry Collector. The response? "Too much legacy already. People know it by this name." I was wrong about the timing, but right about the need. Eventually, the community came around, and OpenTelemetry Collector was born. ## Helping Build the Collector Ecosystem That early misstep didn't discourage me. Instead, it pulled me deeper into the community. Over the following years, I implemented authentication support and the first auth mechanism, built the load balancing exporter, created the OpenTelemetry Collector Builder (ocb), developed the OpenTelemetry Operator, and maintained the tail sampling processor. I gave conference talks at events worldwide. More importantly, I talked to users. Hundreds of conversations about their deployments, their challenges, their workarounds. I addressed concerns where I could, but some problems were too big for a single engineer, even one working at larger organizations. ## The Problems I Couldn't Solve Alone The requests came consistently, almost predictably: "Can we get commercial support for the Collector?" Some users had backend vendors who offered Collector support, but only as long as they remained customers. Want to migrate to a different backend? Your Collector support disappears. Companies building custom Collector distributions with proprietary components were left entirely on their own. It was painful. Passionate users consuming my code and projects, and I couldn't offer them the support they needed. The technical challenges were equally frustrating. Teams stuck on ancient Collector versions because upgrades broke their dashboards and alerts when internal telemetry metrics changed. Organizations forced to update configurations for unrelated components just to consume a critical bug fix. Custom distribution maintainers struggling to keep pace with upstream changes while managing their own components. These weren't edge cases. These were real operational pain points affecting production systems at scale. ## Introducing OllyGarden Tulip Today, we're changing that. I'm excited to announce [**OllyGarden Tulip**](https://github.com/ollygarden/tulip), a commercially supported OpenTelemetry Collector distribution that solves the problems I've heard about for years. Tulip provides the stability guarantees, predictable release cycles, and professional support that production systems deserve. It's an open source distribution built using ocb, the same tool I created for the community. You can use it freely, extend it, and build on it. And when you need support, we're here with the deep expertise that comes from years of building and maintaining the Collector itself. This isn't just another distribution. It's the support offering that Collector users have been asking for, delivered by the people who know this codebase intimately. ## Why OllyGarden Tulip Exists When Yuri and I founded OllyGarden at the beginning of this year, our focus was clear: give observability engineers superpowers to understand what's good and what's bad about their telemetry through our Insights platform. That remains our core mission, and we're making significant progress there. But as we've built OllyGarden, I've kept hearing the same pains from Collector users that I've witnessed for years. These aren't problems we can ignore, and they're problems we can solve right now. So we're accelerating our plans and launching **OllyGarden Tulip** today, a commercially supported OpenTelemetry Collector distribution built specifically to address the support and stability challenges that production teams face every day. ## What Makes Tulip Different Tulip provides stability guarantees that match real-world needs. Need a critical bug fix without updating every component? We've got you covered. Upgraded to the latest version and experiencing unexpected performance issues? Throw it at us. Want predictable release cycles that align with your planning? We deliver. Our approach combines flexibility with reliability. We provide quarterly releases tracking upstream versions, starting with v25.11 (November 2025). Every 18 months, we'll release an LTS version, with the first likely at v26.5. The distribution itself is open source, built using ocb. You can use the binaries or container images for free. Need components we don't support yet? Fork our repository and add them. We'll still support the ones in our manifest. Need commercial support? We're here for you. ## Built on Open Source, Backed by Experience OllyGarden Tulip isn't a fork or a proprietary reimagining. It's an open source distribution of the Collector built using the same tools I created, specifically ocb. You can use it freely. You can extend it. You can build on it. What we're offering is something the community has asked for repeatedly: stable, professional support from people who know this codebase intimately. I haven't been as involved in the OpenTelemetry Collector community since January. I've been focused on building our products. But Tulip brings us closer again. More importantly, it provides a support offering that our users deserve. **Give your coding agent Collector knowledge.** OllyGarden's free, open source OpenTelemetry Agent Skills give coding agents like Claude Code and Cursor current Collector configuration, OTTL, and SDK setup knowledge, pointing them at upstream sources of truth. [Get the skills](https://ollygarden.com/resources/agent-skills) ## Who This Is For You should consider OllyGarden Tulip if you run the OpenTelemetry Collector in production and need reliable support, if you build custom Collector distributions and want stable upstream compatibility, if you need predictable upgrade paths that won't break your observability infrastructure, if you want to decouple your Collector support from your backend vendor relationship, or if you value stability and professional support over bleeding-edge features. ## Getting Started OllyGarden Tulip is available now. Our open source manifest and container images are free to use in any environment. For commercial support, contact us to discuss your needs. Visit our documentation site for implementation guides and resources. We're starting this journey with the v25.11 release, and we're committed to the long-term stability that production systems require. ## A Personal Note For six years, I've watched the OpenTelemetry Collector grow from an experimental service to critical infrastructure powering observability at organizations worldwide. I've celebrated its successes and felt the pain of its operational challenges. OllyGarden Tulip represents my commitment to the users who've trusted my code over the years. You've built incredible things. You deserve support that matches your ambition. Let's build something reliable together. **Ready to learn more?** Visit our documentation or contact us to discuss commercial support options. --- # The Variability Principle: How to Decide What Deserves a Span Discover how to manage spans effectively using the Variability Principle to improve trace readability and control costs in OpenTelemetry. Published: 2025-10-06 Canonical URL: https://ollygarden.com/resources/blog/what-deserves-a-span Every team discovers OpenTelemetry the same way. First, excitement—finally, visibility into distributed systems! Then comes the instrumentation party. Spans everywhere. Every function. Every validation. Every calculation gets its own span because "more data is better," right? Three months later, you're staring at a trace with 500 spans trying to figure out why a simple API call took 3 seconds. Your observability bill has grown 10x. And your engineers have given up on traces entirely because they're impossible to read. There's a better way. ## **The Problem: Span Explosion** Most teams create spans like this: ``` func ProcessPayment(ctx context.Context, payment Payment) error { ctx, span := tracer.Start(ctx, "process payment") defer span.End() validateAmount(ctx, payment.Amount) // Another span validateCard(ctx, payment.CardNumber) // Another span calculateFees(ctx, payment.Amount) // Another span formatCurrency(ctx, payment.Total) // Another span // ... 10 more spans for trivial operations } ``` At 10,000 requests per minute with 15 spans each, you're generating 6.5 billion spans per month. At $0.20 per million spans, that's $1,300 monthly just for payment processing traces. But cost isn't the real problem. The real problem is that your traces become unreadable. When everything has a span, nothing stands out. Signal drowns in noise. ## **The Variability Principle: Your New Mental Model** Here's the principle that changed everything for us: **"Is this operation unpredictable?"** If yes, create a span. If no, don't. This simple question cuts through all the complexity. It's not about operation importance or business value—it's about performance predictability. ### **Unpredictable = Create a Span** Operations with unpredictable performance need spans: - **Database queries**: Could take 5ms or 5 seconds depending on locks, data size, indexes - **HTTP calls**: Network latency, retries, timeouts are all variable - **External APIs**: You don't control their performance - **Message queues**: Depends on queue depth, consumer availability - **Cache operations**: Network round-trip to Redis/Memcached - **File I/O**: Disk performance varies, especially with network storage These operations can surprise you. When they're slow, you need to know. ### **Predictable = Skip the Span** Operations with predictable performance don't need spans: - **Validation logic**: Checking if a string contains "@" is always microseconds - **Math calculations**: CPU-bound operations are consistent - **Data transformation**: Mapping objects in memory is deterministic - **String formatting**: Always fast, never the problem - **Getters/setters**: Not worth measuring These operations can't surprise you. They're never the bottleneck. ## **The Pattern in Practice** Let's refactor that payment processing: ``` func ProcessPayment(ctx context.Context, payment Payment) { ctx, span := tracer.Start(ctx, "process payment") defer span.End() // Add context as attributes, not spans span.SetAttributes( attribute.Float64("payment.amount", payment.Amount), attribute.String("payment.currency", payment.Currency), ) // Validation is predictable - no span needed if payment.Amount <= 0 || !isValidCard(payment.CardNumber) { span.RecordError(errors.New("invalid payment")) return } // Database operation is unpredictable - needs a span ctx, dbSpan := tracer.Start(ctx, "INSERT payments") dbSpan.SetAttributes( attribute.String("db.system", "postgresql"), attribute.String("db.collection.name", "payments"), attribute.String("db.operation.name", "INSERT"), ) db.SavePayment(ctx, payment) dbSpan.End() // External API is unpredictable - needs a span ctx, chargeSpan := tracer.Start(ctx, "charge card") paymentGateway.Charge(ctx, payment) chargeSpan.End() } ``` Result: 3 spans instead of 15. Traces are readable. Engineers can actually find problems. **Catch span explosion in code review.** Rose reviews the code in your GitHub repositories for spans that add noise and operations missing telemetry, explains each finding, and prepares source-level fixes for your developers to review. [Meet Rose](https://ollygarden.com/products/rose) ## **What to Use Instead of Spans** When you skip creating a span, you still need to capture information. That's where attributes and events come in. ### **Attributes: Context Without Cost** Attributes add metadata to existing spans. They're perfect for: - Request/response data (user ID, order total, currency) - Configuration values (retry count, timeout settings) - Business context (customer tier, feature flags) ``` span.SetAttributes( attribute.String("user.id", userID), attribute.Float64("order.total", 157.46), attribute.Bool("cache.hit", true), ) ``` Attributes are indexed and searchable. They let you filter traces without creating separate spans. ### **Events: Milestones in Time** Events mark important moments within a span's lifecycle. They're perfect for: - Validation checkpoints - State transitions - Progress markers in loops ``` // Mark validation completion span.AddEvent("validation completed") // Track calculation results span.AddEvent("total calculated", trace.WithAttributes( attribute.Int("line_items.count", 4), attribute.Float64("total", 157.46), )) // Record state changes span.AddEvent("payment saved") // Track retry attempts span.AddEvent("retry attempt", trace.WithAttributes( attribute.Int("attempt", 3), attribute.String("reason", "timeout"), )) ``` Events show you *when* something happened and provide rich context without the overhead of a full span. When debugging, they help you see the timeline of operations within your parent span. ## **The Decision Framework** Before creating any span, ask one question: **"Is this operation unpredictable?"** Yes → Create a span No → Use attributes or events That's it. This single question replaces complex decision trees and eliminates 80% of unnecessary spans. ## **Remember This** Your traces should tell a story, not document every CPU cycle. Each span costs money, performance, and clarity. Create spans only for operations that could surprise you. For everything else, there are attributes and events. The best observability isn't about having all the data—it's about having the right data. --- # How to Name Your Metrics Learn how to name OpenTelemetry metrics: follow semantic conventions, keep the 'who' in attributes, and apply units and naming patterns that scale. Published: 2025-09-09 Canonical URL: https://ollygarden.com/resources/blog/how-to-name-your-metrics Metrics are the quantitative backbone of observability—the numbers that tell us how our systems are performing. This is the third post in our OpenTelemetry naming series, where we've already explored [how to name spans](how-to-name-your-spans) and how to enrich them with meaningful attributes. Now let's tackle the art of naming the measurements that matter. Unlike spans that tell stories about what happened, metrics tell us about quantities: how many, how fast, how much. But here's the thing—naming them well is just as crucial as naming spans, and the principles we've learned apply here too. The "who" still belongs in attributes, not names. ## Learning from Traditional Systems Before diving into OpenTelemetry best practices, let's examine how traditional monitoring systems handle metric naming. Take Kubernetes, for example. Its metrics follow patterns like: - `apiserver_request_total` - `scheduler_schedule_attempts_total` - `container_cpu_usage_seconds_total` - `kubelet_volume_stats_used_bytes` Notice the pattern? **Component name + resource + action + unit**. The service or component name is baked right into the metric name. This approach made sense in simpler data models where you had limited options for storing context. But this creates several problems: - **Cluttered observability backend**: Every component gets its own metric namespace, making it harder to find the right metric among dozens or hundreds of similarly-named metrics - **Inflexible aggregation**: Can't easily sum metrics across different components - **Vendor lock-in**: Metric names become tied to specific implementations - **Maintenance overhead**: Adding new services requires new metric names ## The Core Anti-Pattern: Service Names in Metric Names Here's the most important principle for OpenTelemetry metrics: **Don't include your service name in the metric name**. Let's say you have a payment service. You might be tempted to create metrics like: - `payment.transaction.count` - `payment.latency.p95` - `payment.error.rate` Don't do this. The service name is already available as context through the `service.name` resource attribute. Instead, use: - `transaction.count` with `service.name=payment` - `http.server.request.duration` with `service.name=payment` - `error.rate` with `service.name=payment` Why is this better? Because now you can easily aggregate across all services: ``` sum(transaction.count) // All transactions across all services sum(transaction.count{service.name="payment"}) // Just payment transactions ``` If every service had its own metric name, you'd need to know every service name to build meaningful dashboards. With clean names, one query works for everything. ## OpenTelemetry's Rich Context Model OpenTelemetry metrics benefit from the same rich context model we discussed in our span attributes article. Instead of forcing everything into the metric name, we have multiple layers where context can live: ### Traditional Approach (Prometheus style): ``` payment_service_transaction_total{method="credit_card",status="success"} user_service_auth_latency_milliseconds{endpoint="/login",region="us-east"} inventory_service_db_query_seconds{table="products",operation="select"} ``` ### OpenTelemetry Approach: ``` transaction.count - Resource: service.name=payment, service.version=1.2.3, deployment.environment.name=prod - Scope: instrumentation.library.name=com.acme.payment, instrumentation.library.version=2.1.0 - Attributes: method=credit_card, status=success auth.duration - Resource: service.name=user, service.version=2.0.1, deployment.environment.name=prod - Scope: instrumentation.library.name=express.middleware - Attributes: endpoint=/login, region=us-east - Unit: ms db.client.operation.duration - Resource: service.name=inventory, service.version=1.5.2 - Scope: instrumentation.library.name=postgres.client - Attributes: db.sql.table=products, db.operation=select - Unit: s ``` This three-layer separation follows the OpenTelemetry specification's **Events → Metric Streams → Timeseries** model, where context flows through multiple hierarchical levels rather than being crammed into names. ## Units: Keep Them Out of Names Too Just like we learned that service names don't belong in metric names, **units don't belong there either**. Traditional systems often include units in the name because they lack proper unit metadata: - `response_time_milliseconds` - `memory_usage_bytes` - `throughput_requests_per_second` OpenTelemetry treats units as metadata, separate from the name: - `http.server.request.duration` with unit `ms` - `system.memory.usage` with unit `By` - `http.server.request.rate` with unit `{request}/s` This approach has several benefits: - **Clean names**: No ugly suffixes cluttering your metric names - **Standardized units**: Follow the Unified Code for Units of Measure (UCUM) - **Backend flexibility**: Systems can handle unit conversion automatically - **Consistent conventions**: Aligns with OpenTelemetry semantic conventions The specification recommends using non-prefixed units like `By` (bytes) rather than `MiBy` (mebibytes) unless there are technical reasons to do otherwise. ## Practical Naming Guidelines When creating metric names, apply the same `{verb} {object}` principle we learned for spans, where it makes sense: - **Focus on the operation**: What is being measured? - **Not the operator**: Who is doing the measuring? - **Follow semantic conventions**: Use established patterns when available - **Keep units as metadata**: Don't suffix names with units Here are examples following OpenTelemetry semantic conventions: - `http.server.request.duration` (not `payment_http_requests_ms`) - `db.client.operation.duration` (not `user_service_db_queries_seconds`) - `messaging.client.sent.messages` (not `order_service_messages_sent_total`) - `transaction.count` (not `payment_transaction_total`) ## Real-world Migration Examples | Traditional (Context+Units in Name) | OpenTelemetry (Clean Separation) | Why It's Better | | --- | --- | --- | | payment_transaction_total | transaction.count + service.name=payment + unit 1 | Aggregatable across services | | user_service_auth_latency_ms | auth.duration + service.name=user + unit ms | Standard operation name, proper unit metadata | | inventory_db_query_seconds | db.client.operation.duration + service.name=inventory + unit s | Follows semantic conventions | | api_gateway_requests_per_second | http.server.request.rate + service.name=api-gateway + unit {request}/s | Clean name, proper rate unit | | redis_cache_hit_ratio_percent | cache.hit_ratio + service.name=redis + unit 1 | Ratios are unitless | **Fix metric instrumentation where it lives.** OllyGarden Rose reviews your GitHub repositories for missing or low-value metrics, explains each finding, and prepares source-level fixes for your review, so problems get fixed where the metrics are defined. [Meet Rose](https://ollygarden.com/products/rose) ## Benefits of Clean Naming Separating context from metric names provides specific technical advantages that improve both query performance and operational workflows. The first benefit is cross-service aggregation. A query like `sum(transaction.count)` returns data from all services without requiring you to know or maintain a list of service names. In a system with 50 microservices, this means one query instead of 50, and that query doesn't break when you add the 51st service. This consistency makes dashboards reusable across services. A dashboard built for monitoring HTTP requests in your authentication service works without modification for your payment service, inventory service, or any other HTTP-serving component. You write the query once—`http.server.request.duration` filtered by `service.name`—and apply it everywhere. No more maintaining dozens of nearly-identical dashboards. Some observability vendors now take this further, automatically generating dashboards based on semantic convention metric names—when your services emit `http.server.request.duration`, the platform knows exactly what visualizations and aggregations make sense for that metric. Clean naming also reduces metric namespace clutter. Consider a platform with dozens of services each defining their own metrics. With traditional naming, your metric browser shows hundreds of service-specific variations: `apiserver_request_total`, `payment_service_request_total`, `user_service_request_total`, `inventory_service_request_total`, and so on. Finding the right metric becomes an exercise in scrolling and searching through redundant variations. With clean naming, you have one metric name (`request.count`) with attributes capturing the context. This makes metric discovery straightforward—you find the measurement you need, then filter by the service you care about. Unit handling becomes systematic when units are metadata rather than name suffixes. Observability platforms can perform unit conversions automatically—displaying the same duration metric as milliseconds in one graph and seconds in another, based on what makes sense for the visualization. The metric remains `request.duration` with unit metadata `ms`, not two separate metrics `request_duration_ms` and `request_duration_seconds`. The approach also ensures compatibility between manual and automatic instrumentation. When you follow semantic conventions like `http.server.request.duration`, your custom metrics align with those generated by auto-instrumentation libraries. This creates a consistent data model where queries work across both manually and automatically instrumented services, and engineers don't need to remember which metrics come from which source. ## Common Pitfalls to Avoid Engineers often embed deployment-specific information directly into metric names, creating patterns like `user_service_v2_latency`. This breaks when version 3 deploys—every dashboard, alert, and query that references the metric name must be updated. The same problem occurs with instance-specific names like `node_42_memory_usage`. In a cluster with dynamic scaling, you end up with hundreds of distinct metric names that represent the same measurement, making it impossible to write simple aggregation queries. Environment-specific prefixes cause similar maintenance problems. With metrics named `prod_payment_errors` and `staging_auth_count`, you can't write a single query that works across environments. A dashboard that monitors production can't be used for staging without modification. When you need to compare metrics between environments—a common debugging task—you have to write complex queries that explicitly reference each environment's metric names. Technology stack details in metric names create future migration headaches. A metric named `nodejs_payment_memory` becomes misleading when you rewrite the service in Go. Similarly, `postgres_user_queries` requires renaming if you migrate to something else. These technology-specific names also prevent you from writing queries that work across services using different tech stacks, even when they perform the same business function. Mixing business domains with infrastructure metrics violates the separation between what a system does and how it does it. A metric like `ecommerce_cpu_usage` conflates the business purpose (e-commerce) with the technical measurement (CPU usage). This makes it harder to reuse infrastructure monitoring across different business domains and complicates multi-tenant deployments where the same infrastructure serves multiple business functions. The practice of including units in metric names—`latency_ms`, `memory_bytes`, `count_total`—creates redundancy now that OpenTelemetry provides proper unit metadata. It also prevents automatic unit conversion. With `request_duration_ms` and `request_duration_seconds` as separate metrics, you need different queries for different time scales. With a single `request.duration` metric that includes unit metadata, the observability platform handles conversion automatically. The pattern is clear: context that varies by deployment, instance, environment, or version belongs in attributes, not in the metric name. The metric name should identify what you're measuring. Everything else—who's measuring it, where it's running, which version it is—goes in the attribute layer where it can be filtered, grouped, and aggregated as needed. ## Cultivating Better Metrics Just like the spans we covered earlier in this series, well-named metrics are a gift to your future self and your team. They provide clarity during incidents, enable powerful cross-service analysis, and make your observability data truly useful rather than just voluminous. The key insight is the same one we learned with spans: **separation of concerns**. The metric name describes what you're measuring. The context—who's measuring it, where, when, and how—lives in the rich attribute hierarchy that OpenTelemetry provides. In our next post, we'll dive deep into **metric attributes**—the context layer that makes metrics truly powerful. We'll explore how to structure the rich contextual information that doesn't belong in names, and how to balance informativeness with cardinality concerns. Until then, remember: a clean metric name is like a well-tended garden path—it leads you exactly where you need to go. --- # How to Name Your Span Attributes A practical guide to naming OpenTelemetry span attributes: when to use semantic conventions, how to namespace custom attributes, and pitfalls to avoid. Published: 2025-08-26 Canonical URL: https://ollygarden.com/resources/blog/how-to-name-your-span-attributes Welcome to the second installment in our series on OpenTelemetry naming best practices. In our previous post, we explored how to name spans using the `{verb} {object}` pattern. Today, we're diving into span attributes, the rich contextual data that transforms your traces from simple operation logs into powerful debugging and analysis primitives. This guide targets developers who are: - **Instrumenting their own applications** with custom spans and attributes - **Enriching telemetry** beyond what auto-instrumentation provides - **Creating libraries** that others will instrument The attribute naming decisions you make directly impact the usability and maintainability of your observability data. Let's get them right. ## Start with Semantic Conventions Here's the most important rule that will save you time and improve interoperability: **if an OpenTelemetry semantic convention exists and the semantics match your use case, use it**. This isn't just about convenience—it's about building telemetry that integrates seamlessly with the broader OpenTelemetry ecosystem. When you use standardized attribute names, your data automatically works with existing dashboards, alerting rules, and analysis tools. ### When Semantics Match, Use the Convention | Your Need | Use This Semantic Convention | Why | | --- | --- | --- | | HTTP request method | http.request.method | Standardized across all HTTP instrumentation | | Database collection name | db.collection.name | Works with database monitoring tools | | Service identification | service.name | Core resource attribute for service correlation | | Network peer address | network.peer.address | Standard for network-level debugging | | Error classification | error.type | Enables consistent error analysis | The key principle is **semantic match over naming preference**. Even if you prefer `database_table` over `db.collection.name`, use the semantic convention when it accurately describes your data. ### When Semantics Don't Match, Don't Force It Resist the temptation to misuse semantic conventions: | Don't Do This | Why It's Wrong | | --- | --- | | Using db.collection.name for a file name | Files and database collections are different concepts | | Using http.request.method for business actions | "approve_payment" isn't an HTTP method | | Using user.id for a transaction ID | Users and transactions are different entities | Misusing semantic conventions is worse than creating custom attributes—it creates confusion and breaks tooling that expects the standard semantics. ## The Golden Rule: Domain First, Never Company First When you need custom attributes beyond the semantic conventions, the most critical principle is: **start with the domain or technology, never your company or application name**. This principle seems obvious but is consistently violated across the industry. Here's why it matters and how to get it right. ### Why Company-First Naming Fails | Bad Attribute Name | Problems | | --- | --- | | og.user.id | Company prefix pollutes global namespace | | myapp.request.size | Application-specific, not reusable | | acme.inventory.count | Makes correlation with standard attributes difficult | | shopify_store.product.sku | Unnecessarily ties concept to one vendor | These approaches create attributes that are: - Difficult to correlate across teams and organizations - Impossible to reuse in different contexts - Vendor-locked and inflexible - Inconsistent with OpenTelemetry's interoperability goals ### Domain-First Success Stories | Good Attribute Name | Why It Works | | --- | --- | | user.id | Universal concept, vendor-neutral | | request.size | Reusable across applications | | inventory.count | Clear, domain-specific concept | | product.sku | Standard e-commerce terminology | | workflow.step.name | Generic process management concept | This approach creates attributes that are universally understandable, reusable by others facing similar problems, and future-proof. ## Understanding the Structure: Dots and Underscores OpenTelemetry attribute names follow a specific structural pattern that balances readability with consistency. Understanding this pattern helps you create attributes that feel natural alongside standard semantic conventions. ### Use Dots for Hierarchical Separation Dots (`.`) separate hierarchical components, following the pattern: `{domain}.{component}.{property}` Examples from semantic conventions: - `http.request.method` - HTTP domain, request component, method property - `db.collection.name` - Database domain, collection component, name property - `service.instance.id` - Service domain, instance component, id property ### Use Underscores for Multi-Word Components When a single component contains multiple words, use underscores (`_`): - `http.response.status_code` - "status_code" is one logical component - `system.memory.usage_percent` - "usage_percent" is one measurement concept ### Create Deeper Hierarchies When Needed You can nest further when it adds clarity: - `http.request.body.size` - `k8s.pod.label.{key}` - `messaging.kafka.message.key` Each level should represent a meaningful conceptual boundary. ## Reserved Namespaces: What You Must Never Use Certain namespaces are strictly reserved, and violating these rules can break your telemetry data. ### The `otel.*` Namespace is Off-Limits The `otel.*` prefix is exclusively reserved for the OpenTelemetry specification itself. It's used to express OpenTelemetry concepts in telemetry formats that don't natively support them. Reserved `otel.*` attributes include: - `otel.scope.name` - Instrumentation scope name - `otel.status_code` - Span status code - `otel.span.sampling_result` - Sampling decision **Never create attributes starting with `otel.`** Any additions to this namespace must be approved as part of the OpenTelemetry specification. ### Other Reserved Attributes The specification also reserves these specific attribute names: - `error.type` - `exception.message`, `exception.stacktrace`, `exception.type` - `server.address`, `server.port` - `service.name` - `telemetry.sdk.language`, `telemetry.sdk.name`, `telemetry.sdk.version` - `url.scheme` **Review your span attributes at the source.** OllyGarden Rose connects to your GitHub repositories, reviews spans and attributes in your code for missing or low-value telemetry, explains each finding, and prepares source-level fixes you can review as pull requests. [Meet Rose](https://ollygarden.com/products/rose) ## Semantic Convention Patterns The best way to develop good attribute naming intuition is studying OpenTelemetry's semantic conventions. These represent thousands of hours of design work by observability experts. ### Domain Organization Patterns Notice how semantic conventions organize around clear domains: **Infrastructure Domains** - `service.*` - Service identity and metadata - `host.*` - Host/machine information - `container.*` - Container runtime information - `process.*` - Operating system processes **Communication Domains** - `http.*` - HTTP protocol specifics - `network.*` - Network layer information - `rpc.*` - Remote procedure call attributes - `messaging.*` - Message queue systems **Data Domains** - `db.*` - Database operations - `url.*` - URL components ### Universal Property Patterns Across all domains, consistent patterns emerge for common properties: **Identity Properties** - `.name` - Human-readable identifiers (`service.name`, `container.name`) - `.id` - System identifiers (`container.id`, `process.pid`) - `.version` - Version information (`service.version`) - `.type` - Classification (`messaging.operation.type`, `error.type`) **Network Properties** - `.address` - Network addresses (`server.address`, `client.address`) - `.port` - Port numbers (`server.port`, `client.port`) **Measurement Properties** - `.size` - Byte measurements (`http.request.body.size`) - `.count` - Quantities (`messaging.batch.message_count`) - `.duration` - Time measurements (`http.server.request.duration`) When creating custom domains, follow these same patterns. For inventory management, consider: - `inventory.item.name` - `inventory.item.id` - `inventory.location.address` - `inventory.batch.count` ## Creating Custom Domains Safely Sometimes your business logic requires attributes outside existing semantic conventions. This is normal—OpenTelemetry can't cover every possible business domain. ### Guidelines for Safe Custom Domains - **Choose descriptive, generic names** that others could reuse - **Avoid company-specific terminology** in the domain name - **Follow hierarchical patterns** established by semantic conventions - **Consider if your domain could become a future semantic convention** ### Examples of Well-Designed Custom Attributes | Domain | Good Attributes | Why They Work | | --- | --- | --- | | Business | payment.method, order.status | Clear, reusable business concepts | | Logistics | inventory.location, shipment.carrier | Domain-specific but transferable | | Process | workflow.step.name, approval.status | Generic process management | | Content | document.format, media.codec | Universal content concepts | ## The Rare Exception: When Prefixes Make Sense In rare cases, you might need company or application prefixes. This typically happens when your custom attribute might conflict with attributes from other sources in a distributed system. **Consider prefixes when:** - Your attribute might conflict with vendor attributes in a distributed system - You're instrumenting proprietary technology that's truly company-specific - You're capturing internal implementation details that shouldn't be generalized For most business logic attributes, stick with domain-first naming. ## Your Action Plan Naming span attributes well creates telemetry data that's maintainable, interoperable, and valuable across your organization. Here's your roadmap: - **Always check semantic conventions first** - Use them when semantics match - **Lead with domain, never company** - Create vendor-neutral attributes - **Respect reserved namespaces** - Especially avoid `otel.*` - **Follow hierarchical patterns** - Use dots and underscores consistently - **Build for reusability** - Think beyond your current needs By following these principles, you're not just solving today's instrumentation challenges, you're contributing to a more coherent, interoperable observability ecosystem that benefits everyone. In our next post in this series, we'll shift our focus from spans to metrics, exploring how to name the quantitative measurements that tell us how our systems are performing, and why the same principles of separation and domain-first thinking apply to the numbers that matter most. --- # How to Name Your Spans Good span names make traces readable. Learn the {verb} {object} pattern for naming OpenTelemetry spans and avoid the most common naming mistakes. Published: 2025-08-05 Canonical URL: https://ollygarden.com/resources/blog/how-to-name-your-spans One of the most fundamental yet often overlooked aspects of good instrumentation is naming. This post is the first in a series dedicated to the art and science of naming things in OpenTelemetry. We'll start with spans, the building blocks of a distributed trace, and give you the most important takeaway right at the beginning: how to name the spans that describe your unique business logic. ## Naming your business spans While OpenTelemetry's automatic instrumentation is fantastic for covering standard operations (like incoming HTTP requests or database calls), the most valuable insights often come from the custom spans you add to your own business logic. These are the operations unique to your application's domain. For these custom spans, we recommend a pattern that borrows from basic grammar. Simple, clear sentences often follow a subject -> verb -> direct object structure. The "subject" (the service performing the work) is already part of the trace's context. We can use the rest of that structure for our span name: ## {verb} {object} This pattern is descriptive, easy to understand, and helps maintain low cardinality—a crucial concept we'll touch on later. - **{verb}**: A verb describing the work being done (for example: process, send, calculate, render). - **{object}**: A noun describing what is being acted upon (for example: payment, invoice, shopping_cart, ad). Let's look at some examples: | Bad Name | Good Span Name | Why It's Better | | --- | --- | --- | | process_payment_for_user_jane_doe | process payment | The verb and object are clear. The user ID belongs in an attribute. | | sendinvoice#98765 | send invoice | Aggregable. You can easily find the P95 latency for sending all invoices. | | render_ad_for_campaign_summer_sale | render ad | The specific campaign is a detail, not the core operation. Put it in an attribute. | | calculate_shipping_for_zip_90210 | calculate shipping | The operation is consistent. The zip code is a parameter, not part of the name. | | validation_failed | validate user_input | Focus on the operation, not the outcome. The result belongs in the span's status. | By adhering to the `{verb} {object}` format, you create a clear, consistent vocabulary for your business operations. This makes your traces incredibly powerful. A product manager could ask, "How long does it take to process payments?" and an engineer can immediately filter for those spans and get an answer. ## Why this pattern works So why is `process payment` good and `process*invoice*#98765` bad? The reason is **cardinality**. Cardinality refers to the number of unique values a piece of data can have. A span name should have **low cardinality**. If you include unique identifiers like a user ID or an invoice number in the span name, you will create a unique name for every single operation. This floods your observability backend, makes it impossible to group and analyze similar operations, and can significantly increase costs. The `{verb} {object}` pattern naturally produces low-cardinality names. The unique, high-cardinality details (`invoice\_#98765, user_jane_doe`) belong in **span attributes**, which we will cover in a future blog post. **Get span naming feedback on your code.** OllyGarden Rose reviews your GitHub repositories for missing or low-value instrumentation, like spans that need better names, explains why, and prepares source-level fixes for your developers to review. [Meet Rose](https://ollygarden.com/products/rose) ## Learning from Semantic Conventions This `{verb} {object}` approach isn't arbitrary. It's a best practice that reflects the principles behind the official **OpenTelemetry Semantic Conventions (SemConv)**. SemConv provides a standardized set of names for common operations, ensuring that a span for an HTTP request is named consistently, regardless of the language or framework. When you look closely, you'll see this same pattern of describing an operation on a resource echoed throughout the conventions. By following it for your custom spans, you are aligning with the established philosophy of the entire OpenTelemetry ecosystem. Let's look at a few examples from SemConv. ### HTTP spans For server-side HTTP spans, the convention is `{method} {route}`. - **Example:** `GET /api/users/:ID` - **Analysis:** This is a verb (`GET`) acting on an object (`/api/users/:id`). The use of a route template instead of the actual path (`/api/users/123`) is a perfect example of maintaining low cardinality. ### Database spans Database spans are often named `{db.operation} {db.name}.{db.sql.table}`. - **Example:** `INSERT my_database.users` - **Analysis:** This is a verb (`INSERT`) acting on an object (`my_database`.users). The specific values being inserted are high-cardinality and are rightly excluded from the name. ### RPC spans For Remote Procedure Calls, the convention is `{rpc.service}/{rpc.method}`. - **Example:** `com.example.UserService/GetUser` - **Analysis:** While the format is different, the principle is the same. It describes a method (`GetUser`), which is a verb, within a service (`com.example.UserService`), which is the object or resource. The key takeaway is that by using `{verb} {object}`, you are speaking the same language as the rest of your instrumentation. ## Cultivating a healthy system Naming spans is not a trivial task. It's a foundational practice for building a robust and effective observability strategy. By adopting a clear, consistent pattern like `{verb} {object}` for your business-specific spans, you can transform your telemetry data from a tangled mess into a well-tended garden. A well-named span is a gift to your future self and your team. It provides clarity during stressful outages, enables powerful performance analysis, and ultimately helps you build better, more reliable software. In our next post in this series, we will dig into the next layer of detail: **span attributes**. We'll explore how to add the rich, high-cardinality context to your spans that is necessary for deep debugging, without compromising the aggregability of your span names. --- # 🌱 Cultivating Unique service.instance.id on NGINX Ingress with OpenTelemetry The NGINX Ingress Controller has no knob for the OpenTelemetry service.instance.id attribute. Here's how to inject a unique ID per pod with the Helm chart. Published: 2025-07-31 Canonical URL: https://ollygarden.com/resources/blog/cultivating-unique-serviceinstanceid-on-nginx-ingress-with-opentelemetry A task to set a unique `service.instance.id` on NGINX Ingress Controller using the OpenTelemetry should be simple, right? But as it turned out, the ingress controller doesn't expose all of NGINX’s OTel knobs out of the box, so I had to roll my own tweak garden. ## Why should I care? Without unique instance IDs, your tracing data looks like a tangled tangleweed—hard to trace, difficult to debug, and completely undermines what we’re trying to do with observability. **Check your own traces for gaps like this.** OllyGarden Insights analyzes the traces, metrics, and logs you already send and flags quality problems per service, such as incomplete instrumentation, before they make a debugging session harder. [Explore Insights](https://ollygarden.com/products/insights) ## The solution ### Part 1 - Nginx Ingress Controller - Helm Chart To get `POD_UID` injected into each trace, here’s the minimal yet powerful config snippet I landed on: - Set the POD_UID environment variable, once NGINX Ingress Controller set by default only the POD_NAME and POD_NAMESPACE. ``` extraEnvs: - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid ``` - Set the POD_UID as span-attribute for instance. ``` controller: config: main-snippet: | env POD_UID; server-snippet: | set $pod_uid "unknown"; access_by_lua_block { ngx.var.pod_uid = os.getenv("POD_UID") or "unknown" } opentelemetry_attribute service.instance.id $pod_uid; ``` ### Part 2 - OpenTelemetry Collector Config - Configure the [transform](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) processor to set the resource-attribute. ``` processors: transform: error_mode: ignore trace_statements: - set(resource.attributes["service.instance.id"], span.attributes["service.instance.id"]) - delete_key(span.attributes, "service.instance.id") ``` ### How it works: - Explicitly exposes the `POD_UID` environment variables so NGINX can see them. - Initializes a default `$pod_uid`—my fail-safe in case the env var goes missing. - Uses Lua to pull in the real `POD_UID`. - Finally, sets the OpenTelemetry attribute `service.instance.id` to match the actual UID. - Through OpenTelemetry Collector capture the `service.instance.id` from span-attributes and set it on resource-attributes. ### Benefits to your telemetry: | Without unique service.instance.id | With unique service.instance.id | | --- | --- | | Pods share the same instance ID, causing confusion in telemetry. | Each pod is individually identifiable. | | Difficult to isolate errors and debug effectively. | Allows precise tracing to individual pods. | | Limited visibility into pod-specific issues. | Enables accurate root-cause analysis. | ### Final result By implementing this solution, each NGINX ingress pod is clearly distinguishable in tracing data. This improves observability significantly by providing accurate, pod-specific telemetry, facilitating precise troubleshooting and diagnostics. --- # Introducing OllyGarden OllyGarden comes out of stealth with pre-seed funding to fight bad telemetry: helping teams understand, score, and improve the telemetry they generate. Published: 2025-07-09 Canonical URL: https://ollygarden.com/resources/blog/introducing-ollygarden In the rush for visibility, many organizations find themselves lost in an overgrown jungle of data. Teams generate a constant stream of telemetry, hoping it will sprout into useful insights. Instead, they often end up with "bad telemetry"—data that is noisy, irrelevant, or incomplete, driving up costs and obscuring the very answers they seek. Today, we're coming out of stealth to introduce [OllyGarden](https://ollygarden.com/), a new company dedicated to tending this garden of observability. Backed by a **pre-seed round** led by [DIG Ventures](https://www.dig.ventures/), with investments from observability leaders like [Datadog Ventures](https://www.datadoghq.com/), [Grafana Labs](https://grafana.com/), and [Dash0](https://www.dash0.com/), as well as special angels with deep knowledge in our space, like [Batuhan Uslu](https://www.linkedin.com/in/batuhanuslu/), [Ben Sigelman](https://www.linkedin.com/in/bensigelman/), [Chris Aniszczyk](https://www.linkedin.com/in/caniszczyk/), and [Ian Livingstone](https://www.linkedin.com/in/irlivingstone/), we're ready to get to work. Our mission is simple: to improve the efficiency of telemetry pipelines. We believe the first and most crucial step is to help companies understand and optimize the telemetry they are already generating. ## **The Weeds We’re Tackling: Real-World Telemetry Pains** The problem of bad telemetry isn't theoretical; it's a daily struggle for engineering teams everywhere. During our research, we heard the same stories again and again. These might sound familiar to you: - **Runaway Costs:** An engineer at a company based in Berlin mentioned that they had a single high-cardinality metric generating over **$20,000 worth of telemetry per month** without a clear value. After reducing the cardinality, the cost was reduced to $2,000. How many such metrics are hiding in your pipelines, undetected? - **Broken Insights & Poor User Experience:** What happens when telemetry fails? An engineer working at a company in Australia mentioned that their support team uses distributed tracing to debug user-reported problems, but incomplete traces limit their ability to help their users, directly impacting customer satisfaction. - **The "Instrument Everything" Trap:** Auto-instrumentation tools are powerful, but if left unconfigured they can gather far more information than necessary. This can generate extremely high volumes of data, overloading systems before you even get a chance to analyze the data. - **Vast, Unchecked Data Volumes:** Another company shared that they generate about **3 Petabytes of uncompressed telemetry per month**, acknowledging that among that trove of data, "a lot of it is bad telemetry". With industry experts estimating that up to 90% of telemetry data goes unused, imagine how many CPU cycles, collector instances, and how much egress traffic could be spared if that bad telemetry wasn't generated in the first place. ## **Our First Step: OllyGarden Insights & the Instrumentation Score** You can't improve what you can't measure. For too long, evaluating telemetry quality has been a subjective exercise based on gut feelings and tribal knowledge. We're tackling this problem by giving observability engineers superpowers, allowing them to see exactly how good the telemetry is inside their pipelines. Our first product, **OllyGarden Insights**, analyzes your telemetry streams to give you deep insights into your data quality. It assesses your instrumentation against best practices, identifies services that are over- or under-instrumented, and helps you make informed, data-driven decisions about what to change. To provide a common vocabulary for this, we launched the **Instrumentation Score**, a standardized value to objectively assess OpenTelemetry instrumentation. We believe such a fundamental metric for telemetry health shouldn't be proprietary. That’s why we initiated the Instrumentation Score as an **open-source effort**, with an open governance model and support from partners like Dash0, Datadog, Grafana Labs, Honeycomb, New Relic, and Splunk. This is our first major contribution back to the community we care so much about, and a testament to how we plan to operate. **Fix bad telemetry in the code.** OllyGarden Rose reviews your GitHub repositories for missing or low-value telemetry, explains each finding, and prepares source-level fixes for your review, so noisy or incomplete telemetry gets fixed where it is produced. [Meet Rose](https://ollygarden.com/products/rose) ## **Why OllyGarden? Our Roots in OpenTelemetry** We know the challenges of telemetry because we've been helping build the solutions for years. OllyGarden was founded by OpenTelemetry veteran Juraci Paixão Kröhling and SRE expert Yuri Oliveira Sá. As a member of the OpenTelemetry Governance Committee, creator of the OpenTelemetry Operator and OpenTelemetry Collector Builder (ocb), and one of the project's top contributors, Juraci has been on the front lines of the observability revolution. This deep experience shapes our core philosophy: - **A Vendor-Neutral Approach:** Our business is to make your telemetry more efficient. We are a neutral partner you can trust, acting as a complement to observability backends by helping their customers send them higher-quality data. We succeed when you gain clarity, regardless of where your data is stored. - **The Start of Our Journey:** OllyGarden Insights is our first major step. We are focused on delivering immediate value by helping you understand and improve the telemetry you already have. We are just at the beginning of our journey and are incredibly excited about the future, but our commitment today is clear: to empower engineers with the backend-neutral tools they need to cultivate clarity from their data. ## **Start Cultivating Better Telemetry Today** It's time to move beyond the jungle of bad telemetry and start purposefully cultivating a garden of clear, actionable insights. We are now engaging with **early users**. By joining us, you will not only see the power of purposeful instrumentation firsthand but also have the unique opportunity to help shape our product and secure special early-bird pricing. We'll analyze a sample of your telemetry and provide you with your **Instrumentation Score** and actionable insights for improvement. Ready to see what's really growing in your telemetry pipelines? Contact us at [**contact@olly.garden**](mailto:contact@olly.garden) or visit our website to learn more. --- # Introducing the Instrumentation Score The Instrumentation Score is an open, standardized measure of OpenTelemetry instrumentation quality, scoring OTLP data against best-practice rules. Published: 2025-06-11 Canonical URL: https://ollygarden.com/resources/blog/instrumentation-score *This post announced the Instrumentation Score in June 2025. The current definition, every rule with its weight, and a worked example live on the [Instrumentation Score page](https://ollygarden.com/resources/instrumentation-score), and what the score measures is explained in [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality)* Telemetry data is the foundation of our observability. We gather metrics, traces, and logs, aiming to cultivate a clear understanding of application health. Yet, a persistent question often arises: "Is our telemetry actually *good*?" How do we distinguish valuable insights from data that merely consumes resources? For too long, evaluating instrumentation effectiveness has been a subjective exercise. We've lacked a common language or a standard measure to truly understand if our telemetry is enriching our insights or just overgrowing the plot. At [OllyGarden](https://ollygarden.com/), we recognize this challenge and are building a product to give super powers to observability engineers. We heard from them that our Instrumentation Score was something special. And we believe that [if you want something really special, you share it](https://www.youtube.com/watch?v=vhYMRtqvMg8&t=118s). Today, OllyGarden introduces its first major contribution to the observability ecosystem: the **Instrumentation Score**. The Instrumentation Score is a standardized, numerical value designed to objectively assess the consistency and effectiveness of [OpenTelemetry](https://opentelemetry.io/) instrumentation. It analyzes [OTLP](https://opentelemetry.io/docs/specs/otlp/) (OpenTelemetry Protocol) data streams against a predefined set of rules rooted in OpenTelemetry best practices and [semantic conventions](https://opentelemetry.io/docs/specs/semconv/). It’s a health check for your telemetry, providing a clear, actionable measure of its quality. As [Juraci Paixão Kröhling](https://www.linkedin.com/in/jpkroehling/), Co-founder at OllyGarden, states: "As an OpenTelemetry contributor and enthusiast, I've seen firsthand the project's power to democratize instrumentation. Yet, a persistent question has always been: 'Are we generating good telemetry?' Too often, the answer is unclear, leading to missed insights or wasted resources. The Instrumentation Score, an initiative we're launching from OllyGarden, aims to provide that clarity. It's about establishing a common, actionable language for telemetry quality, built on OpenTelemetry principles, to empower every engineer and organization to confidently improve their observability practices and truly harness the value of their data." The Instrumentation Score provides a common vocabulary for discussing instrumentation effectiveness. For **engineers and SREs**, it offers actionable guidance, highlighting where instrumentation can be improved. For **CTOs and technology leaders**, the strategic value includes improved ROI on observability by focusing on [purposeful telemetry](/resources/blog/purposeful-instrumentation) and reducing [bad telemetry](/resources/blog/theres-a-lot-of-bad-telemetry-out-there). **See your own Instrumentation Score.** OllyGarden Insights analyzes the traces, metrics, and logs you already send, gives you an Instrumentation Score, and flags the quality problems behind it for each service, from noisy signals to missing ownership. [Explore Insights](https://ollygarden.com/products/insights) OllyGarden is committed to OpenTelemetry and the open-source ecosystem. The Instrumentation Score leverages OpenTelemetry Semantic Conventions and analyzes OTLP data. Crucially, we've ensured this initiative is not only open source but is being developed with an open governance model, with support or contributions from key industry players like [Dash0](https://www.dash0.com/), [New Relic](https://newrelic.com), [Splunk](https://splunk.com/), [Datadog](https://datadog.com/), and [Grafana Labs](https://grafana.com/). The specification is an open-source effort, and we are actively requesting observability engineers to contribute with their own rules by opening a pull request against the GitHub repository. Our goal is a collaborative evolution, with the Instrumentation Score eventually finding a home within a neutral foundation. The introduction of the Instrumentation Score is a step towards a future where organizations can confidently understand and improve their telemetry. We invite you to learn more and get involved: - Explore the **Instrumentation Score landing page**: [https://github.com/instrumentation-score/spec](https://github.com/instrumentation-score/spec) - Review the **specification and contribute on GitHub**: [https://github.com/instrumentation-score/spec](https://github.com/instrumentation-score/spec) OllyGarden aims to improve the efficiency of telemetry pipelines. The Instrumentation Score is the first seed we’re planting, hoping that together, we can help everyone grow a more effective observability practice. --- # Concrete Applications of Purposeful Instrumentation From NASA-grade telemetry design to code-level tips: concrete techniques for applying purposeful instrumentation in real-world OpenTelemetry systems. Published: 2025-06-04 Canonical URL: https://ollygarden.com/resources/blog/concrete-applications-of-purposeful-instrumentation In our [Purposeful Instrumentation](/resources/blog/purposeful-instrumentation) blog post, we laid the groundwork for a more disciplined approach to observability. We argued that the goal isn't to merely amass data, but to **cultivate high-quality telemetry signals** – focusing on quality over quantity. The aim is to transform our experience during high-pressure incidents from frantically searching through a "dense thicket of irrelevant data" to confidently navigating a "well-lit path to the root cause." Many of us have experienced the pitfalls of the "instrument everything" mantra. While well-intentioned, it often leads to an "overgrown jungle of telemetry data," where critical signals are drowned out by noise. Purposeful instrumentation, in contrast, encourages us to strategically gather the *right* data. This isn't just about digital decluttering; it yields tangible benefits: **reduced noise, faster troubleshooting, and improved clarity and maintainability** in our systems. This post moves from philosophy to practice. We'll dive into concrete examples and techniques, showcasing how to apply purposeful instrumentation in real-world scenarios—from initial telemetry design to ongoing pipeline adjustments and even code-level optimizations. For the dimensions these techniques improve, see [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality) ## **Designing Telemetry with NASA's Rigor** When we think about systems operating under the most severe limitations, spacecraft telemetry, particularly from missions like NASA's Mars rovers, offers [profound inspiration](https://www-robotics.jpl.nasa.gov/media/documents/Flight_Software_Case_Study_Spacecraft_Telemetry.pdf). The extreme constraints of space exploration—limited bandwidth, power, and processing capabilities—force engineers to meticulously justify and optimize every single bit of data transmitted. For observability engineers on Earth, even without such stark limitations, these practices offer invaluable lessons in **cultivating efficiency**. Here are some key takeaways: - **Data Type Optimization**: Spacecraft systems often convert 64-bit floating-point numbers to 32-bit or even 16-bit integers. Sometimes, scaled integers (like centi-degrees Celsius) are used to preserve essential precision while drastically reducing data volume. For our enterprise systems, this prompts a critical question: Do we *really* need microsecond precision for every timer, or would seconds suffice for certain metrics, thereby reducing storage and processing overhead? - **Bit Packing and Enumerated Types**: To save space, boolean flags and enumerated values with a limited set of states are often packed into smaller integer types on spacecraft. For example, 15 distinct safety checks might be encoded into a single 16-bit integer. This principle is directly applicable to software telemetry, particularly in how we design attributes to **reduce cardinality** and data volume. Instead of verbose string representations for statuses, can an enumerated integer suffice? - **Configurable Data Collection**: Spacecraft aren't static in their data collection. They possess "knobs" that allow operators to increase data verbosity for anomaly investigations, switching between "Brief records" for nominal operations and "Verbose records" when digging deeper. This mirrors the need in our systems for dynamic control over telemetry, perhaps adjusting log levels or sampling rates based on operational context rather than maintaining a constant, high-volume stream. - **Summary Data and Compression**: Reporting small, high-level summary data packets independently from detailed diagnostic data products allows for quick operational decision-making. If summaries are nominal, large, detailed data products might even be discarded to save precious bandwidth. Lossless compression is also a standard practice, always balancing the CPU cost of compression/decompression against bandwidth savings. - **The "Very Small Products" Problem**: Interestingly, generating a multitude of tiny data products can be inefficient, consuming storage slots and impacting system performance, as was observed with the Mars 2020 rover's packetizer. This highlights the importance of **batching and aggregation** not just for network efficiency but also for processing and storage optimization within our telemetry pipelines. The OpenTelemetry Collector’s batch processor is a prime example of applying this principle. These extreme examples from NASA underscore a fundamental discipline: diligently asking, "What data do I *really* need?" and "What is the cost versus the value?" This scrutiny is crucial for building sustainable and effective telemetry strategies, ensuring we're not just collecting data, but harvesting actionable insights. ## **Tuning Automatic Instrumentation for Precision with OpenTelemetry** OpenTelemetry's [auto-instrumentation](https://opentelemetry.io/docs/zero-code/) agents are a massive boon, offering broad telemetry coverage for popular libraries and frameworks with minimal upfront effort. It’s tempting to see this as "zero code, zero thought." However, this convenience doesn't absolve us from the need for **purposeful configuration**. Blindly enabling instrumentation for every conceivable library can quickly lead back to that "overgrown jungle of telemetry data," swamping your systems with noise and incurring unnecessary costs. - **Review Default Configurations**: Auto-instrumentation defaults are often tuned for maximum coverage, which might not align with your specific observability goals or the critical paths of your application. As [Elena Kovalenko of Delivery Hero noted](https://youtu.be/QzStkLbA7Qk), unconfigured auto-instrumentation can generate extremely high cardinality and massive data volumes, potentially overloading collectors and backend systems. It’s vital to treat the default settings as a starting point, not a final destination. - **Selectively Disable Unnecessary Instrumentation**: Most OpenTelemetry auto-instrumentation agents allow for fine-grained control, enabling you to disable instrumentation for components that are irrelevant to your critical diagnostic paths or those known to produce excessive, low-value data. **Concrete Example: Suppressing JDBC Telemetry**: If your primary diagnostic focus is at the service interaction level, the verbose telemetry generated by JDBC instrumentation (tracing every database call) might be more noise than signal. With the OpenTelemetry Java agent, for instance, you can easily [disable this by setting](https://opentelemetry.io/docs/zero-code/java/agent/disable/) the environment variable `OTEL_INSTRUMENTATION_JDBC_ENABLED=false`. This targeted **pruning** ensures that resources aren't wasted collecting, processing, and storing data that doesn't contribute significantly to your understanding of system health. Auto-instrumentation plants the seeds of visibility; purposeful configuration helps you cultivate the desired crop, ensuring a healthy yield of actionable insights rather than a field of weeds. ## **Optimizing Data Flow with the OpenTelemetry Collector: Pipeline Adjustments** The OpenTelemetry Collector is more than just a telemetry forwarder; it's a powerful, vendor-agnostic control plane. It’s a great place to implement purposeful telemetry strategies by filtering, sampling, enriching, and transforming data *before* it even reaches your observability backends. Let's look at how sophisticated organizations are leveraging the Collector. ### **eBay's Journey: Scaling Distributed Tracing with Cost Optimization** [Handling telemetry at eBay's scale](https://youtu.be/qq8hTct8zm4?si=t5pnl-tyV9-WQ6VP)—ingesting 6.5 million spans per second—necessitates highly judicious instrumentation and aggressive optimization. They faced challenges with broken call chains due to context propagation issues and the difficulty of applying uniform sampling across APIs with vastly different traffic volumes. Their approach to sampling evolved: - **Initial Strategy**: They started with head sampling at the client (e.g., 2% of requests) combined with parent-based sampling to ensure entire traces were captured if any part was sampled. - **Adding Tail Sampling**: After that, they employed a tail-sampling strategy to retain "interesting" traces—those with errors, high latency, or specific critical attributes—along with a baseline 1% of successful traces, storing these for 14 days. This allowed them to focus retention on the most valuable diagnostic data. - **Evolving Tail Sampling with OTel Collector**: Recognizing the significant memory and complexity challenges of performing in-memory tail sampling within the OpenTelemetry Collector for long-duration traces or requests spanning multiple clusters, eBay pivoted. They now leverage **exemplars from metrics** to identify traces of interest. These traces are then copied from a raw trace table to a sampled table after a 10-15 minute delay. This innovative, storage-based tail sampling approach demonstrates a mature balance between comprehensive diagnostic capability and cost control. ### **TomTom's Centralized Control: Enforcing Governance and Flexibility** [TomTom implemented](https://engineering.tomtom.com/opentelemetry-simpifying-observability/) a **centralized OpenTelemetry Collector Service** that acts as a gateway between their internal applications and various SaaS observability platforms. This central hub provides several advantages: - **Governance and Standardization**: It allows them to enforce authentication, manage general configurations like batching and encryption consistently, and, crucially, handle **data enrichment and manipulation** centrally. - **Filtering and PII Redaction**: They use the filterprocessor to drop noisy or irrelevant logs (e.g., from specific Kubernetes namespaces). For sensitive data, a combination of the transformprocessor and attributesprocessor is used to redact Personally Identifiable Information (PII) before telemetry leaves their trust boundary. - **Telemetry Enrichment**: Data is enriched with valuable metadata, such as an "owner" label, which provides better context during troubleshooting and improves accountability. - **Strategic Benefits**: This centralized model offers flexibility in switching telemetry backends, enforces data governance policies, and has proven critical for cost control and maintaining data quality at an enterprise scale. These real-world examples illustrate the power of the OpenTelemetry Collector as a central point for **cultivating telemetry quality**. **See where your own telemetry is noisy.** OllyGarden Insights analyzes the traces, metrics, and logs you already send and flags noisy signals, PII leaks, and high cardinality per service, so you can see where filtering and redaction would pay off. [Explore Insights](https://ollygarden.com/products/insights) ## **Crafting Intentional Manual Instrumentation: OllyGarden’s Example** While auto-instrumentation provides breadth, manual instrumentation offers depth and precision. But even here, more isn't always better. A common pitfall is "over-spanning": creating an excessive number of highly granular spans for minor, sequential internal operations within a single logical unit of work. This can obscure the true flow of a request, add unnecessary overhead, and make traces harder to interpret—akin to "wandering aimlessly in the woods" instead of following a clear path. For example, a single logical `onTraces` operation might be fragmented into several child spans for `processResourceSpans`, cluttering the trace view and inflating span counts unnecessarily. Here’s the original Go code we wrote and landed in production: ``` ctx, span := telemetry.Tracer().Start(ctx, "tendril.processResourceSpans") defer span.End() // Extract service information from resource svcName := getResourceString(rs.Resource(), attrServiceName) span.SetAttributes(attribute.String("service.name", svcName)) svcVersion := getResourceString(rs.Resource(), attrServiceVersion) span.SetAttributes(attribute.String("service.version", svcVersion)) svcEnv := getResourceString(rs.Resource(), attrEnvironmentName) span.SetAttributes(attribute.String("deployment.environment.name", svcEnv)) ``` **The Purposeful Solution: Leveraging Span Events** Instead of creating distinct child spans for every micro-step, it's often far more effective to **consolidate these internal milestones as span events within a single, overarching span** that represents the larger logical operation. This aligns with the core principle of choosing the "most effective signal type for your defined purpose." Logs provide detailed context for discrete occurrences, metrics track aggregatable trends, and traces show flow; span events offer a way to add rich, contextual markers to a span without creating new ones. And here’s the code after the fine-tuning: ``` span := trace.SpanFromContext(ctx) // Extract service information from resource svcName := getResourceString(rs.Resource(), attrServiceName) svcVersion := getResourceString(rs.Resource(), attrServiceVersion) svcEnv := getResourceString(rs.Resource(), attrEnvironmentName) span.AddEvent("processing resource spans for service", trace.WithAttributes( attribute.String("service.name", svcName), attribute.String("service.version", svcVersion), attribute.String("deployment.environment.name", svcEnv), )) ``` **Benefits of Using Span Events Over Excessive Child Spans:** - **Clearer Trace Representation**: A single span with well-defined events provides a **cleaner, more focused view** of a component's internal workings within the context of the larger trace. This gives a "well-lit path" to understanding that component's behavior. - **Reduced Overhead and Cost**: Span events are generally lighter-weight than full spans. This translates to **reduced data volume** and consequently **lower processing and storage costs** in your observability backend. - **Enhanced Context**: Events, with their associated attributes, allow you to capture crucial details (e.g., input size, processing duration for a specific sub-task, success/failure flags) at precise points within the operation, without fragmenting the trace into many tiny pieces. ## **Conclusion: Towards Insightful and Economical Observability** Moving from indiscriminate data collection to **purposeful software telemetry** is more than an engineering exercise; it's a strategic imperative. It ensures that our substantial investments in observability deliver tangible business value—faster incident resolution, optimized performance, and controlled costs—rather than just overwhelming data lakes. This journey of **continuous cultivation** is not a one-off task. It requires ongoing review, governance, and a feedback loop where insights from incidents, performance anomalies, and cost reports are fed back into your instrumentation design and data pipeline policies. As your systems evolve, so too must your telemetry strategy. The guiding questions we discussed in our previous post remain your most valuable tools: - "What question are we trying to answer with this data?" - "What data do we *truly* need, and at what precision?" - "Why *this specific signal type* (metric, log, trace, event)?" - "How will this data actually be *used* and by whom?" - "And critically, what is its ongoing *cost versus its value*?" By consistently applying this critical lens, engineering teams can cultivate an observability practice that is not only powerful and insightful but also sustainable and economically sound. This deliberate, adaptive, and insight-driven approach is the future of effective software observability. OllyGarden is committed to being a neutral and valuable partner in this ecosystem, helping you analyze, optimize, and manage your OpenTelemetry pipelines to harvest the richest insights efficiently. --- # Purposeful Instrumentation Purposeful instrumentation means gathering the right telemetry, not all of it. Cut noise, troubleshoot faster, and lower observability costs. Published: 2025-05-01 Canonical URL: https://ollygarden.com/resources/blog/purposeful-instrumentation It’s the middle of the night. An alert jolts you awake – a critical service is sputtering. Your mind races as you dive into a labyrinth of dashboards, logs, and traces. Are you navigating a well-lit path to the root cause, or are you lost in a dense thicket of irrelevant data? In these high-pressure moments, the quality – not just the quantity – of your observability instrumentation is what truly counts. Many teams, in their quest for visibility, fall into the trap of "instrument everything." The intention is good, but the result is often an overgrown jungle of telemetry data: noisy metrics, verbose logs, and sprawling traces that obscure rather than illuminate, or perhaps a monoculture of one of those telemetry data types. This is where the practice of Purposeful Instrumentation comes in – a disciplined approach to cultivating high-quality observability signals. It's about moving beyond simply collecting data to strategically gathering the right data to understand system health, optimize performance, and troubleshoot effectively. Think of it as tending a garden: you don't just let everything grow wild; you carefully select, nurture, and prune to ensure a healthy and productive yield. It's fundamentally about quality over quantity, having the telemetry you need, without excess. For what quality means in practice, from service identity to sensitive data, see our guide [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality) ## Why Prune the Noise? Adopting a purposeful approach isn't just about tidiness; it delivers tangible benefits that directly impact your team's effectiveness and your organization's bottom line. - **Reduced Noise & Increased Signal:** Over-instrumentation creates a cacophony. Imagine trying to hear a single bird's song in the middle of a roaring stadium. Purposeful instrumentation acts like a filter, silencing the distracting roar and amplifying the signals that truly indicate system behavior and potential issues. You focus your resources on telemetry that provides genuine insight, making it easier to spot anomalies and trends. - **Faster Troubleshooting & Resolution:** When an incident occurs, time is critical. Sifting through irrelevant data wastes precious minutes, if not hours. With instrumentation designed to answer specific questions or diagnose common failure modes, you have targeted data trails leading you directly towards the problem's source. It’s the difference between wandering aimlessly in the woods and following a clearly marked trail. - **Significant Cost Optimization:** Telemetry data isn't free. Storage, processing, and analysis all incur costs, which can escalate rapidly with high data volumes and cardinality. Instrumenting only what provides clear value ensures you're not paying to store noise. This optimizes your observability spend and demonstrably increases the return on your investment (ROI). Think of it as allocating water and fertilizer only to the plants you intend to grow. - **Improved Clarity & Maintainability:** Code cluttered with arbitrary instrumentation is harder to read, understand, and maintain. When instrumentation is added with clear intent, documented appropriately (even if informally via commit messages or code comments), it serves as a form of living documentation. Future engineers (including your future self!) can readily grasp *why* a particular metric, span, or log statement exists and how it contributes to understanding the system. ## Guiding Questions for Purposeful Instrumentation Before adding *any* new metric, span, span event, or log line, pause and cultivate intention by asking critical questions: - **What question am I trying to answer?** This is the cornerstone. Are you trying to understand latency distribution, error rates under specific conditions, resource consumption patterns, or the flow of requests across services? Defining the question sharpens the focus of your instrumentation. Don't aim to predict every *possible* future question, but consider the *types* of questions most likely to arise based on the service's function and history. What are the known failure modes or performance bottlenecks for this component? - **What data do I *really* need to answer this?** Challenge the defaults. Do you need millisecond precision, or would seconds suffice? Do you need the full user ID (potentially creating high cardinality), or could you use a user *type* or a randomized cohort ID? Can data be aggregated at the source to reduce volume and cardinality? For example, instead of logging every request, could you use metrics instead with enough labels to distinguish between outcomes? - **Why *this* type of signal (Metric, Trace, Log)?** Each signal type has strengths. Metrics are great for aggregatable trends and alerting (e.g., overall request rate). Traces excel at illustrating request flows and latency breakdowns across distributed systems. Logs provide detailed, event-specific context, especially for non-transaction data (configuration changes, connections/disconnections to databases, …). Are you choosing the most effective signal type for your defined purpose? Adding high-cardinality attributes to metrics intended for aggregation, for instance, is often an anti-pattern. Creating a span when a simple event on an existing span would suffice adds unnecessary overhead. - **How will this data actually be *used*?** Will this feed a critical dashboard panel? Trigger an alert? Be used primarily for ad-hoc debugging during incidents? Understanding the consumption pattern helps determine the required granularity, retention, and format. How do you envision it being visualized or queried? Instrumenting data that no one knows how to use or interpret is like planting seeds you never intend to water. Again, don’t aim to predict exactly how things will be used, but having an idea helps set the direction. - **What is the *cost* versus the *value*?** Consider the compute resources needed to generate the data, the network bandwidth to transmit it, and the storage/processing costs in your observability backend. Is the potential insight or troubleshooting value gained worth this ongoing cost? Regularly reassess this balance, especially for verbose or high-frequency telemetry. You’ll never get the perfect balance on the first shot. In fact, even if you get into a perfect state today, it won’t be suitable anymore tomorrow as systems evolve. Keep an open mind and add things that are going in the direction of what you believe you’ll need. Adding too much fertilizer does hurt your garden. ## Applying Purposefulness in Practice Purposeful instrumentation isn't just a theoretical concept; it's applied through conscious choices during development and operation. - **Manual Instrumentation:** When you manually add code to emit telemetry (e.g., using OpenTelemetry APIs), be explicit. Add enough details explaining the 'why' behind non-obvious metrics or attributes. Document the intended use case, especially for custom, high-value signals. This foresight is invaluable during incident response or later refactoring, and writing about them is a great exercise to reason about them in the first place. - **Use semantic conventions to your favor:** not only does it tell how things should be named, but it also helps brainstorming what kind of instrumentation can be used. For instance, are you adding `deployment.environment.name` to your resource attributes? - **Auto-Instrumentation:** Tools like OpenTelemetry's auto-instrumentation agents (e.g., the Java agent) are powerful, providing broad coverage with minimal effort. However, "zero code" doesn't mean "zero thought." Don't blindly enable every single instrumentation library offered. Review the default configuration. Can you disable instrumentation for components irrelevant to your critical paths (e.g., verbose JDBC logging if you primarily diagnose issues at the service level)? Can you configure sampling decisions more intelligently? Tune or suppress instrumentation known to generate excessive noise or high-cardinality data that bloats costs without commensurate value. Auto-instrumentation provides the seeds; purposeful configuration helps you cultivate the desired crop. - **Regular Review & Weeding:** Instrumentation needs aren't static. Systems evolve, code gets refactored, and priorities shift. Schedule periodic reviews (e.g., quarterly) of your existing telemetry. It might still be a bit early, but consider using [OTel Weaver](https://github.com/open-telemetry/weaver) to help you here. Ask: Are there metrics, logs, or trace attributes that haven't been queried or looked at in months? Be ruthless about pruning unused or redundant instrumentation. This ongoing "weeding" keeps your observability garden healthy, cost-effective, and focused on yielding insights. **Put purpose into every span and metric.** Rose reviews the code in your GitHub repositories for missing or low-value telemetry, explains why each finding matters, and prepares source-level fixes your developers can review. [Meet Rose](https://ollygarden.com/products/rose) ## Reaping the Benefits of Critical Scrutiny Consistently applying a critical, purposeful lens to your instrumentation strategy, whether manual or automatic, transforms observability from a potential data swamp into a beautiful field full of data ready to harvest. It ensures your telemetry remains: - **Focused:** Directly addressing key operational questions and business KPIs. - **Relevant:** Aligned with current system architecture and troubleshooting needs. - **Cost-Effective:** Providing maximum insight for the resources invested. - **Actionable:** Enabling swift diagnosis, resolution, and performance optimization. By consciously choosing *what* to plant in your observability garden and *why*, you cultivate a rich harvest of insights. ## Next up What does purposeful instrumentation *actually* look like in real code, and how do we correct existing instrumentation that might not be that useful? Stay tuned for our next article, where we'll expose concrete examples of common instrumentation pitfalls and walk you through the precise steps to fix them, both directly at the source and with the OTel Collector. ## Conclusion In today's complex, distributed systems, observability is non-negotiable. But the path to enlightenment isn't paved with sheer data volume. It's built on the foundation of **purposeful instrumentation** – the deliberate act of gathering the *right* signals to illuminate system behavior. By embedding the practice of asking "Why this signal? Why now? How will it help?" into our development workflow, we shift from reactive data collection to proactive insight generation. We reduce noise, accelerate troubleshooting, control costs, and ultimately, build more reliable and performant software. So, the next time you reach for that instrumentation library or add that log line, take a moment. Pause. Ask yourself: "What is my purpose?". Cultivate clarity, and you'll reap the rewards of truly effective observability. *Acknowledgement: The concept of intentional instrumentation gained prominence for me through a conversation with Adriel Perkins, which evolved into purposeful instrumentation.* --- # There's a Lot of Bad Telemetry Out There Most telemetry is never queried. Learn what separates good telemetry from bad, what bad data costs you, and how to start improving quality at the source. Published: 2025-03-28 Canonical URL: https://ollygarden.com/resources/blog/theres-a-lot-of-bad-telemetry-out-there Ninety percent. That's the number the founder of an observability company mentioned some time ago when talking about telemetry data that withers away unused. It is data created, collected, transmitted, and stored, without ever blooming into a dashboard, alert, or query result. Our monitoring tools keep planting more and more data seeds, hoping they'll sprout into useful insights during a 2am production incident. Like planting without proper soil analysis, these generic instrumentation tools create low-quality telemetry that isn't suited for our environment. Traces from health check endpoints are like weeds taking up valuable space. Don’t get me wrong, this is still better than nothing! However, many companies have matured past this stage and need to cultivate higher quality telemetry. Telemetry is essential for us, observability engineers, to understand how our systems are performing. It's the soil from which observability and modern monitoring blossom. Despite its importance, a significant amount of telemetry data out there is, to put it bluntly, bad. This post will explore what good telemetry looks like, digging into the problems caused by bad telemetry, examine the root causes, and plant a thought on how to cultivate higher quality data. ## What is Good Telemetry? Good telemetry is characterized by providing an **accurate**, **relevant**, **timely**, and **actionable** picture of a system's health and performance. It offers **just enough** information — not too much or too little — to allow for improved troubleshooting, quick identification and resolution of issues, and a deeper understanding of the system. Additionally, good telemetry facilitates faster incident response, minimizing downtime and service disruptions, and bears fruit in the form of data-driven decisions that optimize performance. Examples include traces that show the path of a request through a system, metrics that accurately reflect resource utilization, and logs that provide context for errors and warnings. Quite frequently, good telemetry is used to provide business insights along with operational data. Our guide [What is telemetry quality?](https://ollygarden.com/resources/telemetry-quality) keeps an up-to-date definition, with the five dimensions we now measure and the signs of bad telemetry for each one. ## Bad Telemetry Bad telemetry is the opposite of good telemetry: inaccurate, irrelevant, old, non-actionable, or far too much in terms of volume. - **Inaccurate** telemetry happens when we have the wrong values, leading us to wrong conclusions. For instance, concurrently counting the number of leaves in a tree and storing the counter on a non-concurrent data structure will eventually result in the wrong number of leaves being reported - **Irrelevant** telemetry doesn’t provide any meaningful insight. It might be interesting to know that I have three boxes of kiwis, but without knowing the size of the box, that information is meaningless. - **Incomplete** telemetry means that we can’t determine the root cause of a problem. How can I tell whether my strawberries received enough sun if I’m not recording the amount of sun they received? How can I tell why my tulips didn’t grow if I don’t know whether they were planted in the first place? - **Old** data is bad telemetry, meaning that I’m only taking action when it might be too late. There’s no point in employing a scarecrow when the birds ate all the seeds already - Having **too much** data can also make it harder to find the real information we are looking for. It’ll definitely take us longer to find a magic herb in the middle of overgrown bushes. One interesting consideration is that the definition of bad telemetry also varies based on the backend we are using. Rice needs tons of water, purple coneflower will probably complain. For a time-series database, a high-cardinality metric is certainly not desirable. ## Consequences of bad telemetry We, as an industry, have tolerated bad telemetry as a price to pay for having telemetry in the first place. However, bad telemetry isn’t just bad: it’s the weed in our garden. Misleading insights from erroneous data can lead to poor decisions, and time and effort can be wasted analyzing useless data. Additionally, bad telemetry can result in slower problem resolution, due to the difficulty in identifying and fixing issues. It can lead to poor decision-making and incorrect business strategies based on faulty metrics. Perhaps more relevant for today’s economic realities, bad telemetry often means paying way more for observability than you should: high egress costs, overprovisioned infrastructure, complex pipelines, and unreasonably big checks to observability vendors. **See where your telemetry goes bad.** OllyGarden Insights analyzes the traces, metrics, and logs you already send, flags quality problems like noisy signals, PII leaks, and incomplete instrumentation per service, and scores each with an Instrumentation Score. [Explore Insights](https://ollygarden.com/products/insights) ## Root Causes of Bad Telemetry Many developers and engineers lack a full understanding of the principles of good telemetry. Observability is not taught alongside writing operating systems or databases at the university. It's not surprising that engineers don't learn how to properly instrument their applications. Like security, observability is typically a discipline that engineers come across when they get more experienced, once they’ve been burned by bug reports that they couldn’t reproduce, leading to frustrated users. Insufficient planning is another issue, as telemetry is often an afterthought rather than a core part of system design. Poor implementation is also a factor, as instrumentation can be complex and mistakes are easily made. Finally, telemetry systems require ongoing tending and upkeep, and inadequate maintenance can lead to having an infestation of bad telemetry. Instrumenting complex systems can be particularly challenging, as tasks like ensuring that context is propagated correctly across distributed services are crucial for accurate tracing. It doesn’t help that things are still moving really fast in observability, causing best practices to suddenly become anti-patterns, as well as the lack of standardization in new areas. Applications making use of generative AI tools or LLMs need to be instrumented today, but standards to instrument those components are still being worked on. Without clear industry guidance, people often have to make decisions based on their experiences (or their vendor’s suggestions), which quite often means that they’d be at odds with the standard when it gets created. OpenTelemetry semantic conventions definitely help here, and while OTel is in constant progress, we don’t have enough stable conventions for all that matters out there. ## How to Improve Telemetry Quality We are at the beginning of this journey towards high quality telemetry, and while there’s a lot to learn, I believe that we have enough to start changing the status quo. We already know a few anti-patterns, like the ones we mentioned earlier. We can also be opinionated about some solutions, such as OpenTelemetry, and gather insights based on those strong opinions. Perhaps all my telemetry should have a [`service.name`](http://service.name) and `service.version` field? Perhaps some resource attributes, like `process.executable.path`, should be filtered out at the source when going to a time-series metrics backend? Improving telemetry quality is a long-term activity, and I believe we’ll never have perfectly good telemetry. New services come and go every day, new versions are deployed all the time, and they all bring new telemetry with them, which is likely to be imperfect. It’s our job as observability engineers to understand what is good and what is bad telemetry, knowing our tools and how they’ll behave with the telemetry we are sending to them. Once we know and have our own recipes of what’s good and what’s bad, it’s a matter of looking into our telemetry and applying our knowledge, cross-pollinating with the engineers doing the instrumentation. Very likely, they are the ones doing the actual changes, or at least reviewing the pull requests we send their way improving the instrumentation of their code. The question that remains is: how do we show “progress”? Are we really better off with good telemetry, or can we just survive with bad telemetry? Instinctively, we know that good telemetry is more efficient than bad telemetry, but we should be ready to measure the impact. ## Conclusion Bad telemetry remains a pervasive challenge with significant consequences, one that we, as observability engineers, have often tolerated or underestimated. This is partly due to a lack of clear understanding of what constitutes good telemetry and partly due to the absence of robust tools and processes for detection and remediation. However, by actively defining and implementing standards for good telemetry, identifying the root causes of poor data quality (like pulling out weeds), and leveraging tools like OpenTelemetry, we can unlock the full potential of observability. This means reduced alert fatigue, faster incident resolution, and ultimately, more reliable and performant systems. Pioneering teams are already demonstrating the value of this approach, implementing semantic conventions and data quality pipelines that yield healthy, actionable insights. It's time for each of us to assess our own telemetry landscape, advocate for better instrumentation practices within our teams, and ensure that the data we rely on is truly serving our needs. ## Credits Dan Blanco, my colleague on the OpenTelemetry Governance Committee, is the author of the quote “There’s a lot of bad telemetry out there,” and he used it when I was describing what we are building at OllyGarden. He immediately understood our value proposition, wrapping it up with this quote. I like it: it’s blunt, it’s real, and it’s blameless.