KUBICAST 182 - Observability BR with Lara Xavier

Today we welcomed Lara Xavier for a direct, technical, and humorous chat about how to actually do observability in Brazil. We talked about what changes when we move from "monitoring pages" to designing telemetry that answers product questions and reduces MTTR. And, of course, we discussed the pain points: cost, alert noise, and shared responsibility between dev and SRE.

Throughout the episode, we went through logs, metrics, and tracing, gave examples of instrumentation with OpenTelemetry, and compared approaches between tools and stacks. We also jumped into the reality of teams: when to start, how to prioritize, how not to turn dashboards into wallpaper, and why poorly defined SLIs sabotage any strategy.

We wrapped up with a tour of the Grafana stack (Loki and Tempo), best practices for observability data governance, and links for those who want to dive deeper. It’s perfect for both those taking their first steps and those who are already tuning signals and alerts in production.

About the Guest

Lara Xavier is a reference in the observability community, working daily with applied telemetry and is part of the Grafana Champions program. In this episode, she shares lessons learned from incidents, migrations, and building SLIs in teams moving from "putting out fires" to continuous improvement.

What We Discussed

Observability without Glamour

We broke down the idea that observability is just "having a pretty dashboard." We discussed how actionable signals start by defining questions (hypotheses) and designing events, metrics, and traces that answer them. We talked about alerts with less noise and how to separate symptoms from causes before opening an incident.

Logs, Metrics, and Tracing

We talked about metrics as aggregated signals that support SLIs, logs as a detailed narrative that shouldn't be an unrestricted dump, and tracing to stitch together the request journey. We reinforced criteria for retention, cardinality, and sampling.

OpenTelemetry in Practice

We showed why OpenTelemetry is the neutral base for instrumentation and provided an incremental adoption path: start with business metrics and latency, add contexts, propagate IDs, and only then expand coverage. We commented on common mistakes: creating useless spans, collecting everything without a budget, and not defining stable attributes.

Legacy Monitoring vs. Cloud-Native

We contrasted classic monitoring with Zabbix and observability practices in distributed systems. The discussion focused less on tools and more on modeling signals that answer questions about availability, performance, and business flow.

Grafana Stack: When and Why

We took a spin through the Grafana stack for teams that prefer open-source/managed solutions: Loki for logs and Tempo for traces. Honest comparisons with "Elastic" came up, with pros and cons regarding cost and operation. Lara also commented on the Grafana Champions program and study materials.

Costs, Priority, and Responsibility

We debated cost and prioritization. Observability without limits becomes a financial bottleneck; with limits and purpose, it becomes an accelerator. We talked about shared responsibility: dev instruments and takes care of signal quality; SRE defines standards, governs, and facilitates the platform.

Key Takeaways

• Starting with questions and SLIs avoids collections of metrics that don't help during incidents.

• Less is more: controlled cardinality, clear retention, and sampling protect the budget.

• Alerts must point to an action. If nobody knows what to do when they wake up to an alert, that alert is poorly defined.

• Tracing shines in cross-service debugging; use spans with purpose and stable attributes.

• OpenTelemetry reduces lock-in and creates a common foundation among teams and tools.

• Grafana Stack (Loki/Tempo) is a solid option when your requirements match the stack's profile; choose by the problem, not by the trend.

• Culture matters: blame-free post-mortems and continuous signal reviews.

Important Links:

- Lara Xavier - https://www.linkedin.com/in/lara-xavier-bb389788/

- Lara's Links - https://linktr.ee/Larasxavier

- João Brito - https://www.linkedin.com/in/juniorjbn

- Watch the Movie TEArapia - https://youtu.be/M4QFmW_HZh0?si=HIXBDWZJ8yPbpflM

- Become a Grafana Champion - https://grafana.com/community/champions/

- Join our early access program and have a safer environment in moments! https://getup.io/zerocve

🎧 Also listen to Kubicast on Spotify, and share it with the whole crowd who doesn't even know what a service bus is!