4 posts tagged with "sre"

An SRE lead at a mid-size fintech told me the quote that defines 2026 observability decisions: "Datadog is the iPhone of observability — expensive, polished, and I wish I had a choice." The market has three credible positions now: Datadog as the integrated default, Grafana as the open-source-first alternative, and Honeycomb as the wide-events specialist. Each is optimized for a different failure mode, and picking the wrong one doesn't show up in the first quarter — it shows up as a $2M annual bill and a team that still can't answer "why was latency spiky on Tuesday?"

CNCF's 2024 Annual Survey reported that 86% of cloud-native organizations use OpenTelemetry in some form — which sounds like the market is standardizing. In practice OTel is a pipeline, not a destination; every shop running it still picks one of these three stacks (or Splunk, New Relic, Dynatrace — we'll touch those briefly) to actually store, query, and visualize the data. Honeycomb's own observability maturity research shows that teams adopting wide-events cut investigation time on novel incidents by 40-60%, but only when the culture adapts — tooling alone doesn't deliver the lift.

Your best SRE quit last quarter. She didn't say "burnout" in the exit interview, but her last three months included 14 after-hours pages, 2 weekend incidents, and a 3am call on her birthday. A 2021 Catchpoint / DevOps Institute survey of 500+ on-call engineers found 67% reported burnout symptoms tied directly to paging load. Google's SRE book sets an internal ceiling of 2 incidents per on-call shift before a rotation is declared unhealthy — most teams we measure blow past that in week one.

On-call is fixable. It's a scheduling and sociotechnical problem, not a personality flaw in the people who can't hack it. Here's a 9-rule playbook that keeps your SLA intact and keeps your best engineers on the team past their second rotation.

The average post-mortem takes 4 hours to write and generates zero action items the team actually completes within 30 days. We looked at 120 post-mortem documents from three of our on-prem customers before rebuilding this template. 83% of action items were still "open" six months later. That's not an incident review — that's a document graveyard.

A post-mortem is worth writing only if it changes something. Everything else is CYA.

Google's Site Reliability Engineering book (2016) popularized a counterintuitive principle: accept failure as inevitable and invest in recovery speed. The DORA research confirmed it with data — the difference between elite and low-performing teams isn't that elite teams have fewer incidents. It's that they recover in under an hour instead of under a week. Every engineering organization invests in preventing failures. Fewer invest in recovering from them quickly. The data says this is backwards.