// DevOps
Structured logs, RED metrics, OpenTelemetry traces and SLOs with error budgets — and why you should alert on symptoms, not causes.
Md Nayeem HossainNov 3, 20253 min readFor years my debugging workflow in production was grep across log files and hope. It works until a request touches five services, a queue and two databases. Then a slow checkout becomes an archaeology project.
Observability is the ability to ask new questions about your system from the outside, without shipping new code to answer them. It rests on three signals — logs, metrics and traces — and on service level objectives that tell you which questions matter.
Unstructured log lines are for humans reading one at a time. At scale you search and aggregate, so log JSON with consistent fields:
1logger.info({2 event: "payout.completed",3 payoutId,4 affiliateId,5 amountCents,6 durationMs,7 traceId: context.traceId,8});Rules I enforce in code review:
payout.completed instead of "Payout finished successfully!".error fires a hundred times an hour and nobody acts, it is not an error.Metrics are cheap, aggregated numbers over time. They answer "is something wrong?" quickly. For every service, start with the RED method:
Averages lie. If your p50 latency is 80 ms and your p99 is 4 seconds, one in a hundred users is having a terrible time and the average will hide it. Use histograms and alert on high percentiles.
Watch out for cardinality. A metric labeled by userId creates one time series per user and will blow up your metrics backend. Labels should have a small, bounded set of values: route, status class, region.
A distributed trace follows a single request across every service it touches. Each unit of work is a span with a start time, a duration and attributes. Put together, they show exactly where the time went.
OpenTelemetry has become the standard way to instrument code without locking into a vendor:
1import { trace } from "@opentelemetry/api";2 3const tracer = trace.getTracer("payout-service");4 5export async function processPayout(id: string) {6 return tracer.startActiveSpan("payout.process", async (span) => {7 span.setAttribute("payout.id", id);8 try {9 await chargeProvider(id);10 return await markCompleted(id);11 } catch (err) {12 span.recordException(err as Error);13 throw err;14 } finally {15 span.end();16 }17 });18}Auto-instrumentation for HTTP clients, database drivers and queues covers most spans for free. Add manual spans around the business steps you care about.
The real power comes from correlation: put the trace ID in every log line. When a metric spikes, you jump to an example trace, and from the trace to its exact logs.
Without targets, every graph is just a graph. A service level indicator is a measurement of user experience, for example "the share of checkout requests that succeed in under 500 ms". A service level objective is the target: 99.9% over 30 days.
The gap between 100% and your objective is your error budget. At 99.9%, that is about 43 minutes of failure per month. The budget turns reliability into a shared, objective decision:
The most common observability mistake is alerting on everything: CPU over 80%, one failed job, a disk at 70%. People learn to ignore pages, and the real incident gets lost in the noise.
Page a human only when users are affected or about to be, which in practice means when the error budget is burning too fast. Everything else goes to a dashboard or a ticket. Every page should be actionable, urgent and real.
Structured logs tell you what happened, metrics tell you that something is wrong, traces tell you where, and SLOs tell you whether it matters. Instrument with OpenTelemetry, correlate everything by trace ID, and alert on user-facing symptoms. The next incident will take minutes to understand instead of hours.