Week 11 — Observability & Service Mesh

The Three Pillars of Observability

Observability = ability to understand a system’s internal state from external outputs.

1. Metrics

Numeric measurements aggregated over time. Use the RED method per service:

• Rate — requests per second
• Errors — error rate (%)
• Duration — latency (p50, p95, p99)

Why percentiles matter more than averages:

At 100K requests/day with p99 = 10 seconds: 1,000 users hit a 10-second response every day. Average would hide this.

Always alert on p95 or p99, never average.

2. Structured Logging

{
  "timestamp": "2025-01-15T10:23:45Z",
  "level": "ERROR",
  "service": "payment-api",
  "trace_id": "abc-123-def",
  "user_id": 456,
  "message": "Payment gateway timeout",
  "duration_ms": 5001,
  "gateway": "stripe"
}

• Structured JSON — queryable by field, not just grep
• Correlation ID / trace_id — follow a request across multiple services
• Never log passwords, tokens, or PII

3. Distributed Tracing

Visualise how a single request flows through multiple services:

Browser request [total: 450ms]
  └─ API Gateway [12ms]
  └─ Order Service [380ms]
      └─ DB query [320ms] ← bottleneck
      └─ Inventory check [40ms]
  └─ Response [8ms]

Tools: Jaeger, Zipkin, OpenTelemetry (vendor-neutral standard).

Add a trace_id to every log line — link logs and traces together.

Circuit Breaker

Prevents cascading failures when a downstream service is unhealthy.

CLOSED (normal) → OPEN (fail fast) → HALF-OPEN (test recovery)

• CLOSED: requests pass through normally. Count failures.
• OPEN: threshold exceeded → immediately return error without calling downstream. Saves resources.
• HALF-OPEN: after timeout, allow one request through. Success → CLOSED. Failure → OPEN.

Service Mesh

A layer of sidecar proxies (e.g. Envoy) deployed alongside each service instance.

Service A pod:   [App container] + [Envoy sidecar]
                                       ↕ mTLS
Service B pod:   [App container] + [Envoy sidecar]

What sidecars handle transparently (no app code changes):

• mTLS between services
• Automatic retries and timeouts
• Load balancing
• Circuit breaking
• Metrics and tracing

Examples: Istio, Linkerd, Consul Connect.

Trade-off: adds latency (one extra hop per request), increases operational complexity.

Production Checklist

• /metrics endpoint (Prometheus format)
• /health and /ready endpoints
• Structured JSON logging with correlation IDs
• Distributed tracing instrumented
• Alerts on p95 latency + error rate
• Circuit breakers on all external service calls
• Runbook for every alert

Self-check

• Why is alerting on average latency dangerous?
• A downstream service starts failing. Without a circuit breaker, what happens to your service?
• What’s the difference between a health check and a readiness check?