Finding out about a production incident from a customer email is not a monitoring strategy.

APM instrumentation that shows where request latency is coming from at the level of individual service calls, database queries, and external API calls -- not aggregate CPU that tells you something is wrong without telling you why. Instrumentation approach: Datadog APM agent for Node.js, Python, Java, and Ruby services (auto-instrumentation wraps HTTP frameworks, ORM calls, and Redis/memcache operations with minimal code changes); OpenTelemetry SDK for language-agnostic instrumentation that avoids vendor lock-in (traces exported to Datadog, Grafana Tempo, or Jaeger depending on your stack). Request trace shape: every inbound HTTP request generates a root span with child spans for each downstream call -- the database query that took 800ms out of a 1,200ms total response time is visible in the trace waterfall without log correlation or guesswork. Database query performance: slow query detection threshold configurable per database (flag queries above 100ms in production), query plan analysis for repeated slow queries, and N+1 query detection for ORM-heavy services where 50 queries are executing per request instead of 1. External API monitoring: latency histogram and error rate per third-party API call (Stripe, SendGrid, AWS S3, etc.) so degradation in a vendor's performance is immediately distinguishable from a regression in your own code. Performance baseline: 14-day rolling baseline established per endpoint and per time-of-day so anomaly detection compares against typical behaviour for that hour rather than the overall average, reducing false-positive alerts during expected load peaks.

Infrastructure metrics coverage across the full stack -- compute, database, network, and serverless -- with alert thresholds calibrated to your services' actual behaviour rather than arbitrary percentages that create alert fatigue. AWS metric collection: Datadog agent on EC2 and ECS task definitions collects CPU, memory, disk I/O, and network metrics at 15-second intervals; CloudWatch native metrics ingested for Lambda (invocation count, duration, error rate, concurrent executions, throttle count), RDS (CPU, IOPS, connection count, replica lag), SQS (queue depth, oldest message age), and EKS nodes. GCP metric collection: Datadog or Grafana Agent deployed to GKE nodes and Cloud Run services with GCP Cloud Monitoring as the supplementary metrics source. Alert calibration process: baseline each metric over a 14-day normal operation window; identify the 95th percentile of normal values; set alert thresholds at statistically meaningful deviations above that baseline; suppress alerts during scheduled maintenance windows. Alert severity routing: critical (page immediately, wake up on-call) for conditions like service down, database connection exhaustion, or disk full within 1 hour; warning (notify via Slack, no page) for conditions approaching thresholds but not yet urgent; informational (log only) for metrics useful in postmortems but not requiring immediate action. PagerDuty integration with on-call schedule rotation so the right engineer is paged without manual escalation; OpsGenie as alternative for teams already using it. Alert-to-runbook linking: every alert configured with a runbook_url annotation pointing to the relevant incident response procedure.

Structured JSON logging standardised across every service so incident investigation is a query against consistent fields rather than a grep through unformatted text across six different log format conventions. Log schema design: timestamp, level, service, trace_id, span_id, user_id, request_id, and error.message/error.stack as standard fields on every log line -- the schema that enables correlation between logs and traces and between logs across services for a single user request. Log library configuration: Winston (Node.js), structlog (Python), Logback with Logstash encoder (Java), or Zerolog (Go) configured to emit JSON at every log call without requiring code changes at individual log sites. Log aggregation destination: Datadog Log Management for teams already on Datadog APM (unified query across traces and logs); Grafana Loki for open-source stacks (label-based indexing at lower storage cost than full-text search solutions); AWS CloudWatch Logs with CloudWatch Insights for teams on AWS who want to avoid an additional SaaS tool. Log-based metrics derived from structured fields: error count per error code per service; request count per HTTP status code; specific business events (payment processed, signup completed) counted as metrics from log events rather than requiring separate metric instrumentation. Log retention policy: production logs retained at full resolution for 30 days; archived to S3 in compressed format for 12 months for compliance and forensic investigation; configurable per environment (dev/staging logs retained for 7 days). Sensitive field redaction: PII fields (email, credit card, national ID) identified in the log schema and automatically redacted before logs leave the service, preventing PII from appearing in the log aggregation platform.

End-to-end trace correlation that follows a single user request across every microservice, database, and message queue it touches -- so a slow checkout that spans the API gateway, product service, inventory service, payment service, and order database is a single trace showing exactly where the 3,200ms went rather than six separate logs to correlate manually. OpenTelemetry instrumentation: the W3C traceparent header propagated across all HTTP service calls and injected into Kafka message headers and SQS message attributes so trace context survives async boundaries; OpenTelemetry Collector deployed as a DaemonSet on EKS or as a sidecar on ECS, exporting spans to Datadog APM, Grafana Tempo, or Jaeger depending on your existing stack. Trace sampling: head-based sampling at 10% for healthy traffic to control storage costs; tail-based sampling that ensures 100% of traces containing errors or latency above the p99 threshold are retained even when overall sampling is reduced -- the sampling strategy that keeps storage costs manageable without losing the traces you need most for incident investigation. Service dependency map generated from span data: a live graph of which services call which, with request volume and error rate on each edge -- the map that makes it immediately visible when a new deployment adds an unexpected dependency or removes a required one. Trace search for incident investigation: filter by service, endpoint, error status, latency range, and any span attribute (user ID, order ID, customer tier) to find the exact traces relevant to a reported incident without reading through log files.

Service Level Objectives defined with measurement methodology agreed before implementation -- because an SLO that measures the wrong thing creates false confidence and an SLO with an impossible target creates alert fatigue from the first week. SLO definition process: identify the user-visible reliability dimension that matters most (availability, latency, or throughput); define the measurement query (percentage of requests returning 2xx status codes within 500ms over a 28-day rolling window); agree the target (99.5% for an internal service, 99.9% for a customer-facing API, 99.99% for a payment-critical endpoint); and document the measurement gap between the SLO indicator and the actual user experience -- so the team understands what events the SLO does and does not capture. Error budget calculation: 100% minus the SLO target expressed as a count of allowed failures per window (99.9% availability on a service receiving 10M requests per month = 10,000 allowed failures); remaining budget displayed as a percentage and as an absolute count remaining in the current window. Burn rate alerting using Google SRE-recommended multi-window burn rate rules: alert when the 1-hour and 5-minute burn rates both indicate the SLO will be exhausted before the window ends -- the alerting pattern that catches fast burns (sudden outage) and slow burns (gradual degradation) without flooding on-call with notifications for every 0.1% dip. Error budget policy documented with the SLO: what the team does when budget is low (feature freeze, reliability sprint), when budget is exhausted (incident response priority), and when budget is healthy (normal feature velocity) -- the governance process that makes SLOs meaningful rather than decorative dashboard elements.

Runbook development that makes on-call effective rather than exhausting -- each runbook written so a newly on-call engineer who has never seen the relevant alert before can investigate and resolve the most common causes without waking a senior engineer. Runbook structure per alert: what the alert is measuring and why it matters; the initial investigation steps in order (check the Datadog APM trace for the highest-latency requests, run the specific CloudWatch Insights query, check whether a recent deployment correlates with the alert onset); the three most common root causes ranked by frequency, each with its resolution procedure; the escalation contact and the information to gather before calling them. Runbook linkage: every PagerDuty and OpsGenie alert configured with a runbookUrl field pointing to the relevant Confluence or Notion page -- the link that appears in the notification the on-call engineer receives at 2 AM before they have had time to remember where the runbook lives. Major incident playbook covering the first 30 minutes: incident commander designation, communication channel setup (#incident-YYYY-MM-DD-description in Slack), customer communication timing, investigation team assembly, and status page update cadence (every 15 minutes for a P1 incident). Postmortem template and facilitation process: timeline reconstruction, contributing factors (with root cause distinguished from symptoms), impact quantification (users affected, error count, revenue impact), action items with owners and due dates -- the format that produces specific remediation rather than "improve monitoring" as the only output. SLO impact assessment section in every postmortem: how much error budget the incident consumed and whether the rate of budget consumption suggests the reliability target needs revision.