Database storage latency high
Read 0ms, write 135.9ms
What this likely means
The rule fired on RDS write latency (135.9ms, threshold 100) while read latency is 0ms. Every other signal is healthy: API returns 200 in 28ms, ALB-observed API latency is 8.6ms across 6 requests with zero 4xx/5xx, one healthy host, ECS at ~2% CPU / 15% memory with 1/1 tasks running, RDS at 3.9% CPU with 3 connections. That combination — near-zero traffic plus a high per-operation latency average — most likely means the figure is an average over a very small number of write IOPS, where one slow or cold operation (checkpoint flush, autovacuum, log/WAL write, or an EBS burst-credit/gp2 cold-block hit) dominates the average. At 6 requests in the window there is almost certainly no user-visible impact; nothing in the data shows failed requests or errors. Read latency of exactly 0ms corroborates low I/O volume (reads served from cache or no reads at all), so this is likely a low-denominator artefact rather than a storage fault. Two caveats: freeable memory is 186MB, which is low in absolute terms and could indicate memory pressure pushing work to disk — worth confirming against the instance class before dismissing it. And the workflow metrics (registrations, applications, email failures) plus staff/talent activity are not instrumented, so I cannot confirm that candidate-facing writes — registrations, application submissions, stage transitions and their transactional emails — actually succeeded during this window. Absence of evidence there is not evidence of health. If this alert repeats under real write load, it would point at a genuine storage or IOPS problem.
Suggested checks
- Pull the RDS WriteIOPS / WriteThroughput CloudWatch metric for the same minute — if IOPS is in the single digits, the 135.9ms average is a small-sample artefact and can be downgraded.
- Chart WriteLatency over the last 6–24h and check whether spikes correlate with traffic or occur at idle; repeated idle spikes point to checkpoints/autovacuum, load-correlated spikes point to real I/O saturation.
- Check the RDS volume type and burst state: if gp2, look at BurstBalance; if gp3, compare provisioned IOPS against peak WriteIOPS. Also check EBSIOBalance%/EBSByteBalance% for the instance.
- Confirm the instance class and whether 186MB freeable memory is normal for it — if it is a small class near its ceiling, check SwapUsage and ReadIOPS for signs of cache thrash forcing disk access.
- In Postgres, check pg_stat_bgwriter (checkpoints_timed vs checkpoints_req) and log_checkpoints output for a checkpoint overlapping 10:04 UTC; a requested checkpoint or long write/sync time explains the spike.
- Look for autovacuum/VACUUM activity at that timestamp via pg_stat_activity history or the RDS log, especially on high-churn tables such as applications and pipeline stage transitions.
- Verify the user-facing path independently of metrics: query recent rows in the registrations/applications/stage-transition tables for the 10:00–10:10 window to confirm writes were committing, since these workflows are not instrumented.
- Check the transactional email sending path (provider dashboard or API logs) for failures around that window — emailsFailed is not instrumented, so it cannot be assumed to be zero.
- Review the alert's evaluation config — consider requiring a minimum write IOPS or a multi-datapoint breach so idle-period averages stop paging.
- Backlog item, not for now: instrument the workflow and activity counters that are currently reporting not_instrumented, so the next storage alert can be assessed against actual user impact.
Suggestions only — this system has no ability to act on them.