TaliLink Monitoring

What this likely means

A single email send failed because the recipient address is on the SES suppression list. This is almost certainly a per-recipient issue, not a platform fault: the address was previously suppressed (bounce or complaint), so SES rejected the send before delivery. Everything around it looks healthy — API reachable at 200 in 33ms, 0 HTTP 5xx, 3/3 healthy hosts on both ALBs, ECS 3/3 tasks at 2.5% CPU / 23% memory, RDS at 4% CPU with 9 connections and sub-millisecond write latency, and 19/19 registrations plus 3/3 applications succeeding with no recorded failure reasons. So the mail path itself and the stage-transition workflow that triggers it are working for everyone else in this window. The user impact is narrow but real: one candidate did not receive a transactional email (likely a stage-transition or registration/verification notice) and has no in-product signal that it was dropped — they may be blocked on a verification link or unaware their application moved stages. Note the alert threshold is 0, so this rule will fire on any single suppressed recipient; that is expected noise unless the count climbs. Two things are unknown from this payload: which specific email type and recipient failed, and whether there is a retry/dead-letter path — neither is instrumented here, so do not assume the message will be re-sent. The 5 HTTP 4xx on the API are likely unrelated routine client errors, but they are not broken down here.

Suggested checks

• 1.Pull the application log line for this failed send to identify the recipient address, the email template/type, and the stage transition or event that triggered it.
• 2.Query the SES suppression list for that address (account-level and global) to see the suppression reason (BOUNCE vs COMPLAINT) and the date it was added.
• 3.If it was a hard bounce, verify the address is valid — check the candidate's profile for a typo or a dead domain, and confirm with them out-of-band before removing the suppression.
• 4.Determine whether the candidate is blocked: if this was a verification or password-reset email, they may be stuck mid-registration and need manual intervention or a corrected address.
• 5.Check whether your code has a retry or dead-letter path for failed sends — confirm explicitly rather than assuming; if none exists, the email is simply lost.
• 6.Look at emailsFailed over the last 24–72 hours to see whether suppressed recipients are a one-off or a rising trend; a rising bounce/complaint rate risks your SES reputation and eventual sending throttles.
• 7.Review the SES account reputation dashboard (bounce rate, complaint rate) to confirm you are well clear of AWS thresholds.
• 8.Consider whether the threshold of 0 on this rule is right — recipient-side suppressions are expected background noise and may warrant a separate, lower-severity signal from genuine send-path failures (SES API errors, throttling, credential problems).

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: ALB target-group average latency for the API is 2645.8ms over 84 requests in the window, while a direct synthetic probe of the API returned 200 in 23ms. Infrastructure is idle: ECS CPU 4.4%, memory 31.8%, 3/3 tasks running and healthy; RDS CPU 5.2%, 9 connections, 961MB freeable, read latency 0ms / write 0.3ms. No 5xx, 4 x 4xx, web tier at 5.3ms. Business workflows in the window all succeeded: 12 registrations, 5 applications, 0 failures, 0 email failures, with 15 candidates and 0 staff active. Most likely interpretation: this is a latency average skewed by a small number of genuinely slow requests rather than a platform-wide slowdown. With only 84 requests, roughly 5-10 requests taking 20-30s would produce this mean on their own. The fact that CPU, memory, DB latency and connection count are all near-idle argues strongly against resource contention, connection-pool exhaustion or a DB hot spot; a saturated pool or slow disk would show up in RDS connections or latency. The endpoints candidates hit in this window are registration and application submission, both of which typically do synchronous outbound work - transactional email at stage transitions, file/resume upload to storage, or an external enrichment call. Slow or retrying outbound I/O is the explanation most consistent with 'high wall-clock latency, no CPU, no DB pressure, no errors'. A second, weaker possibility is cold-start or first-request compilation cost on a recently deployed task, or a few unindexed list/search queries returning large result sets (these would show as low DB CPU but non-trivial query duration, which is not instrumented here). User impact: likely limited but real - a minority of candidates experienced multi-second waits on register or apply, while most requests were fast. Nothing failed: no registration, application, or email failures were recorded, and no 5xx. Note the 4 x 4xx are unexplained by the workflow counters, and staff activity is 0, so staff-side latency is unmeasured in this window. Unknowns - do not assume healthy: there is no p50/p95/p99 breakdown, no per-route or per-endpoint latency, no application-level trace or slow-query log, no outbound email/SMTP latency metric, and no DB connection-pool utilisation metric. All of these are simply unknown, not zero. The 23ms probe only proves one lightweight path was fast at one instant.

Suggested checks

• 1.Pull ALB target-group latency percentiles (p50/p90/p99) for this window and compare to the 2645.8ms mean - if p50 is low and p99 is tens of seconds, this is a tail-latency problem in a few requests, not a platform slowdown. This single check decides the whole investigation.
• 2.Break latency down per route/endpoint (ALB access logs in Athena, or APM if instrumented). Sort by target_processing_time descending and identify the specific slow requests and their paths - expect POST /auth/register, POST /applications, or a candidate list/search endpoint.
• 3.For those slow requests, check the NestJS application logs at the matching timestamps for the phase that consumed the time: outbound email send, S3/storage upload, external HTTP call, or query execution. Look for retry or timeout messages.
• 4.Check email/transactional provider latency and status for 18:40-18:55 UTC. emailsFailed is 0, but slow-then-successful sends would still block the request and would not appear in that counter.
• 5.Query pg_stat_statements on RDS ordered by mean_exec_time and total_exec_time for the statements touched by those endpoints, and check for sequential scans on applications/candidates. RDS CPU is idle, so look for long-but-cheap queries (lock waits, large result serialisation) rather than CPU-bound ones.
• 6.Check pg_locks and pg_stat_activity for sessions in a waiting state - low CPU with high wall-clock time is consistent with blocking on a row or advisory lock during pipeline stage transitions.
• 7.Confirm the ECS deployment/task start times. If tasks started shortly before 18:51, the slow requests are plausibly cold-start or JIT/first-hit cost and should not recur.
• 8.Identify the 4 x 4xx in the ALB logs (status code, path, client). The workflow counters report zero failures, so these are unaccounted for and may be timeout-driven client retries or aborts.
• 9.Verify whether this is a one-off spike or sustained: re-check API latency over the last 60 minutes and the prior 24 hours to see if 2645.8ms is an outlier window or a trend.
• 10.Instrument what is missing before the next occurrence: per-route latency percentiles, outbound email/HTTP client duration, and DB connection-pool utilisation. All three are currently blind spots in this dataset.

Suggestions only — this system has no ability to act on them.

What this likely means

The ALB is reporting a 2263.7ms average API response time over the window, but every resource metric that would explain a broad slowdown is idle: ECS CPU 2.7%, memory 31%, 3/3 tasks running and healthy; RDS CPU 5.4%, 9 connections, 956MB freeable, read latency 0ms and write latency 0.7ms. A direct synthetic probe at detection time returned 200 in 33ms, and the Next.js frontend behind the same infrastructure averaged 8.3ms. No 5xx, only 2 4xx out of 57 requests, and all workflows succeeded (20 registrations, 7 applications, 0 failures, 0 email failures) with 11 talents active. This pattern is most consistent with a small number of individually slow requests skewing the average over a low-volume window (57 requests), not a platform-wide degradation — one or two 30-60s requests would be enough to lift the mean past 2000ms. The likely culprits are a specific slow endpoint or code path rather than capacity: an unindexed or N+1-heavy query (RDS aggregate latency can look fine while one query blocks on locks or a seq scan), a synchronous outbound call such as the transactional email provider on a stage transition, or a cold-start/first-request penalty on a rarely-hit route. Note what is not instrumented here: there are no per-endpoint or percentile latency breakdowns, no ALB target_processing_time vs request/response time split, no queue depth, and no email-provider latency — so the slow path cannot be identified from this payload alone, and its absence should not be read as healthy. User impact is likely narrow: most candidates are seeing normal response times, but whoever hits the slow path is waiting seconds (possibly tens of seconds) or may perceive a hang, and staff activity is 0 so no internal users are currently affected.

Suggested checks

• 1.Pull ALB access logs for the 5-minute window and sort by target_processing_time descending — identify the specific request URLs, methods, and durations of the top few slowest requests. This alone will usually confirm whether it is 1-3 outliers or a broad shift.
• 2.Compare p50 vs p95/p99 target response time in CloudWatch for the same window. A normal p50 with a spiked p99 confirms the outlier hypothesis; an elevated p50 means the whole API is genuinely slow and the synthetic 33ms probe hit a fast path.
• 3.Separate ALB target_processing_time from request_processing_time and response_processing_time in the logs — this distinguishes slow application work from slow client upload/download or connection-level issues.
• 4.Check pg_stat_activity and pg_stat_statements on RDS for long-running or lock-waiting queries around 16:11-16:17Z. Aggregate RDS CPU and IO latency being near zero does not rule out a single query blocked on a lock or doing a seq scan.
• 5.Check the transactional email provider's latency and any API timeouts around this window, and confirm whether stage-transition emails are sent synchronously in the request path. 7 applications were submitted with 0 email failures, so a slow-but-successful send is plausible.
• 6.Review NestJS application logs for the window for slow-request warnings, timeout handling, or requests without a logged completion — correlate by request ID with the slow URLs found in the ALB logs.
• 7.Confirm whether the slow requests concentrated on one ECS task. Per-task metrics are not in this payload; if one task is degraded while the other two are fine, task-level request counts and latency will show it.
• 8.Check whether any of the slow endpoints are rarely hit (admin, reporting, export routes) — a cold connection pool or JIT/first-call cost on a low-traffic route is a benign explanation consistent with 57 requests in 5 minutes.
• 9.Instrument per-endpoint latency (route-level histograms) if not already present, so the next occurrence resolves in seconds rather than requiring log archaeology.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 8 email sends failed in the window, all with the same reason — the recipient address is on the SES suppression list. Everything else is healthy: API returns 200 in 29ms, 3/3 ECS tasks running at 3% CPU / 26% memory, RDS at 4% CPU with 9 connections, zero 4xx/5xx on both ALBs, and 12 registrations plus 6 application submissions all succeeded with no recorded failure reasons. So this is not an outage, a capacity problem, or a broken integration — the API accepted the sends and SES rejected them at the recipient level. What this likely means: suppression-list entries are created by prior hard bounces or complaints (account-level suppression) or by an explicit suppression list you manage. With 13 talents active and 12 registrations in 5 minutes, the plausible readings are (a) genuinely invalid/typo'd addresses from self-registration that previously hard-bounced, or (b) a batch of addresses — e.g. seeded, synthetic, or load-test accounts such as @example.com or SES simulator addresses — that were suppressed earlier and are now being re-mailed. A third, worse possibility is that a recent bounce spike put real domains into account-level suppression; that would be a deliverability/reputation issue, not just bad data. User impact: those 8 candidates did not receive their transactional email (registration confirmation or a stage-transition notice). The pipeline state change itself almost certainly still committed — nothing in the workflow counters failed — so those candidates are silently advanced or registered without notification. Whether the platform retries, dead-letters, or drops these sends is not instrumented here, so treat the eventual delivery state as unknown. Also unknown: bounce and complaint rates, SES reputation-dashboard status, and whether sending quota is constrained — none of those appear in this payload, so do not assume they are fine. This is a data-quality / deliverability issue, not an infrastructure incident. It is unlikely to escalate on its own, but if the underlying cause is a rising hard-bounce rate it can threaten the whole sending domain's reputation, which would be a much broader outage.

Suggested checks

• 1.Pull the 8 failed sends from application logs for this window and list the actual recipient addresses — this single step separates 'synthetic/test accounts' from 'real candidates' and determines whether this matters at all.
• 2.Check the SES account-level suppression list for those addresses (aws sesv2 get-suppressed-destination) and note the Reason (BOUNCE vs COMPLAINT) and the date each was added. Recent additions point to a live problem; old ones point to stale accounts being re-mailed.
• 3.Look at the SES account-level reputation dashboard: current bounce rate, complaint rate, and whether the account is under review or sending is paused. This is not in the payload and is the main thing that could turn this into a real incident.
• 4.Determine which email type failed — registration confirmation vs stage-transition notification — by correlating the 8 failures against the 12 registrations and 6 applications. If they are registration emails, the addresses came straight from self-registration and the signup flow may lack validation.
• 5.Check whether the email sender has a retry or dead-letter path, or whether these 8 sends were dropped silently. Confirm whether a failed send is surfaced anywhere to staff (0 staff were active, so nobody saw this in-product).
• 6.Query the DB for recently registered candidates whose email domain is obviously non-deliverable (example.com, test domains, SES simulator addresses like bounce@simulator.amazonses.com) to see if a seeding or load-test job introduced them.
• 7.If the addresses turn out to be real candidates with recent BOUNCE suppression, check DNS/SPF/DKIM/DMARC for the sending domain and recent changes to it before deleting anything from the suppression list.
• 8.Only after the above: decide per-address whether to delete from the suppression list (aws sesv2 delete-suppressed-destination) and re-send. Do not bulk-clear the list — re-mailing genuine hard bounces will worsen the bounce rate.
• 9.Consider adding email-format and MX validation at registration, and an operator-visible counter for suppressed-recipient failures, so this is caught at signup rather than at send time.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: the ALB reports a 5-minute average API latency of 2839.5ms across 58 requests, but every other indicator is healthy. The direct health probe returned 200 in 21ms. ECS is at 3% CPU / 26% memory with 3/3 tasks running and 3 healthy hosts behind the load balancer. RDS is at 4.6% CPU, 9 connections, 970MB freeable, read latency 0ms. There are no 5xx, only 7 4xx out of 58 requests, and no workflow failures: 25 registrations and 8 applications succeeded, zero failed, zero email failures. The frontend is fast (11.7ms). 12 candidates and 0 staff were active in the window. Most likely interpretation: this is not a capacity or saturation problem. With idle CPU everywhere and a fast health check, a ~2.8s average over a small sample (58 requests) is most consistent with a handful of individually slow requests dragging the mean up — an average, not a percentile, is very sensitive at this volume. Two candidate causes fit best. (1) A few slow endpoints doing work that blocks on something external rather than on CPU: the RDS write latency of 31.6ms is the one genuinely off-looking number (reads are 0ms), so write-heavy paths — registration inserts, application submission, stage transitions — are plausible contributors, likely compounded by N+1 queries or missing indexes that are slow in wall-clock but cheap in CPU. (2) Synchronous outbound calls on the request path, most plausibly transactional email sends at stage transitions; if those are awaited inline, a slow SMTP/provider response would produce exactly this signature: high latency, no errors, no resource pressure. Note the traffic mix is unusual — 25 registrations in a 5-minute window from 12 active candidates is a burst, so the slow requests are likely concentrated in the registration/application path (which is write-heavy and email-triggering) rather than spread evenly. User impact: likely a subset of candidates experiencing multi-second waits on specific actions (probably signup or apply), while most browsing stays fast. Nothing is failing — no failed registrations, applications, or emails — so this is currently a degradation, not an outage. Important unknowns: per-endpoint and per-percentile latency breakdowns are not in this payload, so I cannot confirm which routes are slow or whether this is a few outliers versus a broad shift. Email send latency, external provider health, and ECS task-level (as opposed to service-average) metrics are not instrumented here — treat all of these as unknown, not healthy. The 7 4xx responses are uncharacterised and may or may not be related.

Suggested checks

• 1.Break the ALB TargetResponseTime down by percentile (p50/p90/p99) over this window — if p50 is low and p99 is multi-second, confirm this is a few outliers rather than platform-wide degradation, which changes the urgency entirely.
• 2.Get per-route latency from APM or NestJS request logs for the window around 11:14 UTC. Identify the specific slow endpoints; the registration and application-submission paths are the first places to look given 25 registrations in 5 minutes.
• 3.Check whether transactional email sending is awaited inline in the request handler at stage transitions and on registration. If it is synchronous, check the email provider's response times and status page — a slow provider would explain high latency with zero errors and zero CPU.
• 4.Inspect RDS Performance Insights / pg_stat_statements for the slowest statements by total and mean time. The 31.6ms write latency with 0ms read latency points at write paths specifically — look for long-running INSERT/UPDATE transactions, lock waits, or commit stalls.
• 5.Check for blocking locks and long-running transactions in pg_stat_activity (state, wait_event_type, query_start). A transaction held open across a slow external call would produce both the write latency and the request latency.
• 6.Confirm whether the latency is spread across all 3 ECS tasks or isolated to one. Service-average CPU can hide a single degraded task; check per-task metrics and ALB per-target response times.
• 7.Identify the 7 4xx responses — route, status code, and client. Rule out whether they share a source with the slow requests (e.g. validation failures on the same burst of registrations) or are unrelated background noise.
• 8.Confirm whether the registration burst is legitimate human traffic or automated/bot signups, since 25 registrations from 12 active candidates in 5 minutes is disproportionate and could be driving abnormal load on a specific write path.
• 9.Re-sample the ALB latency metric now. If it has returned to baseline without intervention, this was a transient burst and the priority shifts to instrumenting per-route and email-send latency so the next occurrence is diagnosable.

Suggestions only — this system has no ability to act on them.

What this likely means

Two email sends failed in the window, both rejected because the recipient address is on the SES account-level suppression list. This is a per-recipient delivery problem, not a platform fault: SES accepted nothing for those two addresses because they previously hard-bounced or produced a complaint. Everything around it is healthy and corroborates that reading — API reachable at 200 in 27ms, 0 HTTP 5xx across API and web, 3/3 healthy hosts and 3/3 ECS tasks, ECS CPU ~4.7% / memory ~32%, RDS CPU ~4.3% with 9 connections and sub-millisecond latency, and 24 registrations plus 6 applications all succeeding with no recorded failure reasons. So the email path itself is up and the rest of the system is transacting normally; the failures are scoped to specific addresses. User impact is narrow but real: those two candidates did not receive their transactional email (likely a registration confirmation or stage-transition notice, given 24 registrations and 12 active talents in the window), and they will silently keep missing every future email while they remain suppressed — they may believe their application has stalled. Note the 10 HTTP 4xx on the API; that is not obviously connected and is plausibly ordinary client-side noise, but it is unverified. Also unknown: whether the platform retries or dead-letters suppressed sends, whether the candidate-facing UI surfaces any error, and whether other emails in this window succeeded (only the failure count is instrumented, so I cannot infer a success rate). Treat those as unknown rather than fine. The thing that would change the severity assessment is the trend: two suppressed recipients is routine, but a rising count would point at a bad address source (typo-prone signup form, no email verification, or imported/seeded data) or at a reputation problem generating new bounces.

Suggested checks

• 1.Pull the two failed sends from the application logs and get the actual recipient addresses, the template/stage that triggered them, and the SES message IDs.
• 2.Look those addresses up in the SES suppression list (GetSuppressedDestination) to see the reason (BOUNCE vs COMPLAINT) and the date added — an old hard bounce is benign; a complaint added today is not.
• 3.Inspect the addresses themselves for obvious typos (gmial.com, missing TLD) or test/seed data — that distinguishes a signup-validation gap from genuine dead mailboxes.
• 4.Check the SES account reputation dashboard for current bounce and complaint rates against the 5% / 0.1% thresholds, to confirm this is isolated and not the leading edge of a reputation issue.
• 5.Graph emailsFailed over the last 24–48 hours and compare against registration volume — establishes whether 2 is the steady-state background rate or a new spike.
• 6.Confirm whether the send path records successes, not just failures; without a denominator you cannot tell 2 failures out of 30 from 2 out of 2.
• 7.Trace what the platform does after a suppressed send: is it retried, dead-lettered, or dropped, and does the candidate's pipeline stage still advance? Determines whether these two candidates are merely uninformed or actually stuck.
• 8.Decide per address whether to delete it from the suppression list (only if the mailbox is known good and the bounce was transient/misclassified) — re-sending to a genuine hard bounce will further damage sender reputation.
• 9.If no email verification exists at registration, check whether adding confirm-address or syntax validation is warranted, since 24 registrations in 5 minutes is a high enough rate to keep feeding bad addresses into SES.
• 10.Separately, sample the 10 API 4xx responses to confirm they are unrelated client errors and not failing email-preference or profile-update calls.

Suggestions only — this system has no ability to act on them.

What this likely means

Three transactional emails failed, all with the same SES reason: the recipient address is on the SES account-level suppression list. This is a recipient-scoped problem, not a platform or infrastructure problem. Every surrounding signal is healthy and corroborates that: API returns 200 in 23ms with 3/3 healthy hosts and no 5xx; ECS is at 3% CPU / 25% memory with 3/3 tasks running; RDS is idle (4.5% CPU, 9 connections, sub-millisecond latency); 20 registrations and 4 applications succeeded with zero failures. So the send path itself is working — SES accepted the API call and then refused delivery for specific addresses. A suppression-list entry means SES previously recorded a hard bounce or complaint for that address (or it was added manually/by a bulk import), so SES silently drops sends to it to protect the account's sending reputation. User impact: the affected candidates did not receive their stage-transition email and will not receive any future email until the suppression is removed. They may be stuck waiting on an interview invite or assessment link with no idea it was sent. Note the 3 http4xx on the API in the same window — likely unrelated (nothing in recentFailureReasons), but worth a glance in case they are the same requests surfacing the failure. Also treat the count as a floor: only failures the platform observed are counted here, and asynchronous bounces or complaints that arrive after the send are not visible in this data, nor is any retry/dead-letter state — those are unknown, not zero. Whether this is three unrelated stale addresses or an early sign of a reputation problem (rising bounce rate, a bad import, a domain-auth regression) cannot be determined from these metrics alone.

Suggested checks

• 1.Identify the 3 recipient addresses from the email-failure logs, and check which candidates and which pipeline stage transitions they correspond to — this tells you who is silently stranded and whether anyone needs manual follow-up.
• 2.Query the SES suppression list for those addresses (aws sesv2 get-suppressed-destination) to see the reason (BOUNCE vs COMPLAINT) and the date added. COMPLAINT means the user marked mail as spam — do not re-send. BOUNCE with an old date on a still-valid address is a candidate for removal.
• 3.Inspect the addresses for typos or obviously invalid domains (gmial.com, trailing whitespace, internal test addresses). Self-registration means candidates type their own address, so bad data at signup is the most common source of hard bounces.
• 4.Check the SES account-level reputation dashboard: current bounce rate and complaint rate against the 5% / 0.1% thresholds, and whether the account is in a warning or review state. This distinguishes 'three stale addresses' from 'reputation trending toward a sending pause'.
• 5.Check whether the suppression list is account-level or configuration-set-level, and whether automatic suppression is enabled for BOUNCE, COMPLAINT, or both — this determines whether entries will keep accumulating.
• 6.Look at the trend in emailsFailed over the last 24–48 hours. A flat baseline of a few per day is normal attrition; a step change points to a recent import, a code change in the send path, or a DKIM/SPF/DMARC regression causing broad bounces.
• 7.Verify the platform's retry and dead-letter behaviour for failed sends: are these 3 messages queued for retry (which will fail again while suppressed), dropped silently, or surfaced to staff? This is not visible in the current metrics.
• 8.Glance at the 3 http4xx on the API in this window to confirm they are unrelated to the email failures — recentFailureReasons is empty, so they are likely routine client errors, but rule it out.
• 9.If addresses are confirmed valid and were bounced for a transient reason, delete the suppression entries (aws sesv2 delete-suppressed-destination) and trigger a resend of the missed stage-transition emails. Do not bulk-remove entries without checking reasons first.

Suggestions only — this system has no ability to act on them.

What this likely means

Scope is almost certainly a single recipient, not a platform problem. One email delivery failed in the window, and SES reported the reason as 'Recipient is on the SES suppression list' — that is SES refusing to attempt delivery to one address that was previously hard-bounced or marked as a complaint. It is not an outage of the email path: other emails in the window were not reported as failed, and every surrounding metric is healthy (API 200 in 34ms, 0 5xx, 3/3 ECS and ALB hosts, ECS CPU ~4% / memory ~25%, RDS CPU ~6% with 9 connections and sub-ms write latency, 21 registrations and 6 applications succeeded with zero failures and no recorded failure reasons). The alert fired only because the threshold is 0. User impact: one candidate silently did not receive a stage-transition or registration email and will not receive any future email at that address until the suppression entry is removed. If it was a verification or stage-change notification, that candidate may appear stalled in the pipeline from their perspective. The suppression is a symptom of an earlier event — a hard bounce (typo'd or non-existent address) or a complaint/spam report — so the real question is which, because complaints affect sender reputation and repeated hard bounces suggest an address-validation gap at registration. Unknowns to respect: the alert payload does not include the recipient address, the template/stage that triggered the send, or whether the app records and retries suppressed sends. SES account-level bounce and complaint rates, and reputation/sending-pause status, are not present in this data — treat them as unknown rather than fine. Staff activity is 0 in the window, so nobody is watching this live.

Suggested checks

• 1.Pull the failing send from the application email log for ~17:41–17:46 UTC to get the recipient address, the template, and which stage transition or registration triggered it.
• 2.Query the SES suppression list for that address (account-level and global) to see the suppression reason and timestamp — BOUNCE vs COMPLAINT. BOUNCE with a malformed/nonexistent domain means a bad address; COMPLAINT means the candidate marked mail as spam and should not be re-sent to.
• 3.Check whether the address looks like a typo or disposable domain; if so, confirm whether registration does any MX/format validation — a gap there would explain recurring hard bounces.
• 4.Check SES account-level bounce rate, complaint rate, and reputation/sending-pause status in the SES console or CloudWatch (not in this payload). Bounce >5% or complaint >0.1% escalates this from a one-off to a deliverability risk.
• 5.Confirm whether the application retries or dead-letters suppressed sends, or drops them silently — determines whether this candidate's notification can be replayed after the suppression is cleared.
• 6.Look at the candidate's pipeline state: if the suppressed email was a verification or interview-invite, they may be blocked and need out-of-band contact.
• 7.Review emailsFailed over the last 24–48h to distinguish a single stale address from a growing pattern; if it is flat at ~1 occasional failure, consider raising the alert threshold above 0 or splitting suppression-list failures into a lower-severity signal.
• 8.If the suppression was a BOUNCE on an address the candidate has since corrected, a human can delete the suppression entry and re-trigger the send — do not do this for COMPLAINT suppressions.

Suggestions only — this system has no ability to act on them.

What this likely means

Three email deliveries failed in the window, all with the same reason: the recipient address is on the SES suppression list. Everything else looks healthy — API 200 at 19ms, 0 HTTP 5xx, ECS 3/3 tasks at ~4% CPU and 23% memory, RDS at 4% CPU with sub-millisecond write latency and 9 connections, and 17 registrations plus 4 applications all succeeded with no recorded failure reasons. So this is almost certainly not an infrastructure, capacity, or code-path problem: the application is calling SES successfully and SES is deliberately refusing the send for those three specific recipients. SES suppression entries come from prior hard bounces or complaints against those addresses (either your account-level suppression list or the AWS global list). Likely causes: candidates who registered with a typo'd or defunct address, addresses that previously bounced during an earlier stage transition, or someone who marked a past transactional email as spam. Because 17 talents were active and only 3 emails failed, this reads as a per-recipient data-quality issue rather than a systemic sending failure — but I'm inferring that from the failure count and the uniform reason, not from a confirmed list of affected recipients. User impact: those specific candidates silently received no stage-transition email. They will not know their application moved to review/assessment/interview, and if the pipeline assumes email is the notification channel they may stall. The failure is silent to them and, unless the app surfaces it, silent to staff too — note staff activity is 0 in this window, so no one is watching. Worth flagging: whether these sends are retried, whether the failure is persisted against the candidate record, and whether a stage transition is committed even when its email fails are all things I cannot tell from this data. Also unknown here: the account-wide SES bounce and complaint rates. If suppression entries are accumulating steadily, the underlying bounce rate could be trending toward the SES reputation thresholds, which would be a much more serious problem than three missed emails — that needs to be checked directly, not assumed healthy.

Suggested checks

• 1.Pull the three failed sends from application logs for this window and get the actual recipient addresses and the stage transition each was tied to — this immediately tells you whether it's 3 distinct candidates or 1 candidate retried 3 times.
• 2.Query the SES account-level suppression list for those addresses (aws sesv2 get-suppressed-destination) to see the reason (BOUNCE vs COMPLAINT) and the date added. A recent BOUNCE suggests bad address entry; a COMPLAINT is a deliverability concern.
• 3.Check the SES account reputation dashboard for current bounce rate and complaint rate against AWS thresholds (5% / 0.1%). This is the escalation question: three failures are trivial, an upward-trending bounce rate is not.
• 4.Inspect those candidates' email addresses for obvious typos (gmial.com, missing TLD, trailing whitespace). If they're malformed, the real fix is registration-time validation, not SES.
• 5.Determine whether the stage transitions for those candidates committed in the database despite the email failing — confirm the pipeline state is correct and only the notification was lost.
• 6.Check whether the application retries failed sends or dead-letters them, and whether the failure is recorded against the candidate record so staff can see it. Silent loss is the bigger risk here.
• 7.Review the suppression-list growth rate over the last 7–30 days to distinguish a one-off from an accumulating pattern.
• 8.Decide per-address whether to delete the suppression entry (only if the address is genuinely valid and the bounce was transient/mistaken) or to contact the candidate through another channel and correct their address.

Suggestions only — this system has no ability to act on them.

What this likely means

Three transactional emails failed, all for the same reason: the recipient address is on the SES suppression list. This is a per-recipient delivery problem, not a platform fault. Everything around it is healthy and corroborates that: API reachable at 200 in 25ms, ECS 3/3 tasks at 2.6% CPU / 28.6% memory, RDS at 4.3% CPU with 9 connections and 949MB freeable, zero 5xx on both API and web, 3 healthy hosts each. Registrations (28) and applications (6) all succeeded with no recorded failure reasons, so the pipeline that generates these emails is working — the messages were produced and handed to SES, and SES declined to send to those specific addresses. SES suppression normally means those addresses previously hard-bounced or filed a complaint, either on your account-level suppression list or the global AWS one. User impact is narrow but real and silent: up to 3 candidates did not receive a stage-transition email (e.g. assessment or interview invite) and have no way to know it. They will appear in the pipeline as unresponsive. Note the 8 HTTP 4xx on the API in this window — likely unrelated client/validation noise given 18 active candidates, but worth a glance to confirm none are from the email-sending path. Also note RDS write latency of 23.8ms, which is elevated relative to a read latency of 0ms; it is not causing this alert and may simply reflect low sample volume, but it is the one metric that is not obviously idle. Unknowns: the actual recipient addresses, whether the suppression is account-level or global, the original bounce/complaint reason, and whether any retry or dead-letter mechanism exists for failed sends are not in this data. Bounce and complaint rates for the account are also not instrumented here — do not assume they are healthy.

Suggested checks

• 1.Pull the 3 failed send records from your application logs or SES event destination and get the actual recipient addresses, message IDs, and which stage transition each was for — this tells you who was affected and what they missed.
• 2.Check each address against the SES suppression list (aws sesv2 get-suppressed-destination --email-address <addr>) to see the suppression reason (BOUNCE vs COMPLAINT) and the date it was added.
• 3.Determine whether these are account-level suppressions (yours, removable) or the AWS global suppression list — the remediation differs, and global entries you cannot delete directly.
• 4.Look at the pattern: are all 3 addresses on the same domain or from the same signup batch? A shared domain points at a receiving-server or reputation issue rather than 3 independent bad addresses.
• 5.Check your SES account reputation dashboard for current bounce and complaint rates — if these are rising, a handful of suppressions is an early symptom of a larger deliverability problem and sending could be paused.
• 6.Confirm whether failed sends are retried or dead-lettered, or silently dropped. If dropped, these 3 candidates need manual outreach regardless of how you fix the suppression.
• 7.Verify that registration email-address validation is catching typos and disposable domains — hard bounces from invalid addresses at signup are the most common source of suppression entries.
• 8.Sanity-check the 8 API 4xx in this window to confirm none originate from the email-sending or notification endpoints.
• 9.If the addresses look legitimate and the bounce was transient, a human can remove the suppression entry and re-trigger the stage-transition email for those candidates; do not bulk-delete suppressions without reading each bounce reason first.

Suggestions only — this system has no ability to act on them.

What this likely means

Scope is one email, not a delivery outage. A single send failed with SES reason 'Recipient is on the SES suppression list' — that is SES refusing to send to one specific address, not TaliLink failing to send. Every surrounding metric is healthy and corroborates a narrow, recipient-specific issue rather than a platform fault: API reachable (200, 33ms), 0 HTTP 5xx across both API and web, ECS 3/3 tasks at 5% CPU / 23% memory, RDS at 4.5% CPU with 9 connections and sub-millisecond write latency, and the workflows that trigger emails are succeeding (42 registrations, 12 applications, zero failures, no recent failure reasons). So the alert fired on a threshold of 0, which any single suppressed recipient will breach. User impact is likely limited to one candidate who will not receive their transactional email (e.g. a stage-transition notice) and has no in-product signal that it was never delivered. Addresses land on the SES suppression list after a prior hard bounce or complaint, so the underlying cause is most likely a bad/typo'd or abandoned address, or a candidate who previously marked TaliLink mail as spam — possibly from an earlier send. Note what the data does not tell you: the recipient address, the email type/stage, and the candidate ID are not in this payload, and there is no metric here for the account-wide SES bounce or complaint rate or sending-quota state. Those are unknown, not healthy. The one thing worth ruling out quickly is whether this is the first of a cluster — a rising suppression count would point at a list-hygiene or reputation problem that could eventually threaten SES sending privileges for all candidates. A single event on a zero threshold does not yet support that reading. The 12 HTTP 4xx on the API are almost certainly unrelated at this traffic level (124 requests, 22 active candidates, normal for validation errors and auth challenges), but nothing in this data links them either way.

Suggested checks

• 1.Pull the application/email-service log line for this failed send to get the recipient address, email type, stage transition, and candidate ID — without it you cannot tell whether this is a typo'd signup address or a real candidate silently missing a pipeline notification.
• 2.Query the SES suppression list for that address (account-level and, if used, the global list) to see the suppression reason (BOUNCE vs COMPLAINT) and the timestamp it was added. COMPLAINT is more serious than BOUNCE and should not be blindly removed.
• 3.Check the account-wide SES bounce rate, complaint rate, and sending quota in the SES reputation dashboard — not in this payload, so currently unknown. This is the fastest way to confirm whether this is one isolated address or the leading edge of a reputation problem.
• 4.Count suppression/bounce failures over the last 24–72 hours rather than this 5-minute window. A flat baseline of ~1 means this alert's threshold of 0 is too tight; a rising trend changes the diagnosis entirely.
• 5.Inspect the candidate record for that address: is it a real, active candidate mid-pipeline, or a spam/typo registration? If real and mid-pipeline, they need the notification delivered by another route and the pipeline stage may need a manual nudge.
• 6.Verify whether the send failure is retried or silently dropped by the email code path, and whether staff have any visibility that a candidate's notification failed. A silent drop is the more damaging defect here than the single suppressed address.
• 7.If the suppression was a BOUNCE on a verified-good address (e.g. transient mailbox-full that has since resolved), consider removing it from the suppression list and re-sending — but confirm the address is genuinely deliverable first, since re-sending to a hard-bouncing address further damages sender reputation.
• 8.Review registration-time email validation. 42 successful registrations in this window with no address verification step is a plausible upstream source of future hard bounces and suppressions.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: the NestJS MailService is repeatedly failing to authenticate its secondary/failover SMTP transport against Google Workspace with '535-5.7.8 Username and Password not accepted' — roughly 16 occurrences logged between 07:05:29 and 07:05:48, i.e. a tight, ongoing retry loop rather than a one-off. Everything else is healthy: API 200 in 27ms, ECS 3/3 tasks at 6% CPU / 30% memory, RDS 5% CPU with 9 connections and sub-1ms write latency, no 5xx on either ALB, 6 healthy API hosts and 3 healthy web hosts. Workflows are succeeding normally in the window — 44 registrations, 13 applications, zero failures — and emailsFailed is reported as 0, which is consistent with the primary path (SES) still delivering. This is almost certainly a credential problem, not an infrastructure problem: a Google Workspace app password that was revoked, expired, rotated, or invalidated (e.g. by a password change, 2FA/app-password policy change, or admin-enforced revocation), and the env var SMTP_USER/SMTP_PASS in the running task definition still holds the old value. Impact right now is latent, not user-visible: candidates are still receiving stage-transition, invite and password-reset email via SES, so no one is blocked today. The real exposure is that redundancy is gone — if SES is throttled, suspended, or hits a sending/reputation problem, transactional mail fails outright, which on this platform means candidates cannot complete password resets or receive invites and stage-transition notices. Two caveats: emailsFailed=0 covers only the observed window and may only count primary-path sends, so it is weak evidence that no mail is being lost, not proof; and I cannot see SES-side metrics (bounce rate, send quota, suppression list) in this payload, so SES health is unknown rather than confirmed good. A secondary concern is the retry loop itself — if it is unbounded it can generate log volume and, on Google's side, repeated failed logins can trigger further account lockout.

Suggested checks

• 1.Confirm the Workspace account state directly: in Google Admin, check whether the mailbox used by SMTP_USER is suspended, had its password changed recently, or had app passwords revoked; also check whether 2FA/app-password policy changed for that OU. This single check usually resolves the cause.
• 2.Read the actual SMTP_USER value from the running ECS task definition / Secrets Manager or SSM parameter and confirm which account is being used — a mismatch between the account that owns the app password and the account in the env var produces exactly this 535-5.7.8.
• 3.Check whether the secret was rotated in Secrets Manager/SSM but the ECS service was never redeployed — Fargate tasks pick up secret values at task start, so a rotated secret with no new deployment leaves stale credentials in memory. Compare secret last-modified time against the task start time.
• 4.Verify SES is genuinely healthy, since it is now the only path: check SES send quota vs 24h usage, bounce and complaint rates, account sending-enabled status, and whether the sending identity is still verified. Do not assume SES is fine just because emailsFailed is 0.
• 5.Confirm what emailsFailed actually counts. If it only increments on primary-path failures, it tells you nothing about failover-path mail, and some invite/password-reset sends may already be silently dropped rather than retried. Check the mail queue/outbox for stuck or dead-lettered messages since ~07:00.
• 6.Check the retry/backoff behaviour in MailService — ~16 auth attempts in 20 seconds suggests little or no backoff. Repeated failed logins can cause Google to lock the account further, and will bloat CloudWatch log ingest.
• 7.Once a new app password is generated, update the secret and force a new ECS deployment, then verify by watching for the absence of the MailService error and by triggering a test password-reset through the failover path specifically.
• 8.Lower priority: the 2 HTTP 4xx on the API and the 468ms average API response time against a 27ms health-check response are worth a glance, but with 0 5xx and 24 requests in the window they are most likely unrelated background noise, not part of this incident.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests a performance bottleneck affecting user experience. Despite all components reporting healthy usage levels, with low CPU and memory usage across ECS and RDS, and no HTTP 5xx errors indicating server errors, the average API response time remains high at 2493.3ms. The absence of increased database latency and the fact that CPU and memory are not maxed out across resources suggest the bottleneck may reside in application processing or network latency. The elevated API response time, despite low traffic levels (130 requests), further points to an internal inefficiency or misconfiguration rather than resource exhaustion.

Suggested checks

• 1.Examine API and application logs for signs of long-running processes or errors that could cause delays.
• 2.Check network latency or issues within the VPC that might affect API performance.
• 3.Inspect recent deployment or configuration changes that may have introduced inefficiencies.
• 4.Review application-level tracing or profiling to identify slow endpoints or handlers.
• 5.Ensure that any third-party services or external APIs that the backend depends on are performing optimally.

Suggestions only — this system has no ability to act on them.

What this likely means

The high write latency of 160.2ms in the PostgreSQL RDS instance likely indicates a performance bottleneck in write operations to the database. Despite this, the API is reachable with a response status of 200 and normal error rates, suggesting that the immediate impact on the platform's functionality and user experience might not yet be severe. However, elevated write latency could lead to delays in operations such as candidate registrations or job applications, particularly if they involve writing data to the database. The low CPU utilization (4.29%) and modest number of connections (9) suggest that the database instance is not CPU-bound. The platform's Next.js frontend also shows normal operation, with low response times and no HTTP errors, indicating that current user interactions are likely unaffected. Despite the limited immediate impact, prolonged high write latency can degrade performance over time, especially under heavier loads.

Suggested checks

• 1.Check database I/O performance metrics to identify possible disk issues or IOPS bottlenecks.
• 2.Review recent slow query logs to identify if specific queries are causing the latency spike.
• 3.Investigate any recent changes to database schema or queries that could affect write operations.
• 4.Ensure that the database's storage throughput provisioning is adequate for current workload demands.
• 5.Verify the RDS instance's instance type and storage configuration to ensure it is adequate for current and future demand.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests latency issues affecting user interactions with the platform, likely resulting in slow page loads or timeouts. Despite the high response times, there are no elevated 5xx errors, indicating that the API is reachable and functional, but experiencing delays. Both ECS and RDS show low CPU and memory utilization, implying they are not bottlenecks. The error logs indicate some POST requests to /api/uploads have encountered errors, though these do not appear to be widespread. The transactional and workflow activities, such as registrations and applications, are completing successfully without failures, supporting that the latency might be isolated or affecting a subset of functions like file uploads.

Suggested checks

• 1.Examine the network latency between the load balancer and ECS tasks to verify if there are any networking issues.
• 2.Review the logs and metrics related to the /api/uploads endpoint to identify any recurring issues or bottlenecks.
• 3.Verify if there are any latency issues or errors in the Storage or file processing systems that might affect uploads.
• 4.Check AWS CloudWatch or equivalent monitoring for any underlying infrastructure alarms or events during the time of elevated response times.
• 5.Investigate the load balancer for any potential issues or misconfigurations that could affect HTTP routing.
• 6.Examine recent deployments or code changes to see if they correlate with the onset of latency issues.
• 7.Check for any unusual traffic patterns or spikes that could temporarily be contributing to higher than usual latency.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests a bottleneck impacting users' experience. Despite the API's healthy status, the increased response time indicates potential issues in the request handling pipeline. With low CPU and memory usage on ECS and RDS, system resource exhaustion is unlikely. The absence of elevated 5xx errors or failed workflows reinforces this. The discrepancy between the reported responseMs in apiTraffic (2422.3ms) and the API's quicker responseMs (45ms) suggests intermittent network latency or another external factor prolonging API requests as aggregated by apiTraffic.

Suggested checks

• 1.Check network latency and performance between the load balancer and ECS.
• 2.Review application logs on ECS containers to find any unusual delays.
• 3.Analyze ALB logs for long-lived API requests.
• 4.Examine slow database queries or contention causing delays.
• 5.Inspect third-party service dependencies for latency contributing to longer response times.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests there is a bottleneck or a delay in the API processing, affecting user experience. Considering that the ECS and RDS resource usage is low (CPU 2.07%, memory 29.37%, RDS CPU 4.93%), the issue is likely not due to resource exhaustion. API traffic shows 8 requests with an average response time of 2135.6ms, but there are no 5xx errors, which indicates the API is functioning but slowly. With only 1 HTTP 4xx error, this points to a likely application-level slowness rather than an influx of invalid requests or server errors. The ECS tasks are running as desired, and the database latency is low, so network issues or an isolated slow query could also contribute to this delay.

Suggested checks

• 1.Review recent deployments or code changes in the API that might increase execution time.
• 2.Check for any recent configuration changes in NestJS or AWS ECS that might impact performance.
• 3.Investigate specific API endpoints or operations that correlate with the elevated response time.
• 4.Analyze the database queries executed during this period for any anomalies or slow executions.
• 5.Check the API load balancer logs for unusual traffic patterns or delays in request routing.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests a performance issue, but specific metrics show the underlying systems are performing within normal parameters. The ECS and RDS instances have low CPU usage, indicating they aren't overwhelmed. Network latency to or from the API could be a factor, or potential blockage in between steps of processing HTTP requests. Also, the slightly elevated HTTP 4xx errors suggest user or input errors, but no 5xx implies server-side errors aren't present. Given the low number of requests, possible issues with the load balancer or network configuration might be causing delayed responses despite available resources.

Suggested checks

• 1.Check network latency or packet loss between the load balancer and ECS instances.
• 2.Inspect any recent changes to network, security, or load balancing configurations.
• 3.Review recent code changes in API endpoints that could cause processing bottlenecks.
• 4.Cross-check if the current response time is anomalous with historical data for similar loads.
• 5.Check logs for individual request paths showing elevated response times to isolate specific API actions or queries.
• 6.Inspect if the Next.js frontend has reported any client-side errors affecting API interactions.

Suggestions only — this system has no ability to act on them.

What this likely means

The API response time of 4488ms, which has breached the threshold of 2000ms, indicates a performance bottleneck. The data shows the ECS CPU and memory usage are low, ruling out resource exhaustion on the application servers. The RDS metrics show low CPU usage and latency, suggesting the database is not currently a bottleneck. The report of 0 HTTP 5xx errors suggests the API is error-free but slow. Given the low traffic (10 requests) relative to the elevated response time, this is consistent with an issue in backend processing or a third-party service the API relies on being slow, rather than server overload. The absence of collection errors also suggests that data input isn't disrupted. Users may experience delays in their requests being processed, affecting interactions like job applications and stage transitions.

Suggested checks

• 1.Check for any third-party service dependencies that might be slow or experiencing issues.
• 2.Review API logs for any unusual processing delays or errors not captured as HTTP 5xx.
• 3.Examine the code or configuration changes recently deployed, which could have introduced processing slowdowns.
• 4.Investigate if there are scheduled background tasks or jobs running longer than usual.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time suggests that users experience slower-than-expected interactions with the platform. However, the system metrics show low resource usage: ECS CPU is at 2.05%, memory at 27.62%, and RDS CPU at 4.14%. Network or concurrency issues may contribute, considering the low traffic (20 API requests in 5 minutes) and no signs of system overload. Given zero API 5xx errors, current delays are more likely due to queuing, load balancing inefficiencies, or transient spikes in resource demand. It is worth checking if certain endpoints are particularly slow or if external dependencies, such as transactional email operations, introduce latency.

Suggested checks

• 1.Inspect detailed API endpoint metrics to identify which endpoints are experiencing slow responses.
• 2.Review AWS ECS Fargate task logs for any signs of intermittent bottlenecks.
• 3.Check the Application Load Balancer for uneven request distribution across healthy hosts.
• 4.Analyze recent trends in API response time to find possible spikes or patterns.
• 5.Investigate any indirect RDS issues, such as increased query complexity or unexpected query volume.
• 6.Look into transactional email processing or external API dependencies impacting API response times.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time of 2529.7ms is primarily affecting the user experience by making interactions with the platform slower than expected. Despite the high average response time, the system is reachable and returning 200 status codes, indicating it is functioning but with degraded performance. The ECS CPU and memory usage are low, suggesting that compute resource limitations in ECS are not the root cause. Similarly, the RDS instance shows low CPU usage and normal read/write latency, ruling out database processing as a bottleneck. The presence of recent errors related to the AnthropicService and multiple schema validation failures might imply issues in auxiliary service calls or business logic handling that could be contributing to the increased API response time. No indication of exhausting AI feature limits or API 5xx errors; thus, the platform's core processing is healthy, but latency issues are likely rooted in specific logic or external service calls.

Suggested checks

• 1.Review any recent changes or deployments in the NestJS application for potential logic causing delays.
• 2.Investigate the AnthropicService errors in the logs to identify any external dependencies causing latency.
• 3.Examine database query performance and ensure any long-running queries are optimized or indexed correctly.
• 4.Check for network latency issues between the ECS Fargate instances and the RDS instance or any external APIs.
• 5.Verify that transaction email services related to stage transitions are not delaying API responses.

Suggestions only — this system has no ability to act on them.

What this likely means

The platform is experiencing high write latency on the PostgreSQL RDS instance, recorded at 117.2ms against a threshold of 100ms. While other metrics such as API response times and error rates appear normal, the elevated write latency could result in slower performance for operations involving data submission or updates, such as candidates applying for jobs or moving through the pipeline stages. This might slightly degrade user experience but is not causing outages or visible errors. The low CPU usage and connections on RDS, and absence of application-level errors, suggest the issue is not due to an overloading condition or increased traffic demands, but may be related to I/O performance issues on the RDS instance itself.

Suggested checks

• 1.Investigate RDS instance storage performance and IOPS, particularly focusing on recent changes or stressors that could affect write efficiency.
• 2.Check for any recent configuration changes or maintenance activities on RDS that could impact disk performance.
• 3.Examine RDS CloudWatch metrics for any signs of disk throughput limitations or throttling.
• 4.Verify if there has been an increase in demand for write operations that could be affecting latency.
• 5.Review any recent updates or deployments on the platform that might coincide with the onset of write latency issues.

Suggestions only — this system has no ability to act on them.

What this likely means

The platform is experiencing elevated database write latency at 101.9 ms, just above the threshold of 100 ms, which could lead to slower performance during write operations. This can affect the speed at which user actions like submitting applications or registering are processed. Despite the high write latency, the overall system appears to be functioning without errors (e.g., no elevated 5xx errors or failed workflows). The CPU usage on both ECS and RDS is low, indicating that the issue is likely not related to CPU resource constraints.

Suggested checks

• 1.Examine disk IOPS performance on the RDS instance to check for any bottlenecks.
• 2.Inspect RDS instance storage for potential saturation or issues (e.g., disk space available, burst credits if using burstable storage).
• 3.Review recent changes to database schema or queries that could increase write latency.
• 4.Check for any recent spike in write operations or large batch processes that could be impacting performance.

Suggestions only — this system has no ability to act on them.

What this likely means

The elevated API response time of 2244.4ms despite a healthy platform indicates a localized slowdown. With CPU and memory usage low on both ECS and RDS, hardware resource saturation is unlikely. The absence of API errors (0 HTTP 5xx) suggests stability, but 7 HTTP 4xx errors indicate some bad requests affecting performance. Traffic levels are modest (35 API requests in 5 mins), ruling out volume bottlenecks. The issue may lie in specific request patterns or payloads causing slowness, or an unoptimized process during specific API calls.

Suggested checks

• 1.Examine recent logs for slow endpoints or large payloads.
• 2.Review payload sizes and complexity of queries during slow requests.
• 3.Check for data processing inefficiencies or round trips in the API logic.
• 4.Investigate retry logic if malformed requests (HTTP 4xx) are causing stalls.
• 5.Ensure no unusual ASG configuration changes influencing performance latency.

Suggestions only — this system has no ability to act on them.

What this likely means

The average API response time is slightly elevated at 2126ms, above the threshold of 2000ms. Despite this, the ECS and RDS metrics are within normal operating ranges, indicating that neither the application server nor the database is heavily overloaded. The network and traffic load is relatively low with only 6 API requests and 3 web requests during the monitoring period. This suggests the latency might be due to specific API endpoints or certain requests rather than general system load. The API requests have 3 instances of HTTP 4xx errors but no 5xx errors, meaning there are some client-related issues. Two upload errors were recorded but do not directly contribute to API response latency because they involve file uploads which are separate from the average response time latency being observed. Therefore, it most likely means the latency may be related to specific processes or endpoints within the API that are not heavily impacted by server or database performance.

Suggested checks

• 1.Check if specific endpoints have increased processing times or inefficient queries.
• 2.Review the API logs for error patterns or slow query logs around the time of increased latency.
• 3.Verify the recent changes in the API codebase or configuration that might affect response times.
• 4.Examine detailed request logs for requests with high response times to identify any patterns.
• 5.Investigate any new integrations or third-party services that might add latency.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 21 of 300 API requests (7%) returned 5xx in the window. Every captured failure reason is the same endpoint — GET /api/me/applications -> 500, thrown through AllExceptionsFilter — with occurrences spread across 08:14 to 08:17, so this is ongoing rather than a single blip. Infrastructure looks healthy and is unlikely to be the cause: ECS 7% CPU / 28% memory with 3/3 tasks running, RDS 4.8% CPU, 9 connections, 1032 MB freeable, sub-3ms write latency, 3 healthy hosts on both target groups, and the direct API health probe returned 200 in 23ms. The frontend is clean (287 requests, zero 4xx/5xx). Write paths are fine too: 228 registrations succeeded with 0 failures and 0 email failures. So this is almost certainly an application-level bug on one read endpoint, not saturation, not a deploy-wide outage, and not a database availability problem. User impact (inferred): 83 talents were active in the 5-minute window and 0 staff. /api/me/applications is the candidate's "my applications" view, so affected candidates likely see a broken or empty applications dashboard after logging in or registering. Notably applicationsSubmitted is 0 in this window while registrations are high — consistent with a burst of new signups whose application list is empty or whose data is in an unexpected shape (e.g. a null join on a pipeline stage, missing profile row, or an unhandled case for a user with zero applications). That is a plausible hypothesis, not established: the log lines carry no stack trace or exception class, so the actual throw site is unknown from this data. Caveats: the aggregated failure reasons account for 8 occurrences, not 21 — the remainder are not represented in this sample, so I cannot rule out a second failing endpoint. The 352ms mean API latency vs 7.6ms on web is elevated but not alarming, and per-endpoint latency is not instrumented here. Whether the 12 4xx are related is unknown.

Suggested checks

• 1.Pull full ECS task logs around 08:14–08:18 for AllExceptionsFilter entries and get the actual stack trace / exception class for GET /api/me/applications — that alone will likely identify the throw site.
• 2.Confirm whether all 21 5xx are this one endpoint or whether other routes are failing: group ALB/access logs by path and status for the window. The listed reasons only cover 8 of 21.
• 3.Reproduce directly: call GET /api/me/applications with a token for (a) a brand-new user with zero applications and (b) an established user with applications. The 228 registrations / 0 applications submitted pattern points at the empty or newly-created-user case.
• 4.Check recent deploys or migrations — get the current running task definition revision and its deploy time, and compare against the first occurrence of these 500s (before 08:14). A schema change with a nullable/missing column on the applications or stage join is a prime suspect.
• 5.Inspect the query behind that endpoint against the DB directly (joins to pipeline stage, profile, job). RDS is idle so this is a correctness issue, not contention — look for a null-dereference or a row the code assumes exists.
• 6.Verify whether the failures are spread across all 3 tasks or concentrated on one; if one task, compare its image/config, since a single bad task would also explain an intermittent ~7% rate.
• 7.Check error rate by user: if a small set of candidate accounts generate all the 500s, dump their applications/profile rows to find the data shape that breaks the handler.
• 8.Review the 12 4xx separately to confirm they are ordinary auth/validation noise and not a related symptom.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 458 API requests in the window, 37 of them 5xx (8.1%), while infrastructure is essentially idle — ECS 6.5% CPU / 27% memory with 3/3 tasks running, RDS 4.4% CPU, 9 connections, 1034 MB freeable, write latency 1.6 ms, and all 3 API targets healthy. A synthetic probe against the API returned 200 in 30 ms. So this is not capacity, not database saturation, and not a host/task outage — it is consistent with a specific code path throwing on specific requests. The captured failure reasons point almost entirely at one endpoint: GET /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility?email=..., failing repeatedly across different candidate emails on a single job ID (cmtvbo4hj001m...), spanning 7:54 to 7:58. One unrelated GET /api/me/applications -> 500 also appears. Note the sample of reasons shown is only 5 lines, not all 37 errors, so the endpoint attribution is inferred from that sample, not proven for all failures. Likely user impact: candidates viewing or applying to that one job are hitting a broken sponsor-eligibility check. Registrations are unaffected (148 succeeded, 0 failed) and 66 talents were active in the 5-minute window, so traffic is real user traffic, not a probe artifact. Notably applicationsSubmitted is 0 for this window with 0 failures — that may be normal for a 5-minute slice, or it may mean candidates are stalling before submit because the eligibility call fails; the data cannot distinguish these. Mean API latency of 294 ms is unremarkable and the frontend is clean (310 requests, 0 5xx, 6.5 ms), so the Next.js app itself is serving fine; users would see an error state or a stuck widget inside an otherwise working page. Email failures are 0, but emailsFailed only covers attempted sends — it says nothing about emails never triggered. Most likely cause, stated as inference: an unhandled exception in the sponsor-eligibility handler — e.g. a null/missing field on that particular job record (a sponsorship or visa attribute that other jobs have), an unhandled result from a downstream/external eligibility lookup, or a query returning no row where the code assumes one. The fact that it fails for many different emails but one job ID points at job-record data rather than per-candidate data. Unknown from this data: the actual stack trace, whether all 37 5xx share this endpoint, and whether any write path is affected.

Suggested checks

• 1.Pull the full AllExceptionsFilter log lines with stack traces for the 07:53–07:59 window in CloudWatch Logs — confirm how many of the 37 5xx are sponsor-eligibility vs. other routes, and read the actual exception type and throwing line.
• 2.Call GET /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility?email=<test address> directly to confirm it reproduces deterministically, then call the same endpoint against two or three other job IDs to confirm the failure is scoped to this one job.
• 3.Inspect the job row cmtvbo4hj001mw001u4prr8tq in Postgres and diff its sponsorship/visa/eligibility-related columns against a job whose endpoint returns 200 — look for NULLs, empty arrays, or an enum value the handler doesn't branch on.
• 4.Check whether sponsor-eligibility calls an external service or third-party API; if so, check that dependency's error rate and timeouts for the same window (not covered by any metric here).
• 5.Separately reproduce GET /api/me/applications as an authenticated candidate — confirm whether that 500 is the same root cause or an independent bug that this alert is masking.
• 6.Check whether this job was created or edited shortly before 07:54, and correlate against the most recent ECS task deployment time, to distinguish bad data from a bad deploy.
• 7.Confirm whether applicationsSubmitted = 0 is abnormal by comparing against the same endpoint's submission counts in preceding windows — this determines whether the broken eligibility check is actually blocking applications or just degrading a page element.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 22 of 422 API requests (5.2%) returned 5xx in the window, and API p-average latency is 736ms while the health-check probe itself answered in 20ms with a 200. The named failures cluster on GET /api/me/applications (6 occurrences across 07:49-07:53, from the AllExceptionsFilter) plus one GET /api/public/jobs/:id/sponsor-eligibility. Infrastructure is not strained: ECS 3/3 tasks running, CPU ~6%, memory ~25%; RDS CPU 4.55%, 9 connections, 1030MB freeable, read latency 0ms. All 3 ALB targets are healthy on both API and web, and the Next.js frontend is returning zero 4xx/5xx with 6.5ms responses. Registrations are succeeding (125, 0 failed) and 0 emails failed. Most likely interpretation: this is an application-level fault on specific read endpoints, not a capacity, deploy-wide, or database-down event. If it were resource exhaustion or a bad task, you would expect elevated CPU/memory, unhealthy hosts, or failures spread across registration and other write paths — none of which are present. The concentration on /api/me/applications (a candidate-facing read of their own applications) is consistent with a query or serialisation error triggered by particular data — e.g. a null/orphaned relation, a candidate with an unusual pipeline stage, or an unhandled edge case — rather than a systemic outage. One corroborating oddity: RDS write latency is 32.2ms against 0ms read latency, which is high relative to the near-idle CPU; that may be unrelated background write activity, but it is worth a glance since the 736ms average API latency suggests something is slow somewhere. User impact: likely narrow but sharp. 56 talents were active in the 5-minute window and 125 registrations succeeded, so signup is working. But candidates hitting their applications dashboard are getting errors — for an affected user this probably fails every time, not intermittently, since it is likely data-dependent. applicationsSubmitted is 0 in this window; with 0 applicationsFailed, that is most plausibly just no submission attempts, but I cannot rule out that broken dashboard navigation is suppressing submissions. Note also that staff activity is 0, so no one internal is likely to notice from the inside. The 21 4xx responses are not broken out by route or status and are unexplained. Not instrumented / unknown here: the actual exception stack traces and messages (the log lines captured only the route and status), per-route request counts and latency breakdown, which user IDs hit the errors, recent deploy timestamps, and whether the 22 5xx are all from these logged routes — the sampled reasons account for only 7, so 15 of the 22 have no attached reason in this payload.

Suggested checks

• 1.Pull the full stack trace for GET /api/me/applications 500s from CloudWatch Logs around 07:49-07:53 — the AllExceptionsFilter line here has no message or exception class, and the actual error type (Prisma/TypeORM query error vs TypeError on a null relation vs timeout) immediately separates a data-edge-case from an infrastructure fault.
• 2.Account for the other ~15 5xx not covered by the 5 sampled failure reasons: break down 5xx by route and target in ALB/ECS logs to confirm the errors really are concentrated on these two endpoints rather than spread wider.
• 3.Check whether the 5xx are concentrated on one ECS task or spread across all three. One bad task (stale code, exhausted connection pool, leaked state) looks very different from a code bug hitting all replicas evenly.
• 4.Check for a deploy, migration, or config change in the last 1-2 hours. A schema change that left orphaned rows or a nullable column the query does not handle would produce exactly this pattern — healthy infrastructure, one read endpoint failing on a subset of data.
• 5.Identify the specific candidates whose requests failed (from request logs / auth context) and try to reproduce by loading their applications record. Look for anomalies: an application referencing a deleted job, a null stage, or a stage value outside submitted/under review/assessment/interview/decision.
• 6.Inspect the sponsor-eligibility failure separately (email omane_4@icloud.com, job cmtvbo4hj001mw001u4prr8tq). It may share a root cause with /api/me/applications if both join the same application/job tables, which would point at one bad row rather than two bugs.
• 7.Investigate the 736ms average API latency against near-idle ECS and RDS CPU. Check per-route latency and slow-query logs — if the failing endpoint is also the slow one, a query timeout is the likely mechanism; if latency is broadly elevated, look for an external dependency or connection-pool wait.
• 8.Look at RDS write latency of 32.2ms versus 0ms read latency on a database at 4.55% CPU. Probably benign, but check for lock contention, a long-running transaction, or autovacuum activity that could be stalling queries.
• 9.Confirm whether applicationsSubmitted = 0 is normal for this time of day by comparing against the prior few windows. If submissions normally occur here, the failure may be blocking the apply flow and the impact is wider than the error count suggests.
• 10.Check the 21 4xx responses by route and status code — they are uncategorised in this data, and a spike in 401/403 alongside the 500s would suggest an auth or token issue upstream of the failing endpoint rather than a data bug.

Suggestions only — this system has no ability to act on them.

What this likely means

A single endpoint is failing, not the platform. All 5 captured failure reasons are GET /api/me/applications -> 500 from the NestJS AllExceptionsFilter, spread across 07:42-07:46, so the 29 5xx (8.4% of 345 requests) are most likely concentrated on the candidate 'my applications' list. Infrastructure looks healthy and is not the constraint: ECS 3/3 tasks, CPU 5.8%, memory 24.7%; RDS CPU 5.8%, 9 connections, 1030MB freeable, write latency 1.4ms; ALB reports 3 healthy hosts on both API and web; the direct API health probe returned 200 in 27ms. That pattern is consistent with an application-level bug (unhandled exception in the handler/query/serialisation) rather than saturation, a bad deploy of the whole service, or a database outage. Two things deserve attention beyond the obvious. First, API p-average latency is 508.6ms versus 8.1ms on the web tier, which is high for a system this idle - either the failing path is timing out/retrying before throwing, or a slow query is affecting more than just this endpoint; the data does not tell us which. Second, applicationsSubmitted is 0 while 100 registrations succeeded and 50 talents were active in the 5-minute window. That may simply mean nobody happened to submit in this window, but it is also consistent with the application flow being broken upstream of submission - the log sample only covers a GET, so we have no evidence either way. applicationsFailed is 0, which means no failure was recorded, not that submissions are working. User impact: candidates who log in and open their applications page likely see an error or empty state, which is high-visibility even though registration and the frontend are unaffected. Note the failure reasons are raw log lines with no statusCode field and no stack trace, so the actual exception type is unknown from this data.

Suggested checks

• 1.Pull the full stack trace for GET /api/me/applications from ECS task logs around 07:42-07:46 - the AllExceptionsFilter line is truncated; the underlying exception (TypeError, query error, timeout) is what identifies the fix.
• 2.Confirm the blast radius: break down the 29 5xx by path/route in ALB or APM logs. If all 29 are /api/me/applications, it is one handler; if other routes appear, look for a shared dependency or a bad deploy.
• 3.Check whether all three tasks are erroring or only one - if errors are isolated to a single task, consider draining it and compare its image/env to the others.
• 4.Check recent deploys/migrations to the API and DB in the last hour. A schema change (e.g. a column the applications query reads) is the most common cause of a single-endpoint 500 with healthy infrastructure.
• 5.Reproduce as a candidate with applications: call GET /api/me/applications with a real token and see if it fails for all users or only for accounts with particular data (null stage, missing job, orphaned row). Intermittency here - 29 of 345 - suggests data-dependent failure.
• 6.Investigate the 508.6ms average API latency separately: check pg_stat_statements / RDS Performance Insights for slow queries. Read latency reporting 0ms with 508ms API responses suggests the time is spent in the app or on a query not reflected in that metric.
• 7.Verify whether application submissions are actually working, independently of the 0 count - try a submission in a test account or query the applications table for recent inserts. applicationsFailed=0 does not prove the path is healthy.
• 8.Check the transactional email path for stage transitions - emailsFailed=0 but with no submissions recorded in this window it may simply be untested; confirm the queue/provider is draining.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 16 of 318 API requests (5.03%) returned 5xx in the window, just over the 5% threshold. Infrastructure is not the constraint — ECS is at ~7% CPU / 24% memory with 3/3 tasks running, RDS at 4% CPU, 9 connections, 1032MB freeable, sub-2ms write latency, and all 3 API hosts healthy. A direct health probe returned 200 in 24ms. So this is almost certainly application-level, not capacity or availability. The named failures cluster on two candidate-facing GET endpoints: /api/public/jobs/{id}/sponsor-eligibility (4 occurrences, all against the same job ID cmtvbo4hj001mw001u4prr8tq, different candidate emails) and /api/me/applications (2 occurrences). The sponsor-eligibility failures repeating on one job ID with varying emails is consistent with bad or missing data on that specific job record, or an unhandled case in the eligibility logic — not a generic outage. Note the listed reasons only account for 6 of the 16 5xx; the other 10 are unattributed in this payload, so the blast radius may be wider than these two routes. Corroborating signals worth weighting: API average response time is 945ms against a web tier at 6.8ms and a health check at 24ms — that gap suggests some API requests are slow (possibly timing out or retrying) beyond the ones that hard-fail. Also, 93 registrations succeeded but applicationsSubmitted is 0 with 53 talents active in a 5-minute window. That is a plausible funnel break — candidates arriving and registering but not completing applications — though applicationsFailed is also 0, so this could equally be a cohort that has not reached the apply step yet. Treat it as a lead, not a conclusion. Staff activity is 0, so no internal user is currently reporting this. User impact: likely a subset of candidates hitting an error page when checking sponsor eligibility on one particular job, and some candidates unable to load their own applications list. emailsFailed is 0 and registrations are all succeeding, so signup and transactional email appear intact.

Suggested checks

• 1.Pull the full stack traces for the 5xx on /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility from CloudWatch Logs — the AllExceptionsFilter line gives no exception type. This is the fastest single step to a root cause.
• 2.Inspect the job record cmtvbo4hj001mw001u4prr8tq directly in RDS (sponsorship fields, any nullable columns the eligibility handler dereferences, related rows it joins to). Four failures on one ID across different emails points at that row's data.
• 3.Break down all 16 5xx by route and status to identify the ~10 not represented in the sample reasons — confirm whether this is confined to these two endpoints or broader.
• 4.Get the stack trace for GET /api/me/applications -> 500 and check whether it shares a code path or query with the eligibility endpoint (both may touch the same application/job join).
• 5.Check the p50/p95/p99 latency distribution on the API target group, not just the 945ms mean, and correlate slow requests with the failing routes — the health check at 24ms says the mean is being dragged by a specific subset.
• 6.Verify whether the 0 applicationsSubmitted is real or an instrumentation gap: query the applications table for rows created in the last 15–30 minutes and compare against the 53 active talents.
• 7.Check recent deploys or migrations to the API in the hours before 07:38 UTC — first failure timestamps cluster tightly at 07:38, which is consistent with a change or with a specific job being published/edited around then.
• 8.Confirm whether sponsor-eligibility calls any external or third-party service; if so, check that dependency's health, since it is not represented in these metrics and would read as unknown here.
• 9.Review ALB target group 5xx vs. ELB 5xx to confirm errors originate from the application and not the load balancer layer.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 20 of 280 API requests (7.1%) returned 5xx in the window, while infrastructure is idle — ECS 4% CPU / 23% memory with 3/3 tasks running, RDS 4.6% CPU, 9 connections, 1 GB freeable memory, read latency 0 ms / write 1.8 ms, and all 3 API targets healthy. A direct health probe returned 200 in 26 ms. So this is not capacity, not a crashed task, and not an unreachable database. The frontend is completely clean (248 requests, 0 errors, 8.4 ms), so users can load pages; the failures are in API calls made from those pages. The named failures are concentrated on GET /api/me/applications (4 of the 5 sampled log lines, spread across 07:28-07:32) and one GET /api/public/jobs/{id}/sponsor-eligibility. Both are read paths, both are candidate-facing. Note the sample only accounts for 5 of the 20 5xx responses — the other 15 are not itemised here, so do not assume they share the same cause. The logs are raw AllExceptionsFilter lines with no stack trace or exception type captured, so the actual defect is unknown from this data alone. Most likely interpretation (inference, not established): an unhandled exception in a specific read handler, triggered by particular data or particular user accounts rather than by load — consistent with a repeating failure at low CPU with a healthy DB. A plausible variant is a slow/failing query on that endpoint: mean API latency is 464.7 ms versus 8.4 ms on web, which is high for an idle system and could reflect a handful of very slow requests dragging the average, though RDS-side latency counters look fine. User impact: 31 candidates were active in the 5-minute window and 57 registrations succeeded with zero failures, and zero applications were submitted (with zero application failures — this may simply be a quiet window, or may mean candidates are stalling before submitting; the data cannot distinguish). The concrete visible harm is that logged-in candidates hitting their applications list are likely seeing an error or empty state, and at least one candidate hit an error on the sponsor-eligibility check during a job application flow. Registration and email are unaffected (emailsFailed 0). Unknowns to respect: no per-endpoint request counts or latency breakdown, no exception types or stack traces, no per-task error distribution, and no visibility into whether the remaining 15 5xx responses are the same route. Treat all of those as unknown, not healthy.

Suggested checks

• 1.Pull the full AllExceptionsFilter log lines for 07:28-07:33 with stack traces (CloudWatch Logs Insights on the ECS log group) — get the exception class and message for GET /api/me/applications. This is the single fastest step; the provided sample is truncated to the route line only.
• 2.Group all 20 5xx responses in the window by route and by ECS task ID, to confirm whether this is one buggy endpoint or several, and whether failures are isolated to a single task (suggesting a bad deploy/partial rollout or one poisoned instance).
• 3.Identify which user(s) hit /api/me/applications and failed — check whether it is a small set of accounts. If so, inspect their application rows for the data shape that breaks the handler (null stage, orphaned job reference, missing relation) rather than chasing infrastructure.
• 4.Reproduce directly: authenticated GET /api/me/applications for an affected candidate, and GET /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility?email=saud72119@gmail.com. The sponsor-eligibility one needs no auth and is reproducible right now.
• 5.Check deploy/migration history against 07:28 — was a task definition rolled out or a schema migration applied shortly before the first observed error? A column rename or nullable change would produce exactly this pattern (idle infra, one route throwing).
• 6.Look at RDS slow query logs / pg_stat_statements for the window to explain the 464.7 ms mean API latency and to see if the applications query is timing out at the ORM layer (which would surface as a 500 while RDS-side metrics still look fine).
• 7.Confirm whether applicationsSubmitted=0 is normal for this hour by comparing against the same window on prior days — if candidates normally submit here, the sponsor-eligibility failure may be blocking the application funnel and this is more urgent than the 7.1% figure suggests.
• 8.Verify error handling in the affected handler: if it swallows a DB/driver error into a generic 500, add the exception type and route-level structured logging so the next occurrence is diagnosable without this guesswork.

Suggestions only — this system has no ability to act on them.

What this likely means

Infrastructure is healthy: ECS 3/3 tasks at 3% CPU / 19% memory, RDS at 5% CPU with 9 connections and sub-2ms read/write latency, 3 healthy hosts on both target groups, and a direct API probe returned 200 in 28ms. So this is not capacity, deploy-wide failure, or database exhaustion — it is a specific endpoint throwing. Four of the five captured failure reasons are GET /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility?email=... returning 500, all against the same job ID with different candidate emails, and one is GET /api/me/applications -> 500. The most likely reading is a handler-level bug on the sponsor-eligibility route — something about that job record or the eligibility logic (e.g. missing/malformed sponsorship field, null deref, or an unhandled lookup for candidates with no existing profile) — reproducing every time a candidate hits that job's page. Note the log lines only show five distinct reasons while the window counted 8 5xx, so 3 more errors are unattributed; assume they are more of the same until confirmed rather than absent. User impact is real but narrow: 20 talents were active and 33 registrations succeeded with zero failures, and zero registration/application/email failures are recorded, so signup and mail are working. However applicationsSubmitted is 0 for this window — that is consistent with candidates being blocked at the eligibility check before they can apply, but with no baseline for a 5-minute window it could equally be normal low volume; do not treat it as confirmed impact. Aggregate API latency of 246ms vs 4.9ms on the frontend is elevated but not alarming, and is plausibly inflated by the failing requests themselves. Overall: a localised endpoint defect degrading the apply funnel for one job, not a platform outage.

Suggested checks

• 1.Pull the full stack trace for one sponsor-eligibility 500 from the ECS task logs (AllExceptionsFilter logs the path but the exception body is what identifies the bug) — this alone will likely name the cause.
• 2.Curl /api/public/jobs/cmtvbo4hj001mw001u4prr8tq/sponsor-eligibility with one of the failing emails and again with a known-good email, to determine whether the failure is job-specific or email/candidate-specific.
• 3.Inspect the job row cmtvbo4hj001mw001u4prr8tq in RDS for null or malformed sponsorship/eligibility fields, and compare against a job whose eligibility checks are succeeding.
• 4.Account for the 3 unexplained 5xx: query ALB/access logs for all non-2xx in the 07:21–07:26 window grouped by path, to confirm whether the blast radius is only these two routes.
• 5.Check whether /api/me/applications -> 500 shares the same code path or DB join as sponsor-eligibility (both touch candidate↔job relations) — a shared query or serializer fault would explain both.
• 6.Correlate against the most recent deploy/migration timestamp: if the 500s start at a deploy boundary, roll-back is the fastest mitigation; if they predate it, it is data-triggered.
• 7.Confirm whether applications are genuinely blocked by comparing applicationsSubmitted over prior 5-minute windows today — this distinguishes funnel breakage from normal low traffic and sets the urgency.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 2 of 25 API requests in the window returned 5xx (8%), and both logged failures are the same endpoint — GET /api/assessments -> 500, at 07:10:37 and 07:10:38, caught by AllExceptionsFilter. Nothing else looks stressed: API health check returns 200 in 36ms, avg API latency 18.2ms, 3/3 ECS tasks running and healthy, ECS CPU 1.9% / memory 17.6%, RDS CPU 5.2% with 9 connections, 1046MB freeable, sub-1ms write latency and 0ms read latency. Web tier has zero 5xx. No 4xx at all, no email failures, no collection errors. Most likely reading: this is a localised application-level bug on the assessments read path, not a capacity, database, or deployment-wide outage. The two errors are one second apart and probably the same actor retrying, which matches the activity data (1 staff user, 0 talents in the 5-minute window). Because the denominator is only 25 requests, a single failing endpoint hit twice is enough to push the percentage over a 5% threshold — the rate is statistically fragile at this volume, so treat 8% as 'two errors', not 'a 1-in-12 failure rate'. User impact is likely narrow: a staff user viewing the assessments list is seeing a failure; candidate-facing flows show no evidence of harm. But note the limits — registrations, applications, and talent activity are all zero in this window, so those paths were not exercised and their health is unknown, not confirmed good. The log lines carry no stack trace, exception type, or status code field, so the underlying cause (null data, bad query, missing relation, unhandled edge case in a specific assessment record) cannot be determined from this payload alone.

Suggested checks

• 1.Pull the full ECS/CloudWatch log entries around 07:10:37–07:10:38 for the AllExceptionsFilter events — you need the stack trace and exception class, which are not in this payload. That alone likely identifies the fault.
• 2.Check whether both 500s came from the same ECS task or different ones. Same task suggests a task-local issue (bad state, stale connection); spread across tasks suggests a genuine code or data bug.
• 3.Reproduce: call GET /api/assessments yourself with the same auth context/query params the staff user used. If it fails consistently, it's deterministic; if it succeeds, suspect a data-dependent or transient case.
• 4.Identify the caller — correlate ALB access logs on /api/assessments at 07:10:37–38 to get the user, query string, and whether it was a retry of one action.
• 5.Inspect the assessments query path for unguarded joins or nullable relations (e.g. an assessment referencing a deleted candidate/stage). Data-shape bugs match the clean infra metrics.
• 6.Confirm whether a deploy or migration landed shortly before 07:10 — nothing in this payload rules that out, and a recent change is the cheapest thing to eliminate.
• 7.Widen the query window to 30–60 minutes and check whether /api/assessments 500s recur or were a one-off pair. At 25 requests the 8% figure is noisy; the trend matters more than this sample.
• 8.Separately verify the candidate-facing paths (registration, application submit) actually work — they recorded zero events here, so they are untested in this window, not proven healthy.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 2 of 25 API requests in the window returned 5xx (8%), tripping a threshold on a very small denominator — at this traffic level a single extra error moves the rate by 4 points, so the percentage is statistically noisy. The recent error log lines name two specific read endpoints: GET /api/analytics/dashboard (bu=all and bu=BU%20ARA, at 06:15:56-58) and GET /api/assessments (06:15:58 and 06:18:07). All are unhandled exceptions caught by AllExceptionsFilter. Infrastructure looks healthy: API probe returned 200 in 30ms, ECS 3/3 tasks running with CPU ~1.8% and memory ~17%, RDS CPU 4.3% with 9 connections and 1042MB freeable, 3 healthy hosts on both target groups, average API latency 9.2ms, zero 4xx on the API, and no registration, application, or email failures recorded. So this is not capacity, not a crashed task, and not a broad outage — it is consistent with a localised bug or data-dependent failure on two specific staff-facing read endpoints. One caveat worth checking rather than dismissing: RDS writeLatencyMs is 40.7 while readLatencyMs is 0, which is unusually asymmetric; it may just be near-zero write volume producing a noisy average, but it is the only infrastructure number out of character. User impact: likely narrow. Activity in the window was 1 staff user and 0 talents, so the failing dashboard and assessments calls are almost certainly that one staff member's console retrying — probably seeing a broken or empty analytics dashboard and assessments list. Candidate-facing flows show no evidence of harm: zero registration failures, zero application failures, zero email failures, and no 5xx on the web tier. Note these workflow counters are all zero partly because no candidates were active, so 'zero failures' here means 'no traffic to fail', not 'verified working'. Nothing confirms the candidate path is healthy — it is simply untested in this window. Also note the log sample shows only 5 recent failure reasons and may be truncated; error counts before 06:15 are unknown from this payload. I have not changed anything and cannot; the following are checks for you to run.

Suggested checks

• 1.Pull the full stack trace for GET /api/assessments and GET /api/analytics/dashboard from CloudWatch Logs around 06:15:56-06:18:07 — AllExceptionsFilter logs the route but the exception class and trace will name the cause directly (null deref, query error, serialisation failure).
• 2.Reproduce both endpoints by hand: GET /api/assessments and GET /api/analytics/dashboard?bu=all and ?bu=BU%20ARA with a valid staff token. Confirm whether the failure is persistent or was a transient burst that has already stopped — the errors span 06:15:56 to 06:18:07 with none after, which may mean it self-resolved or that the operator stopped retrying.
• 3.Compare bu=all vs bu=BU%20ARA — if only one business unit fails, it is data-dependent (bad or missing row, division by zero, a BU with no records) rather than a code-wide break. The URL-encoded space in 'BU ARA' is also worth checking as a parameter-parsing suspect.
• 4.Check whether a deploy, migration, or config change landed shortly before 06:15:56 — a fresh task definition revision or schema change is the most common cause of two previously-working read endpoints failing simultaneously.
• 5.Query the Postgres logs for errors or statement failures in the 06:15-06:19 window, and check whether the analytics and assessments queries share a table, view, or join. A single missing column or broken view would explain both endpoints failing together.
• 6.Verify whether the 40.7ms RDS write latency against 0ms read latency is normal for this instance by comparing against the last 24h baseline — probably an artefact of near-zero write volume, but rule out storage-level slowness before dismissing it.
• 7.Confirm the failures are confined to these two routes: grep the ALB or application logs for any other 5xx paths in the last hour. The five log lines shown may be a truncated sample, so absence of other routes in this payload is not proof they are clean.
• 8.Synthetically exercise one candidate registration and one job application in a test context. The zero-failure workflow counters reflect zero candidate traffic, not verified health, so the candidate path is currently unproven.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 2 of 25 API requests in the window returned 5xx (8%), and both recorded failure reasons are the same endpoint — GET /api/assessments -> 500, logged at 06:06:50 and 06:06:51, about 35 seconds apart and ~35s before detection. Everything else looks healthy: API probe returned 200 in 23ms, ECS 3/3 tasks running at 1.8% CPU and 17% memory, RDS at 5% CPU with 9 connections, 1045MB freeable and sub-ms read/write latency, 3 healthy hosts on both target groups, zero 4xx, and the web tier shows no 5xx at all. No registrations or applications failed, and no emails failed — though with 0 talents and only 1 staff user active in the 5-minute window, those counters being zero reflects near-zero traffic, not proven health. Most likely interpretation: this is an endpoint-specific application bug or unhandled exception in the assessments route, not an infrastructure or capacity problem. There is no resource pressure, no database stress, and no host removed from the load balancer that would explain a systemic failure. The 8% figure is also an artifact of a tiny denominator — 2 errors out of 25 requests during an almost-idle window crosses a percentage threshold that would not have fired under normal traffic. The alert is likely real but low-severity in blast radius. User impact: likely confined to whoever was hitting the assessments endpoint — plausibly the single active staff user, or a frontend page polling that route. Since talents was 0, it is unlikely a candidate was affected. I cannot confirm from this data whether the two 500s came from the same caller or the same underlying cause; the log lines do not include a request ID, user, or stack trace. Whether the failure is intermittent or would reproduce on every call to that endpoint is unknown from this snapshot.

Suggested checks

• 1.Pull the full stack trace for the two AllExceptionsFilter entries at 06:06:50 and 06:06:51 from CloudWatch Logs — the reason strings here are truncated to the summary line. The exception class and originating file will usually identify the cause immediately.
• 2.Call GET /api/assessments yourself with a valid token to see whether it reproduces deterministically or was transient. This single check separates 'persistent broken endpoint' from 'one-off blip'.
• 3.Check whether both 500s came from the same ECS task or different ones (log stream name per task). Same task points to a task-local issue such as a stale connection or bad in-memory state; different tasks points to a code or data problem.
• 4.Identify the caller — was it a Next.js page/component polling the endpoint, or a direct API call? Correlate against the 1 active staff user's session to confirm blast radius.
• 5.Look at the query the assessments handler runs: check for a null-dereference on optional relations or a missing record, which is the common cause of a 500 on a read endpoint when the DB itself is healthy (RDS metrics here are clean).
• 6.Review recent deploys or migrations to the API preceding 06:06:50 — a schema change that left an assessments-related column or relation inconsistent would fit the endpoint-specific pattern.
• 7.Widen the metrics window to the last few hours for this endpoint. With only 25 requests in the window, confirm whether /api/assessments has been failing at a steady low rate or whether these two errors are genuinely new.
• 8.Consider tuning the alert to require a minimum request count (or absolute error count) alongside the 5% rate, so near-idle windows stop triggering on 2 errors.

Suggestions only — this system has no ability to act on them.

What this likely means

All 5xx errors in the window trace to a single endpoint: GET /api/assessments, which threw unhandled exceptions caught by AllExceptionsFilter at least 7 times between 05:48:57 and 05:49:17. This is an application-level bug or data-dependent failure on one route, not an infrastructure problem. Corroboration: ECS is healthy (3/3 tasks, CPU 1.9%, memory 17%), RDS is idle and fast (CPU 4%, 9 connections, write latency 0.4ms, read latency 0ms), the ALB reports 3 healthy hosts on both target groups, the API health probe returns 200 in 30ms, and average API latency is 33ms - so the failures are fast rejections, not timeouts or resource exhaustion. Note the denominator is tiny: 39 requests in the window, so 4 errors trips a 5% threshold. The percentage is statistically noisy, but the repeated identical stack signature is real and worth investigating. User impact is likely narrow: activity shows 3 staff and 0 candidates active, so the errors are most likely hitting internal/staff users loading an assessments view rather than candidates. Candidate-facing workflows show no failures (0 registrations failed, 0 applications failed, 0 emails failed) - though note that registrations and applications submitted were also 0, so those paths were simply not exercised and their health is unverified, not proven good. The error log lines have no status code field populated and no stack trace or exception message in the captured data, so the actual cause (null dereference, bad query, missing relation, serialisation error) is not determinable from this payload.

Suggested checks

• 1.Pull the full NestJS stack trace for the AllExceptionsFilter entries around 05:48:57-05:49:17 UTC in CloudWatch Logs - the exception class and top frame will usually identify the cause immediately. The monitoring payload truncated everything after the '-> 500'.
• 2.Call GET /api/assessments yourself with a staff token to see if it reproduces consistently or only for certain callers. Consistent failure points to a code/schema bug; intermittent points to data-specific rows or a race.
• 3.Check whether the request had query params or a specific assessment/candidate id - an id-specific 500 suggests a bad or orphaned row (e.g. an assessment referencing a deleted job or candidate) rather than a broken endpoint.
• 4.Confirm whether errors came from all 3 ECS tasks or one. If isolated to a single task, suspect a partial/stale deploy or a task with a bad connection; if spread across all three, it is code or data.
• 5.Check recent deploy and migration history for the assessments module - if a deploy or schema migration landed shortly before 05:48, that is the prime suspect and a rollback is the fastest mitigation.
• 6.Verify the errors are still occurring. The last logged failure is 05:49:17 and detection was at 05:50:27 - roughly a minute of silence. This may already be a self-limited burst from a few retries by one user session.
• 7.Check whether the assessment stage of the pipeline is blocked for any candidate - if the endpoint backs a staff screen used to advance candidates from assessment to interview, stalled transitions also mean the corresponding transactional emails never fire.
• 8.Look at the 1 x 4xx on the API and 3 x 4xx on the web tier to confirm they are unrelated routine noise (auth, 404s) and not a second symptom of the same fault.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: 4 of 34 API requests in the window returned 5xx (11.8%), and all four map to just two staff-facing read endpoints — GET /api/analytics/dashboard?bu=all (2 errors, 05:38:46 and 05:38:47) and GET /api/assessments (2 errors, 05:38:48 and 05:38:49). They occurred in a 4-second burst about 45 seconds before detection. Everything else is healthy: API reachable (200 in 36ms), 3/3 ECS tasks running at ~2% CPU and 17% memory, 3 healthy hosts on both target groups, RDS at 4% CPU with 9 connections and 1044MB freeable, no 4xx, no web-tier 5xx, and average API latency of 9.8ms. Only 1 staff user and 0 candidates were active in the 5-minute window. Most likely interpretation: this is an endpoint-specific application fault (unhandled exception in the analytics dashboard and assessments handlers), not a capacity, infrastructure, or dependency-saturation problem. The AllExceptionsFilter caught the exceptions, so the stack traces exist in the logs but are not in this payload — the actual exception type and message are unknown from this data. The 11.8% rate is inflated by a tiny denominator: 34 requests total, so 4 failures crosses a 5% threshold that would need ~2 failures at this volume. Treat the percentage as low-confidence; the repeated failures on the same two routes are the real signal. User impact: narrow but real. The single active staff user likely saw a broken analytics dashboard and a failing assessments list, probably retrying (hence the duplicate calls one second apart). No candidates were active, and registrations, applications, and email sends all show zero failures — but they also show zero attempts, so those paths are unexercised rather than proven healthy. Candidate-facing signup/apply flows are untested by this data, not confirmed working. One metric worth noting: RDS writeLatencyMs is 45.4 against readLatencyMs of 0. That is elevated in absolute terms, but both failing endpoints are reads, and read latency is flat, so this is probably unrelated background write activity rather than the cause. Worth a glance, not a priority.

Suggested checks

• 1.Pull the full stack traces from the ECS task logs for the four AllExceptionsFilter entries at 05:38:46-05:38:49 — the exception class and message will most likely identify the cause immediately. This is by far the fastest path.
• 2.Check whether all four errors came from the same ECS task or were spread across the three tasks. A single bad task points to a per-instance issue (stale connection, bad config, memory state); spread across all three points to a code or data defect.
• 3.Manually re-request GET /api/analytics/dashboard?bu=all and GET /api/assessments to see whether the failure is persistent or a transient blip already resolved. The API is currently returning 200 on its health path, so the fault may be intermittent.
• 4.Inspect what the analytics/dashboard and assessments handlers have in common — a shared service, repository, DTO, or query helper is a likely single root cause given two unrelated routes failed within four seconds.
• 5.Look for a recent deploy, migration, or config change to the API immediately before 05:38. A sudden onset on previously working endpoints is consistent with a recent change.
• 6.If the traces point at the database, check for a schema/migration mismatch or a query error (missing column, null constraint, failed join) — RDS resource metrics are healthy, so saturation is ruled out but query correctness is not.
• 7.Check whether bu=all specifically triggers the failure by testing the dashboard with a specific business unit value. The 'all' aggregation path may hit a data condition the scoped path does not.
• 8.Verify candidate-facing flows (registration, application submit, stage-transition email) with a synthetic transaction. Those counters are zero because nobody attempted them, not because they were verified healthy.
• 9.Secondary: review RDS write latency of 45.4ms against its recent baseline. Likely unrelated to these read failures, but confirm it is not the start of a separate storage issue.
• 10.Consider whether the 5% error-rate threshold is appropriate at 34-request volumes; adding a minimum-request-count or absolute-error-count condition would reduce noise from low-traffic windows.

Suggestions only — this system has no ability to act on them.

What this likely means

Four 5xx responses out of 55 requests tripped the 5% threshold, but the absolute volume is tiny and the rest of the platform looks healthy: the API health check returns 200 in 36ms, average API latency is 10.1ms, all 3 ECS tasks are running with ~1.8% CPU and ~17% memory, RDS is at 4% CPU with 9 connections and sub-millisecond write latency, and the web tier shows zero 5xx. There is no sign of a saturation or dependency-capacity problem. The error log lines are specific and clustered: two failures on GET /api/analytics/dashboard?bu=all (05:31:17, 05:31:18) and two on GET /api/assessments (05:31:20, 05:31:21) — four consecutive seconds, roughly 80 seconds before detection. All were caught by the NestJS AllExceptionsFilter, which is consistent with an unhandled application exception on those two read endpoints rather than infrastructure failure. Activity shows 1 staff user and 0 talents in the 5-minute window, so this is most likely a single staff member loading the dashboard and assessments views (possibly with retries or a component re-render) and hitting an error on both. Because traffic is so low, one user's failed page load is enough to push the percentage over threshold — the rate is statistically fragile at n=55. Candidate-facing impact is likely minimal right now: zero talents were active, registrations and applications show zero attempts (not failures), and emailsFailed is 0. But note that zero attempts means those workflows are untested in this window, not proven healthy. The affected endpoints are staff-facing analytics and assessments, so the practical impact is a staff user seeing a broken dashboard/assessments view. The underlying defect — whatever exception those two handlers throw — would affect any user hitting the same code path, so it should not be dismissed as noise just because the count is low. The exception type and stack trace are not in this payload, so the actual cause is unknown from the data available here.

Suggested checks

• 1.Pull the full stack traces from the ECS/CloudWatch logs around 05:31:17-05:31:21 for the AllExceptionsFilter entries — the exception class and the throwing line will identify the cause faster than anything else here. The monitoring payload only carries the one-line summary.
• 2.Reproduce directly: call GET /api/analytics/dashboard?bu=all and GET /api/assessments with a staff token and see whether they still 500 or whether this was transient. This immediately separates an ongoing bug from a momentary blip.
• 3.Check whether both failures share a dependency — do the dashboard and assessments handlers both query the same table, view, or aggregate? Two unrelated endpoints failing within 4 seconds suggests a common cause (a shared query, a null in shared data, a migration mid-flight).
• 4.Look for a recent deploy or migration: compare the ECS task definition revision / image tag start time against 05:31. A new revision shipped shortly before would make a code regression the leading hypothesis.
• 5.Inspect the bu=all parameter path specifically — 'all' aggregation branches are a common source of unhandled nulls or empty-result errors when a business unit has no data.
• 6.Check PostgreSQL logs for query errors or statement timeouts at 05:31. RDS metrics look clean, but a failing query (bad SQL, missing column, constraint) would not show up as CPU or latency pressure.
• 7.Confirm whether all 4 errors came from one ECS task or were spread across the 3 tasks. A single-task pattern points to one bad container or stale state; spread across all points to a code-level defect.
• 8.Verify candidate-facing paths are actually working — registration and application submission recorded zero attempts, so they are unverified rather than confirmed healthy. A synthetic test application would settle whether the blast radius reaches candidates.
• 9.Consider the alert threshold itself: at 55 requests, 3 errors is 5.5%. If this endpoint pair keeps producing occasional errors, decide whether to add a minimum-request-count condition so low-traffic windows do not fire on single-user incidents.

Suggestions only — this system has no ability to act on them.

What this likely means

What the data shows: RDS write latency is 112ms (threshold 100), read latency 0ms. Everything else is idle and healthy — RDS CPU 4.6%, 9 connections, 1062MB freeable memory; ECS 3/3 tasks at 2% CPU and 17% memory; API responding 200 in 34ms; only 4 API requests in the window with 0 4xx/5xx and 3 healthy hosts; zero staff and zero talent activity in the last 5 minutes; no registrations, applications, or email failures recorded. Web traffic response time is reported unavailable(no_data), so frontend latency is unknown, not healthy — though 0 web requests is consistent with genuine idleness. Most likely interpretation: this is a low-volume measurement artefact rather than a user-facing problem. RDS latency metrics are per-operation averages; with almost no write traffic, a handful of writes (an autovacuum, a checkpoint flush, a WAL sync, an automated backup snapshot, or a burst-balance-limited EBS I/O) can dominate the average and push it over a fixed 100ms threshold. Read latency of exactly 0ms corroborates that the volume is tiny — the sample is too small to be statistically meaningful. The near-zero activity counts mean essentially no user is currently experiencing this. User impact right now: likely none observable. No candidate registrations, applications, or stage-transition emails failed or were attempted in this window, and the API is fast. The risk worth ruling out is that this is the leading edge of a real storage problem (depleted gp2/gp3 burst credits, a failing volume, or a backup window overlapping traffic) that would bite once real load returns — that cannot be distinguished from the current snapshot alone.

Suggested checks

• 1.Pull RDS WriteIOPS / WriteThroughput alongside WriteLatency for the same window in CloudWatch — if IOPS is near zero, the 112ms is an average over a few operations and is almost certainly noise.
• 2.Check whether an automated backup, snapshot, or maintenance window overlapped 12:49 UTC; also check RDS events for storage or maintenance activity.
• 3.Check EBS burst balance / BurstBalance and DiskQueueDepth on the RDS instance — exhausted burst credits are the main way this becomes a real problem under load.
• 4.Look at the WriteLatency trend over the past 24–48 hours: a recurring spike at idle periods points to autovacuum/checkpoint/backup; a rising baseline points to a genuine storage issue.
• 5.Confirm whether this alert has fired before during low-traffic windows — if so, the threshold likely needs a minimum-traffic guard (e.g. suppress when WriteIOPS or API requests are below a floor) rather than investigation.
• 6.Verify the frontend is actually reachable, since web response time came back unavailable(no_data) — 0 requests is consistent with idle, but the metric gap means frontend health is unknown here.
• 7.If concerned about real impact, run a synthetic write path (test registration or application submission in a non-production-affecting way) and time it end-to-end to confirm writes complete normally.

Suggestions only — this system has no ability to act on them.

What this likely means

A write-latency threshold breach (118.4ms vs 100ms) on RDS, with essentially no load anywhere else in the system. Corroboration: RDS CPU 4.4%, 9 connections, 1060MB freeable memory, read latency 0ms; ECS at 2% CPU / 17% memory with 3/3 tasks running; only 4 API requests in the window with a 14.9ms average and zero 4xx/5xx; health check returned 200 in 40ms; zero staff and zero talent activity in the last 5 minutes; no registrations, applications, or email failures recorded. No user impact is visible in this snapshot — nobody was on the platform to be affected. The most likely reading is a measurement artifact rather than a storage fault: at near-zero write volume, RDS WriteLatency is an average over a handful of I/O operations, so one slow or cold-cache write (checkpoint flush, autovacuum, log rotation, EBS burst-credit or gp2/gp3 baseline behaviour, or a background snapshot) dominates the average and inflates it. Read latency of exactly 0ms is consistent with 'no read IOPS in the interval' rather than 'reads are instant', which supports the low-sample interpretation. That said, this cannot be ruled out as an early sign of real storage degradation — write IOPS, queue depth, burst balance, and disk throughput are not present in this payload, so their state is unknown and must not be assumed healthy. The key discriminator is whether this latency persists or repeats once real write traffic arrives; a single breach during an idle window is weak evidence of a problem, whereas the same latency under load would be a genuine incident affecting registration, application submission, and stage-transition writes.

Suggested checks

• 1.In CloudWatch, overlay WriteLatency with WriteIOPS and WriteThroughput for the same 5-minute window — if WriteIOPS is in the single digits, the latency average is statistically meaningless and this is almost certainly an artifact.
• 2.Check whether this is a one-off or a pattern: pull WriteLatency for the last 24-48 hours and see if breaches cluster during idle periods (artifact) or during traffic peaks (real).
• 3.Check DiskQueueDepth and, for gp2, BurstBalance / for gp3, whether provisioned IOPS are being approached — none of these are in the alert payload, so their state is currently unknown.
• 4.Confirm no RDS maintenance, automated backup/snapshot, or storage modification was in progress at 10:34 UTC (RDS Events console).
• 5.Query pg_stat_activity and pg_stat_progress_vacuum for a long-running autovacuum, checkpoint, or WAL-heavy background job at that timestamp.
• 6.Check checkpoint behaviour: pg_stat_bgwriter for checkpoints_req vs checkpoints_timed and buffers_checkpoint — frequent forced checkpoints produce write-latency spikes.
• 7.Once real traffic resumes, re-check write latency under load; that is the decisive test for whether this affects candidate registrations, application submissions, and stage-transition writes.
• 8.Consider tuning the alert to suppress or require a minimum write IOPS/sample count before firing, so idle-window averages do not generate noise.

Suggestions only — this system has no ability to act on them.

What this likely means

A write-latency threshold breach (101.4ms vs 100ms) on the RDS instance during what appears to be an idle period. Almost certainly no current user impact, and likely a metric artefact rather than a real storage problem. Corroborating evidence: RDS CPU is 4%, only 9 connections, 1053MB freeable memory, and read latency is 0ms. The API is reachable (200, 37ms), ALB-observed API latency is 13.4ms across 3 healthy hosts, no 4xx/5xx, and ECS is at 2% CPU / 17% memory with 3/3 tasks running. Activity shows 0 staff and 0 talents in the last 5 minutes, 0 registrations, 0 applications, 0 email failures, and only 4 API requests in the window (likely health checks). Web traffic is 0 requests with response time unavailable (no_data) — unknown, not healthy. The most likely mechanism: RDS read/write latency is reported as total latency divided by number of IOPS, so at near-zero write volume a handful of slow or large I/O operations (e.g. a checkpoint flush, autovacuum, a WAL/backup write, or an EBS burst-credit hiccup) produces a very high per-operation average. This is a classic low-denominator artefact. A read latency of exactly 0ms alongside 101ms writes is consistent with that reading — essentially no read I/O happened, so nothing is corroborating a genuine volume-wide storage slowdown. Less likely but worth ruling out: real EBS degradation, exhausted gp2 burst balance or gp3 IOPS/throughput limit, or an automated backup/snapshot window overlapping this timestamp. Those would show sustained elevation and would matter once real write traffic returns. Note that disk queue depth, EBS burst balance, and write IOPS/throughput are not present in this snapshot — treat them as unknown. For users: no observable degradation right now. There is no evidence of failed registrations, applications, or stage-transition emails. The genuine risk is that this is an early signal of storage constraint that would surface as slow application submissions and stage transitions under load.

Suggested checks

• 1.Pull WriteLatency, WriteIOPS, and WriteThroughput for this RDS instance over the last 6–24 hours in CloudWatch. If the IOPS count at 07:54Z was in the single digits, this is an averaging artefact and can be deprioritised. Sustained high latency with meaningful IOPS is a real problem.
• 2.Check whether an automated backup, snapshot, or maintenance window overlapped 07:54Z UTC (RDS events, Maintenance & Backups tab). Backup-induced write spikes on a quiet instance are a common cause of this exact pattern.
• 3.Check DiskQueueDepth for the same period. Near-zero queue depth with high reported latency strongly supports the low-volume-average explanation; elevated queue depth points to genuine I/O saturation.
• 4.Check the storage type and EBS burst balance: if gp2, look at BurstBalance; if gp3, compare against provisioned IOPS/throughput. A depleting burst balance is a real, escalating problem even if today's trigger was noise.
• 5.Correlate against the alert history — how many times has this fired, and always during low-traffic windows? A repeating idle-hours-only pattern confirms the threshold is mistuned for this workload.
• 6.Once real traffic resumes (check activity.talents and apiTraffic.requests), re-read WriteLatency under load. That is the measurement that actually tells you whether candidate-facing writes are slow.
• 7.Look at pg_stat_activity and RDS Performance Insights around 07:54Z for long-running writes, autovacuum, or checkpoint activity that could account for a burst of expensive I/O.
• 8.Consider changing the rule to require a minimum write IOPS floor or a sustained duration (e.g. 3 consecutive breaches) before firing, so idle-period averaging noise does not page anyone.

Suggestions only — this system has no ability to act on them.

What this likely means

RDS write latency is 123.5ms against a 100ms threshold, but every corroborating signal says the platform is idle and healthy. Traffic is 4 API requests in the window, 0 web requests, 0 staff and 0 talents active in the last 5 minutes, 9 DB connections, RDS CPU at 4%, ECS CPU 2% / memory 17%, all 3 tasks running and healthy, API responding 200 in 31ms (ALB-observed 12.1ms), no 4xx/5xx, and no registration, application or email failures. Read latency is reported as 0ms, which is what you'd expect when there are essentially no read IOs in the interval rather than evidence of fast reads. The most likely reading: with almost no write IO happening, the per-operation average is computed over a tiny denominator, so one slow or unusual write (a checkpoint flush, autovacuum, a WAL/backup-related IO, or an EBS burst-credit / volume-level latency blip) dominates the average and produces a misleadingly large number. This is a classic low-traffic metric artefact, not a user-visible incident — no user was on the platform to be affected, and no workflow failed. The alternative, which the data cannot rule out, is genuine storage-layer degradation (EBS gp2/gp3 throughput or burst exhaustion, an ongoing snapshot/backup, or a maintenance operation) that simply has not been exercised by real load yet; that would become visible as soon as write traffic returns. Note that write IOPS/throughput, queue depth, burst balance, and any maintenance/backup schedule are not present in this payload, so they are unknown, not healthy.

Suggested checks

• 1.In CloudWatch, plot RDS WriteLatency alongside WriteIOPS and WriteThroughput over the last few hours — if WriteIOPS is near zero at the spike, this is an averaging artefact from a near-empty denominator and needs no action beyond tuning the rule.
• 2.Check DiskQueueDepth and, for gp2 or burstable storage, BurstBalance / EBSIOBalance% and EBSByteBalance% over the same window — a draining balance would indicate real storage exhaustion rather than a blip.
• 3.Check whether an automated backup, snapshot, or maintenance window overlapped 07:39 UTC on 2026-09-18 (RDS events log) — backup IO commonly inflates write latency during idle periods.
• 4.Look at whether the spike is a single isolated datapoint or a sustained trend; compare against the same hour on previous days to see if it is a recurring nightly/low-traffic pattern.
• 5.Correlate with Postgres-side activity: pg_stat_bgwriter checkpoint counts and autovacuum/analyze runs around that timestamp — a checkpoint flush on an otherwise idle instance fits this signature.
• 6.Re-check the same metrics during a period of real user traffic; if write latency stays under threshold under load, the alert is a low-traffic false positive.
• 7.If it is confirmed as an artefact, consider gating this rule on a minimum write IOPS or minimum request volume, and/or evaluating over a longer window, so idle periods don't page.

Suggestions only — this system has no ability to act on them.

What this likely means

A single RDS write-latency sample of 107.5ms crossed the 100ms threshold, but essentially nothing else on the platform is under stress, and there is almost no load to be affected. Corroboration: RDS CPU 4.2%, only 9 connections, 1044MB freeable memory, read latency 0ms; ECS at 1.9% CPU / 15% memory with 3/3 tasks running; API healthy (200, 45ms probe, 19.2ms average over just 4 requests, zero 4xx/5xx); 3 healthy hosts on both target groups. Activity in the last 5 minutes is zero staff and zero talents, zero registrations, zero applications, zero email failures. Read latency of exactly 0ms alongside a non-zero write latency is typical of a near-idle database: the metric is an average per-I/O figure, so with only a handful of writes (likely just autovacuum, WAL/checkpoint flushes, or a log/backup task) one slow flush dominates the average. This is most likely benign low-sample-count noise rather than a storage problem, and no user is currently experiencing anything — web traffic is zero and the frontend response time is unavailable (no_data), so frontend latency is unknown, not healthy. The residual risk worth ruling out: if this is real storage degradation (EBS burst-balance exhaustion, gp2 IOPS starvation, or an underlying volume issue), it would stay invisible while idle and then surface as slow writes on registrations, application submissions, and stage transitions once traffic returns. Note also that disk free space, IOPS/throughput, burst balance, queue depth, and any background job or email-queue health are not present in this snapshot and should be treated as unknown.

Suggested checks

• 1.Pull the RDS WriteLatency graph in CloudWatch over the last 24–48h at 1-minute resolution alongside WriteIOPS/WriteThroughput. If the 107.5ms spike coincides with a near-zero write count, it is a low-sample artefact; if latency is elevated across periods with real write volume, treat it as genuine.
• 2.Check EBSByteBalance% and EBSIOBalance% (or BurstBalance for gp2) on the RDS instance. A draining or exhausted balance is the most common cause of real write-latency degradation and is not in this snapshot.
• 3.Check DiskQueueDepth for the same timestamp. A queue depth near zero with high latency points to a single slow I/O, not saturation.
• 4.Check FreeStorageSpace and recent storage growth — a volume approaching full can degrade write performance and is unknown here.
• 5.Check whether an automated backup/snapshot, maintenance window, or major-version/parameter change ran at ~14:49:55Z; snapshot I/O commonly produces isolated write-latency spikes on idle instances.
• 6.In Postgres, check for checkpoint and autovacuum activity around that time (pg_stat_bgwriter, log_checkpoints output, pg_stat_progress_vacuum). A checkpoint flush on an idle DB explains a one-off write spike with zero reads.
• 7.Confirm the alert has not re-fired since. If it was a single sample, consider requiring 2–3 consecutive breaches and a minimum write-IOPS floor before alerting, to suppress idle-window noise.
• 8.Once real traffic returns, re-check write latency and watch registration/application submission success rates and stage-transition email sends — those are the user-facing paths that would degrade first if the storage issue is real.
• 9.Separately, confirm why webTraffic requests are 0 and frontend response time is no_data. Expected for an off-hours idle window, but verify against the normal traffic pattern for this time of day so a genuine frontend outage is not being masked by the DB alert.

Suggestions only — this system has no ability to act on them.

What this likely means

Write latency of 178ms on the RDS volume is the only abnormal signal; everything around it looks idle and healthy. API is up (200, 39ms), ALB p50 is 14.6ms across 5 requests in the window, no 4xx/5xx, 3/3 ECS tasks healthy with CPU at 2% and memory at 15%, RDS CPU at 4% with 9 connections and 1043MB freeable memory. Read latency is 0ms and there were zero registrations, applications, staff or talent sessions in the 5-minute window. That combination is most consistent with a low-volume measurement artefact rather than a user-facing problem: WriteLatency is an average per-write figure, so when only a handful of writes occur in the interval, a single slow or cold I/O (e.g. an EBS burst-credit stall, a checkpoint/WAL flush, an autovacuum write, a backup snapshot, or first-touch of cold gp2/gp3 blocks) dominates the average and produces a large number with no real throughput behind it. No user is currently observed to be affected — there is no traffic and no failed workflow to correlate with the latency. The risk to weigh is the alternative reading: genuine storage degradation that is only visible now because there is no load masking it, which would surface as slow stage transitions, slow application submits and email-triggering writes once traffic returns. The metrics present cannot distinguish these two on their own; disk queue depth, IOPS/throughput, burst balance and DB-side query timings are not in this payload and should be treated as unknown, not healthy.

Suggested checks

• 1.Pull RDS WriteIOPS / WriteThroughput for the same minute in CloudWatch — if write count is near zero, the 178ms average is a small-sample artefact and can be deprioritised.
• 2.Check DiskQueueDepth and, for gp2, BurstBalance / EBSIOBalance% and EBSByteBalance% over the last few hours — a draining burst balance points to real, sustained storage exhaustion rather than a blip.
• 3.Look at WriteLatency over a 24-48h window rather than one datapoint: decide whether this is an isolated spike or a rising trend, and whether spikes align with a repeating schedule.
• 4.Correlate against the RDS automated backup/snapshot window and any maintenance window — snapshot I/O is a common cause of one-off write latency spikes on an otherwise idle instance.
• 5.Check for autovacuum/checkpoint activity around 09:04 UTC: pg_stat_activity, pg_stat_bgwriter (checkpoint_write_time, buffers_checkpoint) and Postgres logs for 'checkpoint complete' lines with long write/sync times.
• 6.Confirm no user impact by replaying a real write path once traffic resumes — time a candidate registration and a pipeline stage transition end-to-end, since both write and then trigger transactional email.
• 7.Verify the alert rule's own behaviour: whether it requires a minimum write count or multiple consecutive breaching datapoints. A single-datapoint threshold on an average-latency metric will fire repeatedly on idle instances.
• 8.If write latency stays elevated once traffic returns, check instance storage type and provisioned IOPS against actual demand, and review recent schema/index changes that could have increased write amplification.

Suggestions only — this system has no ability to act on them.

What this likely means

RDS write latency of 107.4ms crossed the 100ms threshold, but essentially every corroborating signal is idle or healthy, so this is most likely a low-traffic measurement artefact rather than a live user-facing problem. Evidence: read latency is 0ms, RDS CPU is 4.5%, only 9 connections, 1046MB freeable memory; ECS is at 1.8% CPU / 14.9% memory with 3/3 tasks running; the API answered the health probe in 34ms and the ALB saw only 5 requests at 13.9ms average with zero 4xx/5xx; zero staff and zero talents active in the last 5 minutes; zero registrations, applications, or email failures. At ~08:34 UTC with no user activity, CloudWatch's WriteLatency is an average over a very small number of write IOPS, so a single slow or cold-cache write (or a background task such as autovacuum, checkpoint, a backup snapshot, or log flush) can dominate the average and produce a large number that says little about throughput. Note what is NOT known: there is no instrumentation here for disk queue depth, write IOPS/throughput, burst balance, EBS credits, or per-query timings, so the underlying cause cannot be confirmed from this payload, and web responseMs is unavailable (no_data) rather than zero. Practical impact right now: no evidence any user was affected — no failed workflows, no errors, no latency in the API path. The genuine risk to rule out is a storage-level degradation (exhausted gp2/gp3 burst credits, an EBS volume issue, or a backup/maintenance window) that would become user-visible once traffic returns during business hours.

Suggested checks

• 1.Pull the RDS WriteIOPS / WriteThroughput metric for the same one-minute window — if writes are in the single digits, the 107.4ms average is statistically meaningless and the alert can be dismissed as a low-volume artefact.
• 2.Chart WriteLatency over the last 24-48 hours to see whether this is a recurring overnight/idle-window spike or a new and sustained rise. A pattern that only appears at low traffic strongly supports the artefact explanation.
• 3.Check DiskQueueDepth and, for gp2 volumes, BurstBalance (or gp3 provisioned IOPS headroom) — a rising queue depth or a burst balance trending toward zero would indicate a real storage constraint that will bite under load.
• 4.Confirm whether an automated backup, snapshot, or maintenance/patching window overlapped 08:34 UTC; these routinely inflate write latency on the primary.
• 5.Check whether this instance is Multi-AZ and look at the replication/standby write path, since synchronous replication to the standby is a common contributor to write-latency spikes.
• 6.Look at pg_stat_activity and pg_stat_progress_vacuum for long-running autovacuum, checkpoint, or bulk write activity around that time, plus PostgreSQL logs for checkpoint-frequency or slow-fsync warnings.
• 7.Re-run the same health snapshot during business-hours traffic. If write latency stays under threshold once real write volume arrives, the alert is confirmed as a low-traffic averaging effect.
• 8.If write latency remains elevated under real load, verify that stage-transition writes and transactional email sends are still completing by watching registrationsFailed, applicationsFailed, and emailsFailed alongside API p95 latency.
• 9.Consider tuning the alert rule to require a minimum write IOPS floor or a sustained multi-datapoint breach, so idle-window averages stop paging.

Suggestions only — this system has no ability to act on them.