Got rejected from 3 companies in a row on the system design round. my architectures were fine — right components, right data flow. but every time the interviewer asked "what happens when [component] goes down?" i froze.

turns out the happy path is only 30% of the system design score. the other 70% is failure handling. heres what i wasn't thinking about:

"what if your cache (redis) goes down?"
→ do you have a fallback? (read from DB, slower but alive). is the system designed to survive without cache, just degraded?

"what if payment succeeds but your server crashes before confirming the booking?"
→ idempotency keys. payment gateway returns same result on retry. reconciler job matches unconfirmed payments with pending bookings every 5 minutes.

"what if one item goes viral and 100K users hit it simultaneously?"
→ hot key problem. shard the cache by key prefix. or replicate the hot key across N read replicas. rate limit writes at the edge.

"what if kafka consumer dies mid-processing?"
→ offset not committed until processing succeeds. consumer restart replays from last committed offset. workers must be idempotent (running twice = same result).

"what if the DB is slow during peak?"
→ circuit breaker: after N timeouts, stop calling and return a degraded response. queue requests for retry. alert ops.

the mental model that finally helped: for every arrow in your architecture diagram, ask "what if this fails?" every connection between two boxes is a potential failure point. have an answer for each.

the resource that taught me to think this way: each design on systemcraft.in includes failure scenarios in their deep dives — the "Bad/Good/Great" format specifically shows what breaks before showing the fix. their BookMyShow design covers "payment succeeds but hold expired" and their Job Scheduler covers "worker dies mid-execution" with lease-based detection.

the full set of designs (28 total) with failure handling baked into every deep dive: systemcraft.in/hld

after internalizing this pattern, i stopped getting caught off-guard. now i proactively say "here's what happens when this fails" BEFORE the interviewer asks. they love that.

what failure question caught you completely off guard in an interview?