Understanding the HA stack — what happens when a container goes down

Walk through failure scenarios for each part of the stack and understand what users experience during each type of outage.

Scenario 1: An app container fails

Impact: None for users (assuming at least one other healthy app container is running). What happens: Nginx detects the failed health check within seconds and stops routing to the failed container. Traffic is redistributed to the remaining containers. The autoscaler notices a container is down and may spin up a replacement if total capacity is below threshold. User experience: No error. Some in-flight requests may need to be retried by the browser, which Nginx handles transparently.

Scenario 2: All app containers fail simultaneously

Impact: The site is down. Users see a 502 Bad Gateway or 503 Service Unavailable error from Nginx. What happens: Nginx has no healthy upstreams and returns an error. The autoscaler should detect this and restart containers. Docker Swarm's restart policy will also reschedule failed replicas automatically. Recovery: Usually within 30–60 seconds as containers restart and pass health checks.

Scenario 3: The Nginx container fails

Impact: The site is completely unreachable from the internet — all traffic enters via Nginx. What happens: Docker Swarm's restart policy kicks in immediately and reschedules the nginx replica. The stack already runs two nginx replicas (nginx-ha-1 and nginx-ha-2), so the second instance continues serving traffic while the failed one is replaced. Recovery: Usually within 10–20 seconds. For even higher availability, place the nginx replicas behind a hardware or cloud load balancer (e.g. AWS ALB or Cloudflare).

Scenario 4: The Redis master fails

Impact: Briefly degraded — users may see errors on actions that require the job queue (new transcriptions) for 10–30 seconds during failover. What happens: Redis Sentinel detects the master is down, reaches quorum, and promotes a replica to master. The app's Redis client library discovers the new master via Sentinel and reconnects. User experience: An error may appear if the user submits a transcription during the failover window. Refreshing and retrying will work once the new master is elected.

Scenario 5: The Supabase database is unavailable

Impact: Significant. Most user-facing features require the database (loading transcripts, authentication checks, billing validation). What happens: The app cannot serve most requests and will return error pages. Redis-only operations (e.g. checking if a job is in progress) may still work briefly. Recovery: Supabase runs on managed infrastructure with its own HA setup. If Supabase is unavailable, monitor their status page at status.supabase.com. There is no self-recovery action available on our side.