CockroachDB Node Down or Cluster Unavailable: Fix

How to fix it

// try these in order, the most common fix is first

1. FIX 1Check cluster health

   Copy steps
   1. 01Open DB Console > Cluster Overview and look at node liveness and unavailable ranges
   2. 02Run cockroach node status --ranges
   Did this fix it? Yes No

2. FIX 2Bring nodes back

   Copy steps
   1. 01Restart stopped nodes, free disk space, fix out-of-memory causes
   2. 02Once a quorum returns, ranges recover automatically
   Did this fix it? Yes No

3. FIX 3Fix clock sync

   Copy steps
   1. 01Run NTP or chrony on every node
   2. 02Watch for 'clock synchronization' errors in logs
   Did this fix it? Yes No

4. FIX 4Prevent a repeat

   Copy steps
   1. 01Decommission nodes one at a time and wait for replication to finish
   2. 02Use replication factor 3+ and survival goals matching your regions
   Did this fix it? Yes No

Symptoms

Queries hang; 'result is ambiguous'; 'range unavailable'; nodes shown as suspect or dead; under-replicated or unavailable ranges.

Why it happens

• C-01Majority of replicas of a range are on dead or partitioned nodes
• C-02Disk full or node out of memory
• C-03Network partition between regions
• C-04Clock offset beyond the max (default 500ms) causing nodes to shut down
• C-05Decommissioning nodes before replicas are moved

// this guide also covers

• cockroachdb node down
• cockroachdb cluster unavailable
• cockroachdb range unavailable
• cockroachdb result is ambiguous
• cockroachdb under-replicated ranges

Other CockroachDB errors

• CockroachDB 'restart transaction' Error (40001) Fix
• CockroachDB Cluster Throttled: License Expired or No Telemetry
• Why Is CockroachDB Slow? Slow Query Fixes
• CockroachDB Connection Refused / Cannot Connect Fix
• CockroachDB High Memory, CPU or OOM Crash Fix
• CockroachDB 'Not Supported' / Postgres Compatibility Errors Fix
• CockroachDB Cloud Bill Too High: How to Cut Costs
• How to Cancel or Downgrade CockroachDB Cloud