Solving Split-Brain Scenarios in High-Availability Distributed Clusters

Quorum arithmetic is the first defence

A cluster can only avoid split-brain if at most one partition can assemble a majority. That requires an odd number of voting members and a topology where no single network fault produces two equal halves. Even-numbered clusters split cleanly down the middle, which is exactly the condition to avoid.

Two-site deployments deserve particular scepticism: whatever the node count, a link failure between sites leaves two groups, and only a third location can break the tie.

Witness placement decides real behaviour

A witness or arbiter carries a vote without carrying data, and its location determines which failures the cluster survives.

• Place the witness in a third failure domain, never alongside a data node
• Verify the witness has independent network paths to both sites
• Monitor witness reachability as a first-class alert, not a secondary metric
• Test the cluster's behaviour when the witness alone is unreachable

Fencing turns detection into safety

Quorum tells a node it should not be leader; fencing makes it impossible for it to act as one. STONITH-style power fencing, storage-level reservations, and network isolation all serve the same purpose: ensuring a demoted node cannot continue accepting writes because it has not yet learned it lost the election.

Applications need their own guard. Leases with monotonically increasing fencing tokens, validated at the storage layer, prevent a paused process from resuming and writing with stale authority.

Recovery is a decision, not a script

When divergence has already occurred, someone must decide which timeline survives. Prepare that decision in advance: identify the authoritative side by transaction volume, business criticality, or region of record, and keep tooling that can extract and replay the losing side's writes for review. Rehearse the procedure on a staging cluster, because the first attempt should never happen during an incident.