In distributed system design, one of the main issues to address is the single point of failure (SPOF). To ensure the system can continue to operate normally even if a node fails, multiple replicas are typically deployed for each node, with some acting as leaders and others as followers. An election protocol is used to select a leader among the replicas, and if the leader fails, the protocol automatically switches to a follower.
The leader is referred to as the Leader, and the followers are referred to as Followers.
A well-functioning election protocol in a distributed system should meet two expectations:
Correctness
That is, when a replica considers itself a leader, no other replica should also consider itself a leader. Having two replicas considering themselves leaders at the same time is called a "brain split." The election mechanism in the Raft protocol avoids brain splits by ensuring only one leader is assigned per term. However, in native Raft, multiple replicas can consider themselves leaders at the same time, even if they are handling different terms and the leader of the smaller term has already failed but is unaware of it. To ensure the data read is the most recent, the native Raft protocol requires reading from the majority of replicas. OceanBase Database avoids this by using the lease mechanism, ensuring that only one replica considers itself a leader at any given time.
Liveness
That is, at any time, if the leader fails, as long as the majority of replicas in the cluster are still alive, a replica should be able to become the new leader within a limited time.
In addition to correctness and liveness, OceanBase Database's election protocol also provides a priority mechanism and a switch leader mechanism. The priority mechanism selects the replica with the highest priority as the leader when no leader exists. The switch leader mechanism allows for seamless switching of the leader to a specified replica when a leader exists.
Lease duration
If the leader cannot communicate with the majority of replicas for more than one lease duration, a new election will be initiated, and the next leader will be selected. The default lease duration for elections in OceanBase Database is 4 seconds.
Priority
During the election process, the leader will renew its lease to detect changes in the priority of the follower replicas. If the priority of a follower replica is higher than that of the leader, the leader will proactively switch to the follower replica.
Currently, the priorities are ranked as follows:
User-specified leader.
Primary zone.
Switch leader
Users can control the distribution of log stream leaders using the ALTER SYSTEM SWITCH REPLICA command. The system will smoothly switch the leader to the specified replica without external awareness.
Users can also change the primary zone. The election process will detect the change in the primary zone and switch the leader to the target replica.