This document describes the principles, preparations, and server operations of the high availability mode
Theory
TuGraph provides high availability mode (HA mode) through multiple server hot backups. In high availability mode, writes are synchronized to all servers so that service availability is not affected even if some servers go down.
In high availability mode, multiple TuGraph servers form a backup group. Each backup group consists of three or more TuGraph servers, one of which serves as the 'leader' and the other replication group servers serve as the 'follower'. Write requests are serviced by the 'leader' who replicates each request to the 'follower' and responds to the client only after the request has been synchronized to the server. This way, if any server fails, the other servers will still have all the data that has been written so far. If the 'leader' server fails, other servers will automatically select a new 'leader'.
The high availability mode is only available in the enterprise edition. The open-source Community Edition does not include this feature.
Preparation
To enable high availability mode, users need to:
Three or more instances of TuGraph servers.
To obtain a license file with high availability, please contact our distributor for details.
To enable high availability mode when starting lgraph_server, the 'enable_ha' option can be set to 'true' using a configuration file or the command line.
Set the correct rpc_port through the configuration file or command line
Start the initial backup group
The initial data is consistent
When the data in all servers is the same or there is no data at startup, the user can specify --conf host1:port1,host2:port2 to start the server. In this way, all prepared TuGraph instances can be added to the initial backup group at one time, All servers in the backup group elect leader according to the RAFT protocol, and other servers join the backup group with the role of follower.
An example command to start an initial backup group is as follows:
$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090,172.22.224.16:9090,172.22.224.17:9090
After the first server is started, it will elect itself as the 'leader' and organize a backup group with only itself.
The first server is started with the '--master "" or' --master BOOTSTRAP 'options, depending on whether the first server already has data.
'--master "":' If there is no data in the first server, the user can start the server directly with the '--master ""' option.
'--master BOOTSTRAP:' if the data already exists in the first server (imported using lgraph_import or transferred from a server not in high availability mode) and has not been used in high availability mode before, Then the user should use the 'BOOTSTRAP' option to start the server in boot mode. In boot mode, the server copies its data to the new server before adding the new server to the backup group, so that the data in each server is consistent.
The following is an example of the command line for starting the first server that already has data:
$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --master BOOTSTRAP
Inconsistent initial data
If there is already data in the first server (imported by the lgraph_import tool or transferred from a server in non-high availability mode), And it has not been used in high availability mode before, the user should use the enable_bootstrap parameter Start the server in boot mode. In bootstrap mode, the server will add its own The data is copied to the new server so that the data in each server is consistent.
An example command to start a data server is as follows:
$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090 --enable_bootstrap true
Other servers without data do not need to specify the enable_bootstrap parameter, just specify the leader through the conf parameter, the command example is as follows
$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090
Scale out other servers
After starting the initial backup group, if you want to scale out the backup group, add new servers to the backup group, The --conf HOST:PORT option should be used, where HOST can be the IP address of any server already in this backup group, And PORT is its RPC port. E.g:
./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090
This command will start a TuGraph server in high availability mode and try to add it to the backup group containing the server 172.22.224.15:9090. Note that joining a backup group requires a server to synchronize its data with the backup group's leader server, and this process may take a considerable amount of time, depending on the size of the data.
Stopping the Server
When a server goes offline via 'CTRL-C', it will notify the current 'leader' server to remove the server from the backup group. If the leader server goes offline, it will pass the leader identity permission to another server before going offline.
If a server is terminated or disconnected from other servers in the backup group, the server is considered a failed node and the leader server will remove it from the backup group after a specified time limit.
If any server leaves the backup group and wishes to rejoin, it must start with the '--master {HOST: PORT}' option, where 'HOST' is the IP address of a server in the current backup group.
Restarting the Server
Restarting the entire backup group is not recommended because it can interrupt service. You can shut down all servers if you need to. However, on a restart, the last server that went down must be started first.
Server Status
The current status of the backup group can be obtained from the TuGraph visualization tool, REST API, and Cypher query.
In the TuGraph visualization tool, you can find the list of servers and their roles in the backup group in the DBInfo section.
With the REST API, you can use GET /info/peers to request information.
In Cypher, the CALL dbms.listServers() statement is used to query the status information of the current backup group.
Data synchronization in high availability mode
In high availability mode, different servers in the same backup group may not always be in the same state. For performance reasons, if a request has been synchronized to more than half of the servers, the leader server will consider the request to be in the committed state. Although the rest of the servers will eventually receive the new request, the inconsistent state of the servers will persist for some time. A client may also send a request to a server that has just restarted, thus having an older state and waiting to join a backup group.
To ensure that the client sees consistently continuous data, and in particular to get rid of the 'reverse time travel' problem, where the client reads a state older than it has seen before, each TuGraph server keeps a monotonically increasing data version number. The mapping of the data version number to the database state in the backup group is globally consistent, meaning that if two servers have the same data version number, they must have the same data. When responding to a request, the server includes its data version number in the response. Thus, the client can tell which version it has seen. The client can choose to send this data version number along with the request. Upon receiving a request with a data version number, the server compares the data version number to its current version and rejects the request if its own version is lower than the requested version. This mechanism ensures that the client never reads a state that is older than before.