Restart a node

Restarting a node is a common O&M action. It is suitable for brief maintenance of a server or when a system configuration item is modified and needs to take effect. The duration of a restart must be shorter than the time specified in the server_permanent_offline_time parameter. Otherwise, the node will be permanently offline. If the server needs long-term maintenance, you must replace the server. For more information, see Replace a node.

Background information

As a distributed database, OceanBase Database is typically deployed with multiple replicas (for example, three replicas in a region with three IDCs, or five replicas across three regions with five IDCs each). The Paxos protocol is used to achieve majority consensus among the replicas during transaction commit to maintain data consistency among replicas, and to ensure RPO=0 SLA in the case of failures of minority replicas.

The STOP SERVER command achieves lossless restart in a multi-replica architecture. The STOP SERVER command performs the following operations:

1. Removes all leaders from the restart node and ensures that the majorities of replicas are maintained on other nodes.

2. Marks the restart node as stopped (with the node status set to ACTIVE and the stop_time field greater than 0) in the Root Service. The client recognizes the node as stopped and does not route business requests to the node.

After the STOP SERVER command succeeds, the restarted node does not cause leader elections or client errors, and remains transparent to the business traffic. If the STOP SERVER command fails, you need to stop the restart and check the cause. Some possible causes include insufficient replicas, delayed redo logs, and fewer than the majority of voting members.

Procedure

The main process of restarting a node is: stop services -> perform a minor compaction -> stop the process -> start the process -> start services.

This topic will guide you through the procedure of restarting a node in a cluster. If you want to restart multiple nodes, you can perform the same operation multiple times.

1. Log in to the sys tenant of the cluster as the root user.

   Note that you must specify the corresponding parameters in the following sample code based on your database connection string.

   obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -A
   

   For more information about how to connect to the database, see Overview (MySQL mode) and Overview (Oracle mode).

2. Run the following command to isolate the node.

   During the node restart, the service continuity may be interrupted. For example, if the cluster contains only one or two nodes or the data of a tenant is distributed on only two nodes, the system cannot provide services during the restart of the node. The Stop Server operation ensures that the interruption to service continuity is minimized. After the node is isolated, the system will perform a safety check to ensure that the node can be restarted without interrupting the continuity of system services. After the node is successfully isolated, it will no longer provide services. If the Stop Server operation fails, check the deployment of the cluster or resolve the issue based on the error message, and then run the Stop Server operation again; or if you can tolerate the stop of the node, you can skip this step.

   obclient [(none)]> ALTER SYSTEM STOP SERVER 'svr_ip:svr_port';
   

   The parameters are described as follows:

   • svr_ip: the IP address of the node to be stopped.

   • svr_port: the RPC port of the node to be stopped. The default value is 2882.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM STOP SERVER '172.xx.xx.xx:2882';
   

   After the execution is successful, query the STATUS column of the oceanbase.DBA_OB_SERVERS view for the server to be restarted. You will find that the value of this column remains ACTIVE but the value of the STOP_TIME column changes from NULL to the service-stop time.

   For more information about how to query the oceanbase.DBA_OB_SERVERS view, see View a node.

3. Run the following command to perform a minor compaction on the node to shorten the redo log replay time after the node is restarted and accelerate the restart process.

   obclient [(none)]> ALTER SYSTEM MINOR FREEZE SERVER = ('svr_ip:svr_port');
   

   The parameters are described as follows:

   • svr_ip: the IP address of the node to be restarted.

   • svr_port: the RPC port of the node to be restarted. The default value is 2882.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM  MINOR FREEZE SERVER = ('172.xx.xx.xx:2882');
   

   After the minor compaction is completed, proceed to the next step. For more information about how to check the minor compaction progress, see View minor compaction information.

   For more information about minor compactions, see Minor compaction.

4. Stop the observer process.

   1. Log in to the server where the observer process to be stopped is located as the admin user.

   2. Use the CLI tool to navigate to the /home/admin/oceanbase directory of the server.

      [admin@xxx /]$ cd /home/admin/oceanbase
      

      For more information about the installation directories of OceanBase Database, see OBServer installation directory structure.

   3. Run the following command to view and obtain the process ID of the node.

      [admin@xxx oceanbase]$ ps -ef | grep observer | grep -v grep
      admin    103364      1 99  2022 ?        51-17:24:41 /home/admin/oceanbase/bin/observer
      

      In this example, 103364 is the process ID of the node.

   4. Stop the observer process.

      The command is as follows:

      [admin@xxx oceanbase]$ kill -9 pid
      

      Here, pid is the observer process ID of the node to be stopped.

      Here is an example:

      [admin@xxx oceanbase]$ kill -9 103364
      

      Notice

      You can stop only one observer process in a deployment directory. If you want to stop observer processes on multiple nodes, you need to log in to each server in sequence.

   5. Run the following command to check whether the observer process has stopped.

      [admin@xxx oceanbase]$ ps aux | grep observer
      

      If no information is returned after the command execution, the observer process has been stopped successfully.

5. (Optional) If you want to perform maintenance on the server, perform the maintenance in this step.

6. Start the observer process.

   1. Log in to the server where the observer process to be started is located as the admin user.

   2. Run the following command to start the observer process.

      [admin@xxx oceanbase]$ cd /home/admin/oceanbase  &&  ./bin/observer
      

      Notice

      You can start only one observer process in a deployment directory. If you want to start observer processes on multiple nodes, you need to log in to each server in sequence.

      For more information about the installation directories of OceanBase Database, see OBServer installation directory structure.

      After the observer process is successfully started, check the value of the START_SERVICE_TIME column in the oceanbase.DBA_OB_SERVERS view. If this value is not NULL, the observer process has been started successfully.

7. Run the following command to start the services of the node.

   obclient [(none)]> ALTER SYSTEM START SERVER 'svr_ip:svr_port';
   

   where:

   • svr_ip: the IP address of the node to be started.

   • svr_port: the RPC port of the node to be started. The default value is 2882.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM START SERVER '172.xx.xx.xx:2882';
   

   After the execution is successful, query the STOP_TIME column in the oceanbase.DBA_OB_SERVERS view. If this value is NULL, the node has been restarted and can now provide services.

   For more information about how to query the oceanbase.DBA_OB_SERVERS view, see View a node.

References

For more information about node O&M, see the following topics:

• View a node

• Add a node

• Delete a node

• Isolate a node

• Replace a node