Check the NTP offset

Network Time Protocol (NTP) synchronizes clocks between computer systems through the network. The NTP server keeps the clocks of all servers in an organization synchronized.

OceanBase Database is a distributed database. Therefore, the clocks of servers in a cluster and OceanBase Cloud Platform (OCP) servers must be configured with a clock synchronization service, either NTP or Chrony, to ensure that the clock offset of all servers is within 100 ms.

The system time of an OBServer node is synchronized by NTP. However, if the time offset is too large, NTP does not change or correct the system time. If the system time of OBServer nodes within an OceanBase cluster is not synchronized, the election module of the cluster may be affected, resulting in split-brain or the absence of a leader. This topic describes how to adjust the system time of an OBServer node.

View the NTP offset

You can view the NTP offset by using either of the following methods.

View the NTP offset on the host

Run the following command to query the NTP offset:

[root@hostname /]# ntpq -p|grep -E "\*|\=|remote"
   remote         refid           st t when poll reach    delay     offset     jitter
==========================================================================
xxxxxxxxxxx       xxxxx           2 u   16  64  377       1.333     0.046   0.033

Adjust the NTP offset

The system time of an OBServer node is synchronized by NTP. However, if the time offset is too large, NTP does not change or correct the system time. If the system time of OBServer nodes within an OceanBase cluster is not synchronized, the election module of the cluster may be affected, resulting in split-brain or the absence of a leader. This section describes how to adjust the system time of an OBServer node.

Offset threshold

The default threshold of NTP offsets is 500 ms. If an NTP offset exceeds the threshold, you must adjust the NTP offset.

Troubleshooting for OBServer node time earlier than standard time

In this case, you can directly change the system time of the OBServer node to the standard time. For example, the current standard time is 20:00, whereas the system time of the OBServer node is 19:00, which is 1 hour earlier.

1. Stop an OBServer node.

2. Log on to the host by using SSH and change the system time.

   Here, ntp_server_ip specifies the IP address of the NTP server.

   [root@hostname /]# ntpdate <ntp_server_ip>
   

3. Start the OBServer node.

Troubleshooting for OBServer node time later than standard time

In this case, you cannot directly change the system time of the OBServer node. Instead, you must stop the OBServer node, wait until the standard time exceeds the system time of the OBServer node, and then change the system time of the OBServer node. For example, the current standard time is 20:00, whereas the system time of the OBServer node is 21:00, which is 1 hour later.

1. Stop an OBServer node.

2. Wait until the standard time exceeds 21:00. For example, wait until the standard time is 21:05. Then, log on to the host by using SSH and change the system time.

   Here, ntp_server_ip specifies the IP address of the NTP server.

   [root@hostname /]# ntpdate ntp_server_ip
   

3. Start the OBServer node.