Network Time Protocol (NTP) synchronizes clocks between computer systems through the network. The NTP server keeps the clocks of all servers in an organization synchronized.
OceanBase Database is a distributed database service. Therefore, the clocks of servers in a cluster and OceanBase Cloud Platform (OCP) servers must be configured with a clock synchronization service, either NTP or Chrony, to ensure that the clock offset of all servers is within 100 ms.
The system time of an OBServer node is synchronized by NTP. However, if the time offset is too large, NTP does not change or correct the system time. If the system time of OBServer nodes within an OceanBase cluster is not synchronized, the election module of the cluster may be affected, resulting in split-brain or the absence of a leader. This topic describes how to adjust the system time of an OBServer node.
View the NTP offset
You can view the NTP offset by using either of the following methods.
View the NTP offset in OCP
OceanBase clusters strongly depend on NTP. You can monitor the NTP offset in OCP through health inspection.
Log on to the OCP console. The Cluster Overview page appears.
In the Clusters list on the Cluster Overview page, find the target cluster and click its name.
In the left-side navigation pane of the page that appears, click Health Inspection.
In the Inspection Results section, view the information about the NTP offset.
The following table describes the parameters displayed in the inspection results.
Parameter Description Module The object on which the inspection is performed. Valid values: Cluster, Host, OB_SEVER, and OB_PROXY. Metric The metric of the cluster, host, OBServer node, or ODP. For more information, see the following Inspection metrics table. Health Status The health status of the object. Valid values: Critical, Warning, and Healthy. Result Description The detailed inspection result of the object, such as the information about the host, cluster, ODP, zone, or OBServer node. Inspected At The time when the inspection was initiated. Example: 17:04:00 on November 27, 2020. NTP The NTP offset.
View the NTP offset on the host
Run the following command to query the NTP offset:
[root@hostname /]# ntpq -p|grep -E "\*|\=|remote"
remote refid st t when poll reach delay offset jitter
==========================================================================
xxxxxxxxxxx xxxxx 2 u 16 64 377 1.333 0.046 0.033
Notice
You can run the ntpq command to query the NTP offset. An acceptable NTP offset must be less than 50 ms.
Adjust the NTP offset
The system time of an OBServer node is synchronized by NTP. However, if the time offset is too large, NTP does not change or correct the system time. If the system time of OBServer nodes within an OceanBase cluster is not synchronized, the election module of the cluster may be affected, resulting in split-brain or the absence of a leader. This section describes how to adjust the system time of an OBServer node.
Offset threshold
The default threshold of NTP offsets is 500 ms. If an NTP offset exceeds the threshold, you must adjust the NTP offset.
Troubleshooting for OBServer node time earlier than standard time
In this case, you can directly change the system time of the OBServer node to the standard time. For example, the current standard time is 20:00, whereas the system time of the OBServer node is 19:00, which is 1 hour earlier.
Stop an OBServer node.
Log on to the OCP console.
In the left-side navigation pane, click Clusters to go to the Cluster Overview page. In the Clusters section, click the name of the cluster to which the required OBServer node belongs.
In the OBServer node list, find the required OBServer node and click Stop in the Actions column.
Log on to the host by using SSH and change the system time.
ntp_server_ipspecifies the IP address of the NTP server.[root@hostname /]# ntpdate <ntp_server_ip>Start an OBServer node.
Log on to the OCP console.
In the left-side navigation pane, click Clusters to go to the Cluster Overview page. In the Clusters section, click the name of the cluster to which the required OBServer node belongs.
In the OBServer node list, find the required OBServer node and click Start in the Actions column.
Troubleshooting for OBServer node time later than standard time
In this case, you cannot directly change the system time of the OBServer node. Instead, you must stop the OBServer node, wait until the standard time exceeds the system time of the OBServer node, and then change the system time of the OBServer node. For example, the current standard time is 20:00, whereas the system time of the OBServer node is 21:00, which is 1 hour later.
Stop an OBServer node.
Log on to the OCP console.
In the left-side navigation pane, click Clusters to go to the Cluster Overview page. In the Clusters section, click the name of the cluster to which the required OBServer node belongs.
In the OBServer node list, find the required OBServer node and click Stop in the Actions column.
Wait until the standard time exceeds 21:00. For example, wait until the standard time is 21:05. Then, log on to the host by using SSH and change the system time.
ntp_server_ipspecifies the IP address of the NTP server.[root@hostname /]# ntpdate ntp_server_ipStart an OBServer node.
Log on to the OCP console.
In the left-side navigation pane, click Clusters to go to the Cluster Overview page. In the Clusters section, click the name of the cluster to which the required OBServer node belongs.
In the OBServer node list, find the required OBServer node and click Start in the Actions column.