When you verify whether the primary cluster can be switched to the standby role or the other way around, you need to check the value of the SWITCHOVER_INFO field in the V$OB_CLUSTER view. This topic describes the meanings of the different values of this field and how to fix the related errors.
Value 1: CHECK PRIMARY CLUSTER SERVER STATUS
Description
Some OBServer nodes may be inactive. You must check the status of OBServer nodes in the primary cluster.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK PRIMARY CLUSTER SERVER STATUS |
+-------------------+-------------------------------------+
Parameters:
SWITCHOVER_STATUS: the switchover status. If the returned value of this field isNOT ALLOWED, no switchover is allowed.The cause of the switchover failure is included in the
SWITCHOVER_INFOfield. The valueCHECK PRIMARY CLUSTER SERVER STATUSindicates that some OBServer nodes may be inactive. You must check the status of OBServer nodes in the primary cluster.
Solutions
Execute the following statement to query the status of OBServer nodes in the primary or standby cluster. If STATUS is not ACTIVE, the OBServer node is not active and you need to restore it.
Example:
obclient> SELECT SVR_IP, STATUS FROM oceanbase.__ALL_SERVER WHERE STATUS != 'ACTIVE';
+-------------+----------+
| SVR_IP | STATUS |
+-------------+----------+
| xxx.xx.x.x5 | inactive |
+-------------+----------+
Parameters:
SVR_IP: the IP address of the OBServer node.STATUS: the status of the OBServer node. If the returned value isinactive, the OBServer node is inactive.
Value 2: CHECK MERGE STATUS
Description
The major compaction task is abnormal. You must check the major compaction task for errors.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK MERGE STATUS |
+-------------------+-------------------------------------+
Solutions
Execute the following statement on the current cluster to query the status of the major compaction task. If an error occurs, contact OceanBase Technical Support for assistance.
obclient> SELECT ZONE, NAME, INFO FROM oceanbase.__ALL_ZONE WHERE NAME ='MERGE_STATUS';
+-------+--------------+------+
| ZONE | NAME | INFO |
+-------+--------------+------+
| | merge_status | IDLE |
| zone1 | merge_status | IDLE |
| zone2 | merge_status | IDLE |
| zone3 | merge_status | IDLE |
+-------+--------------+------+
INFO: the major compaction status. Valid values:
IDLE: No major compaction is in progress.MERGING: The major compaction is in progress.TIMEOUT: The major compaction times out. After the timeout, the major compaction continues with a TIMEOUT flag.ERROR: An error has occurred during the major compaction and must be handled with top priority.
Value 3: NONE SYNCED STANDBY CLUSTER
Description
The primary cluster cannot be switched to the standby role because no synchronized standby cluster is available.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | NONE SYNCED STANDBY CLUSTER |
+-------------------+-------------------------------------+
Solutions
On the primary cluster, execute the following statement to query synchronization details of standby clusters and causes of asynchronization:
Example:
obclient> SELECT CLUSTER_ID, SYNCHRONIZATION_STATUS FROM oceanbase.V$OB_STANDBY_STATUS;
+------------+------------------------+
| CLUSTER_ID | SYNCHRONIZATION_STATUS |
+------------+------------------------+
| 2 | OK |
+------------+------------------------+
Parameters:
CLUSTER_ID: the cluster ID.SYNCHRONIZATION_STATUS: the synchronization status. If the returned value isOK, synchronization succeeded. Otherwise, the cause of asynchronization is returned.
Value 4: CHECK SYS SCHEMA SYNC STATUS
Description
Schemas are not synchronized. You must ensure that the schemas of the sys tenant in the standby cluster are synchronized.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK SYS SCHEMA SYNC STATUS |
+-------------------+-------------------------------------+
Solutions
Execute the following statement on the primary and standby clusters to query the tenant_id with asynchronous schemas. You need to wait until schemas of the tenant are synchronized. If the synchronization is stuck, contact OceanBase Technical Support for assistance.
Example:
obclient> SELECT *, USEC_TO_TIME(MIN_SYS_TABLE_SCN), USEC_TO_TIME(MIN_USER_TABLE_SCN) FROM oceanbase.V$OB_CLUSTER_STATS;
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
| tenant_id | refreshed_schema_version | ddl_lag | min_sys_table_scn | min_user_table_scn | USEC_TO_TIME(MIN_SYS_TABLE_SCN) | USEC_TO_TIME(MIN_USER_TABLE_SCN) |
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
| 1 | 1638945061379584 | 0 | 1638965918418375 | 1638965918418375 | 2021-12-08 20:18:38.418375 | 2021-12-08 20:18:38.418375 |
| 1001 | 1638945068536016 | 0 | 1638965918418766 | 1638965918418766 | 2021-12-08 20:18:38.418766 | 2021-12-08 20:18:38.418766 |
| 1002 | 1638945068659384 | 0 | 1638965918418912 | 1638965918418912 | 2021-12-08 20:18:38.418912 | 2021-12-08 20:18:38.418912 |
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
Parameters:
tenant_id: the tenant ID. If the returned value is1, the tenant is the sys tenant. Otherwise, it is a user tenant.MIN_SYS_TABLE_SCN: the synchronization progress of the system table.MIN_USER_TABLE_SCN: the synchronization progress of the user table.If the value of
MIN_SYS_TABLE_SCNfor the sys tenant withtenant_idbeing1is inconsistent between the primary cluster and standby cluster, schemas are not synchronized.
Note
MIN_SYS_TABLE_SCNandMIN_USER_TABLE_SCNspecify the latest snapshot of the sys tenant.
Value 5: CHECK USER SCHEMA SYNC STATUS
Description
Schemas are not synchronized. You must ensure that the schemas of the user tenant in the standby cluster are synchronized.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK USER SCHEMA SYNC STATUS |
+-------------------+-------------------------------------+
Solutions
Execute the following statement on the primary and standby clusters to query the tenant_id with asynchronous schemas. You need to wait until schemas of the tenant are synchronized. If the synchronization is stuck, contact OceanBase Technical Support for assistance.
obclient> SELECT *, USEC_TO_TIME(MIN_SYS_TABLE_SCN), USEC_TO_TIME(MIN_USER_TABLE_SCN) FROM oceanbase.V$OB_CLUSTER_STATS;
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
| tenant_id | refreshed_schema_version | ddl_lag | min_sys_table_scn | min_user_table_scn | USEC_TO_TIME(MIN_SYS_TABLE_SCN) | USEC_TO_TIME(MIN_USER_TABLE_SCN) |
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
| 1 | 1638945061379584 | 0 | 1638965918418375 | 1638965918418375 | 2021-12-08 20:18:38.418375 | 2021-12-08 20:18:38.418375 |
| 1001 | 1638945068536016 | 0 | 1638965918418766 | 1638965918418766 | 2021-12-08 20:18:38.418766 | 2021-12-08 20:18:38.418766 |
| 1002 | 1638945068659384 | 0 | 1638965918418912 | 1638965918418912 | 2021-12-08 20:18:38.418912 | 2021-12-08 20:18:38.418912 |
+-----------+--------------------------+---------+-------------------+--------------------+---------------------------------+----------------------------------+
tenant_id: the tenant ID. If the returned value is1, the tenant is the sys tenant. Otherwise, it is a user tenant.MIN_SYS_TABLE_SCN: the synchronization progress of the system table.MIN_USER_TABLE_SCN: the synchronization progress of the user table.If the value of
MIN_USER_TABLE_SCNfor a user tenant is inconsistent between the primary cluster and the standby cluster, schemas are not synchronized.
Value 6: CHECK FREEZE INFO SYNC STATUS
Description
The frozen information is not synchronized. You must ensure that the frozen information of the standby cluster is synchronized.
Example:
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK FREEZE INFO SYNC STATUS |
+-------------------+-------------------------------------+
Solutions
Execute the following statement on the primary and standby clusters to query the asynchronous frozen information. You need to wait until the frozen information is synchronized. If the synchronization is stuck, contact OceanBase Technical Support for assistance.
Example:
obclient> SELECT * FROM oceanbase.__ALL_VIRTUAL_FREEZE_INFO;
+----------------+------------------+----------------+
| frozen_version | frozen_timestamp | schema_version |
+----------------+------------------+----------------+
| 1 | 1 | NULL |
| 2 | 1637548600183610 | NULL |
| 3 | 1637604010868012 | NULL |
+----------------+------------------+----------------+
Parameters:
frozen_version: the freeze version. When the value offrozen_versionis consistent on the primary cluster and standby cluster, the frozen information is synchronized.frozen_timestamp: the freeze timestamp.
Value 7: CHECK ENOUGH REPLICA
Description
The replicas are insufficient. You must ensure that the standby cluster has sufficient replicas.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK ENOUGH REPLICA |
+-------------------+-------------------------------------+
Solutions
Insufficient replicas mean that the number of active replicas is not enough to for a majority. This is probably caused by a failure of the OBServer node or network isolation. You can check the status of all OBServer nodes in the standby cluster to verify whether each partition has sufficient active replicas.
SQL statement for checking:
obclient> SELECT SVR_IP, STATUS FROM oceanbase.__ALL_SERVER WHERE STATUS != 'ACTIVE';
+-------------+----------+
| SVR_IP | STATUS |
+-------------+----------+
| xxx.xx.x.x5 | inactive |
+-------------+----------+
SVR_IP: the IP address of the OBServer node.STATUS: the status of the OBServer node. If the returned value isinactive, the OBServer node is inactive. If the returned value isactive, the OBServer node is normal.
Value 8: CHECK REDO LOG SYNC STATUS
Description
The redo logs are not synchronized. You must ensure that the redo logs of the standby cluster are synchronized.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+-------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+-------------------------------------+
| NOT ALLOWED | CHECK REDO LOG SYNC STATUS |
+-------------------+-------------------------------------+
Solutions
Execute the following statement on the standby cluster to check for replicas with failed log synchronization and check the replica status.
obclient> SELECT * FROM oceanbase.__ALL_VIRTUAL_CLOG_STAT WHERE IS_IN_SYNC=0;
If you find replicas with unfinished log synchronization, wait until the logs are synchronized. If the log synchronization keeps lagging or gets stuck, contact OceanBase Technical Support for assistance.
Value 9: CAN NOT ACCESS CLUSTERS: cluster1, cluster2
Description
You are not allowed to access some standby clusters, and you must verify their status.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | CAN NOT ACCESS CLUSTERS: cluster1, cluster2 |
+-------------------+------------------------------------------------------+
cluster1 and cluster2 specify the inaccessible standby clusters.
Solutions
Execute the following statement on the primary cluster to query the synchronization details of standby clusters.
obclient> SELECT CLUSTER_ID, SYNCHRONIZATION_STATUS FROM oceanbase.V$OB_STANDBY_STATUS;Check the value of the
SYNCHRONIZATION_STATUSfield and take corresponding actions.
Value 10: NO CLUSTER IN SWITCHING
Description
You cannot switch a standby cluster to the primary cluster as no cluster is in the SWITCHING state. This error may occur when the standby cluster is in the NOT ALLOWED state.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | NO CLUSTER IN SWITCHING |
+-------------------+------------------------------------------------------+
Solutions
Execute the following statement on the primary cluster to query the status of standby clusters and check for the record of successful switchover operations.
obclient> SELECT SWITCHOVER_STATUS FROM oceanbase.V$OB_CLUSTER;
+-------------------+
| SWITCHOVER_STATUS |
+-------------------+
| NOT ALLOWED |
+-------------------+
Value 11: STANDBY CLUSTER NOT IN SYNC MODE
Description
The log transfer mode is not set to SYNC mode.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | STANDBY CLUSTER NOT IN SYNC MODE |
+-------------------+------------------------------------------------------+
In MAXIMUM PROTECTION and MAXIMUM AVAILABILITY modes, you can only switch a standby cluster in SYNC mode to the primary cluster. The log transfer parameter of the standby cluster is not set to SYNC, so you cannot switch it to the primary role.
Solutions
Select a standby cluster in SYNC mode and switch it to the primary role.
Value 12: PRIMARY CLUSTER HAS REPLICA IN RESTORE
Description
Switchover is not allowed because some replicas on the primary cluster are not restored.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | PRIMARY CLUSTER HAS REPLICA IN RESTORE |
+-------------------+------------------------------------------------------+
Solutions
You must delete these unrestored replicas to proceed with the role switchover.
You can use the following syntaxes to query the status of replicas on the primary cluster:
Check the status of the system table replica that is being restored.
obclient> SELECT * FROM oceanbase.__ALL_ROOT_TABLE WHERE IS_RESTORE != 0;Check the status of user table replicas that are being restored.
obclient> SELECT * FROM oceanbase.__ALL_VIRTUAL_META_TABLE WHERE IS_RESTORE != 0;
Value 13: STANDBY CLUSTER HAS REPLICA IN RESTORE
Description
Switchover is not allowed because some replicas on the standby cluster are not restored.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | STANDBY CLUSTER HAS REPLICA IN RESTORE |
+-------------------+------------------------------------------------------+
Solutions
You can continue the role switching only after replicas in the standby cluster are restored.
Execute the following statement on the standby cluster to check the replica state:
Check the status of the system table replica that is being restored.
obclient> SELECT * FROM oceanbase.__ALL_ROOT_TABLE WHERE IS_RESTORE != 0;Check the status of user table replicas that are being restored.
obclient> SELECT * FROM oceanbase.__ALL_VIRTUAL_META_TABLE WHERE IS_RESTORE != 0;
Value 14: PRIMARY CLUSTER DOING BACKUP
Description
The switchover is not allowed because the primary cluster is in the process of backup. Backup is not supported for standby clusters, so you must stop the backup task first.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | PRIMARY CLUSTER DOING BACKUP |
+-------------------+------------------------------------------------------+
Solutions
Execute the following statement to stop the backup task on the primary cluster.
obclient> ALTER SYSTEM CANCEL BACKUP;
Value 15: CHECK OTHER PRIMARY CLUSTER
Description
Due to the existence of another primary cluster, the switchover is not allowed to avoid the circumstance of two primary clusters.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | CHECK OTHER PRIMARY CLUSTER |
+-------------------+------------------------------------------------------+
Solutions
Execute the following statement on the current cluster to query information of other clusters. Confirm the status of the clusters and select a proper cluster for the switchover.
obclient> SELECT * FROM oceanbase.V$OB_STANDBY_STATUS;
Value 16: INNER ERROR
Description
An internal error occurred.
obclient> SELECT SWITCHOVER_STATUS, SWITCHOVER_INFO FROM oceanbase.V$OB_CLUSTER;
+-------------------+------------------------------------------------------+
| SWITCHOVER_STATUS | SWITCHOVER_INFO |
+-------------------+------------------------------------------------------+
| NOT ALLOWED | INNER ERROR |
+-------------------+------------------------------------------------------+
Solutions
Contact OceanBase Technical Support for assistance.