Switchover and failover issues

OceanBase Database may return an error message for the execution of a switchover or failover command, such as SWITCHOVER TO STANDBY VERIFY, SWITCHOVER TO STANDBY, SWITCHOVER TO PRIMARY VERIFY, SWITCHOVER TO PRIMARY, ACTIVATE STANDBY VERIFY, or ACTIVATE STANDBY. The returned error message shows the reason why the switchover or failover command cannot be successfully executed. This topic describes how to troubleshoot these switchover and failover issues.

Error message 1

• ERROR HY000: log restore source is primary, switchover to primary is not allowed

• ERROR HY000: log restore source is not in normal switchover status, switchover to primary is not allowed

• ERROR HY000: log restore source is not in normal status, switchover to primary is not allowed

Possible causes

When you perform a switchover to switch a standby tenant to the PRIMARY role, the tenant of the log restore source must meet the following conditions so that the switchover can be successful:

1. The value of TENANT_ROLE is PRIMARY.

2. The value of STATUS is NORMAL.

3. The value of SWITCHOVER_STATUS is NORMAL.

If the tenant fails to meet any one of the above conditions, this error may be reported in a network-based Physical Standby Database scenario.

Troubleshooting procedure

1. Log in to the cluster where the archive source or restore source is located and query the status of the tenant of the archive source or restore source.

   sys tenant

   User tenant

   You can execute the following SQL statement to query the status of the tenant of the archive source or restore source from the sys tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'tenant_name';
   

   Replace tenant_name with the name of the tenant of the restore source or archive source.

   You can execute the following SQL statement to query the status of the current MySQL tenant that serves as the tenant of the archive source or restore source:

   obclient [oceanbase]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS;
   

   You can execute the following SQL statement to query the status of the current Oracle tenant that serves as the tenant of the archive source or restore source:

   obclient [SYS]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM SYS.DBA_OB_TENANTS;
   

2. Check whether the tenant status meet the requirements. If not, log in to the tenant of the log restore source or archive source and perform a switchover. Then retry the previous switchover operation that is stuck due to the incorrect tenant status.

3. If the issue persists, contact OceanBase Technical Support.

Error message 2

• ERROR HY000: tenant status is not normal, switchover to primary is not allowed

• ERROR HY000: tenant status is not normal, failover to primary is not allowed

• ERROR HY000: tenant status is not normal, switchover to standby is not allowed

Possible causes

When you perform a switchover, the STATUS value of the tenant must be NORMAL. If not, this error may be reported.

Troubleshooting procedure

1. Query the status of the tenant.

   sys tenant

   User tenant

   You can execute the following SQL statement to query the status of the tenant from the sys tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, STATUS FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'tenant_name';
   

   Replace tenant_name with the name of the tenant on which the switchover command is executed.

   You can execute the following SQL statement to query the tenant status from the current MySQL tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, STATUS FROM oceanbase.DBA_OB_TENANTS;
   

   You can execute the following SQL statement to query the tenant status from the current Oracle tenant:

   obclient [SYS]> SELECT TENANT_NAME, STATUS FROM SYS.DBA_OB_TENANTS;
   

2. Check whether the tenant status is NORMAL. If the status is not NORMAL, the tenant may be being created or restored. Wait until the creation or restore is complete. If the issue persists, contact OceanBase Technical Support.

Error message 3

• ERROR HY000: switchover status not match, switchover to primary is not allowed

• ERROR HY000: switchover status not match, failover to primary is not allowed

• ERROR HY000: switchover status not match, switchover to standby is not allowed

Possible causes

When you perform a switchover on a tenant, the current tenant must not be in an intermediate role switching state. Otherwise, this error is reported due to an unexpected SWITCHOVER STATUS value.

For example, in the following two scenarios, the tenant is in an intermediate role switching state:

• The previous switchover to the PRIMARY role performed on the tenant has failed, causing the tenant to stay in the SWITCHING TO PRIMARY state or another intermediate switchover state.

• The previous switchover to the STANDBY role performed on the tenant has failed, causing the tenant to stay in the SWITCHING TO STANDBY state or another intermediate switchover state.

Troubleshooting procedure

1. Query the SWITCHOVER_STATUS value of the current tenant.

   sys tenant

   User tenant

   You can execute the following SQL statement to query the SWITCHOVER_STATUS value of a specified tenant from the sys tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'tenant_name';
   

   Replace tenant_name with the name of the tenant on which the switchover command is executed.

   You can execute the following SQL statement to query the SWITCHOVER_STATUS value from the current MySQL tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS;
   

   You can execute the following SQL statement to query the SWITCHOVER_STATUS value from the current Oracle tenant:

   obclient [SYS]> SELECT TENANT_NAME, SWITCHOVER_STATUS FROM SYS.DBA_OB_TENANTS;
   

   For more information about the values of the SWITCHOVER_STATUS field, see DBA_OB_TENANTS.

2. Determine the ongoing switchover process of the current tenant based on the SWITCHOVER_STATUS value. Then, retry the switchover.

3. If the issue persists, contact OceanBase Technical Support.

Error message 4

ERROR HY000: Incorrect arguments to tenant name, only support operating user tenant

Possible causes

When you perform a tenant role switchover, the current tenant must be a user tenant. If it is not a user tenant, this error is reported.

Troubleshooting procedure

1. Check the type of the current tenant.

   sys tenant

   User tenant

   You can execute the following SQL statement to query the type of a specified tenant from the sys tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, TENANT_TYPE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'tenant_name';
   

   Replace tenant_name with the name of the tenant on which the switchover command is executed.

   You can execute the following SQL statement to query the type of the current MySQL tenant:

   obclient [oceanbase]> SELECT TENANT_NAME, TENANT_TYPE FROM oceanbase.DBA_OB_TENANTS;
   

   You can execute the following SQL statement to query the type of the current Oracle tenant:

   obclient [SYS]> SELECT TENANT_NAME, TENANT_TYPE FROM SYS.DBA_OB_TENANTS;
   

   In the query result, check whether the value of TENANT_TYPE is USER.

2. If it is not USER, select a user tenant and retry the switchover.

Error message 5

ERROR HY000: Incorrect arguments to tenant name, please don't specify tenant name

Possible causes

You can perform a switchover only on the current user tenant. You cannot specify a tenant name in the switchover command. If you specify a tenant name in the switchover command, this error is reported.

Troubleshooting procedure

Remove TENANT tenant_name from the command and retry the switchover.

Error message 6

• ERROR HY000: the tenant has LS replicas without leader, switchover to primary is not allowed

• ERROR HY000: the tenant has LS replicas without leader, failover to primary is not allowed

Possible causes

When you perform a switchover to the PRIMARY role on a standby tenant, all log streams of the standby tenant must have leaders. If a log stream on the standby tenant has no leader, this error is reported.

Troubleshooting procedure

1. Log in to the sys tenant of the cluster where the current tenant resides, execute the following SQL statement to obtain the list of log streams that do not have a leader:

   obclient [oceanbase]> SELECT DISTINCT A.TENANT_ID, A.LS_ID
   FROM CDB_OB_LS A LEFT JOIN oceanbase.GV$OB_LOG_STAT B ON A.LS_ID = B.LS_ID
   AND A.TENANT_ID = B.TENANT_ID AND B.ROLE='LEADER'
   WHERE B.LS_ID IS NULL
     AND A.STATUS NOT IN ('CREATING',
                         'CREATED',
                         'TENANT_DROPPING',
                         'CREATE_ABORT',
                         'PRE_TENANT_DROPPING')
     AND A.TENANT_ID IN (user_tenant_id, user_tenant_id - 1);
   

   Replace user_tenant_id with the tenant ID of the current tenant.

   If the query result is not empty, the tenant has log streams without leaders.

2. In this case, Contact OceanBase Technical Support.

Error message 7

• ERROR HY000: the tenant has units on temporary offline servers, switchover to primary is not allowed

• ERROR HY000: the tenant has units on temporary offline servers, failover to primary is not allowed

Possible causes

When you perform a switchover to the PRIMARY role on a standby tenant, all OBServer nodes of the tenant are required to be online by default. If an OBServer node is offline due to a downtime or network failure, the OBServer node must be permanently offline and has no valid replicas on it so that the switchover can be successful.

The preceding error message indicates that some OBServer nodes of the tenant are temporarily offline, in which case a switchover to PRIMARY or failover to PRIMARY is not allowed.

Troubleshooting procedure

1. Log in to the sys tenant of the cluster where the standby tenant resides.

2. Query temporarily offline OBServer nodes in the units for the tenant and obtain the LAST_OFFLINE_TIME values of the OBServer nodes.

   obclient [oceanbase]> SELECT SVR_IP, SVR_PORT, LAST_OFFLINE_TIME, NOW() FROM oceanbase.DBA_OB_SERVERS WHERE LAST_OFFLINE_TIME IS NOT NULL AND (SVR_IP, SVR_PORT) IN ( SELECT DISTINCT SVR_IP, SVR_PORT FROM oceanbase.DBA_OB_UNITS WHERE TENANT_ID = user_tenant_id);
   

   Replace user_tenant_id with the tenant ID of the current tenant.

   A sample query result is as follows:

   +----------------+----------+----------------------------+---------------------+
   | SVR_IP         | SVR_PORT | LAST_OFFLINE_TIME          | NOW()               |
   +----------------+----------+----------------------------+---------------------+
   | xxx.xx.xxx.212 |    13326 | 2023-12-07 10:44:53.928742 | 2023-12-07 15:58:26 |
   +----------------+----------+----------------------------+---------------------+
   

3. Query the server_permanent_offline_time value of each temporarily offline OBServer node, and determine the time when the OBServer node goes permanently offline based on the obtained value and the LAST_OFFLINE_TIME value.

   The server_permanent_offline_time parameter specifies the time threshold for heartbeat missing. When the heartbeat of an OBServer node is missing for the specified period of time, the OBServer node is considered permanently offline. Data replicas on a permanently offline OBServer node must be automatically supplemented. For more information about this parameter, see server_permanent_offline_time.

   obclient [oceanbase]> SHOW PARAMETERS LIKE 'server_permanent_offline_time';
   

   Based on the query results, if LAST_OFFLINE_TIME + server_permanent_offline_time is less than or equal to NOW(), it indicates that the OBServer node is permanently offline. Otherwise, the OBServer node is temporarily offline.

4. If the OBServer node is temporarily offline, take the following steps:

   1. Check whether the observer process is running on the OBServer node. If not, manually start the observer process and retry the switchover to PRIMARY.

      If the switchover remains unsuccessful, perform the next step.

   2. Decrease the value of server_permanent_offline_time to make the temporarily offline OBServer node permanently offline, and then retry the switchover to PRIMARY.

      We recommend that you set the server_permanent_offline_time parameter to a value greater than 20s, as shown in the following example:

      obclient [oceanbase]> ALTER SYSTEM SET server_permanent_offline_time = '30s';
      

      If the switchover retry fails and an error message is returned, indicating that a replica is on a permanently offline OBServer node, refer to Error message 8 for further steps. For other issues, contact OceanBase Technical Support.

Error message 8

• ERROR HY000: the tenant has LS replicas on at least one of the permanent offline servers, switchover to primary is not allowed

• ERROR HY000: the tenant has LS replicas on at least one of the permanent offline servers, failover to primary is not allowed

Possible causes

When you perform a switchover to the PRIMARY role on a standby tenant, all OBServer nodes of the tenant are required to be online by default. If an OBServer node is offline due to a downtime or network failure, the OBServer node must be permanently offline and has no valid replicas on it so that the switchover can be successful.

The preceding error indicates that the tenant has a replica on a permanently offline OBServer node, in which case a switchover to PRIMARY or failover to PRIMARY is not allowed.

Troubleshooting procedure

1. Log in to the sys tenant of the cluster where the standby tenant resides.

2. Check the information of the permanently offline OBServer nodes in the units for the tenant.

   1. Query the IP address information and the LAST_OFFLINE_TIME values of the permanently offline OBServer nodes.

      obclient [oceanbase]> SELECT SVR_IP, SVR_PORT, LAST_OFFLINE_TIME, NOW() FROM oceanbase.DBA_OB_SERVERS WHERE LAST_OFFLINE_TIME IS NOT NULL AND (SVR_IP, SVR_PORT) IN ( SELECT DISTINCT SVR_IP, SVR_PORT FROM oceanbase.DBA_OB_UNITS WHERE TENANT_ID = user_tenant_id);
      

      Replace user_tenant_id with the tenant ID of the current tenant.

   2. Query the server_permanent_offline_time value of each permanently offline OBServer node, and determine the time when the OBServer node goes permanently offline based on the obtained value and the LAST_OFFLINE_TIME value.

      The server_permanent_offline_time parameter specifies the time threshold for heartbeat missing. When the heartbeat of an OBServer node is missing for the specified period of time, the OBServer node is considered permanently offline. Data replicas on a permanently offline OBServer node must be automatically supplemented. For more information about this parameter, see server_permanent_offline_time.

      obclient [oceanbase]> SHOW PARAMETERS LIKE 'server_permanent_offline_time';
      

   3. Based on the results of the two queries, if LAST_OFFLINE_TIME + server_permanent_offline_time is less than or equal to NOW(), it indicates that the OBServer node is permanently offline.

3. Query the interval for dropping replicas.

   obclient [oceanbase]> SHOW PARAMETERS LIKE '%balancer_idle_time%';
   

   Wait until the current time is at least twice the balancer_idle_time value later than the sum of LAST_OFFLINE_TIME + server_permanent_offline_time, and then perform the next step.

4. Query the CDB_OB_LS_LOCATIONS view and check the MEMBER_LIST information of the leader of each log stream to identify permanently offline OBServer nodes.

   obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE TENANT_ID = user_tenant_id AND ROLE = 'LEADER' AND MEMBER_LIST LIKE "%permanent_offline_server_ip:permanent_offline_server_port%";
   

   Take note of the following considerations:

   • Replace user_tenant_id with the tenant ID of the current tenant.
   • Replace permanent_offline_server_ip with the IP address of the permanently offline OBServer node.
   • Replace permanent_offline_server_port with the RPC port of the permanently offline OBServer node.

   Check the query result.

   • If the query result is not empty, execute the following statement to check whether a task has been initiated to drop the permanently offline OBServer node from the MEMBER_LIST of the leader of the log stream.

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE MODULE = 'disaster_recovery' AND EVENT LIKE "%remove_ls_paxos_replica%" ORDER BY TIMESTAMP DESC;
     

     In the query result, each log stream should have a task that drops the permanently offline OBServer node from the MEMBER_LIST of its leader. If the corresponding task is not initiated or not completed for a log stream, contact OceanBase Technical Support.

   • If the query result is empty, no permanently offline OBServer nodes exist in the MEMBER_LIST of the leader of each log stream. In this case, take the next step to check the LEARNER_LIST information.

5. Query the CDB_OB_LS_LOCATIONS view. Check the LEARNER_LIST information of the leader of each log stream for permanently offline OBServer nodes.

   obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE TENANT_ID = user_tenant_id AND ROLE = 'LEADER' AND LEARNER_LIST LIKE "%permanent_offline_server_ip:permanent_offline_server_port%";
   

   Take note of the following considerations:

   • Replace user_tenant_id with the tenant ID of the current tenant.
   • Replace permanent_offline_server_ip with the IP address of the permanently offline OBServer node.
   • Replace permanent_offline_server_port with the RPC port of the permanently offline OBServer node.

   Check the query result.

   • If the query result is not empty, execute the following statement to check whether a task has been initiated to drop the permanently offline OBServer node from the LEARNER_LIST of the leader of the log stream.

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE MODULE = 'disaster_recovery' AND EVENT LIKE "%remove_ls_non_paxos_replica%" ORDER BY TIMESTAMP DESC;
     

     In the query result, each log stream should have a task that drops the permanently offline OBServer node from the LEADER_LIST of its leader. If the corresponding task is not initiated or not completed for a log stream, contact OceanBase Technical Support.

   • If the query result is empty, no permanently offline OBServer nodes exist in the LEARNER_LIST of the leader of each log stream. In this case, take the next step to check whether the tenant has extra replicas on the permanently offline OBServer node.

6. If no permanently offline OBServer nodes exist in either the MEMBER_LIST or LEARNER_LIST of the leader of each log stream, wait for a moment and execute the following statement to check whether the tenant has extra replicas on permanently offline OBServer nodes:

   obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE TENANT_ID = user_tenant_id AND SVR_IP = 'permanent_offline_server_ip' AND SVR_PORT = permanent_offline_server_port;
   

   Take note of the following considerations:

   • Replace user_tenant_id with the tenant ID of the current tenant.
   • Replace permanent_offline_server_ip with the IP address of the permanently offline OBServer node.
   • Replace permanent_offline_server_port with the RPC port of the permanently offline OBServer node.

   If the query result is empty, the tenant does not have extra replicas on the permanently offline OBServer node. In this case, retry the switchover to PRIMARY. If the switchover remains unsuccessful, contact OceanBase Technical Support.

   If the query result is not empty, the tenant has extra replicas on the permanently offline OBServer node. In this case, wait for a moment and check for extra replicas on the permanently offline OBServer node again. If the tenant no longer has any extra replicas on the permanently offline OBServer node, retry the switchover to PRIMARY. If the tenant still has extra replicas on the permanently offline OBServer node, contact OceanBase Technical Support.

7. For other issues, contact OceanBase Technical Support.

Error message 9

ERROR HY000: wait tenant sync to latest failed(original error code: -4012), switchover to primary is not allowed

Possible causes

When you perform a switchover to PRIMARY on a standby tenant, the log streams of the standby tenant must be synchronized with the restore source. If any log stream is unsynchronized, this error is reported.

Troubleshooting procedure

• Network-based Physical Standby Database scenario

  1. Check the network connection status between the primary and standby tenants.

     If the network is connected, continue with the next step.

  2. Log in to the sys tenant of the cluster where the standby tenant resides and check the restore status of the standby tenant.

     obclient [oceanbase]> SELECT LS_ID, SYNC_SCN, SYNC_STATUS, ERR_CODE, COMMENT FROM oceanbase.V$OB_LS_LOG_RESTORE_STATUS WHERE TENANT_ID = user_tenant_id;
     

     Replace user_tenant_id with the tenant ID of the current tenant.

     In the query result, if the value of SYNC_STATUS is NORMAL, check whether the value of SYNC_SCN is growing continuously. If yes, or if SYNC_STATUS has another value, contact OceanBase Technical Support.

• Archive-based Physical Standby Database scenario

  1. Check whether the role of the log archive tenant is STANDBY.

     sys tenant

     User tenant

     You can execute the following SQL statement to query the status of the corresponding tenant from the sys tenant:

     obclient [oceanbase]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME = 'tenant_name';
     

     Replace tenant_name with the name of the tenant of the restore source or archive source.

     You can execute the following SQL statement to query the tenant status from the current MySQL tenant:

     obclient [oceanbase]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS;
     

     You can execute the following SQL statement to query the tenant status from the current Oracle tenant:

     obclient [SYS]> SELECT TENANT_NAME, TENANT_ID, TENANT_ROLE, STATUS, SWITCHOVER_STATUS FROM SYS.DBA_OB_TENANTS;
     

     In the query result, if the value of TENANT_ROLE is not STANDBY for the current tenant, you need to switch the tenant to STANDBY first.

     If the query result is as follows, the tenant status is normal. Continue with the next step.

     +-------------+--------+-------------------+
     | TENANT_ROLE | STATUS | SWITCHOVER_STATUS |
     +-------------+--------+-------------------+
     | STANDBY     | NORMAL | NORMAL            |
     +-------------+--------+-------------------+
     

  2. Log in to the sys tenant of the cluster where the standby tenant resides and check the restore status of the standby tenant.

     obclient [oceanbase]> SELECT LS_ID, SYNC_SCN, SYNC_STATUS, ERR_CODE, COMMENT FROM oceanbase.V$OB_LS_LOG_RESTORE_STATUS WHERE TENANT_ID = user_tenant_id;
     

     Replace user_tenant_id with the tenant ID of the current tenant.

     In the query result, if the value of SYNC_STATUS is NORMAL, check whether the value of SYNC_SCN is growing continuously. If yes, or if SYNC_STATUS has another value, contact OceanBase Technical Support.

Error message 10

ERROR HY000: log restore source LS state not match, switchover to primary not allowed

Possible causes

For a switchover to PRIMARY on a standby tenant in a network-based Physical Standby Database scenario, the ACCESS_MODE value of all log streams of the tenant of the restore source must be RAW_WRITE. Otherwise, this error is reported.

Troubleshooting procedure

1. Log in to the sys tenant of the cluster where the tenant of the log restore source resides.

2. Check the value of ACCESS_MODE for all log streams of the tenant.

   obclient [oceanbase]> SELECT LS_ID, ACCESS_MODE FROM oceanbase.GV$OB_LOG_STAT WHERE TENANT_ID = user_tenant_id AND ROLE = 'LEADER';
   

   Replace user_tenant_id with the ID of the tenant of the log restore source.

   In the query result, if the ACCESS_MODE value of a log stream is not RAW_WRITE, the log stream may be waiting for garbage collection (GC). After GC is performed, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.

Error message 11

ERROR HY000: query primary failed(original error code: -2002), switchover to primary is not allowed

Possible causes

For a switchover to PRIMARY on a standby tenant in a network-based Physical Standby Database scenario, the current standby tenant must be able to connect to the tenant of the log restore source and have query privileges on necessary system views in the tenant.

If the OBServer node where the tenant of the log restore source resides is down or the network is disconnected, this error is reported.

Troubleshooting procedure

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the IP_LIST=xxxx attribute of the VALUE field displays the IP address and port of the OBServer node where the tenant of the log restore source resides.

2. Check the network connectivity of the OBServer node where the tenant of the log restore source resides or manually start the observer process on it. Then, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.

Error message 12

Possible causes

For a switchover to PRIMARY on a standby tenant in a network-based Physical Standby Database scenario, the current standby tenant must be able to connect to the tenant of the log restore source and have query privileges on necessary system views in the tenant.

Otherwise, this error is reported. The preceding error message indicates that the user connected to the log restore source (the dedicated user specified when the log restore source is configured) does not have query privileges on system views or some necessary system views.

Troubleshooting procedure

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the USER=xxx attribute of the VALUE field displays the information about the tenant of the log restore source. In this example, mytest is the dedicated username specified for the log restore source, that is, the username used to connect to the log restore source, and backup_mysql_tenant is the name of the tenant of the log restore source.

2. Use the dedicated user specified for the log restore source to log in to the current tenant.

3. Check the privileges of the current user and verify whether the current user has query privileges on necessary system views in the tenant of the log restore source.

   • For more information about how to view the privileges of the current user in a MySQL tenant, see View user privileges.

   • For more information about how to view the privileges of the current user in an Oracle tenant, see View user privileges.

   For more information about the user privileges required for a dedicated view access user in a network-based Physical Standby Database scenario, see Specify a log restore source. If the current user does not have the required privileges, grant the corresponding privileges to the current user.

4. After granting the corresponding privileges, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.

Error message 13

ERROR HY000: query primary failed(original error code: -1045), switchover to primary is not allowed

Possible causes

For a switchover to PRIMARY on a standby tenant in a network-based Physical Standby Database scenario, the current standby tenant must be able to connect to the tenant of the log restore source and have query privileges on necessary system views in the tenant.

Otherwise, this error is reported. The preceding error message indicates that the username or password of the account used to connect to the log restore source (the dedicated user specified for the log restore source) is incorrect.

Troubleshooting procedure

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the USER=xxx attribute of the VALUE field displays the information about the tenant of the log restore source. In this example, mytest is the dedicated username specified for the log restore source, that is, the username used to connect to the log restore source, and backup_mysql_tenant is the name of the tenant of the log restore source.

2. Check whether the username and password of the account used to connect to the log restore source are correct.

   If both the username and password are correct, contact OceanBase technical Support.

Error message 14

query primary failed(original error code: -xxxx), switchover to primary is not allowed

If the preceding error message is returned for a switchover to PRIMARY on a standby tenant of a network-based physical standby database, with an error code other than any of the error codes mentioned in this topic, use the following troubleshooting procedure:

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the IP_LIST=xxxx attribute of the VALUE field displays the IP address and port of the OBServer node where the tenant of the log restore source resides.

2. Check whether all OBServer nodes of the restore source tenant can provide services.

   If an OBServer node fails to provide services, wait a while until the OBServer node resumes services. Then, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.

   If all OBServer nodes can provide services, contact OceanBase Technical Support.

Error message 15

ERROR HY000: primary tenant's user account is locked, switchover to primary is not allowed

Possible causes

For a switchover to PRIMARY on a standby tenant of a network-based physical standby database, the user account for connecting to the log restore source (the dedicated user specified for the log restore source) must not be locked. If the user account is locked, this error is reported.

Troubleshooting procedure

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the USER=xxx attribute of the VALUE field displays the information about the tenant of the log restore source. In this example, mytest is the dedicated username specified for the log restore source, that is, the username used to connect to the log restore source, and backup_mysql_tenant is the name of the tenant of the log restore source.

2. Log in to the tenant of the log restore source as the administrator.

3. Execute the following statement to check whether the user connecting to the log restore source is locked.

   MySQL mode

   Here is an example:

   obclient [oceanbase]> SELECT user_name,is_locked FROM oceanbase.DBA_OB_USERS WHERE user_name='mysql_tenant';
   

   tab Oracle mode

   Here is an example:

   obclient [SYS]> SELECT user_name,is_locked FROM SYS.ALL_VIRTUAL_USER_REAL_AGENT WHERE user_name='oracle_tenant';
   

   :::

   In the query result, if the value of is_locked is YES, the user is locked.

4. Execute the following statement to unlock the user:

   obclient> ALTER USER user_name ACCOUNT UNLOCK;
   

   Replace user_name with the username of the account for connecting to the log restore source.

   For more information about how to lock or unlock a user, see Lock or unlock a user (MySQL mode) and Lock or unlock a user (Oracle mode).

5. After unlocking the user, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.

Error message 16

ERROR HY000: primary tenant is locked, switchover to primary is not allowed

Possible causes

For a switchover to PRIMARY on a standby tenant of a network-based physical standby database, the tenant of the log restore source must not be locked. If the tenant of the log restore source is locked, this error is reported.

Troubleshooting procedure

1. Obtain information about the tenant of the log restore source.

   1. Log in to the current standby tenant or to the sys tenant of the cluster where the standby tenant resides.

   2. Execute the following statement to obtain information about the tenant of the log restore source.

      • Query from the sys tenant

        obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LOG_RESTORE_SOURCE WHERE tenant_id = xxxx\G
        

        Replace tenant_id with the ID of the current standby tenant.

      • Query from the current user tenant

        MySQL mode

        Oracle mode

        Here is an example:

        obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LOG_RESTORE_SOURCE\G
        

        Here is an example:

        obclient [oracle]> SELECT * FROM SYS.DBA_OB_LOG_RESTORE_SOURCE\G
        

      A sample query result is as follows:

      *************************** 1. row ***************************
               TENANT_ID: 1004
                      ID: 1
                    TYPE: SERVICE
                   VALUE: IP_LIST=100.xx.xx.xxx:13359;100.xx.xxx.xxx:13361,USER=mytest@backup_mysql_tenant,PASSWORD=*********************************************,TENANT_ID=1002,CLUSTER_ID=1,COMPATIBILITY_MODE=MYSQL,IS_ENCRYPTED=true
      RECOVERY_UNTIL_SCN: 4611686018427387903
      1 row in set
      

      In the query result, the USER=xxx attribute of the VALUE field displays the information about the tenant of the log restore source. In this example, mytest is the dedicated username specified for the log restore source, that is, the username used to connect to the log restore source, and backup_mysql_tenant is the name of the tenant of the log restore source.

2. Log in to the sys tenant of the cluster where the tenant of the log restore source resides.

3. Execute the following statement to check whether the tenant is locked:

   obclient [oceanbase]> SELECT LOCKED FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_ID = user_tenant_id;
   

   Replace user_tenant_id with the ID of the tenant of the log restore source.

   In the query result, if the value of the LOCKED field is YES, the tenant is locked.

4. Execute the following statement to unlock the tenant:

   obclient [oceanbase]> ALTER TENANT tenant_name UNLOCK;
   

   Replace tenant_name with the name of the tenant of the log restore source.

   For more information about how to unlock a tenant, see Lock or unlock a tenant.

5. After unlocking the tenant, retry the switchover to PRIMARY on the standby tenant. If the retry fails all the same, contact OceanBase Technical Support.