Failover

When the primary tenant becomes unavailable, you can switch a standby tenant to the PRIMARY role to provide services.

Limitations

• When you perform a failover operation, all log stream replicas of the standby tenant must be online. Otherwise, you must wait until the offline replicas are permanently offline.

  You can log in to the standby tenant or to the sys tenant of the cluster where the standby tenant resides and query the DBA_OB_LS or CDB_OB_LS view to check whether all log stream replicas are online. For more information about log stream replicas, see Overview of replicas.

• The failover operation does not check the status of the primary tenant. Therefore, in addition to disaster recovery upon a failure of the primary tenant, the failover operation also applies to a snapshot of the primary tenant created at a specific point in time for subsequent business verification.

• You cannot execute the failover statement in a session created based on a service name. For more information about services, see Create a service.

  Notice

  If the tenant where the failover statement is to be executed has a service name, the system automatically deletes the service name of the tenant.

Background information

The failover operation modifies log files but not data files.

Each OceanBase Database tenant has multiple log streams, and the failover operation must ensure data consistency. Therefore, the system selects a timestamp with the smallest system change number (SCN) from the synchronization timestamps of all log streams as the failover timestamp. After the failover, all log streams of the tenant are rolled back to the timestamp.

Procedure

1. Log in to the standby tenant or to the sys tenant of the cluster where the standby tenant resides as the administrator.

2. Execute the ACTIVATE STANDBY VERIFY statement to verify whether the ACTIVATE STANDBY statement can succeed.

   • Execute the ACTIVATE STANDBY VERIFY statement in the sys tenant of the cluster where the standby tenant resides

     ALTER SYSTEM ACTIVATE STANDBY TENANT [=] tenant_name VERIFY;
     

     Here is an example:

     ALTER SYSTEM ACTIVATE STANDBY TENANT = mysql VERIFY;
     

   • Execute the ACTIVATE STANDBY VERIFY statement in the standby tenant

     ALTER SYSTEM ACTIVATE STANDBY VERIFY;
     

     After the statement is executed, if OK is returned, the verification succeeds. In this case, you can proceed to the next step.

     If an error is returned, perform troubleshooting based on the error message by referring to Switchover and failover issues. Then, execute the statement again.

3. After the verification succeeds, execute the ACTIVATE STANDBY statement to switch the standby tenant to the PRIMARY role.

   • Execute the following statement in the sys tenant of the cluster where the standby tenant resides to switch the standby tenant to the PRIMARY role:

     ALTER SYSTEM ACTIVATE STANDBY TENANT = tenant_name;
     

   • Execute the following statement in the standby tenant to switch it to the PRIMARY role:

     ALTER SYSTEM ACTIVATE STANDBY;
     

4. Query the DBA_OB_TENANTS view to check whether the original standby tenant is switched to the PRIMARY role.

   • Query the view from the sys tenant of the cluster where the standby tenant resides

     obclient [oceanbase]> SELECT TENANT_NAME, TENANT_TYPE, TENANT_ROLE, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS;
     

   • Query the view from the standby tenant

     MySQL mode:

     obclient [oceanbase]> SELECT TENANT_NAME, TENANT_TYPE, TENANT_ROLE, SWITCHOVER_STATUS FROM oceanbase.DBA_OB_TENANTS;
     

     Oracle mode:

     obclient [SYS]> SELECT TENANT_NAME, TENANT_TYPE, TENANT_ROLE, SWITCHOVER_STATUS FROM SYS.DBA_OB_TENANTS;
     

   A sample query result is as follows:

   +-----------------+-------------+-------------+-------------------+
   | TENANT_NAME     | TENANT_TYPE | TENANT_ROLE | SWITCHOVER_STATUS |
   +-----------------+-------------+-------------+-------------------+
   | standby_tenant  | USER        | PRIMARY     | NORMAL            |
   +-----------------+-------------+-------------+-------------------+
   1 row in set
   

   In the query result, the value of TENANT_ROLE is PRIMARY and that of SWITCHOVER_STATUS is NORMAL, which indicates that the standby tenant is switched to the PRIMARY role.

Considerations

• When executing the Failover command to switch a tenant to a primary tenant, the system will automatically delete the service name associated with that tenant if it has one. However, the service name of the original primary tenant will not be deleted.

  For example, under the service name service1, there is a primary tenant tenantA and its corresponding standby tenant tenantB. If the Failover command is executed on tenantB to switch it to a primary tenant, the system will delete the service name service1 associated with tenantB, as it is no longer part of the original primary/standby relationship. Meanwhile, the service name service1 associated with tenantA will be retained.

• The failover operation restores the synchronized data of all log streams to the consistency timestamp. This ensures that the data of all log streams before the timestamp is complete. Therefore, after a failover operation is performed, the following rules apply:

  • The original primary tenant cannot be a standby tenant of the new primary tenant.

  • If the logs of a third-party standby tenant are synchronized to a timestamp that exceeds the consistency timestamp, you cannot use the third-party standby tenant as a standby tenant of the new primary tenant. For ease of management, we recommend that you do not use a third-party standby tenant as a standby tenant of the new primary tenant.