Restore other clusters after a lossy failover

After a lossy failover, data is inconsistent between the new primary cluster and the original primary cluster and between the new primary cluster and other standby clusters. In this case, you cannot connect the original primary cluster and other standby clusters to the new primary cluster. You need to rebuild the primary/standby relationship.

Procedure

1. Delete the original primary cluster or other standby clusters from the new primary cluster, and clear the data of the original primary cluster or other standby clusters.

   The following two methods are available:

   • After OceanBase Cloud Platform (OCP) is recovered, delete the original primary cluster from OCP, and OCP will automatically clear the data of the cluster.

     For more information, see Delete a standby cluster.

   • If OCP is unavailable, you can execute statements to delete clusters, and then manually clear the processes and data of the clusters.

     1. In the new primary cluster, execute the following statement to query information about other clusters:

        obclient> SELECT CLUSTER_ID, CLUSTER_ROLE, CLUSTER_STATUS FROM V$OB_STANDBY_STATUS;
        +------------+------------------+----------------+
        | CLUSTER_ID | CLUSTER_ROLE     | CLUSTER_STATUS |
        +------------+------------------+----------------+
        |          1 | PHYSICAL STANDBY |       DISABLED |
        +------------+------------------+----------------+
        1 row in set
        

     2. In the new primary cluster, execute the following statement to delete information about other clusters:

        obclient> ALTER SYSTEM REMOVE CLUSTER obcluster CLUSTER_ID = 1;
        Query OK, 0 rows affected
        

     3. Manually clear the processes and data of the other clusters. You can kill cluster processes directly.

2. Create standby clusters on the OBServer nodes of other clusters and rebuild the primary/standby relationship.

   Notice

   After a lossy failover, the system automatically initiates a major compaction. You need to wait for the major compaction to complete and then create a standby cluster. For more information about how to view the major compaction status, see View the major compaction process.

   You can add a standby cluster on OCP, and OCP will automatically allocate a globally unique cluster ID (cluster_id) to the cluster. For more information, see Deploy a standby cluster in OCP.

3. After the new standby cluster is synchronized in real time, switch the primary and standby roles. We recommend that you choose and switch a standby cluster with smaller synchronization latency to the new primary role.

   For more information, see Switchover.

   You can execute the following statement on the new primary cluster to view the synchronization status of the new standby clusters:

   obclient> SELECT SYNCHRONIZATION_STATUS FROM V$OB_STANDBY_STATUS WHERE CLUSTER_ID = standby_cluster_id;
   

   You can also query the SYNCHRONIZATION_STATUS field in the V$OB_STANDBY_STATUS view to check the synchronization status of the standby clusters. If the returned value of the SYNCHRONIZATION_STATUS field is OK, the standby clusters are synchronized to the new primary cluster.