Correct the data after a failover

After you perform a failover, the new primary cluster may lose some data compared with the original one regardless of the lossy or lossless nature of the failover. OceanBase Database provides a read-only mode for the cluster. This mode allows you to correct the data for the new and original primary clusters.

Background

A cluster in read-only mode is in the DISABLED WITH READ ONLY state. Comparison between a cluster in this state and a cluster in the DISABLED state:

• The log transfer feature is disabled for clusters in both states.

• Only the original or new primary cluster can be set to the DISABLED WITH READ ONLY state. A standby cluster cannot be set to the DISABLED WITH READ ONLY state even if it starts in read-only mode.

• A cluster in the DISABLED WITH READ ONLY state supports strong consistency reads for user tenants, in which the logs of different partitions will change. For a cluster in the DISABLED state, logs of user tenants do not change. Therefore, it can be connected to the new primary cluster after a lossless failover. For more information, see Connect the original primary cluster after a lossless failover.

The read-only mode has the following features and limitations:

• Only the PRIMARY cluster can be set to the read-only mode.

• Strong and weak consistency reads are supported for user tenants in a cluster under the read-only mode.

• Writes and DDL operations are not supported for user tenants in a cluster under the read-only mode.

• You cannot create a tenant in a cluster under the read-only mode.

• When no VALID primary cluster is available or exists, you can execute the failover statement to switch a read-only cluster to the VALID state.

Procedure

1. Specify the special parameter -m disabled_with_readonly_cluster on all OBServer nodes in the original primary cluster to start the observer process.

   The read-only mode requires that all OBServer nodes in the original primary cluster must be started with the special parameter. After all OBServer nodes are started, the cluster enters the DISABLED WITH READ ONLY state. The state persists in the cluster, so you do not need to specify the special parameter again when you restart an OBServer node later.

   bin/observer -m disabled_with_readonly_cluster
   

2. After you start the original primary cluster, execute the following statement to check whether the cluster is in the DISABLED WITH READ ONLY state.

   obclient> SELECT CLUSTER_ROLE, CLUSTER_STATUS FROM V$OB_CLUSTER;
   +--------------+-------------------------+
   | CLUSTER_ROLE | CLUSTER_STATUS          |
   +--------------+-------------------------+
   | PRIMARY      | DISABLED WITH READ ONLY |
   +--------------+-------------------------+
   1 row in set
   

   After the original primary cluster is started in read-only mode, it isolates itself to disable data synchronization with the new primary cluster or other standby clusters, which ensures data security. A user tenant in the cluster only supports read operations. It does not support data writes or DDL operations. This ensures that the data is frozen in the snapshot version before the failover.

3. Log on to the user tenant of the cluster and correct the data for the new and original primary clusters.

   OceanBase Database enables strong consistency read by default, so you can correct the data directly. You can perform the following steps to set strong consistency read on the new primary cluster and the original primary cluster:

   • Set the session variable ob_read_consistency.

     obclient> SET ob_read_consistency =STRONG;
     

   • Specify a hint in an SQL statement.

     obclient> SELECT /*+read_consistency(strong) */  FROM table_name;