Perform failover using standard O&M procedures

This topic describes how to perform a failover for an OceanBase database by using the standard O&M procedures (SOP) of OCP.

Application scenarios

In the daily operation and maintenance of OceanBase Database, some fault recovery operations involve multiple steps with dependencies. Manual execution can easily lead to omissions or sequence errors. The Standard Operation Procedure (SOP) in OCP orchestrates these complex operations into standardized process templates, supporting a combination of automatic execution and manual confirmation, helping you complete fault recovery safely and in compliance.

SOPs are applicable to the following typical scenarios:

• Recover data from an abnormal standby tenant: When standby tenant data is corrupted or synchronization is abnormal, and the standby tenant needs to be rebuilt from primary tenant data, you can use the "Rebuild standby tenant" procedure.

• Enable arbitration service upon replica failure: When a replica in the cluster fails and the availability of the tenant needs to be guaranteed through the arbitration service, you can use the "Enable arbitration service upon replica exception" procedure.

• Recover from an OBServer node failure: When an OBServer node fails and cannot be recovered, and the node needs to be replaced and data rebuilt, you can use the "Rebuild data after OBServer exception" procedure.

Prerequisites

• Ensure the current OCP user has the following permissions:

  • Any resource permission for Cluster Management, Tenant Management, or Host Management (the target resource for the operation must have Management or Maintenance permission).
  • Permission to access the Standard Procedures menu in Operations Management.

• The SOP feature is only available in OCP Enterprise Edition; OCP Community Edition does not provide this feature.

• Different SOP types also require meeting the following conditions:

  SOP Type	Prerequisite
  Rebuild standby tenant	The primary tenant is available and its data is intact; the standby tenant to be rebuilt is the downstreammost one in the log synchronization chain; the cluster to which the standby tenant belongs is in a normal state; if the new standby tenant is to be built based on the archive mode, and "Back up primary tenant" is to be executed in the SOP task, the primary tenant must have a backup strategy configured.
  Enable arbitration service upon replica exception	The target cluster is in a normal state; there is an available arbitration service or arbitration service group.
  Rebuild data after OBServer exception	The target cluster is in a normal state; there are other available OBServer nodes in the cluster; it is recommended to prepare a replacement host in advance.

Procedure

Rebuild a standby tenant

When a standby tenant's data is abnormal or synchronization is interrupted and cannot be restored, you can rebuild the standby tenant by following the SOP.

1. Log in to OCP.

2. In the left navigation bar, click O&M > Standard workflow.

3. Click Create. In the Create task dialog box, configure the following information:

   1. Task Name: Enter a task name (2 to 32 characters, starting with a Chinese, English, or Japanese kana character, ending with a Chinese, English, number, or Japanese kana character, and can contain underscores).

   2. Task Type: Select Rebuild standby tenant.

   3. Remarks (Optional): Enter task remarks.

   4. Execute Now: Choose whether to execute immediately after creation as needed. It is recommended not to select this option for the first operation. Confirm the step configuration first, then start manually.

   5. Click Create.

4. On the step configuration page, confirm and adjust the auto-execution switch for each step:

   Step	Description	Recommended Configuration
   Confirm Operation Object	Confirm the information of the standby tenant and primary tenant to be rebuilt	Automatic Execution
   Backup Primary Tenant	Perform data backup for the primary tenant	Disable Auto Execution, execute after manual confirmation
   Decouple Primary/Standby	Disconnect the synchronization relationship between the standby tenant and primary tenant	Automatic Execution
   Lock Original Standby Tenant	Lock the standby tenant to prevent new writes	Automatic Execution
   Delete Original Standby Tenant	Delete the original standby tenant	Automatic Execution
   Create New Standby Tenant	Create a new standby tenant. Creation parameters must be configured before this step	Disable Auto Execution, execute after manual confirmation
   Verify Primary/Standby Synchronization	Verify the synchronization status between the new standby tenant and primary tenant	Automatic Execution

   Note

   The "Create Standby Tenant" step requires setting the createStandbyTenantParamJson parameters before execution, including the new standby tenant name, compatibility mode, Zone replica configuration, etc. It is recommended to set this step to manual confirmation and proceed after the parameters are configured.

5. Click Start to begin execution.

6. Before executing the "Create Standby Tenant" step, update its parameters on the execution details page:

   • Click Edit on the right side of the step. In the dialog box that appears, configure the createStandbyTenantParamJson parameters, including the new standby tenant name, primary tenant ID, standby cluster ID, primary/standby synchronization method, Zone replica configuration, and resource pool specifications.

7. After confirming the parameters are correct, click Execute Next to proceed to the "Create Standby Tenant" step.

8. Subsequent steps are automatically executed until "Verify Primary/Standby Synchronization" is completed.

Enable arbitration service for abnormal replicas

Use this procedure to enable the arbitration service for a tenant when a replica in the cluster fails and tenant availability needs to be guaranteed.

1. Log in to OCP.

2. In the left-side navigation pane, click O&M > Standard workflow.

3. Click Create. In the Create standard O&M workflow dialog box, configure the following information:

   1. Task Name: Enter a task name.
   2. Task Type: Select Enable arbitration service after a replica failure.
   3. Select the target cluster and choose either an arbitration service or an arbitration service group (choose one). When you select an arbitration service group, the system automatically selects the associated arbitration service based on the priority within the service group.
   4. Click Create.

4. After confirming the step configurations, click Start to begin execution.

   This process includes steps such as confirming the operation object, adding an arbitration service, deleting tenant replicas, deleting OBServer nodes, enabling tenant arbitration service, and verifying the availability of the tenant arbitration service. All steps are automatically executed by default, but you can adjust them as needed.

Data reconstruction after OBServer failure

When an OBServer node fails and cannot be recovered, you can use this procedure to replace the node and reconstruct the data.

1. Log in to OCP.

2. Identify the failed OBServer node and an available replacement host.

3. In the left-side navigation pane, click O&M > Standard workflow.

4. Click Create. In the Create standard O&M workflow dialog box, configure the following information:

   1. Task Name: Enter a task name.
   2. Task Type: Select Data Reconstruction after OBServer Failure.
   3. Select the target cluster. If a replacement host is preselected, you can include it here; otherwise, you can set it up before the "Add OBServer" step.
   4. Click Create.

5. After confirming the step configurations, click Start to begin execution.

   Step	Description	Recommended configuration
   Confirm Operation Object	Confirm information about the failed OBServer	Automatically executed
   Delete Tenant Replica	Delete tenant replicas from the failed OBServer	Automatically executed
   Delete OBServer	Remove the failed OBServer node from the cluster	Automatically executed
   Force Delete Host (Optional)	Permanently remove the failed host from OCP. This step is optional and can be enabled when creating or editing a task.	It is recommended to disable automatic execution and proceed after manual confirmation.
   Add OBServer	Add a new OBServer node. You need to specify a replacement host.	Disable automatic execution and proceed after configuring the replacement host.
   Add Tenant Replica	Add tenant replicas to the new OBServer	Automatically executed
   Verify Successful Replica Reconstruction	Verify whether replica reconstruction was successful	Automatically executed

   Note

   "Force Delete Host" is an optional step that can be enabled when creating or editing a task. This step permanently removes the failed host record from OCP. The operation is irreversible. After enabling it, it is recommended to disable automatic execution and proceed with manual confirmation. The "Add OBServer" step requires selecting a replacement host. Ensure the replacement host has been added to OCP and is in a normal state.

6. Before you perform the "Add OBServer" step, update its parameters on the execution details page:

   • Click Edit on the right side of the step and select an alternative host. Ensure that the alternative host has been added to OCP and is in a normal state.

7. After confirming the parameters, click Execute Next to proceed to the "Add OBServer" step.

8. The subsequent steps are executed automatically until "Verify Tenant Replica Reconstruction Successful" is completed.

FAQ

Q: What should I do if a step fails during the execution of an SOP task?

A: When a step fails, the task status changes to Failed. You can view the error information for the failed step on the execution details page. After troubleshooting based on the error cause:

• To retry only a single failed step, click Retry for that step.
• To retry all failed steps, click Retry All Failed Steps on the page.

Q: Can I modify the execution order of steps in an SOP task?

A: When creating or editing a task, you can move steps marked as userMovable=true within the template to allowed sequence numbers. However, the step identifier (stepKey) and steps with fixed enablement cannot be changed.

Q: Can I edit an SOP task after it starts?

A: Once a task starts, it cannot be edited. Tasks in the Not Started (Pending Execution) state can have their name, target parameters, step configuration, and remarks edited. If you need to modify the auto-execution switch or step parameters for unexecuted steps in a started task, you can use the step update feature.

Q: What is the difference between network-based and log-backup-based primary-standby synchronization methods in the "Rebuild Standby Tenant" process?

A:

• Network-based (NETWORK): The primary and standby tenants directly synchronize logs over the network, resulting in lower latency, but requires network connectivity between the primary and standby clusters.
• Log-backup-based (LOG_BACKUP): The standby tenant pulls archived logs from backup storage for replay, which does not require direct network connectivity, but typically results in higher synchronization latency.

Q: Can I delete an SOP task after it is completed?

A: Yes. Tasks in the Finished or Failed state can be deleted. Tasks in the Pending Execution state can also be deleted directly. However, tasks in the Pending Confirmation or Executing state cannot be deleted. Deleting a task will cascade clean up the execution instances and step execution records.

References

• Overview of Standard O&M Processes
• Create a Standard O&M Process
• View Standard O&M Process Execution Details
• Handle Host Failures in OceanBase Clusters
• Achieve High Availability Using the Arbitration Service