OMS HA

The high available (HA) module of OceanBase Migration Service (OMS) ensures the stability and continuity of data transmission. When a server on which OMS is deployed or a task process fails, the HA module can promptly detect the failure and then take steps to restore the service.

Disaster recovery levels supported by the HA module

HA capabilities of components

The following table describes the HA capabilities of four OMS components.

HA triggering scenarios

From the perspective of HA, IDC-level disaster recovery is the same as server-level disaster recovery. An IDC-level disaster recovery involves the disaster recovery of multiple servers at the same time. The following table describes the two scenarios in which HA is triggered:

Server failure

Each OMS server runs an oms_drc_supervisor component, which works like an agent and periodically reports heartbeats to the OMS MetaDB. If a server fails to report a heartbeat in a period longer than the interval specified by the checkHostDownIntervalSec parameter during a background OMS inspection, OMS marks the server as failed.

HA behaviors after a server failure

1. OMS queries Incr-Sync tasks on the failed server and classifies the Incr-Sync threads into two categories, Running and Other, based on their status.
2. OMS attempts to terminate Incr-Sync tasks in the Running state. Usually, this operation cannot succeed. This is because a failed server cannot receive commands.
3. To ensure the global uniqueness of the Incr-Sync registration information in the MetaDB, regardless of whether the Incr-Sync tasks on the failed server were running, OMS recreates all these tasks on a healthy server where OMS is deployed in the same region and deletes the registration information about these tasks on the failed server.
4. For Incr-Sync tasks that were running on the failed server, OMS starts them on the new server. The status of Incr-Sync tasks that were not running remains unchanged after they are migrated to the new server.

Considerations

• When the HA module migrates an Incr-Sync task, all configurations of the task are copied.
• If the failed server is restored, you will have two Incr-Sync tasks performing incremental synchronization. When you query Incr-Sync processes on the restored server in the OMS console, you cannot get any result because the Incr-Sync registration information was deleted when HA recreated the Incr-Sync tasks on a new server.

Component exceptions

HA logic of the Store component

Multiple Store tasks may exist at the same data source to support downstream consumption. This section describes the complex HA logic, based on which the Store component runs.

The HA module checks the number and status of Store tasks at each data source and performs the following operations:

1. If no Store tasks exist, the HA module does not create one.
2. If at least one Store task exists and you manually stopped all Store tasks, the HA module does not create one. Scenarios 1 and 2 are usually the results of manual operations by an O&M engineer. To ensure the continuity of Store operations, the HA module does not create a Store task in those scenarios. To perform a large-scale O&M operation, we recommend that you disable the HA module of OMS in advance and enable it again after the O&M operation is completed.
3. If the number of Store tasks reaches the maximum value that is specified by the subtopicStoreNumberThreshold parameter, the HA module does not create more Store tasks. An excessive number of Store tasks is one of the most common causes of HA failure. This is to prevent the HA module from continuously creating Store tasks in some extreme cases, which depletes the server resources.
4. If no Store task is in the Running state (works normally and reports heartbeats at the specified interval), the HA module will attempt to create a Store task.
6. If the data held by running Store tasks cannot meet downstream consumption requirements and you have set the perceiveStoreClientCheckpoint parameter to true, the HA module will attempt to create a new Store task. For example, when a Store task holds only data of the past 8 hours, but the downstream consumers need data of the past 10 hours, the HA module attempts to create a new Store task.

When the HA module creates a Store task, it must determine the start timestamp, which is a unix timestamp, of the Store task. The start timestamp is the point in time from which the new Store task starts to pull the database logs, The start timestamp is determined based on the following rules:

1. If you set the perceiveStoreClientCheckpoint parameter to false, the HA module does not detect the downstream consumption timestamp. In this case, the start timestamp = ${Current time} - refetchStoreIntervalMin.
2. If you set the perceiveStoreClientCheckpoint parameter to true, the HA module checks the earliest consumption timestamp of all downstream consumers. In this case, the start timestamp = ${The earliest downstream consumption timestamp} - refetchStoreIntervalMin.

Considerations

• OMS does not support Store sharing between migration projects. If you have created multiple migration projects based on the same source database ID, the HA module creates, manages, and maintains a Store task for each project.
• OMS supports Store sharing between synchronization projects. If you have created multiple synchronization projects based on the same source database ID and enabled Store sharing, the HA module creates and maintains the shared Store task for all those synchronization projects.
• If you have set the perceiveStoreClientCheckpoint parameter to true, but the HA module does not detect a downstream consumer, the HA module cannot obtain a downstream consumption timestamp. In this case, the HA module does nothing.

HA parameters

The following table describes the system parameters that control the behavior of the HA module. To view and edit the parameters, log on to the OMS console, choose System Management > System Parameters, and then search for ha.config on the page that appears.