Migration, replication, and rebuild tasks|V4.3.0| docs|Distributed Database

Migration, replication, and rebuild tasks

Last Updated：2024-04-19 08:42:49 Updated

You can check the migration, replication, and rebuild tasks scheduled by RootService by querying the oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY view.

obclient>
select TIMESTAMP,MODULE,EVENT,VALUE1 tenant_id,VALUE2 ls_id,NAME3,VALUE3,NAME4,VALUE4,NAME5,VALUE5,NAME6,VALUE6 from DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%disaster%';
+----------------------------+-------------------+-----------------------+-----------+-------+---------+-----------------------------------+-------------+------------------+-------------+-------------------+----------------+-----------------------------------------+
| TIMESTAMP                  | MODULE            | EVENT                 | tenant_id | ls_id | NAME3   | VALUE3                            | NAME4       | VALUE4           | NAME5       | VALUE5            | NAME6          | VALUE6                                  |
+----------------------------+-------------------+-----------------------+-----------+-------+---------+-----------------------------------+-------------+------------------+-------------+-------------------+----------------+-----------------------------------------+
| 2023-06-07 16:12:58.254734 | disaster_recovery | start_add_ls_replica  | 1         | 1     | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | destination | "x.x.xx.43:2882" | data_source | "x.x.xx.106:2882" | comment        | add paxos replica according to locality |
| 2023-06-07 16:14:04.202678 | disaster_recovery | finish_add_ls_replica | 1         | 1     | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | destination | "x.x.xx.43:2882" | data_source | "x.x.xx.106:2882" | execute_result | ret:0; elapsed:65947960;                |
+----------------------------+-------------------+-----------------------+-----------+-------+---------+-----------------------------------+-------------+------------------+-------------+-------------------+----------------+-----------------------------------------+
2 rows in set (0.04 sec)

The preceding table displays information about the tasks that have been scheduled by RootService. You need to pay attention to the key fields such as event, source, and destination.

You can view the execution result of a task in the value6 field. In the execution result, ret indicates whether the task succeeded, and elapsed indicates the execution duration of the task. If the task fails, a ret_comment field may be displayed to indicate the failed step.

The oceanbase.DBA_OB_SERVER_EVENT_HISTORY view allows you to check whether the tasks scheduled by RootService are scheduled and executed on the corresponding nodes. The following sample code displays the query results of a task whose task_id is YB4206013B6A-0005FC571364EFE0-0-0:

obclient>
select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE5='YB4206013B6A-0005FC571364EFE0-0-0';
+----------------------------+-----------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+-------------------+-------+------------------+---------+-----------------------------------+-------------------+--------+------------+
| TIMESTAMP                  | SVR_IP    | SVR_PORT | MODULE     | EVENT                           | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3            | NAME4 | VALUE4           | NAME5   | VALUE5                            | NAME6             | VALUE6 | EXTRA_INFO |
+----------------------------+-----------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+-------------------+-------+------------------+---------+-----------------------------------+-------------------+--------+------------+
| 2023-06-07 16:12:58.324433 | x.x.xx.43 |     2882 | storage_ha | ls_ha_start                     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:12:58.325473 | x.x.xx.43 |     2882 | storage_ha | initial_migration_task          | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:12:58.616840 | x.x.xx.43 |     2882 | storage_ha | start_migration_task            | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | data_tablet_count | 830    | ADD_LS_OP  |
| 2023-06-07 16:12:58.642320 | x.x.xx.43 |     2882 | storage_ha | sys_tablets_migration_task      | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:13:03.754195 | x.x.xx.43 |     2882 | storage_ha | data_tablets_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:13:04.348244 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.360507 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.382052 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.422866 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.425482 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.426223 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.432973 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.433154 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 101    | ADD_LS_OP  |
| 2023-06-07 16:13:04.519100 | x.x.xx.43 |     2882 | storage_ha | tablet_group_migration_task     | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | tablet_count      | 22     | ADD_LS_OP  |
| 2023-06-07 16:13:20.152015 | x.x.xx.43 |     2882 | storage_ha | migration_finish_task           | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:13:20.152624 | x.x.xx.43 |     2882 | storage_ha | initial_complete_migration_task | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:14:04.198183 | x.x.xx.43 |     2882 | storage_ha | start_complete_migration_task   | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:14:04.198518 | x.x.xx.43 |     2882 | storage_ha | finish_complete_migration_task  | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
| 2023-06-07 16:14:04.203593 | x.x.xx.43 |     2882 | storage_ha | ls_ha_finish                    | tenant_id | 1      | ls_id | 1      | src   | "x.x.xx.106:2882" | dst   | "x.x.xx.43:2882" | task_id | YB4206013B6A-0005FC571364EFE0-0-0 | is_failed         | 0      | ADD_LS_OP  |
+----------------------------+-----------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+-------------------+-------+------------------+---------+-----------------------------------+-------------------+--------+------------+
19 rows in set (0.40 sec)

The following example describes how to check whether a task is stuck by querying the oceanbase.DBA_OB_SERVER_EVENT_HISTORY view.

Execute the following statement to obtain the lists of tenant_id and ls_id of tasks that have been scheduled by RootService but have not been completed:

(select value1, value2 from DBA_OB_ROOTSERVICE_EVENT_HISTORY where event like '%add_ls%') except (select value1, value2 from DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and event like '%finish_complete%');

Execute the following statement to obtain and analyze the migration and replication tasks that have reported errors:
```
select * from DBA_OB_SERVER_EVENT_HISTORY where event like '%migrat%' and name6 like '%fail%' and value6=1;
```

The DBA_OB_LS_REPLICA_TASKS view records all running migration, replication, and rebuild tasks. To confirm a suspicious exception of a task, you can use its task_id value to search for the logs of the task during the corresponding period.

Troubleshoot the error reported by a historical task

You can execute the following SQL statement to view the execution history of migration, replication, and rebuild tasks in a log stream:

select * from DBA_OB_ROOTSERVICE_EVENT_HISTORY
where module like "%disaster%" and value1 = $TENANT_ID and value2 = $LS_ID;

The name6 and value6 fields of a finished event record the execution results of the task. The value 0 of ret indicates that the task was executed, and other values are the error codes for the task. elapsed indicates the execution duration of the task. ret_comment describes the cause of the failure that occurred. If no failure cause is displayed, you can use the corresponding task_id value to search for relevant RootService logs.

grep "$TASK_ID" rootservice.log*