This topic describes how to perform data validation after a data migration task is completed.
Procedure
Go to the page where you create a data validation task.
Log in to the OMS console.
In the left-side navigation pane, click Data Validation.
On the Data Validation page, click Create Task in the upper-right corner.
Enter a custom task name in the task name field.
We recommend that you use a combination of Chinese characters, numbers, and letters. The name must not contain spaces and must be less than 64 characters in length.
On the Select Source and Target step, configure the parameters.
Parameter Description Source Select a data source from the drop-down list. For more information, see Create a data source. Target Select a data source from the drop-down list. Tag (optional) Select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data validation tasks by using tags. Click Next. On the Select Type & Objects step, select the validation type of the current data validation task in the Validation Type section.
At present, the following validation types are supported: Full Validation, Row Count Validation, and Incremental Validation. For more information, see Overview of data validation.
In the Validation Objects section, select the object to be verified for the current data validation task.
You can select the validation object by using Specify Objects and Match by Rule. This topic describes how to select the validation object by using Specify Objects. For more information about how to configure the matching rules, see Configure the matching rules for a validation object.
In the Validation Objects section, select Specify Objects.
Select the object to be verified in the Source Object(s) list.
Note
At present, you can select only a table as the validation object.
Click > to add the object to the Target Object(s) list.
Data validation supports importing objects by using text and supports renaming, setting row filtering, selecting partitions and columns, and removing one or more validation objects from the target.
Action Description Import objects In the right-side list of the selection area, click Import Objects. For more information, see Import a validation object. Rename objects Data validation supports renaming the name of a validation object. For more information, see Rename a database table. Configure settings Full Validation and Incremental Validation support WHEREconditions for row filtering. For more information, see Filter validation objects by using SQL conditions. You can also select the columns to be verified in the Select Columns section. If you set Validation Type to Full Validation, you can also obtain data of a specified partition when validating between a MySQL database and the MySQL compatible mode of OceanBase database, an Oracle database and the Oracle compatible mode of OceanBase database, or between two OceanBase databases.Remove one or all objects Data validation supports removing one or more objects temporarily selected for the target during data mapping. - To remove one object
In the right-side list of the selection area, click the Remove icon next to the target object to remove the object. - To remove all objects
In the right-side list of the selection area, click Remove All. In the dialog box that appears, click OK to remove all objects.
- To remove one object
Click Next. In the Validation Options step, configure the parameters.
Full validation
The following parameters are displayed only if you select Full Validation in the Select Type & Objects step.
Parameter Description Source Concurrency/Target Concurrency The minimum value is 1, and the maximum value is 512. The default value is 4. Validation Mode The following two modes are supported: Adaptive Mode and IN Mode. - Adaptive Mode: data is obtained in batches from the source and target, and the data is compared precisely.
- IN Mode: data is obtained in batches from the source. Data is queried in the target based on the unique key of the target data. Data consistency is verified between the two ends. Note that this mode cannot verify data that exists in the target but not in the source. The validation performance is also degraded. We recommend that you use this mode only when the target has redundant data that is expected and does not require validation, or when the migration scenario involves multi-table aggregation.
Limit BPS The unit is MiB/s, and the maximum value is 1024. If you enable this option, you must configure at least one minimum rate for the source or target. Hidden Column Based Validation During data migration, hidden columns are added and unique keys are created for tables without non-null unique keys in the target to ensure data consistency. Data migration can verify data based on the unique key. For more information, see Hidden column mechanism of data migration service.
This parameter is displayed only when the migration is OB_MySQL to OB_MySQL, OB_Oracle to OB_Oracle/Oracle, or Oracle to OB_MySQL/OB_Oracle.Row count validation
The following parameters are displayed only if you select Row Count Validation in the Select Type & Objects step.
Parameter Description Concurrent Tables for Validation The minimum value is 1, and the maximum value is 100. The default value is 10. This configuration applies to both the source and the target. Query Timeout The unit is minutes. The minimum value is 1, and the maximum value is 1440. The default value is 10. This configuration applies to both the source and the target. Incremental validation
The following parameters are displayed only if you select Incremental Validation in the Select Type & Objects step.
Parameter Description Validation Start Point The start timestamp for incremental validation. By default, the task start time is used. Only incremental changes after the incremental validation start time are verified. Validation End Point The end timestamp for incremental validation. You can set any time after the start time. When incremental change validation for the source and target reaches the specified timestamp, validation is completed. Initial Check Configuration Data is verified based on incremental changes in the source and target. If initial validation finds inconsistent data, validation is re-initiated. Initial check configuration includes: - Hot Data Cooldown Period: specifies the time to wait for the last change to be verified when the same data is frequently changed in the source. The unit is seconds, with a minimum value of 1 and a maximum value of 3600. The default value is 5.
- Mismatch Detection Delay: specifies the time to wait for the data to be verified again after incremental data in the source and target is inconsistent. If the data is still inconsistent, inconsistent data is reported.
The unit is seconds, with a minimum value of 1 and a maximum value of 86400. The default value is 30.
Recheck Configuration If initial validation finds inconsistent data, validation is re-initiated. The latest data is obtained from the source and target databases for comparison. If the data is still inconsistent after repeated validation, the validation task is completed, and inconsistent data is reported. Recheck configuration includes: - Recheck Concurrency: specifies the maximum number of concurrent data queries to the source and target for recheck. The minimum value is 1, and the maximum value is 512. The default value is 16.
- Recheck Timeout: specifies the maximum time for recheck. If the data is still inconsistent after the specified time, the validation task is completed, and inconsistent data is reported. The unit is minutes, with a minimum value of 1 and a maximum value of 720. The default value is 30.
- Limit BPS: specifies the maximum traffic for data queries to the source and target in MiB/s. The maximum value is 1024.
Hidden Column Based Validation During data migration, hidden columns are added and unique keys are created for tables without non-null unique keys in the target to ensure data consistency. Data migration can verify data based on the unique key. For more information, see Hidden column mechanism of data migration service.
This parameter is displayed only when the migration is OB_MySQL to OB_MySQL, OB_Oracle to OB_Oracle/Oracle, or Oracle to OB_MySQL/OB_Oracle.
Click Pre-check to perform a precheck on the data validation task.
After the precheck succeeds, click Start Task.
If you do not want to start the task, click Save to go to the details page of the data validation task. You can manually start the task as needed. For more information, see View details of a data validation task.
Go to the page where you configure a data validation task.
Log in to the OMS console.
In the left-side navigation pane, click Data Validation.
On the Data Validation page, click ... > Configure Validation Task next to the task name of the data validation task.
You can also click the task name of the data validation task on the Data Validation page to go to the details page, and click Configure Validation Task in the upper-right corner.
Enter a custom task name in the task name field. You can also use the default task name, which is in the format of
Data Migration Task Name-verify.We recommend that you use a combination of Chinese characters, numbers, and letters. The name must not contain spaces and must be less than 64 characters in length.
On the Select Source and Target step, confirm that the parameters are correct, and click Next.
On the Select Type & Objects step, select the validation type of the current data validation task in the Validation Type section.
At present, the following validation types are supported: Full Validation, Row Count Validation, and Incremental Validation. For more information, see Overview of data validation.
In the Validation Objects section, select the object to be verified for the current data validation task.
You can select the validation object by using Specify Objects or Match by Rule. You can keep the object selection method of the data migration task or change it. This topic describes how to select the validation object by using Specify Objects. For more information about how to configure the matching rules, see Configure the matching rules for a validation object.
In the Validation Objects section, select Specify Objects.
Select the object to be verified in the Source Object(s) list.
Note
At present, you can select only a table as the validation object.
You can select the same migration object as in the data migration task or select a different one based on the actual situation.
Click > to add the object to the Target Object(s) list.
Data validation supports importing objects by using text and supports renaming, setting row filtering, selecting partitions and columns, and removing one or more validation objects from the target.
Action Description Import objects In the right-side list of the selection area, click Import Objects. For more information, see Import a validation object. Rename objects Data validation supports renaming the name of a validation object. For more information, see Rename a database table. Configure settings Full Validation and Incremental Validation support WHEREconditions for row filtering. For more information, see Filter validation objects by using SQL conditions. You can also select the columns to be verified in the Select Columns section. If you set Validation Type to Full Validation, you can also obtain data of a specified partition.Remove one or all objects Data validation supports removing one or more objects temporarily selected for the target during data mapping. - To remove one object
In the right-side list of the selection area, click the Remove icon next to the target object to remove the object. - To remove all objects
In the right-side list of the selection area, click Remove All. In the dialog box that appears, click OK to remove all objects.
- To remove one object
Click Next. In the Validation Options step, configure the parameters.
Full validation
The following parameters are displayed only if you select Full Validation in the Select Type & Objects step.
Parameter Description Source Concurrency/Target Concurrency The minimum value is 1, and the maximum value is 512. The default value is 4. Validation Mode The following two modes are supported: Adaptive Mode and IN Mode. - Adaptive Mode: data is obtained in batches from the source and target, and the data is compared precisely.
- IN Mode: data is obtained in batches from the source. Data is queried in the target based on the unique key of the target data. Data consistency is verified between the two ends. Note that this mode cannot verify data that exists in the target but not in the source. The validation performance is also degraded. We recommend that you use this mode only when the target has redundant data that is expected and does not require validation, or when the migration scenario involves multi-table aggregation.
Limit BPS The unit is MiB/s, and the maximum value is 1024. If you enable this option, you must configure at least one minimum rate for the source or target. Hidden Column Based Validation During data migration, hidden columns are added and unique keys are created for tables without non-null unique keys in the target to ensure data consistency. Data migration can verify data based on the unique key. For more information, see Hidden column mechanism of data migration service.
This parameter is displayed only when the migration source is OB_MySQL to OB_MySQL, OB_Oracle to OB_Oracle/Oracle, or Oracle to OB_MySQL/OB_Oracle.Row count validation
The following parameters are displayed only if you select Row Count Validation in the Select Type & Objects step.
Parameter Description Concurrent Tables for Validation The minimum value is 1, and the maximum value is 100. The default value is 10. This configuration applies to both the source and the target. Query Timeout The unit is minutes. The minimum value is 1, and the maximum value is 1440. The default value is 10. This configuration applies to both the source and the target. Incremental validation
The following parameters are displayed only if you select Incremental Validation in the Select Type & Objects step.
Parameter Description Validation Start Point The start timestamp for incremental validation. By default, the task start time is used. Only incremental changes after the incremental validation start time are verified. Validation End Point The end timestamp for incremental validation. You can set any time after the start time. When incremental change validation for the source and target reaches the specified timestamp, validation is completed. Initial Check Configuration Data is verified based on incremental changes in the source and target. If initial validation finds inconsistent data, validation is re-initiated. Initial check configuration includes: - Hot Data Cooldown Period: specifies the time to wait for the last change to be verified when the same data is frequently changed in the source. The unit is seconds, with a minimum value of 1 and a maximum value of 3600. The default value is 5.
- Mismatch Detection Delay: specifies the time to wait for the data to be verified again after incremental data in the source and target is inconsistent. If the data is still inconsistent, inconsistent data is reported.
The unit is seconds, with a minimum value of 1 and a maximum value of 86400. The default value is 30.
Recheck Configuration If initial validation finds inconsistent data, validation is re-initiated. The latest data is obtained from the source and target databases for comparison. If the data is still inconsistent after repeated validation, the validation task is completed, and inconsistent data is reported. Recheck configuration includes: - Recheck Concurrency: specifies the maximum number of concurrent data queries to the source and target for recheck. The minimum value is 1, and the maximum value is 512. The default value is 16.
- Recheck Timeout: specifies the maximum time for recheck. If the data is still inconsistent after the specified time, the validation task is completed, and inconsistent data is reported. The unit is minutes, with a minimum value of 1 and a maximum value of 720. The default value is 30.
- Limit BPS: specifies the maximum traffic for data queries to the source and target in MiB/s. The maximum value is 1024.
Hidden Column Based Validation During data migration, hidden columns are added and unique keys are created for tables without non-null unique keys in the target to ensure data consistency. Data migration can verify data based on the unique key. For more information, see Hidden column mechanism of data migration service.
This parameter is displayed only when the migration is OB_MySQL to OB_MySQL, OB_Oracle to OB_Oracle/Oracle, or Oracle to OB_MySQL/OB_Oracle.
Click Pre-check to perform a precheck on the data validation task.
After the precheck succeeds, click Start Task.
If you do not want to start the task, click Save to go to the details page of the data validation task. You can manually start the task as needed. For more information, see View details of a data validation task.