This topic describes how to perform data verification after a data migration task is completed.
Limitations
Only users with the project role of Project Owner, Project Admin, or Data Services Admin can create a data validation task.
Supported source and target database types
Source |
destination |
|---|---|
| MySQL | OceanBase MySQL Compatible |
| OceanBase MySQL Compatible | MySQL |
| Oracle | OceanBase Oracle Compatible |
| OceanBase Oracle Compatible | Oracle |
| OceanBase MySQL Compatible | OceanBase MySQL Compatible |
| OceanBase Oracle Compatible | OceanBase Oracle Compatible |
| TiDB | OceanBase MySQL Compatible |
| PostgreSQL | OceanBase MySQL Compatible |
| PostgreSQL | OceanBase Oracle Compatible |
Procedure
Go to the Create Data Verification Task page.
Log in to the OceanBase Cloud console.
In the left-side navigation pane, click Data Services > Data Validation.
On the Data Validation page, click Create Task in the upper-right corner.
In the Edit Task Name text box, enter a custom name for the verification task.
We recommend that you use a combination of Chinese characters, numbers, and letters. The name cannot contain spaces and must not exceed 64 characters in length.
In the Configure Source & Target step, configure the parameters in the Source section.
If you need to reference an existing data source, click Quick Fill next to Source and select the target data source from the drop-down list. After selection, the configurations for the source section will be automatically populated. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Source area.
You can also click Manage Data Source in the Quick Fill drop-down list to go to the Data Source page to view and manage data sources. This page provides unified management for different types of data sources. For more information, see the documentation in the Data Sources module.
ParameterDescriptionCloud Provider Currently supports AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud. Database Type Currently, the source supports MySQL, Oracle, OceanBase MySQL Compatible, OceanBase Oracle Compatible, TiDB, and PostgreSQL. Select according to your actual needs. Instance Type Select the appropriate instance type as prompted on the page. Region Select the region where the source database is located. Connection Type Includes Endpoint and Public IP. This parameter is displayed only when the database type is MySQL or Oracle. Connection Information - If you select Connection Type as Endpoint, enter the endpoint service name.
- If you select Connection Type as Public IP, enter the IP address and port number of the database host.
Service Name The service name of the Oracle database. This parameter is displayed only when the database type is Oracle. Database Name The database name. This parameter is displayed only when the database type is PostgreSQL. Instance Select an OceanBase instance from the drop-down list. This parameter is displayed only when the database type is OceanBase MySQL Compatible or OceanBase Oracle Compatible. Tenant After selecting an OceanBase instance, choose the OceanBase tenant to use from the drop-down list. Database Account/Password Enter the account and password for the source database. In the Configure Source & Target step, configure the parameters in the Target section.
If you need to reference an existing data source, click Quick Fill next to Target and select the target data source from the drop-down list. After selection, the configurations for the target section will be automatically populated. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Target area.
You can also click Manage Data Source in the Quick Fill drop-down list to go to the Data Source page to view and manage data sources. This page provides unified management for different types of data sources. For more information, see the documentation in the Data Sources module.
ParameterDescriptionCloud Provider Currently supports AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud. You can choose the same cloud provider as the source or perform data verification across different cloud providers. Note
The cross-cloud provider data verification feature is not enabled by default. To use this feature, contact OceanBase Cloud technical support.
Database Type Select the target database type based on the source database type. Instance Type Select the appropriate instance type as prompted on the page. Region Select the region where the target database will be located. Connection Type Includes Endpoint and Public IP. This parameter is displayed only when the database type is MySQL or Oracle. Connection Information - If you select Connection Type as Endpoint, enter the endpoint service name.
- If you select Connection Type as Public IP, enter the IP address and port number of the database host.
Service Name The service name of the Oracle database. This parameter is displayed only when the database type is Oracle. Instance Select an OceanBase instance from the drop-down list. This parameter is displayed only when the database type is OceanBase MySQL Compatible or OceanBase Oracle Compatible. Tenant After selecting an OceanBase instance, select the OceanBase tenant to use from the drop-down list. Database Account/Password Enter the account and password for the target database. Click Test and Continue.
In the Select Type & Objects step, select the verification type for the current data verification task.
Currently supported verification types include Full Validation and Row Count Validation. For details, see Data Verification Overview.
In the Validation Objects section, configure how to select the objects for verification.
You can select verification objects through two methods: Specify Objects and Match by Rule.
In the Select Object section, select the object(s) to be verified.
Note
Currently, only tables can be selected as verification objects.
When you select Specify Objects, select the table object(s) to verify in the left-side selection area and click > to add them to the list on the right.
Data verification supports importing objects via text, and you can rename target objects, filter rows, view column information, and remove one or all verification objects.
OperationDescriptionImport Object In the list on the right side of the selection area, click Import Object in the upper-right corner. For details, see Import a verification object. Rename Data verification supports renaming the name of a verification object. For details, see Rename a verification object. Row Filter Full Validation supports row filtering using WHEREconditions. For details, see Filter verification objects by SQL condition. You can also view the column information of a verification object in the View Column section.Remove/Clear All Data verification supports removing one or more temporarily selected objects to the target during data mapping. - Remove a single migration object
In the list on the right side of the selection area, click the Remove icon next to the target object to remove it. - Remove all migration objects
In the list on the right side of the selection area, click Clear All in the upper-right corner. In the dialog box, click OK to remove all migration objects.
- Remove a single migration object
When you select Match by Rule, for details, see Configure matching rules for verification objects.
Click Next. In the Validation Options step, configure the parameters.
Full Verification
The following parameters are displayed only if you selected Full Validation in the Select Type & Objects step.
ParameterDescriptionSource Concurrency Configuration/Target Concurrency Configuration Minimum value: 1, maximum value: 512, default value: 4. Verification Mode Includes Adaptive Mode and IN Mode. - Adaptive Mode: Retrieves data within the same range from the source and target in batches for precise comparison.
- IN Mode: Retrieves data within a specified range from the source in batches, and queries the target for data with unique keys under the IN condition to check for consistency. Note that this mode cannot verify scenarios where the target contains data not present in the source, and verification performance is somewhat degraded. It is recommended to use this mode only when the target contains expected excess data that does not need verification, or in database/table consolidation migration scenarios.
Limit BPS Rate Unit: MiB/s, maximum value: 1024. If you enable this option, please select at least the lowest rate for either the source or target. Row Count Verification
The following parameters are displayed only if you selected Select Type & Objects in the Select Type & Objects step.
ParameterDescriptionConcurrent Verification Table Count Minimum value: 1, maximum value: 100, default value: 10. This configuration applies to both the source and target. Query Timeout Unit: minutes, minimum value: 1, maximum value: 1440, default value: 10. This configuration applies to both the source and target.
Click Next. The system performs a pre-check on the data verification task.
After the pre-check passes, click Start Task.
If you do not need to start the task immediately, click Save to go to the details page of the data verification task, where you can start it manually as needed. For more information, see View details of a data verification task.
Go to the page for configuring a data verification task.
Log in to the OceanBase Cloud console.
In the left navigation bar, click Data Services > Data Migration.
On the Data Migration page, click the Data Migration tab.
On the Data Migration tab, click the name of the target task to enter its details page.
Click Configure Validation Task in the upper-right corner of the details page.
Note
Data migration tasks in the Ending or Completed state do not support configuring a verification task.
In the Edit Task Name text box, enter a custom verification task name. You can also use the default format
Data Migration Task Name-verify.It is recommended to use a combination of Chinese characters, numbers, and letters. The name must not contain spaces and must not exceed 64 characters in length.
In the Configure Source & Target step, confirm all parameters are correct, then click Test and Continue.
In the Select Type & Objects step, select the verification type for the current data verification task.
Currently, the supported verification types include Full Validation and Row Count Validation. For more information, see Data verification overview.
In the Validation Objects section, configure how to select the verification object.
The methods for selecting a verification object include Specify Objects and Match by Rule. You can keep the method used to select objects in the data migration task or modify it.
In the Select Object section, select the objects to be verified.
Note
Currently, only tables can be selected as verification objects.
The verification objects can be the same as those selected for the data migration task, or you can reselect them based on your actual situation.
Data verification tasks do not support full database verification. For a data migration task that selects an entire database, the data verification task will select all table objects within that database.
To select Specify Objects, select the tables to be verified on the left and click > to add them to the list on the right.
Data verification supports importing objects via text, and allows you to rename target objects, set row filters, view column information, and remove single or all verification objects.
OperationDescriptionImport Objects In the right-side list of the selection area, click Import Object in the upper-right corner. For more information, see Import a verification object. Rename Data verification supports renaming verification objects. For more information, see Rename a verification object. Row Filter Full Validation supports row filtering using WHEREconditions. For more information, see Filter verification objects by SQL condition. You can also view the column information of the verification object in the View Column section.Remove/Clear All Data verification supports removing one or more temporarily selected objects to the target during data mapping. - Remove a single migration object
In the right-side list of the selection area, click the Remove icon next to the target object to remove it. - Remove all migration objects
In the right-side list of the selection area, click Clear All in the upper-right corner. In the dialog box, click OK to remove all migration objects.
- Remove a single migration object
Select Match by Rule. For more information, see Configure matching rules for verification objects.
Click Next. In the Validation Options step, configure the parameters.
Full verification
The following parameters are displayed only if you select Full Validation in the Select Type & Objects step.
ParameterDescriptionSource Concurrency Configuration/Target Concurrency Configuration Minimum value: 1. Maximum value: 512. Default value: 4. Verification Mode Includes Adaptive Mode and IN Mode. - Adaptive Mode: Retrieves data within the same range from the source and target in batches for precise comparison.
- IN Mode: Retrieves data within the specified range from the source in batches, and queries the target with an IN condition based on the unique keys of the target data to check for consistency. Note that this mode cannot verify scenarios where the target contains data not present in the source, and verification performance is somewhat degraded. It is recommended to use this mode only when the target contains expected redundant data that does not need verification, or in migration scenarios involving database/table aggregation.
Limit BPS Rate Unit: MiB/s. Maximum value: 1024. If you enable this option, please set the minimum rate for either the source or target. Row count verification
The following parameters are displayed only if you select Row Count Validation in the Select Type & Objects step.
ParameterDescriptionConcurrent Verification Table Count Minimum value: 1. Maximum value: 100. Default value: 10. This configuration applies to both the source and target. Query Timeout Unit: minutes. Minimum value: 1. Maximum value: 1440. Default value: 10. This configuration applies to both the source and target.
Click Next. The system performs a precheck on the data verification task.
After the precheck passes, click Start Task.
If you do not need to start the task immediately, click Save to go to the details page of the data verification task, where you can start it manually as needed. For more information, see View details of a data verification task.