This topic describes how to validate data after you complete a data migration task.
Limitations
Only users with the Project Owner, Project Administrator, or Data Service Administrator role can create data validation tasks.
Supported source and target database types
| Source | Target |
|---|---|
| MySQL | OceanBase MySQL Compatible |
| OceanBase MySQL Compatible | MySQL |
| Oracle | OceanBase Oracle Compatible |
| OceanBase Oracle Compatible | Oracle |
| OceanBase MySQL Compatible | OceanBase MySQL Compatible |
| OceanBase Oracle Compatible | OceanBase Oracle Compatible |
| TiDB | OceanBase MySQL Compatible |
| PostgreSQL | OceanBase MySQL Compatible |
| PostgreSQL | OceanBase Oracle Compatible |
Procedure
Go to the Create Data Validation Task page.
Log in to the OceanBase Cloud console.
In the left-side navigation pane, select Data Services > Data Validation.
On the Data Validation page, click Create Task in the upper-right corner.
Enter a custom name for the validation task in the Edit Task Name field.
We recommend that you use a combination of letters, numbers, and Chinese characters. The name cannot contain spaces and must be no longer than 64 characters.
In the Configure Source & Target step, configure the parameters in the Source section.
If you want to reference an existing data source, click Quick Fill next to Source and select the desired data source from the drop-down list. The configurations in the Source section will be automatically filled in. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Source section.
You can also click Manage Data Sources in the Quick Fill drop-down list to go to the Data Sources page, where you can view and manage data sources. This page provides unified management for different types of data sources. For more information, see Data Sources module.
Parameter Description Cloud Vendor Currently supports AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud. Database Type Currently supports MySQL, Oracle, OceanBase MySQL Compatible, OceanBase Oracle Compatible, TiDB, and PostgreSQL. Select the appropriate type as needed. Instance Type Select the corresponding instance type as prompted. Region Select the region where the source database is located. Connection Type Includes Endpoint and Public IP. This parameter appears only when the database type is MySQL or Oracle. Connection Information - If Connection Type is set to Endpoint, enter the endpoint service name.
- If Connection Type is set to Public IP, enter the IP address and port number of the database host.
Service Name The service name of the Oracle database. This parameter appears only when the database type is Oracle. Database Name The name of the database. This parameter appears only when the database type is PostgreSQL. Instance Select an OceanBase instance from the drop-down list. This parameter appears only when the database type is OceanBase MySQL Compatible or OceanBase Oracle Compatible. Tenant After selecting an OceanBase instance, select the tenant to use from the drop-down list. Database Account/Password Enter the account and password for the source database. In the Configure Source & Target step, configure the parameters in the Target section.
If you need to reference an existing data source, click Quick Fill on the right side of Target, and select the target data source from the drop-down list. After you select a data source, the configuration parameters in the Target section will be automatically filled in. If you want to save the current configuration as a new data source, click Save in the upper-right corner of the Target section.
You can also click Manage Data Sources in the Quick Fill drop-down list to go to the Data Sources page, where you can view and manage data sources. This page allows you to manage data sources of different types. For more information, see the Data Sources module.
Parameter Description Cloud Vendor Currently, AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud are supported. You can select the same cloud vendor as the source or choose a different cloud vendor for cross-cloud validation. Note
Cross-cloud validation is not enabled by default. To use this feature, contact OceanBase Cloud Technical Support.
Database Type Select the database type corresponding to the target based on the database type of the source. Instance Type Select the instance type as prompted. Region Select the region where the target database is located. Connection Type The value can be Endpoint or Public IP. This parameter is displayed only when the database type is MySQL or Oracle. Connection Information - If you select Endpoint for Connection Type, enter the endpoint service name.
- If you select Public IP for Connection Type, enter the IP address and port number of the database host.
Service Name The service name of the Oracle database. This parameter is displayed only when the database type is Oracle. Instance Select an OceanBase instance from the drop-down list. This parameter is displayed only when the database type is OceanBase MySQL Compatible or OceanBase Oracle Compatible. Tenant After you select an OceanBase instance, select the tenant from the drop-down list. Database Account/Password Enter the account and password of the target database. Click Test and Continue.
In the Select Type & Objects step, select the validation type for the current data validation task.
Currently, you can select Full Validation or Row Count Validation. For more information, see Overview of data validation.
In the Validation Objects section, choose the method for selecting validation objects.
You can select validation objects by using Specify Objects or Match by Rule.
In the Select Objects section, select the object to be validated.
Note
Currently, you can select only tables as validation objects.
If you select Specify Objects. In the left pane, select the table objects to be validated and click > to add them to the right pane.
Data validation supports importing objects through text and allows renaming target objects, setting row filters, viewing column information, and removing individual or all validation objects.
Action Description Import Objects In the right pane of the selection area, click Import in the upper-right corner. For more information, see Import validation objects. Rename Data validation allows renaming validation objects. For more information, see Rename validation objects. Row Filter Full Validation supports WHEREconditions for row filtering. For more information, see SQL condition filtering for validation objects. You can also view column information of validation objects in the View Column area.Remove/Clear All Data validation allows removing temporarily selected objects to the target during data mapping. - Remove a single migration object
Click the Remove icon next to the target object in the right pane of the selection area to remove it. - Remove all migration objects
Click Clear All in the upper-right corner of the right pane of the selection area. In the dialog box that appears, click OK to remove all migration objects.
- Remove a single migration object
If you select Match by Rule. For more information, see Configure matching rules for validation objects.
Click Next. In the Validation Options step, configure the parameters.
Full validation
In the Select Type & Objects step, select Full Validation to display the following parameters.
Parameter Description Source Concurrency/Target Concurrency The minimum value is 1 and the maximum value is 512. The default value is 4. Validation Mode Valid values: Adaptive and IN Mode. - Adaptive: Data is fetched in batches from the source and target, and then compared precisely.
- IN Mode: Data is fetched in batches from the source. Based on the unique keys of the target data, an IN condition query is performed on the target to compare whether the data is consistent. Note that this mode cannot validate scenarios where the target contains data not present in the source. Additionally, the performance of this mode may degrade. We recommend using this mode only when the target contains unexpected extra data that does not need to be validated or in migration scenarios involving database table aggregation.
Limit BPS The unit is MiB/s, and the maximum value is 1024. If you enable this option, please set at least one of the minimum rates for the source or the target. Row count validation
In the Select Type & Objects step, select Row Count Validation to display the following parameters.
Parameter Description Concurrent Tables for Validation The minimum value is 1, the maximum value is 100, and the default value is 10. This configuration applies to both the source and the target. Query Timeout The unit is minutes. The minimum value is 1, the maximum value is 1440, and the default value is 10. This configuration applies to both the source and the target.
Click Next to perform a pre-check on the data validation task.
After the pre-check is successful, click Start Task.
If you do not need to start the task immediately, click Save to go to the details page of the data validation task, where you can manually start the task as needed. For more information, see View details of a data validation task.
Go to the page where you can configure a data validation task.
Log in to the OceanBase Cloud console.
In the left-side navigation pane, select Data Services > Migrations.
On the Migrations page, click the Migrate Data tab.
On the Migrate Data tab, click the name of the target data migration task to go to its details page.
In the upper-right corner of the details page, click Configure Validation Task.
Notice
You cannot configure a validation task for a data migration task that is in the Terminating or Terminated state.
Enter a custom name for the validation task in the Edit Task Name text box. You can also use the default name, which is in the format of
Data Migration Task Name-Verify.We recommend that you use a combination of letters, numbers, and Chinese characters. The name cannot contain spaces and must be no longer than 64 characters.
In the Configure Source & Target step, make sure that all parameters are correctly configured, and then click Test and Continue.
In the Select Type & Objects step, select the validation type for the current data validation task.
Currently, only two types of validation are supported: Full Validation and Row Count Validation. For more information, see Overview of data validation.
In the Validation Objects section, choose the method for selecting validation objects.
You can select validation objects by using Specify Objects or Match by Rule. You can keep the object selection method used in the data migration task or change it.
In the Select Objects section, select the objects to be validated.
Note
At present, only tables can be selected as validation objects.
The validation objects can be the same as those selected in the data migration task, or they can be changed as needed.
A data validation task does not support validating an entire database. If the data migration task selects an entire database, the data validation task will validate all tables in the database.
If you select Specify Objects, select the objects to be validated on the left and click > to add them to the right list.
Data validation supports importing objects through text and allows renaming target objects, setting row filters, viewing column information, and removing individual or all validation objects.
Operation Description Import Objects In the right list of the selection area, click Import in the upper-right corner. For more information, see Import validation objects. Rename Data validation allows renaming validation objects. For more information, see Rename validation objects. Row Filter Full Validation supports WHEREconditions for row filtering. For more information, see SQL condition filtering for validation objects. You can also view column information of validation objects in the View Column section.Remove/Clear All During data mapping, you can remove temporarily selected objects to the target. - Remove a single migration object
In the right list of the selection area, click the Remove icon next to the target object to remove it. - Remove all migration objects
In the right list of the selection area, click Clear All in the upper-right corner. In the dialog box that appears, click OK to remove all migration objects.
- Remove a single migration object
If you select Match by Rule. For more information, see Configure matching rules for validation objects.
Click Next and configure parameters on the Validation Options step.
Full validation
On the Select Type & Objects step, select Full Validation to display the following parameters.
Parameter Description Source Concurrency/Target Concurrency The minimum value is 1 and the maximum value is 512. The default value is 4. Validation Mode Valid values: Adaptive and IN Mode. - Adaptive: Data is fetched in batches from the source and target, and then compared precisely.
- IN Mode: Data is fetched in batches from the source. Based on the unique keys of the target data, an IN condition query is performed on the target to compare whether the data is consistent. Note that this mode cannot validate scenarios where the target contains data not present in the source. Additionally, the performance of this mode may degrade. We recommend using this mode only when the target contains unexpected extra data that does not need to be validated or in migration scenarios involving database table aggregation.
Limit BPS The unit is MiB/s, and the maximum value is 1024. If you enable this option, please set at least one of the minimum rates for the source or the target. Row count validation
On the Select Type & Objects step, select Row Count Validation to display the following parameters.
Parameter Description Concurrent Tables for Validation The minimum value is 1, the maximum value is 100, and the default value is 10. This configuration applies to both the source and the target. Query Timeout The unit is minutes. The minimum value is 1, the maximum value is 1440, and the default value is 10. This configuration applies to both the source and the target.
Click Next to perform a precheck on the data validation task.
If the precheck is successful, click Start Task.
If you do not want to start the task at this time, click Save to go to the details page of the data validation task, where you can manually start the task as needed. For more information, see View the details of a data validation task.