Configure matching rules for migration objects

This topic describes the background information, limits, procedure, scenarios, and troubleshooting tips for configuring matching rules for migration objects.

Background

When you create a data migration project, you must specify the migration objects. The OMS feature supports the synchronization of DDL statements. However, if a new table is created in the source database during migration, data in this table cannot be migrated because this table is not among the specified data migration objects.

The OMS feature allows you to configure matching rules for selecting and mapping objects when you want to migrate a large number of objects. You are also allowed to rename and batch-rename the mapping relationships between the source and the destination. In this way, you can conveniently and quickly create data migration and synchronization projects. This feature helps significantly improve the working efficiency based on simple and efficient string matching rules. The performance in processing a large number of objects is also improved based on matching rules.

Limits

You can configure multiple matching rules. Each rule is displayed in a single line without leading or trailing spaces.
Object migration rules are required and object exclusion rules are optional.
Wildcard characters are not supported in names when you select schemas and databases.
The Schema.* and DataBase.* formats are not supported in object exclusion rules.
If you select Synchronize DDL when you create a data migration project, you can select migration objects only based on wildcard rules.
You can enter specific object names in object migration rules. However, DDL synchronization is not supported for this type of objects.
You cannot change a table name that matches the object exclusion rule to a name that does not match the object exclusion rule by using a DDL operation.
DDL modifications are not supported during schema migration and full migration.
If migration objects are selected based on matching rules, special characters contained in tables will not be displayed. Special characters are . | \ " ' ` ( ) = ; / & \n

Notes

If you select migration objects based on matching rules, we recommend that you create standard CTAS statements and set object exclusion rules , such as a.ctas*, to avoid interrupting the data migration project.
After you configure the object migration rule and the object exclusion rule, if the name of a source table exists in the difference set between the object migration rule and the object exclusion rule, the related objects can be selected.

Note

A difference set between two sets contains all elements that exist in one set but do not exist in the other set.
For the DDL synchronization feature, if the name of a table created or modified in the source database exists in the difference set between the object migration rule and the object exclusion rule, the DDL statements can be synchronized by using a transmission instance to the destination database in real time.
In a multi-table aggregation scenario:
- We recommend that you map objects in the source and destination databases by importing objects and configuring matching rules.
- We recommend that you manually create schemas in the destination database. If you create a schema by using the OMS feature, skip the failed objects in the schema migration step.

Procedure

Create a data migration project and configure it to the Select Migration Objects step.

For more information, see the topic about creating a data migration project between the corresponding data sources in the Data migration chapter.
On the Select Objects page, select the migration objects and migration scope.

You can select Specify Objects or Match Rules to specify the migration rules. This section describes how to configure a matching rule.
1. Select Match Rules.
2. In the Specify Migration Scope section, specify object migration rules in the Object Migration Rule field and object exclusion rules in the Object Exclusion Rule field. The Object Exclusion Rule field is optional. For more information, see Wildcard rules.
3. Click Verify.
  
  To view the matching results, click Preview Objects after the verification succeeds. The matching results are displayed on the Final Objects, New Objects, and Removed Objects tabs. The wildcard rules and exclusion rules apply to tables and views.
Complete subsequent project settings as prompted.

Scenarios

Assume that you have four databases in the production environment: jenkins_api_mysql56, jenkins_api_mysql57, jenkins_my2dh_on, and jenkins_my2dh_one_verify. This section describes how to configure matching rules in different scenarios.

Synchronize all the four databases

To synchronize all the four databases to the destination, configure the matching rules as shown in the following figure.

rule1

Exclude historical tables and log tables

To exclude historical tables whose names begin with "history_" and log tables whose names end with "log" in the jenkins_api_mysql56 database, configure the matching rules as shown in the following figure.

rule2

Set the mapping relationship between the source and destination databases

To synchronize the jenkins_my2dh_one_verify database to the destination database target_test, configure the matching rule as shown in the following figure.

rule-3-en

Aggregate multiple tables

To aggregate the order tables named from order00 to order99 in the jenkins_api_mysql56 and jenkins_api_mysql57 databases to orders in the destination database jenkins_api_mysql59, configure the matching rules as shown in the following figure.

rule-4-en

Troubleshooting

Insufficient privilege

Pay attention to the privilege settings of the source database user. If you do not grant all required privileges to the migration user, some objects are not displayed in the frontend by the OMS feature, and you cannot correctly configure matching rules. In this case, you must add these objects to Object Exclusion Rule to prevent the data migration project from being interrupted because the OMS feature cannot find the target objects.
DML filtering unsupported

If DDL synchronization is disabled, the OMS feature allows you to select objects based on matching rules. If a new table created during incremental synchronization meets a matching rule, the related DDL statements will be ignored but the OMS feature will continue to synchronize DML statements. As a result, the data migration project will be interrupted because objects cannot be synchronized to the destination database. Therefore, you must create the table in the destination database or add the table to the blacklist.