Migrate incremental data from a MySQL-compatible tenant of OceanBase Database to a TiDB Database

You can create a data migration task from a MySQL-compatible tenant of OceanBase Database to a TiDB Database to seamlessly migrate incremental data from the source database to the target database.

Prerequisites

• You have created an instance and a tenant for the target OceanBase Database. For more information, see Create an instance and Create a tenant.

• You have created a target database instance.

• You have created dedicated database users for data migration in the source and target databases and granted the required privileges to these users. For more information, see User privileges.

Limitations

• Only users with the Project Owner, Project Admin, or Data Service Admin project role can create a data migration task.

• Limitations on the source database

  • Do not perform DDL operations that change the schema during schema migration or full data migration. Otherwise, the data migration task may fail.

• At present, the data migration service supports TiDB database V4.x, V5.4, V6.x, and V7.x and OceanBase Database (in the MySQL compatible mode) V2.x, V3.x, and V4.x.

• When you migrate data from a MySQL-compatible tenant of OceanBase Database to a TiDB Database, only incremental synchronization is supported, and DDL synchronization is not supported.

• The Data Transmission Service supports only ASCII characters without special characters (., |, ", ', `, (, ), =, ;, /, &, and line breaks) for the names of databases, tables, and columns.

• The Data Transmission Service supports only the TiCDC open protocol. If you use other protocols, an exception will be thrown when the JDBC-Connector is started because of a null pointer error.

• The Data Transmission Service does not support triggers on the target database. If a trigger exists, the data migration task may fail.

Considerations

• If the character set of the source database is UTF-8, we recommend that you use a character set compatible with the source database (such as UTF-8 or UTF-16) for the target database to avoid garbled characters caused by incompatible character sets.

• If you want to modify the unique index of the target database, you must restart the data migration task. Otherwise, data inconsistency may occur.

• If the clocks of nodes or between the client terminal and server are out of sync, the latency of incremental synchronization may be inaccurate.

  • For example, if the clock is earlier than the standard time, the latency may be negative. If the clock is later than the standard time, the latency may increase.

• Verify whether the precision of the DECIMAL, FLOAT, and DATETIME column types meets your expectations. If the precision of the target field type is less than that of the source field type, truncation may occur, which may cause data inconsistency between the source and target databases.

• When you create a data migration task, you enable only Incremental Synchronization. In this case, the local incremental logs of the source database must be retained for more than 48 hours.

Supported source and target instance types

In the following table, the supported instance types for OceanBase MySQL Compatible include Dedicated (Transactional) and Dedicated (Analytical).

Data type mappings

Procedure

1. Create a data migration task.

   

   1. Log in to the OceanBase Cloud console.

   2. In the left-side navigation pane, choose Services > Migrations.

   3. On the Migrations page, click the Migrate Data tab.

   4. In the upper-right corner of the Migrate Data tab, click Create Task.

2. Enter a custom name for the task in the Edit Task Name field.

   We recommend that you use a combination of Chinese characters, digits, and letters. The name cannot contain spaces and must be no longer than 64 characters.

3. On the Configure Source and Target page, configure the parameters.

   1. In the Source section, configure the parameters.

      If you need to reference an existing data source, click Quick Fill next to Source, select the desired data source from the drop-down list, and the configurations will be automatically filled in. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Source section.

      You can also click Manage Data Sources in the Quick Fill drop-down list to go to the Data Sources page, where you can view and manage data sources. This page provides unified management for different types of data sources. For more information, see Data Sources.

      Parameter	Description
      Cloud vendor	Currently supports AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud.
      Database type	Select the source as OceanBase MySQL Compatible.
      Instance type	Includes Dedicated (Transactional) and Self-managed Database.
      Region	Select the region where the source database is located.
      Connection type	Includes Endpoint and Public IP. If you select Endpoint as the connection method, you need to add the displayed account ID to your endpoint service's whitelist to allow the endpoint to connect to the endpoint service. For more information, see the documentation in the Select Private Network Connection section. If you select Public IP as the connection method, you need to add the displayed data source IP address to the whitelist of your OceanBase database instance to ensure connectivity. For more information, see the documentation in the Select Public Network Connection section. Note This parameter is displayed only when the instance type is set to Self-managed Database. You need to select the source and target regions before the page displayss the data source IP address that needs to be added to the whitelist.
      Connection information	This parameter is displayed only when the instance type is set to Self-managed Database. If you select Connection Type as Endpoint, enter the endpoint service name. If you select Connection Type as Public IP, enter the IP address and port number of the database host.
      Instance	The ID or name of the instance where the MySQL-compatible tenant of OceanBase Database belongs. You can view the ID or name of the target instance on the Instances page. Note If the cloud vendor is Alibaba Cloud, you can also select an Alibaba Cloud main account instance for cross-account authorization. For more information, see Alibaba Cloud Account Authorization.
      Tenant	The ID or name of the MySQL-compatible tenant of OceanBase Database. You can expand the target instance on the Instances page to view the ID or name of the target tenant.
      Database account	The username of the MySQL-compatible tenant of OceanBase Database used for data migration.
      Password	The password of the database user.

      If you select Instance Type as Self-managed Database, you can decide whether to enable advanced settings as needed.

      Notice

      If you create a new migration task that requires incremental synchronization, make sure to enable the sys tenant account and OBLogProxy.

      Parameter	Description
      Sys Tenant Account	If you enable the sys tenant account, you need to enter the sys account and password. Sys Account: the name of the sys user. This user is mainly used to read incremental logs and database object structure information from OceanBase Database. Create this user under the sys tenant of the business cluster. Password: the password of the sys user.
      OBLogProxy	If you enable the incremental log proxy service, you need to fill in OBLogProxy Connection Information. This parameter is the OceanBase Database incremental log proxy service, which provides real-time incremental project intervention and management capabilities in the form of a service, making it convenient for applications to intervene in OceanBase Database incremental logs. At the same time, it can meet the subscription needs of incremental logs under network isolation. The format is OBLogProxy IP: OBLogProxy Port.

   2. In the Configure Source and Target step, configure the parameters in the Target section.

      If you want to reference an existing data source that has been created and saved, click Quick Fill on the right side of the Target section and select the target data source from the drop-down list. After selection, the configurations in the Target section will be automatically filled. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Target section.

      You can also click Manage Data Sources in the Quick Fill drop-down list to go to the Data Sources page, where you can view and manage data sources. This page allows you to centrally manage different types of data sources. For more information, see Data Sources.

      Parameter	Description
      Cloud vendor	Currently supports AWS, Huawei Cloud, Google Cloud, and Alibaba Cloud. You can choose the same cloud vendor as the source or migrate data across cloud vendors. Note By default, cross-cloud data migration is not enabled. To enable this feature, contact OceanBase Cloud technical support.
      Database type	Select TiDB as the database type for the target.
      Instance type	Currently only supports Self-managed TiDB.
      Region	Select the region where the target database is located.
      Connection type	Includes Endpoint and Public IP. If you select Endpoint as the connection type, you need to add the displayed account ID to the allowlist of your endpoint service to allow the endpoint to connect to the endpoint service. For more information, see Select Private Connection. When Cloud Vendor is set to AWS, if you selected Acceptance required for the Require acceptance for endpoint parameter when creating the endpoint service, a prompt will appear when the data migration service first connects via private link, asking you to go to the AWS console and perform the Accept Endpoint Connection Request action. When the cloud vendor is set to Google Cloud, you need to add the authorized project to Published Services. After adding authorization, manual authorization is no longer required during data source testing. If you select Public IP as the connection type, you need to add the displayed data source IP address to the allowlist of your TiDB database instance to ensure connectivity. For more information, see Select Public Connection. Note The page will display the IP address to be added to the allowlist only after you have selected the regions for both the source and target.
      Connection information	If you select Connection Type as Endpoint, enter the endpoint service name. If you select Connection Type as Public IP, enter the IP address and port number of the database host.
      Database account	The username of the TiDB database used for data migration.
      Password	The password of the database user.

4. Click Test and Continue.

5. On the Select Type & Objects page, configure the parameters.

   Note

   Currently, when migrating data from a MySQL-compatible tenant of OceanBase Database to a TiDB Database, only one-way synchronization is supported.

   1. In the Migration Type section, select the migration type for the current data migration task.

      Currently, only Incremental Synchronization is supported for DML Synchronization. For more information, see Customize DML/DDL Settings.

   2. In the Select Migration Objects section, configure the method for selecting the migration object.

      You can select the migration objects by using either Specify Objects or Match by Rule.

   3. In the Select Migration Scope section, select the objects to be migrated.

      • If you select Specify Objects, data migration supports Table-level and Database-level. Table-level migration allows you to select one or more tables or views from one or more databases as migration objects. Database-level migration allows you to select an entire database as a migration object. If you select table-level migration for a database, database-level migration is no longer supported for that database. Conversely, if you select database-level migration for a database, table-level migration is no longer supported for that database.

        After selecting Table-level or Database-level, select the objects to be migrated in the left pane and click > to add them to the right pane.

        Data migration allows you to import objects through text and supports renaming objects at the target, setting row filters, viewing column information, and removing individual or all migration objects.

        

        Note

        If you select Database-level, only the database name is displayed in the right pane, and specific objects cannot be displayed.

        Action	Description
        Import objects	Click Import Objects in the upper-right corner of the right pane. For more information, see Import migration objects.
        Rename	Data migration allows you to rename migration objects. For more information, see Rename database tables.
        Row filter	Data migration allows you to use the WHERE condition to filter rows. For more information, see Filter data by SQL conditions. You can also view column information of migration objects in the View Columns section.
        Remove/Clear All	Data migration allows you to remove temporarily selected objects at the target during data mapping. Remove a single migration object Click the Remove icon next to the target object in the right pane to remove it. Remove all migration objects Click Clear All in the upper-right corner of the right pane. In the dialog box that appears, click OK to remove all migration objects.

      • For more information about how to configure matching rules, see Configure matching rules between databases.

6. Click Next. On the Migration Options page, configure the parameters for incremental synchronization.

   Parameter	Description
   Write Concurrency	This parameter specifies the maximum number of concurrent writes during incremental synchronization, with a maximum limit of 512. Excessive concurrency may overload the target system, impacting business operations.
   Rate Limiting for Incremental Migration	You can choose to enable or disable rate limiting for incremental synchronization. If enabled, set the RPS (maximum number of rows that can be synchronized per second during incremental synchronization) and BPS (maximum data volume that can be synchronized per second during incremental synchronization). Note The RPS and BPS values set here serve as throttling limits. The actual performance of incremental synchronization is influenced by factors such as source and target configurations, instance specifications, etc.
   Incremental Synchronization Start Timestamp	If you selected Full Migration when choosing the migration type, this parameter will not be displayed. If you did not select Full Migration but selected Incremental Synchronization, specify the start time for data migration after a specific point in time. By default, it is set to the current system time. For more information, see Set the incremental synchronization start time.

7. Click Pre-check to initiate a pre-check of the data migration task.

   During the Pre-check phase, the system verifies whether the database user has read/write permissions and whether the network connection meets the requirements. Only after all checks pass can the data migration task be initiated. If an error occurs during the pre-check:

   • You can troubleshoot and resolve the issue, then rerun the pre-check until it succeeds.

   • Alternatively, click Skip in the Actions column of the failed check item. A dialog box will appear, informing you of the specific impact of skipping this operation. After confirming that it's acceptable, click OK in the dialog box.

8. After the pre-check succeeds, click Purchase to go to the Purchase Data Migration Instance page.

   After the purchase succeeds, you can start the data migration task. For more information about how to purchase a data migration instance, see Purchase a data migration instance. If you do not need to purchase a data migration instance at this time, click Save to go to the details page of the data migration task. You can manually purchase a data migration instance later as needed.

   You can click Configure Validation Task in the upper-right corner of the details page to compare the data differences between the source database and the target database. For more information, see Create a data validation task.

   The data migration service allows you to modify the migration objects when the task is running. For more information, see View and modify migration objects. After the data migration task is started, it is executed based on the selected migration types. For more information, see the "View migration details" section in View details of a data migration task.