This topic describes how to use OceanBase Migration Service (OMS) to migrate data from an OBSharding database to a MySQL-compatible tenant of OceanBase Database.
Background information
OBSharding is a unit-based solution developed by OceanBase. OMS supports data synchronization from unit-based OBSharding databases to OceanBase Database, enabling you to aggregate your unit-based sharding into a single OceanBase Database instance to meet the needs of data analysis and other scenarios.
After a data migration task from an OBSharding database to a MySQL-compatible tenant of OceanBase Database is initiated, the task will automatically be deleted. OMS will automatically create a data migration task between two MySQL-compatible tenants of OceanBase Database.
We recommend that you use tags or task names as filtering conditions to perform batch operations such as batch startup, batch suspension, and batch forward switching on these tasks. For more information about batch operations, see Batch operations on data migration tasks.
Prerequisites
You have created dedicated database users for data migration for the source and target databases and granted the necessary privileges to the users. The privileges of the OBSharding database users must be the same as those of the MySQL-compatible tenant users of OceanBase Database when they are used as the source database. For more information, see Create a database user.
The network connection between the OBSharding console and the OMS console is available.
The network connection between the OBSharding physical instance nodes and OMS is available.
Each physical node of the OceanBase Database instance must exist and have the same user privileges.
Limitations
Limitations on the source database
Do not perform DDL operations that change the schema of a database or table during full data migration. Otherwise, the data migration task may be interrupted.
The supported OBSharding database version is V3.2.8.9, and the supported OceanBase Database versions are V3.2.3 and V4.2.1.
OMS does not support triggers on the target database. If a trigger exists, data migration may fail.
Data source identifiers and user accounts must be globally unique in the OMS system.
OMS supports migrating databases, tables, and columns whose names are ASCII characters and do not contain special characters (including line breaks, spaces, and .|"'`()=;/&\).
Procedure
Create a data migration task.
Log in to the OMS console.
In the left-side navigation pane, click Data Migration.
On the Data Migration page, click Create Task in the upper-right corner.
On the Create Task page, specify a name for the migration task.
We recommend that you use a combination of Chinese characters, digits, and letters. The name cannot contain spaces and must be no longer than 64 characters.
Notice
The task name is a unique identifier in the OMS system. Please specify an appropriate task name.
In the Select Source and Target step, configure the parameters.
Parameter Description Source If you have already created an OBSharding data source, select one from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create an OBSharding data source. Target If you have created a physical data source for OceanBase Database in the MySQL compatible mode, select one from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create a MySQL-compatible OceanBase physical data source. Tag Click the text box and select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data migration tasks by using tags. Note
After a data migration task from an OBSharding database to a MySQL-compatible OceanBase database is started, the task is automatically deleted. Please select an appropriate tag for the task.
Click Next. In the Select Migration Type step, select the migration type for the current task.
Migration Type includes Full Migration and Incremental Synchronization.
Migration Type Limitations Full migration After a full migration task is started, OMS migrates the existing data of the source database tables to the corresponding tables in the target database. Incremental synchronization After an incremental synchronization task is started, OMS synchronizes the data that has changed in the source database (added, modified, or deleted) to the corresponding tables in the target database.
Incremental Synchronization includes DML synchronization and DDL synchronization, which you can customize as needed. For more information, see Customize DDL/DML settings.Click Next. In the Select Migration Objects step, select the objects to be migrated for the current data migration task.
Currently, you can only select migration objects by clicking Specify Objects. In the left-side pane, select the objects to be migrated and click > to add them to the right-side list. You can select one or more tables in one or more databases as migration objects.
OMS allows you to remove one or more objects temporarily selected for the target database during data mapping.
Remove a single migration object
In the Target Object(s) list, hover the pointer over the target object and click Remove to remove the migration object.
Remove all migration objects
In the Target Object(s) list, click Remove All in the upper-right corner. In the dialog box that appears, click OK to remove all migration objects.
Click Next. On the Migration Options page, configure the parameters.
Full migration
On the Select Migration Type page, select Full Migration to display the following parameters.
Parameter Description Full Migration Rate Limit You can choose whether to limit the full migration rate as needed. If you choose to limit it, you must specify the RPS and BPS. The RPS specifies the maximum rows of data migrated to the target database per second during full migration, and the BPS specifies the maximum amount of data in bytes migrated to the target database per second during full migration. Note
The RPS and BPS values specified here are only for throttling. The actual full migration performance is subject to factors such as the settings of the source and target databases and the instance specifications.
Full Migration Resource Configuration You can select Small, Medium, or Large to use the corresponding default values of Read Concurrency, Write Concurrency, and Memory. You can also customize the resource configurations for full migration. By setting the resource configuration for the Full-Import component, you can limit the resource consumption of a task in the full migration phase. Notice
In the case of custom configurations, the minimum value is
1, and only integers are supported.Incremental synchronization
On the Select Migration Type page, select Incremental Synchronization to display the following parameters.
Parameter Description Incremental Synchronization Rate Limit You can choose whether to limit the incremental synchronization rate as needed. If you choose to limit it, you must specify the RPS and BPS. The RPS specifies the maximum rows of data synchronized to the target database per second during incremental synchronization, and the BPS specifies the maximum amount of data in bytes synchronized to the target database per second during incremental synchronization. Note
The RPS and BPS values specified here are only for throttling. The actual incremental synchronization performance is subject to factors such as the settings of the source and target databases and the instance specifications.
Incremental Log Pull Resource Configuration You can select Small, Medium, or Large to use the corresponding default value of Memory. You can also customize the resource configurations for incremental log pull. By setting the resource configuration for the Store component, you can limit the resource consumption of a task in log pull in the incremental synchronization stage. Notice
In the case of custom configurations, the minimum value is
1, and only integers are supported.Incremental Data Write Resource Configuration You can select Small, Medium, or Large to use the corresponding default values of Write Concurrency and Memory. You can also customize the resource configurations for incremental data writes. By setting the resource configuration for the Incr-Sync component, you can limit the resource consumption of a task in data writes in the incremental synchronization stage. Notice
In the case of custom configurations, the minimum value is
1, and only integers are supported.Incremental Record Retention Duration The duration that incremental parsed files are cached in OMS. A longer retention duration results in more disk space occupied by the Store component. Incremental Synchronization Start Timestamp - If you have selected Full Migration as the migration type, this parameter is not displayed.
- If you have selected Incremental Synchronization but not Full Migration, specify a point in time after which the data is to be synchronized. The default value is the current system time. For more information, see Set an incremental synchronization timestamp.
If the configuration parameters on the page cannot meet your business requirements, you can click Parameter Configuration at the bottom of the page to configure more parameters. If you have a configured task template or component template, you can also reference it here.
Click Precheck to perform a precheck on the data migration task.
In the Precheck phase, OMS checks whether the network connection to the database meets the requirements. You can start the data migration task only after all precheck tasks are passed. If the precheck fails:
You can troubleshoot and fix the issue, and then perform the precheck again until it succeeds.
You can also click Skip in the Actions column of the failed precheck item. A dialog box pops up to inform you of the specific impact of skipping this operation. After you confirm that you can skip this operation, click OK in the dialog box.
After the precheck succeeds, click Start Task.
If you do not need to start the task, click Save. Subsequently, you can only start the task in Migration Tasks. For more information about how to start a task in batch, see Start a task in batch.
After the task is started, OMS automatically deletes the data migration task from the OBSharding database to the MySQL-compatible tenant of OceanBase Database. OMS saves the physical data migration task between MySQL-compatible tenants of OceanBase Database to the MySQL mode of OceanBase Database. After the task is created, you can click Export CSV in the dialog box to save the task information as a CSV file.
Click OK in the dialog box to go to the Migration Tasks page and start the data migration task between MySQL-compatible tenants of OceanBase Database.
You can click Configure Validation Task in the upper-right corner of the data migration details page to compare the data between the source database and the target database. For more information, see Create a data validation task.
After the data migration task is started, it is executed based on the selected migration types. For more information, see the "View migration details" section in View details of a data migration task.