This topic describes how to use OceanBase Migration Service (OMS) Community Edition to migrate data from OceanBase Database Community Edition to an ElasticSearch database.
Limitations
Limitations on operations in the source database
Do not perform DDL operations that modify database or table schemas during schema migration or full migration. Otherwise, the data migration task may be interrupted.
At present, data migration tasks from OceanBase Database Community Edition to an ElasticSearch database do not support full verification and reverse increment, and do not support migrating tables without primary keys.
The supported versions of Elasticsearch are V7.x and V8.x.
Data source identifiers and user accounts must be globally unique in OMS.
OceanBase Database Community Edition supports only migrating databases, tables, and column objects with ASCII-compliant names that do not contain special characters (including line breaks, spaces, and |"'`()=;/&).
We recommend that you migrate no more than 1,000 tables at a time to avoid impacting the data migration task.
Considerations
If you specify columns to migrate when you create a data migration task, only the specified columns are migrated during schema migration.
In OceanBase Database Community Edition, data of the JSON type is converted to the object type when it is migrated to ElasticSearch. If the task involves full migration and incremental synchronization, make sure that the types of the same-named keys of the JSON data are consistent. If you change the field type in ElasticSearch to text, the data is migrated as a JSON string.
Data type mappings
| OceanBase Database Community Edition | ElasticSearch database |
|---|---|
| tinyint smallint |
short |
| mediumint int |
integer |
| bigint | long |
| float | float |
| decimal numeric double |
double |
| datetime | date |
| timestamp | date |
| time year |
text |
| char varchar |
text |
| tinytext text mediumtext longtext |
text |
| binary varbinary |
binary |
| json | object |
| point | geo_point |
| geometry linestring polygon multipoint multilinestring multipolygon geometrycollection |
geo_shape |
| vector | dense_vector |
Procedure
Create a data migration task.
Log in to the OMS Community Edition console.
In the left-side navigation pane, click Data Migration.
On the Migration Tasks page, click New Task in the upper-right corner.
On the Select Source and Target page, configure the parameters.
Parameter Description Migration Task Name We recommend that you use a combination of Chinese characters, numbers, and letters. The name must not contain spaces and must not exceed 64 characters in length. Tag Click the field and select the target tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data migration tasks by using tags. Source If you have created an OceanBase Community Edition data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create a data source in the dialog box that appears. For more information, see Create an OceanBase-CE data source. Target If you have created an ElasticSearch data source, select it from the drop-down list. If you have not created one, click New Data Source in the drop-down list and create a data source in the dialog box that appears. For more information, see Create an ElasticSearch data source. Click Next. On the Select Migration Type page, configure the parameters.
Migration Type can be Schema Migration, Full Migration, or Incremental Synchronization.
Migration Type Description Schema Migration After a schema migration task is started, OMS Community Edition migrates data object definitions (tables, indexes, constraints, comments, and views) from the source database to the target database and automatically filters out temporary tables.
Manual Table Schema Adjustment Required: OMS Community Edition supports customize table schema. If you select this option, OMS Community Edition only obtains the SQL statements for the table schema. After you modify the schema, the migration proceeds.Full Migration After a full migration task is started, OMS Community Edition migrates existing data from tables in the source database to tables in the target database. If you select Full Migration, we recommend that you collect statistics for OceanBase Community Edition in the source before you migrate data. Incremental Synchronization After an incremental synchronization task is started, OMS Community Edition synchronizes new, modified, or deleted data in the source database to tables in the target database.
Incremental Synchronization is supported only for DML Synchronization. For more information, see DML filtering.Click Next. On the Select Migration Objects page, select the migration objects and the migration scope.
You can select migration objects by using Specify Objects or Match Rules. This topic describes how to select migration objects by using Specify Objects. For more information about how to configure matching rules, see Configure matching rules for migration objects.
Notice
The name of a table to be migrated, as well as the names of columns in the table, must not contain Chinese characters.
The name of the mapped database is
defaultby default after you select a source object.If the name of a database or table contains the characters "$$", an error occurs when you create a data migration task.
OMS Community Edition supports importing objects, renaming table migration objects, configuring row filtering, viewing column information, and removing one or more migration objects.
Operation Steps Import objects - In the right-side list of the Specify Migration Scope section, click Import Objects in the upper-right corner.
- In the dialog box that appears, click OK.
Notice:
Importing objects will overwrite the previously selected objects. Proceed with caution. - In the Import Migration Objects dialog box, import the objects to be migrated.
You can rename databases and tables and configure row filtering by importing CSV files. For more information, see Download and import the settings of migration objects. - Click Validate.
- After the objects pass the validation, click OK.
Rename objects OMS Community Edition allows you to rename table migration objects. For more information, see Rename a database or table. Configure settings OMS allows you to configure row filtering and select columns to be migrated. - In the text box of the
Row Filters section, enter theWHEREclause of a standard SQL statement to configure row filtering. For more information, see Filter data by using SQL conditions. - In the
Select Columns section, select the columns to be migrated.
Remove one or all objects OMS Community Edition allows you to remove a single object or all objects to be migrated to the target database during data mapping. - To remove a single migration object:
In the list on the right of the Specify Migration Scope section, move the pointer over the target object and click Remove. - To remove all migration objects:
In the list on the right of the Specify Migration Scope section, click Remove All in the upper-right corner. In the dialog box that appears, click OK.
Click Next. On the Migration Type page, configure the parameters.
Schema migration
The following parameters are displayed only if you select Full Migration on the Select Migration Type page.
Parameter Description Index Shards (Optional) The number of index shards. This parameter corresponds to the number_of_shardsparameter when you create an index in ElasticSearch.Index Replicas (Optional) The number of index replicas. This parameter corresponds to the number_of_replicasparameter when you create an index in ElasticSearch.Date Format (Optional) The date format. This parameter corresponds to the formatparameter for date fields when you create an index in ElasticSearch.Full migration
The following parameters are displayed only if you select Full Migration on the Select Migration Type page.
Parameter Description Concurrency Speed The migration concurrency. Valid values: Stable, Normal, Fast, and Custom. The performance of full migration varies depending on the migration concurrency. The resources required for a full migration task vary depending on the migration concurrency. If you select Custom, you can set Read Concurrency, Write Concurrency, and JVM Memory. Processing Strategy When Records Exist in Target Object Valid values: Ignore and Stop Migration. - If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Notice
If you select Ignore, data is pulled in IN mode for full verification. In this case, the scenario where the target contains more data than the source cannot be verified, and the verification efficiency will be decreased.
- If you select Stop Migration and a target table contains data, an error is returned during full migration, indicating that the migration is not allowed. In this case, you must clear the data in the target table before you can continue with the migration.
Notice
After an error is returned, if you click Resume in the dialog box, OMS Community Edition ignores this error and continues to migrate data. Proceed with caution.
- If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Incremental synchronization
The following parameters are displayed only if you select Incremental Synchronization on the Select Migration Type page.
Parameter Description Concurrency Speed The migration concurrency. Valid values: Stable, Normal, Fast, and Custom. The performance of incremental synchronization varies depending on the migration concurrency. The resources required for an incremental synchronization task vary depending on the migration concurrency. If you select Custom, you can set Read Concurrency, Write Concurrency, and JVM Memory. Incremental Synchronization Start Timestamp - If you have selected Full Migration as the migration type, this parameter is not displayed.
- If you have selected Incremental Synchronization but not Full Migration, specify the data to be migrated after a specific point in time. The default value is the current system time. For more information, see Set the incremental synchronization point in time.
Incremental Record Retention Time The duration that incremental parsed files are cached in OMS Community Edition. A longer retention period results in more disk space occupied by the Store component of OMS Community Edition.
Click Precheck to perform a precheck on the data migration task.
On the Precheck page, OMS Community Edition checks whether the read and write permissions of the database user and the network connection of the database meet the requirements. You can start a data migration task only after all precheck tasks pass. If an error is returned during the precheck:
You can troubleshoot and fix the error, and then perform the precheck again until the precheck succeeds.
Click Skip in the Actions column of the failed precheck item. In the dialog box that prompts the consequences of the operation, click OK.
Click Start Task to start the data migration task. If you do not need to start the task now, click Save to go to the details page of the data migration task. You can manually start the task as needed.
OMS Community Edition allows you to modify the migration objects in a data migration task during the migration. For more information, see View and modify migration objects. After the data migration task is started, it executes based on the migration type you selected. For more information, see the "View migration details" section in View details of a data migration task.