This topic describes how to use OceanBase Migration Service (OMS) Community Edition to migrate data from a Hive database to OceanBase Database Community Edition.
Background information
OceanBase Migration Service (OMS) Community Edition supports the following modes for data migration from a Hive database:
Hive external table mode: Data is migrated from the Hive database to OceanBase Database in the community edition by using Hive external tables.
Spark local mode: A Spark container is started internally in the synchronization process of OMS Community Edition, and data is written from the Hive database to OceanBase Database in the community edition in the Spark container.
Spark cluster mode: The synchronization process of OMS Community Edition submits a task to the Spark cluster, and data is written from the Hive database to OceanBase Database in the community edition in the task.
Limitations
Limitations on operations in the source database
Do not perform DDL operations for database or schema changes during full data migration. Otherwise, the data migration task may be interrupted.
Only Hive databases of versions V2.1.1 and V3.1.3 are supported.
Only full data migration is supported for migrating data from a Hive database to OceanBase Database in the community edition.
Data source identifiers and user accounts must be globally unique in OMS.
OMS Community Edition supports migrating databases, tables, and column objects with ASCII-compliant names that do not contain special characters (|"'`()=;/& or spaces).
We recommend that you do not migrate more than 1,000 tables at a time to avoid impacting the performance of data migration tasks.
Data type mappings
| Hive database | OceanBase Database Community Edition |
|---|---|
| TINYINT | TINYINT |
| SMALLINT | SMALLINT |
| INT | INT |
| BIGINT | BIGINT |
| BOOLEAN Valid values: true and false. |
BOOLEAN/TINYINT(1) Valid values: 0 and 1. |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| DECIMAL | DECIMAL |
| STRING | LONGTEXT/TINYTEXT/MEDIUMTEXT/TEXT/VARCHAR/CHAR You can map the data type to any supported data type in OceanBase Database Community Edition based on your business requirements. |
| VARCHAR | VARCHAR |
| CHAR | CHAR |
| TIMESTAMP | TIMESTAMP |
| TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP
|
| DATE | DATE |
| BINARY |
|
| STRUCT | JSON This type is supported in external Hive tables. For example, STRUCT<field1:STRING, field2:INT>. The data format is ["example",42]. |
| MAP | JSON This type is supported in external Hive tables. For example, MAP<STRING, INT>. The data format is {"key1":1,"key2":2}. |
| ARRAY | JSON This type is supported in external Hive tables. For example, ARRAY<STRING>. The data format is ["item1","item2","item3"]. |
| UNIONTYPE | JSON This type is supported in external Hive tables. For example, the value in Hive is {1:9.9}, and after migration to OceanBase Database Community Edition, it will be converted to the JSON format {"tag": 1, "object": 9.9}. |
Procedure
Create a data migration task.
Log in to the OMS Community Edition console.
In the left-side navigation pane, click Data Migration.
On the Data Migration page, click New Task in the upper-right corner.
On the Select Source and Target page, configure the related parameters.
Parameter Description Task Name The name can contain Chinese characters, numbers, and underscores. It cannot contain spaces and must be 64 characters or less. Tag (Optional) Click the field and select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data migration tasks by using tags. Source If you have created a Hive data source, select it from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create a Hive data source. Destination If you have created an OceanBase Community Edition data source, select it from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create an OceanBase Community Edition data source. Click Next.
On the Select Migration Type page, select Full Migration.
After you select Full Migration, OMS Community Edition migrates the existing data in the source database tables to corresponding tables in the destination database.
Click Next. On the Select Migration Objects page, select the migration objects and scope.
You can select objects to migrate by using the Specify Objects and Match Rules tabs. This topic describes how to select migration objects by using the Specify Objects tab. For information about how to configure matching rules, see Configure matching rules for migration objects.
Notice
The name of a table to be migrated, as well as the names of columns in the table, must not contain Chinese characters.
If the name of a database or table contains the characters "$$", an error occurs when you create a data migration task.
In the Select Migration Objects section, select Specify Objects.
In the Specify Migration Scope section, select the source database tables on which you want to perform full migration from the Source Object(s) list. You can select tables and views in one or more databases.
Click > to add the selected objects to the Target Object(s) list.
When you migrate data from a Hive database to an OceanBase Community Edition database, OMS Community Edition allows you to import objects by using a text file, rename objects, and remove one or all migration objects.
Operation Steps Import objects - In the right-side list of the Specify Migration Scope section, click the Import Objects icon in the upper-right corner.
- In the dialog box that appears, click OK.
Notice:
The import operation will overwrite the previous selection. Proceed with caution. - In the Import Migration Objects dialog box, select the objects that you want to migrate.
You can rename databases and tables by importing a CSV file. For more information, see Download and import settings of migration objects. - Click Validate.
- If the validation succeeds, click OK.
Rename objects OMS Community Edition allows you to rename migration objects. For more information, see Rename database objects. Remove objects OMS Community Edition allows you to remove one or more objects that have been moved to the destination. - Remove a migration object
In the right-side list of the Specify Migration Scope section, hover the pointer over the target object and click the Remove icon displayed next to the object. - Remove all migration objects
In the right-side list of the Specify Migration Scope section, click the Remove All icon in the upper-right corner. In the dialog box that appears, click OK.
Click Next. On the Migration Type page, configure the related parameters.
Notice
You can select Full Migration on the Select Migration Type page.
Parameter Description Concurrency The concurrency mode. Valid values: Stable, Normal, Fast, and Custom. The performance of a full migration task varies with the concurrency mode. In the Custom mode, you can specify the Read Concurrency, Write Concurrency, and JVM Memory parameters based on your business requirements. Strategy when the destination object contains records The strategy. Valid values: Ignore and Stop Migration: - If you set the value to Ignore, when data exists in the destination table, OMS Community Edition writes data to the destination by logging conflicts and keeping the original data unchanged.
Notice
If you select Skip, the full verification will use the IN mode to fetch data, which cannot verify the cases where the destination contains data that does not exist in the source, and the verification performance will be degraded to some extent.
- If you set the value to Stop Migration, when data exists in the destination table, the full migration task will fail and you need to remove data in the destination before continuing the migration.
Notice
If you click Restore, OMS Community Edition will ignore this setting and continue to migrate table data. Proceed with caution.
Computing Platform The computing platform. Valid values: Local (local runtime mode) and Manage Computing Platform. To add a computing platform, select Manage Computing Platform in the drop-down list. For more information, see Manage computing platforms. Write Method The write method. Valid values: SQL ( INSERTorREPLACEto write data to a table) and Direct Load (direct load mode). Currently, data of vector types cannot be written by using the direct load mode.To view or modify the parameters of the full migration component, click Full Migration > Configuration Details in the upper-right corner. For more information about the parameters, see Component parameters.
- If you set the value to Ignore, when data exists in the destination table, OMS Community Edition writes data to the destination by logging conflicts and keeping the original data unchanged.
Click Precheck. The system checks whether the network connection meets the requirements.
In the Precheck step, OMS Community Edition checks whether the network connection meets the requirements. The data migration task will not start unless all checks succeed. If the precheck fails:
You can troubleshoot the issues and retry the precheck until it succeeds.
You can also click Skip in the Actions column of the failed precheck item. A dialog box will appear, showing the impact of skipping the operation. After you confirm the skip, click OK in the dialog box.
Click Start Task. If you do not want to start the task immediately, you can click Save. The system will redirect you to the details page of the data migration task. You can start the task later as needed.
During the running of a data migration task, you can modify the migration objects. For more information, see View and modify migration objects. After you start a data migration task, the task will perform the operations in sequence based on the selected migration type. For more information, see View task details.