This topic describes how to use OceanBase Migration Service (OMS) Community Edition to migrate data from a Hive database to OceanBase Database Community Edition.
Background information
OceanBase Migration Service (OMS) Community Edition supports the following modes for data migration from a Hive database:
Hive external table mode: Data is migrated from the Hive database to OceanBase Database in the community edition by using Hive external tables.
Spark local mode: A Spark container is started internally in the synchronization process of OMS Community Edition, and data is written from the Hive database to OceanBase Database in the community edition in the Spark container.
Spark cluster mode: The synchronization process of OMS Community Edition submits a task to the Spark cluster, and data is written from the Hive database to OceanBase Database in the community edition in the task.
Limitations
Limitations on operations in the source database
Do not perform DDL operations that modify database or table schemas during full migration. Otherwise, the data migration task may be interrupted.
Only Hive databases of versions V2.1.1 and V3.1.3 are supported.
At present, data migration tasks from a Hive database to OceanBase Database Community Edition support only full migration.
Data source identifiers and user accounts must be globally unique in OMS.
OMS Community Edition supports migrating databases, tables, and column objects with ASCII-compliant names that do not contain special characters (including line breaks, spaces, and |"'`()=;/&).
We recommend that you do not migrate more than 1,000 tables at a time to avoid impacting the performance of data migration tasks.
Data type mappings
| Hive database | OceanBase Database Community Edition |
|---|---|
| TINYINT | TINYINT |
| SMALLINT | SMALLINT |
| INT | INT |
| BIGINT | BIGINT |
| BOOLEAN Valid values: true and false. |
BOOLEAN/TINYINT(1) Valid values: 0 and 1. |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| DECIMAL | DECIMAL |
| STRING | LONGTEXT/TINYTEXT/MEDIUMTEXT/TEXT/VARCHAR/CHAR You can map the data type to any supported data type in OceanBase Database Community Edition based on your business requirements. |
| VARCHAR | VARCHAR |
| CHAR | CHAR |
| TIMESTAMP | TIMESTAMP |
| TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP
|
| DATE | DATE |
| BINARY |
|
| STRUCT | JSON This type is supported in external Hive tables. For example, STRUCT<field1:STRING, field2:INT>. The data format is ["example",42]. |
| MAP | JSON This type is supported in external Hive tables. For example, MAP<STRING, INT>. The data format is {"key1":1,"key2":2}. |
| ARRAY | JSON This type is supported in external Hive tables. For example, ARRAY<STRING>. The data format is ["item1","item2","item3"]. |
| UNIONTYPE | JSON This type is supported in external Hive tables. For example, the value in Hive is {1:9.9}, and after migration to OceanBase Database Community Edition, it will be converted to the JSON format {"tag": 1, "object": 9.9}. |
Procedure
Create a data migration task.
Log in to the OMS Community Edition console.
In the left-side navigation pane, click Data Migration.
On the Data Migration page, click New Task in the upper-right corner.
On the Select Source and Target page, configure the related parameters.
Parameter Description Migration Task Name We recommend that you use a name that contains Chinese characters, numbers, and letters. The name cannot contain spaces and must be 64 characters or less in length. Tag (Optional) Click the field and select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data migration tasks by using tags. Source If you have created a Hive data source, select it from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create a Hive data source. Target If you have created an OceanBase Community Edition data source, select it from the drop-down list. If not, click New Data Source in the drop-down list to create a data source. For more information, see Create an OceanBase Community Edition data source. Click Next.
On the Select Migration Type page, select Full Migration.
After you select Full Migration, OMS Community Edition migrates the existing data in the source database tables to corresponding tables in the target database.
Click Next. On the Select Migration Objects page, select the migration objects and scope.
You can select objects to migrate by using the Specify Objects and Match Rules tabs. This topic describes how to select migration objects by using the Specify Objects tab. For information about how to configure matching rules, see Configure matching rules for migration objects.
Notice
The name of a table to be migrated, as well as the names of columns in the table, must not contain Chinese characters.
If the name of a database or table contains the characters "$$", an error occurs when you create a data migration task.
In the Select Migration Objects section, select Specify Objects.
In the Specify Migration Scope section, select the source database tables on which you want to perform full migration from the Source Object(s) list. You can select tables and views in one or more databases.
Click > to add the selected objects to the Target Object(s) list.
When you migrate data from a Hive database to an OceanBase Community Edition database, OMS Community Edition allows you to import objects by using a text file, rename objects, and remove one or all migration objects.
Operation Steps Import objects - In the right-side list of the Specify Migration Scope section, click the Import Objects icon in the upper-right corner.
- In the dialog box that appears, click OK.
Notice:
The import operation will overwrite the previous selection. Proceed with caution. - In the Import Migration Objects dialog box, select the objects that you want to migrate.
You can rename databases and tables by importing a CSV file. For more information, see Download and import settings of migration objects. - Click Validate.
- If the validation succeeds, click OK.
Rename objects OMS Community Edition allows you to rename migration objects. For more information, see Rename database objects. Remove one or all objects OMS Community Edition allows you to remove one or more objects that have been moved to the target. - Remove a migration object
In the right-side list of the Specify Migration Scope section, hover the pointer over the target object and click the Remove icon displayed next to the object. - Remove all migration objects
In the right-side list of the Specify Migration Scope section, click the Remove All icon in the upper-right corner. In the dialog box that appears, click OK.
Click Next. On the Migration Type page, configure the related parameters.
Notice
You can select Full Migration on the Select Migration Type page.
Parameter Description Concurrency The concurrency mode. Valid values: Stable, Normal, Fast, and Custom. The performance of a full migration task varies with the concurrency mode. In the Custom mode, you can specify the Read Concurrency, Write Concurrency, and JVM Memory parameters based on your business requirements. Processing Strategy When Records Exist in Target Object Valid values: Ignore and Stop Migration. - If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Notice
If you select Ignore, data is pulled in IN mode for full verification. In this case, the scenario where the target contains more data than the source cannot be verified, and the verification efficiency will be decreased.
- If you select Stop Migration and a target table contains data, an error is returned during full migration, indicating that the migration is not allowed. In this case, you must clear the data in the target table before you can continue with the migration.
Notice
After an error is returned, if you click Resume in the dialog box, OMS Community Edition ignores this error and continues to migrate data. Proceed with caution.
Computing Platform The computing platform. Valid values: Local (local runtime mode) and Manage Computing Platform. To add a computing platform, select Manage Computing Platform in the drop-down list. For more information, see Manage computing platforms. Write Method The write method. Valid values: SQL ( INSERTorREPLACEto write data to a table) and Direct Load (direct load mode). Currently, data of vector types cannot be written by using the direct load mode.To view or modify the parameters of the full migration component, click Full Migration > Configuration Details in the upper-right corner. For more information about the parameters, see Component parameters.
- If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Click Precheck. The system checks whether the network connection meets the requirements.
In the Precheck step, OMS Community Edition checks whether the network connection meets the requirements. The data migration task will not start unless all checks succeed. If the precheck fails:
You can troubleshoot the issues and retry the precheck until it succeeds.
Click Skip in the Actions column of the failed precheck item. In the dialog box that prompts the consequences of the operation, click OK.
Click Start Task. If you do not want to start the task immediately, you can click Save. The system will redirect you to the details page of the data migration task. You can start the task later as needed.
During the running of a data migration task, you can modify the migration objects. For more information, see View and modify migration objects. After you start a data migration task, the task will perform the operations in sequence based on the selected migration type. For more information, see View task details.