This topic describes the preparations to be made before you use OceanBase Migration Service (OMS) to migrate data larger than 10 TB. Before data migration, you can preprocess data and properly plan the migration task to improve the migration efficiency and reduce the O&M costs of the migration task.
Specifically, you can identify and classify large tables, partitioned tables, and tables that may require special processing, such as tables with large object (LOB) data.
Plan the migration task
Distribute tables to different links
Evenly distribute large tables to different links, with no more than 2,000 tables on each link to avoid an excessively long allowlist.
Single out HASH- or KEY-partitioned tables
Independent verification rules are required for HASH- or KEY-partitioned tables. If the HASH- or KEY-partitioned tables do not have large amounts of data, you can arrange them on the same link so that you can easily exclude them from the verification link when they are no longer needed. If a source table is a HASH- or KEY-partitioned table and the target table uses a different partitioning algorithm, the same data may be distributed in different partitions in the source and target. In this case, excessive data inconsistency errors may be falsely reported if a regular verification mode is used. We recommend that you use the IN mode for verification.
Migrate cold and hot tables separately.
When OMS is started for the first time, the incremental synchronization can be very slow due to a hot table. If this table is not large, you can shield it from the links and included it in the full synchronization task during cutover.
Distribute tables other than large tables and hash tables to different links.
Create an independent task to migrate tables with LOB data to avoid slowing down the entire migration task.
Separately migrate large tables with data of the character large object (CLOB), *LOB, *text, JSON, or geographic information system (GIS) data type.
The migration of tables with data of the preceding data types is very slow, which will slow down the entire migration progress.
Direct load
At present, only OMS Community Edition V4.2.4 and later support direct load. Direct load is the optimal choice for the migration of large tables. With the direct load mode, you can bypass interfaces in the SQL layer and directly allocate space and insert data into data files, thereby improving the data import efficiency.
Configure direct load
When you configure a migration task in OMS, you can enable direct load for migration.
Notice
In direct load mode, if the target table contains data, you need to empty the table before importing data to it. If the direct load is interrupted due to an error, you still need to empty the table before you restore the migration task.
Verify whether direct load is enabled
If the following information is displayed on the parameter configuration page of the migration task, direct load is successfully enabled:
"sink": { "com.oceanbase.oms.connector.direct.sink.DirectPathSinkBooster" }