This topic describes the basic components of OceanBase Migration Service (OMS).
OMS components include DBCat, Store, Full-Import, Incr-Sync, Full-Verification, and Supervisor. The Supervisor component monitors other components.
DBCat
DBCat is the core schema migration component of OMS. As the native schema conversion engine of OceanBase, the DBCat component can accurately map or convert data based on the source data type, destination data type, and character encoding type. The DBCat component can convert and migrate various database objects such as tables, constraints, indexes, and views.
In addition, the DBCat component can be maximally compatible with the tenant types of OceanBase Database. For example, if a version of OceanBase Database does not support some data types in the source database, the DBCat component selects the most compatible data types for conversion and mapping.
The following problems may occur when you migrate data between heterogeneous databases. In this case, you can use the DBCat component to automatically convert the schema of the source database.
The database engines in the source and destination databases are different.
The definitions of data objects vary greatly between databases.
The destination database does not support some data objects in the source database.
The precision settings of the same data type do not match.
In a complex data migration between heterogeneous databases, you can use the DBCat component to generate basic scripts, which are then manually processed by the database administrator (DBA) to generate a schema in the destination database.
Store
The Store component pulls data in different ways from different types of databases. For example, the Store component of an OceanBase database is implemented based on Liboblog.
Liboblog is an incremental data synchronization tool of OceanBase Database. It pulls the redo logs from each partition of an OceanBase database by using remote procedure calls (RPCs), converts the redo logs into an intermediate data format based on the schema information of each table and column, and then outputs the modified data as a transaction.
Full-Import
The Full-Import component migrates the static data and some incremental data from the source database to the destination database. Each migrated table will undergo a series of steps including data sharding, data reading, data processing, and data application.
The Full-Import component selects a proper sharding method based on the schema of a table to concurrently migrate data in the table to the destination. For most data sources, full data is read in streaming mode to reduce the reading pressure on the source database.
Full data is batch-inserted into the destination database. For partitioned tables in OceanBase Database, data is written and aggregated by partition to improve writing efficiency. The selected tables are concurrently migrated. The sharding timestamp ACK mechanism is used for tables to ensure data integrity.
Incr-Sync
The Incr-Sync component synchronizes the incremental data generated by the Store component to the destination database. OMS abstracts the synchronized incremental records as DML, DDL, and HB operations. DML operations are INSERT, UPDATE, and DELETE. OMS efficiently and correctly synchronizes these operations to the destination database to ensure final data consistency.
For a table with a UNIQUE constraint, OMS ensures final data consistency based on the idempotence of operations performed on the primary key columns.
For a table without a primary key, OMS ensures final data consistency by using the transaction table mechanism to avoid transaction replay.
Currently, OMS does not guarantee final data consistency when you migrate data from a MySQL database to a table without a primary key in a MySQL tenant of OceanBase Database.
Full-Verification
The Full-Verification component reads data from the source and destination databases, compares all fields of the data tables in the source database with those in the destination database based on the index-dependent mappings, and generates files of inconsistent data and revised SQL statements.
The Full-Verification component selects a proper sharding method based on the schema of a table to read data from the database. This way, the component reads data in streaming mode from most data sources and reduces the reading pressure on the source database.
The Full-Verification component supports data comparison between various heterogenous data sources by formatting the data, which ensures the comparison performance and data consistency. The Full-Verification component also supports multiple rounds of data checks to reduce false positives caused by the synchronization latency.
