Overview

Data migration is a core task in data management, involving the transfer of data from one system or storage environment to another to meet business expansion, system upgrades, data integration, or compliance needs. Based on the types of data sources and target systems, data states, and migration scenarios, data migration can be classified into three main types: heterogeneous data offline migration, homogeneous database migration, and big data ecosystem integration migration.

Heterogeneous data migration

Heterogeneous data migration involves migrating data from external data sources (such as MySQL, Oracle, PostgreSQL, and various NoSQL databases) to OceanBase Database. This type of migration requires addressing issues such as data format conversion, schema adaptation, and cross-platform compatibility. In these migration scenarios, the following considerations are important:

• Schema and data type mapping: It is essential to precisely define the data structures and type mapping rules between the source and target (OceanBase). For example, converting MySQL's DATETIME to OceanBase's TIMESTAMP.
• Performance optimization: For large-scale data migration, efficient strategies such as parallel processing, sharding, and batch loading should be adopted to shorten the migration window.
• Data consistency verification: After migration, data integrity and accuracy must be ensured through methods such as hash verification, sampling comparison, or full comparison.

For more information about heterogeneous data migration, see Heterogeneous data migration.

Homogeneous data migration

Homogeneous data migration primarily refers to data migration between OceanBase clusters. This type of migration typically occurs in the following scenarios:

• Version upgrade: Migrating data from an OceanBase cluster of an earlier version to a new version to obtain new features or improve performance.
• Data migration between TP and AP databases: Migrating data from an OceanBase transactional (TP) database to an analytical (AP) database to support real-time analytics or business intelligence (BI) applications.

Since both the source and target are OceanBase, this type of migration usually does not require data format conversion. However, the following considerations are still important:

• Data consistency during migration: Ensuring data consistency during migration through transaction logs or snapshot mechanisms.
• Downtime control: Selecting an appropriate migration solution based on business requirements for RTO and RPO to minimize or avoid business interruptions.
• Version compatibility: Being mindful of subtle differences in SQL syntax, system parameters, or internal implementations between different versions during cross-version migration.

For more information about homogeneous data migration, see Homogeneous data migration.

Big data ecosystem integration

Big data ecosystem integration migration specifically involves data sources that already exist in offline data warehouses (such as Hive, Spark, HBase, or Parquet/ORC files) or within big data platform components, rather than directly from traditional OLTP databases or real-time message queues. The core goal of this type of migration is to efficiently and reliably integrate or migrate the data, which has been processed or stored in the big data ecosystem, to an OceanBase AP database or other target systems to support advanced analytics, real-time queries, or collaboration with other business systems. This type of migration typically occurs in the following scenarios:

• Synchronizing offline data warehouse data to real-time analytics systems: Synchronizing data from offline data warehouses such as Hive and Spark to real-time data processing platforms such as Flink and Kafka to support real-time data processing and streaming analytics.
• Archiving offline data to long-term storage: Archiving historical data from offline data warehouses to long-term storage such as Amazon S3 or Alibaba Cloud OSS to reduce storage costs and meet data retention policies.
• Integrating offline data with external systems: Migrating Hive data to external systems such as Snowflake for cross-team collaboration or integrating big data platform data with business systems.

In these migration scenarios, the following considerations are important:

• Storage format conversion: Converting data formats such as Parquet to Avro requires considering factors such as compression rates, query performance, and compatibility.
• Efficient transfer of large-scale data: Using distributed tools such as DistCp or Spark with parallel processing and network optimization to enhance transfer efficiency.
• Metadata synchronization: Ensuring that source-side metadata such as schema and partition information from the Hive Metastore is correctly synchronized and mapped to the target schema in OceanBase AP.

For more information about big data ecosystem integration, see Big data ecosystem integration.