Data migration overview

Data migration is a core task in data management, involving the transfer of data from one system or storage environment to another to meet business expansion, system upgrades, data integration, or compliance requirements. Based on the types of data sources and target systems, the state of the data, and the migration scenarios, data migration can be categorized into three main types: heterogeneous data offline migration, homogeneous database migration, and big data ecosystem integration migration.

Heterogeneous data migration

Heterogeneous data migration refers to the process of migrating data from external data sources, such as relational databases like MySQL, Oracle, and PostgreSQL, or NoSQL databases, to OceanBase Database. This type of migration requires addressing challenges related to data format conversion, structural adaptation, and cross-platform compatibility. In such scenarios, the following considerations are important:

• Schema and data type mapping: Accurately define the mapping rules between the source and target (OceanBase Database) data structures and types. For example, convert MySQL's DATETIME to OceanBase Database's TIMESTAMP.
• Performance optimization: For large-scale data migration, adopt efficient strategies like parallel processing, sharding migration, and batch loading to minimize the migration window.
• Data consistency verification: After migration, use methods such as hash checks, sampling comparisons, or full comparisons to ensure data integrity and accuracy.

For more information about heterogeneous data migration, see Heterogeneous data migration.

Homogeneous data migration

Homogeneous data migration primarily involves migrating data between OceanBase Database clusters. This type of migration typically occurs in the following scenarios:

• Version upgrade: Migrate data from an older version of OceanBase Database to a newer version to access new features or improve performance.
• TP to AP database migration: Migrate data from OceanBase Database's transactional (TP) database to its analytical (AP) database to support real-time analytics or BI applications.

Since both the source and target are OceanBase Database, data format conversion is usually not required. However, the following considerations remain important:

• Data consistency during migration: Use transaction logs or snapshot mechanisms to ensure data consistency during the migration process.
• Downtime control: Choose an appropriate migration strategy based on business requirements for RTO/RPO to minimize or avoid business interruptions.
• Version compatibility: When migrating across versions, be aware of subtle differences in SQL syntax, system parameters, or internal implementations between versions.

For more information about homogeneous data migration, see Homogeneous data migration.

Big data ecosystem integration

Big data ecosystem integration migration specifically refers to scenarios where data sources already exist in offline data warehouses (such as Hive, Spark, HBase, Parquet/ORC files) or big data platform components, rather than directly from traditional OLTP databases or real-time message queues. The core goal of this type of migration is to efficiently and reliably integrate or migrate data that has already been processed or stored within the big data ecosystem to OceanBase Database's AP database or other target systems. This supports advanced analytics, real-time queries, and collaboration with other business systems. This type of migration typically occurs in the following scenarios:

• Offline data warehouse data needs to be synchronized to a real-time analytics system: Synchronize data from offline data warehouses like Hive or Spark to real-time data processing platforms like Flink or Kafka to support real-time data processing and streaming analytics.
• Offline data needs to be archived to long-term storage: Archive historical data from offline data warehouses to long-term storage solutions like cloud object storage (S3) or Alibaba Cloud OSS to reduce storage costs and meet data retention policies.
• Offline data needs to be integrated with external systems: Migrate Hive data to external systems like Snowflake for cross-team collaboration, or integrate data from big data platforms with business systems.

In these migration scenarios, the following considerations are important:

• Storage format conversion: When converting between formats like Parquet to Avro, consider factors such as compression rates, query performance, and compatibility.
• Efficient large-scale data transfer: Use distributed tools like DistCp or Spark, leveraging parallel processing and network optimization to enhance transfer efficiency.
• Metadata synchronization: Ensure that metadata (such as table structures and partition information) from the source (e.g., Hive Metastore) is correctly synchronized and mapped to the target table structures in OceanBase Database's AP database.

For more information about big data ecosystem integration, see Big data ecosystem integration.