Overview of direct load

OceanBase Database allows you to insert data into the database using direct load. In other words, you can write data directly into data files. Direct load enables you to bypass the SQL layer interface, directly allocate space, and insert data into data files, thereby improving data import efficiency.

Direct load modes

OceanBase Database supports full direct load and incremental direct load.

When you use the LOAD DATA, INSERT INTO SELECT, or CREATE TABLE AS SELECT statement to import data, you can specify direct load for the import task by adding a hint to the statement, or by setting the global parameter default_load_mode.

• Full direct load

  • Full direct load enables you to write an entire dataset directly into a database data file in a single operation. This method bypasses the SQL layer, directly allocates space, and inserts data into the data file, thereby improving data import efficiency.
  • Full direct load is typically used for database initialization, data migration, or quick loading of large amounts of data.

• Incremental direct load

  • Incremental direct load allows you to write new data directly into a database data file, without using SQL interfaces, even when the database already contains a large amount of data. This method bypasses SQL layer processing and writes new data directly to the data file, thus improving data write efficiency.
  • Incremental direct load is typically used in high-throughput data write scenarios, such as large-scale real-time data collection and log write.

Scenarios

The direct load feature can be used in the following scenarios:

• Data migration and synchronization. When migrating and synchronizing data, you often need to move large volumes of data in various formats from different data sources to OceanBase Database. The performance of traditional SQL interfaces may not meet timeliness requirements.

• Traditional ETL. After data is extracted and transformed at the source, large amounts of data often need to be loaded into the destination within a short period. Direct load technology can improve data import performance. During the data loading phase of ETL processes, you can also use direct load to enhance efficiency.

• Load data from text files or other data sources to OceanBase Database. The direct load technology can also improve the data loading efficiency.

References

• Full direct load
• Incremental direct load