Overview

OceanBase Database allows you to insert data into a database by using direct load. In other words, OceanBase Database can directly write data into data files. With the direct load feature, you can get around APIs at the SQL layer and directly allocate space and insert data into data files, thereby improving the data import efficiency.

Direct load methods

OceanBase Database supports two direct load methods: full direct load and incremental direct load.

• Full direct load

  • Full direct load is to write a complete data set into the data file of a database at one time. It can get around APIs at the SQL layer and directly allocate space and insert data in the data file, improving the data import efficiency.
  • Full direct load is generally used for database initialization, data migration, or quick load of a large amount of data.
  • You can use the LOAD DATA and INSERT INTO SELECT statements for full direct load.

• Incremental direct load

  • Incremental direct load is to write new data into the data file of a database when a large amount of data already exists in the file, without the need for SQL APIs. It can get around data processing at the SQL layer and directly write new data into the data file, improving the data write efficiency.
  • Incremental direct load is generally used in data write scenarios with a high throughput, such as large-scale real-time data collection and log write.
  • You can use the LOAD DATA and INSERT INTO SELECT statements for incremental direct load.

Scenarios

The direct load feature applies to the following scenarios:

• Data migration and synchronization: In data migration or synchronization, a large amount of data in different formats must be migrated from different data sources to OceanBase Database. Conventional SQL APIs cannot meet the requirement on timeliness.

• Conventional extract, transform, and load (ETL): After data is extracted and transformed in the source, a large amount of data must be loaded to the destination within a short time. The direct load technology can improve the import performance. As for the ELT technology, the data load efficiency can also be improved by using direct load.

• Data load from text files or other data sources to OceanBase Database: Direct load can accelerate the data load process.

References

• Full direct load
• Incremental direct load