Direct load

Background information

OceanBase Database allows you to insert data into a database by using direct load. In other words, OceanBase Database can directly write data into data files. With the direct load feature, you can get around APIs at the SQL layer and directly allocate space and insert data into data files, thereby improving the data import efficiency.

Scenarios

The direct load feature applies to the following scenarios:

• Data migration and synchronization: In data migration or synchronization, a large amount of data in different formats often needs to be migrated from different data sources to OceanBase Database. Conventional SQL APIs cannot meet the requirement on timeliness.

• Conventional extract, transform, and load (ETL): After data is extracted and transformed in the source, a large amount of data often needs to be loaded to the target within a short time. The direct load technology can improve the import performance.

• Data load from text files or other data sources to OceanBase Database: Direct load can accelerate the data load process.

Considerations

• In direct load, the remote procedure call (RPC) port instead of the SQL port is used to transmit data.

• Data to be imported is submitted at the table level instead of the session or transaction level.

• Retry or resumable transmission is not supported.

• Bit data types are not supported.

• Virtual generated columns are not supported.

• We recommend that you do not import a small amount of data in direct load mode.

• The --replace-data option cannot help address unique index conflicts.

• The differences between the --thread and --parallel options are as follows:

  • --thread specifies the thread pool for connections from the client to the server, and is maintained on the client.

  • --parallel specifies the number of worker threads that can be called by the OBServer node for data writing and sorting.

  • We recommend that you specify consistent values for --thread and --parallel.

Command-line options for direct load

Parameters for direct load

You can configure parameters for direct load in the session.config.json file in the {ob-loader-dumper}/conf directory.

Here is an example:

"direct_path_load": {
    "task_timeout_ms": "25920000", 
    "heartbeat_timeout_ms": "60000000",
    "heartbeat_interval_ms": "30000000" 
}

• task_timeout_ms: the timeout period of an import operation, in ms. If an import operation is not completed within the configured period, it is considered timed out.

• heartbeat_timeout_ms: the heartbeat timeout period in ms, which is used to detect the active status of an import operation.

• heartbeat_interval_ms: the interval between two heartbeats, in ms.