Incremental direct load in obcdc

Background information

Full direct load is supported in OceanBase Database V4.3.0, but it does not deliver the optimal import efficiency. Therefore, incremental direct load is introduced in OceanBase Database V4.3.1. Simply put, incremental direct load is a replacement semantics that directly replaces rows with duplicate primary keys without checking for primary key conflicts. For more information, see Overview of direct load. To synchronize the imported data to downstream services, OceanBase Change Data Capture (obcdc) needs to adapt to the semantics of incremental direct load.

Limitations

obcdc cannot synchronize large object (LOB) data stored in OUTROW mode.

Prerequisites

• You have deployed obcdc V4.3.1 or later.

• You have deployed OceanBase Database V4.3.1 or later.

Procedure

Assume that the A and B tables exist in the OceanBase cluster. The A table contains the following data, with the col1 column as the primary key:

+------+------+
| col1 | col2 |
+------+------+
|    1 |   11 |
|    2 |   22 |
+------+------+

The B table contains the following data, with the col1 column as the primary key:

+------+------+
| col1 | col2 |
+------+------+
|    1 |   21 |
|    3 |   33 |
+------+------+

The following example demonstrates the obcdc operations that you must perform to import data from the B table to the A table in incremental direct load.

1. Modify the configuration file.

   In the configuration file, set enable_direct_load_inc to 1 to enable incremental direct load. For more information about the parameters, see obcdc parameters.

   enable_direct_load_inc=1
   

2. Apply the configuration.

   A restart is required for the enable_direct_load_inc parameter to take effect. You can apply this configuration by using one of the following two methods based on the way you use obcdc.

   • If you connect obcdc to your data consumption link, compile and execute the code again. For more information, see obcdc development instructions.

   • If you use obcdc through obcdc_tailf, run the obcdc_tailf command again. Here is an example:

     [admin@test oceanbase]$ ./bin/obcdc_tailf -f libobcdc.conf -D output.txt
     

     In the preceding example, libobcdc.conf is the relative path of the configuration file, and output.txt is the file where output data is saved. Replace the configuration file and the output data file as needed. For more information, see obcdc_tailf.

3. Log in to OceanBase Database, and import data in incremental direct load mode.

   obclient [test]> insert /*+ enable_parallel_dml parallel(2) direct(true, 0, 'inc_replace') */ into A select * from B;
   

   For more information, see Import data in direct load mode by using the INSERT INTO SELECT statement.

4. View the execution result.

   obclient [test]> SELECT * FROM A;
   

   The following output indicates that data is imported from the B table to the A table:

   +------+------+
   | col1 | col2 |
   +------+------+
   |    1 |   21 |
   |    2 |   22 |
   |    3 |   33 |
   +------+------+
   

5. View the output of obcdc.

   After you import data in incremental direct load mode, obcdc outputs the corresponding messages to the downstream services. Here is a simplified example:

   [EBGIN];
   [EPUT] column_name:col1, old_value:1, new_value:1, column_name: col2, new_value: 21;
   [EPUT] column_name:col1, old_value:3, new_value:3, column_name: col2, new_value: 33;
   [ECOMMIT];