Performance tuning

obbinlog V4.2.0 and later support concurrent log conversion. The following figure shows the overall logic.

As shown in the figure, obbinlog introduces Disruptor-cpp into binlog instances and uses the ring buffer provided by Disruptor-cpp to implement concurrent execution of the binlog conversion, serialization, and release phases. For information about the related parameters, see the Cluster parameters section in the Metadata topic.

Configuration templates

Resource usage of the binlog service consists of the following two parts:

• Log pulling of OceanBase Change Data Capture (obcdc): Memory usage is limited by the binlog_memory_limit parameter. When the binlog_working_mode parameter is set to memory, obcdc stores the data to be consumed in memory, which increases the memory usage. When the binlog_working_mode parameter is set to storage, the data is stored on the disk, which increases the disk usage.

• Log format conversion and client subscription: Converted log files are stored on the disk. Therefore, the disk usage increases over time. Memory usage and CPU utilization are affected by the sizes of the ring buffer, preallocated memory, and thread pool.

When the default values are used for the processing queue size of obcdc and preallocated memory size of the binlog service, the following configuration templates are available for different hardware configurations.

Template 1: 2 CPU cores and 8 GB of memory

Estimated resource usage:

• Memory: 4 GB + 4 GB + 32 MB
• CPU: 2 CPU cores

Estimated performance: 50,000 to 60,000 requests per second (RPS)

The following table describes the parameter settings.

Template 2: 4 CPU cores and 8 GB of memory

Estimated resource usage:

• Memory: 4 GB + 4 GB + 64 MB
• CPU: 4 CPU cores

Estimated performance: 80,000 to 100,000 RPS

The following table describes the parameter settings.

Template 3: 8 CPU cores and 9 GB of memory

Estimated resource usage:

• Memory: 4 GB + 4 GB + 1 GB
• CPU: 8 CPU cores

Estimated performance: 100,000 to 150,000 RPS

The following table describes the parameter settings.

Template 4: 12 CPU cores and 9 GB of memory

Estimated resource usage:

• Memory: 4 GB + 4 GB + 1 GB
• CPU: 12 CPU cores

Estimated performance: 150,000 to 200,000 RPS

The following table describes the parameter settings.

Template 5: 16 CPU cores and 10 GB of memory

Estimated resource usage:

• Memory: 4 GB + 4 GB + 2 GB
• CPU: 16 CPU cores

Estimated performance: 200,000 to 250,000 RPS

The following table describes the parameter settings.

Performance tuning

Binlog format conversion is compute-intensive. We recommend that the number of threads not exceed the number of CPU cores. Take note of the following considerations:

• Number of threads ≈ Number of CPU cores: We recommend that you set the number of threads for compute-intensive tasks to be slightly less than or equal to the number of CPU cores available. This way, CPU resources can be fully utilized, without increasing context switching overheads due to excessive threads.
• Hyper-threading: If hyper-threading is supported, two logical CPU cores can be simulated for each physical CPU core. In this case, you can set the number of threads to be twice the number of physical CPU cores. For example, if 8 physical CPU cores are available and hyper-threading is enabled, you can configure 16 threads.
• Proper parallelism: Excessive threads can cause frequent context switching, which degrades the performance. We recommend that you keep the number of threads consistent with the number of CPU cores.

High CPU utilization in kernel mode is usually caused by frequent context switching. In this case, you can reduce the number of concurrent threads.

obbinlog V4.2.0 and later return the following metrics in the running logs of binlog instances. You can determine the performance status of a binlog instance based on the corresponding metrics. Here are some sample logs:

[2024-12-23 20:44:02.897710] [info] counter.cpp(69): Counter:[Span:2000ms][Delay:123190330,123460630][RCNT:574119][RRPS:287059][RIOS:0][WCNT:2489781][WRPS:1244890][WIOS:93568488,AVG:75][XWIOS:0,AVG:0][RFETCH:1412113][ROFFER:266127][SPOLL:0][SENCODE:0][SSEND:0][ConvertRingBufferQ:1024][NDUMPER:0][NEventQ:49518][RecordQueueSize:44771][ReleaseRingBufferQ:38][SerializeRingBufferQ:8]

The following table describes the metrics.

Scenarios

Scenario 1: ConvertRingBufferQ indicates that accumulation occurs, and SerializeRingBufferQ and NEventQ indicate that no accumulation occurs

This is usually caused by slow conversion. You can increase the values of the binlog_convert_thread_size and binlog_convert_number_of_concurrences parameters. In addition, we recommend that you set the value of the binlog_convert_ring_buffer_size parameter to be no less than that of the binlog_convert_number_of_concurrences parameter. If the number of records is large, we recommend that you decrease the values of the read_wait_num and storage_wait_num parameters in the MetaDB.

ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_number_of_concurrences = '4';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_ring_buffer_size = '16';

Scenario 2: ConvertRingBufferQ indicates that accumulation occurs, and SerializeRingBufferQ and NEventQ indicate that serious accumulation occurs

This is usually caused by slow serialization. We recommend that you increase the values of the binlog_serialize_thread_size and binlog_serialize_parallel_size parameters. Here is an example:

ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_serialize_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_serialize_parallel_size = '4';

Scenario 3: SerializeRingBufferQ indicates that accumulation occurs, and ReleaseRingBufferQ indicates that serious accumulation occurs

This is usually caused by poor performance of asynchronous release. We recommend that you increase the values of the binlog_release_thread_size and binlog_release_parallel_size parameters. Here is an example:

ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_release_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_release_parallel_size = '4';

Scenario 4: ConvertRingBufferQ indicates that no accumulation occurs, and RecordQueueSize indicates that no serious accumulation occurs

This is usually caused by poor performance of data reads by obcdc. Check the libobcdc.log file for further diagnosis.