obbinlog V4.2.0 and later support concurrent log conversion. The following figure shows the overall logic.
As shown in the figure, obbinlog introduces Disruptor-cpp into binlog instances and uses the ring buffer provided by Disruptor-cpp to implement concurrent execution of the binlog conversion, serialization, and release phases. For information about the related parameters, see the Cluster parameters section in the Metadata topic.
Configuration templates
Resource usage of the binlog service consists of the following two parts:
Log pulling of OceanBase Change Data Capture (obcdc): Memory usage is limited by the
binlog_memory_limitparameter. When thebinlog_working_modeparameter is set tomemory, obcdc stores the data to be consumed in memory, which increases the memory usage. When thebinlog_working_modeparameter is set tostorage, the data is stored on the disk, which increases the disk usage.Log format conversion and client subscription: Converted log files are stored on the disk. Therefore, the disk usage increases over time. Memory usage and CPU utilization are affected by the sizes of the ring buffer, preallocated memory, and thread pool.
When the default values are used for the processing queue size of obcdc and preallocated memory size of the binlog service, the following configuration templates are available for different hardware configurations.
Template 1: 2 CPU cores and 8 GB of memory
Estimated resource usage:
- Memory: 4 GB + 4 GB + 32 MB
- CPU: 2 CPU cores
Estimated performance: 50,000 to 60,000 requests per second (RPS)
The following table describes the parameter settings.
| Parameter | Value |
|---|---|
| binlog_convert_ring_buffer_size | 16 |
| binlog_convert_thread_size | 2 |
| binlog_convert_number_of_concurrences | 1 |
| binlog_serialize_ring_buffer_size | 1024 |
| binlog_serialize_thread_size | 3 |
| binlog_serialize_parallel_size | 1 |
| binlog_release_ring_buffer_size | 1024 |
| binlog_release_thread_size | 4 |
| binlog_release_parallel_size | 2 |
Template 2: 4 CPU cores and 8 GB of memory
Estimated resource usage:
- Memory: 4 GB + 4 GB + 64 MB
- CPU: 4 CPU cores
Estimated performance: 80,000 to 100,000 RPS
The following table describes the parameter settings.
| Parameter | Value |
|---|---|
| binlog_convert_ring_buffer_size | 1024 |
| binlog_convert_thread_size | 4 |
| binlog_convert_number_of_concurrences | 3 |
| binlog_serialize_ring_buffer_size | 1024 |
| binlog_serialize_thread_size | 4 |
| binlog_serialize_parallel_size | 2 |
| binlog_release_ring_buffer_size | 1024 |
| binlog_release_thread_size | 4 |
| binlog_release_parallel_size | 2 |
Template 3: 8 CPU cores and 9 GB of memory
Estimated resource usage:
- Memory: 4 GB + 4 GB + 1 GB
- CPU: 8 CPU cores
Estimated performance: 100,000 to 150,000 RPS
The following table describes the parameter settings.
| Parameter | Value |
|---|---|
| binlog_convert_ring_buffer_size | 1024 |
| binlog_convert_thread_size | 6 |
| binlog_convert_number_of_concurrences | 4 |
| binlog_serialize_ring_buffer_size | 1024 |
| binlog_serialize_thread_size | 6 |
| binlog_serialize_parallel_size | 4 |
| binlog_release_ring_buffer_size | 1024 |
| binlog_release_thread_size | 4 |
| binlog_release_parallel_size | 2 |
Template 4: 12 CPU cores and 9 GB of memory
Estimated resource usage:
- Memory: 4 GB + 4 GB + 1 GB
- CPU: 12 CPU cores
Estimated performance: 150,000 to 200,000 RPS
The following table describes the parameter settings.
| Parameter | Value |
|---|---|
| binlog_convert_ring_buffer_size | 512 |
| binlog_convert_thread_size | 8 |
| binlog_convert_number_of_concurrences | 6 |
| binlog_serialize_ring_buffer_size | 512 |
| binlog_serialize_thread_size | 8 |
| binlog_serialize_parallel_size | 6 |
| binlog_release_ring_buffer_size | 1024 |
| binlog_release_thread_size | 4 |
| binlog_release_parallel_size | 2 |
Template 5: 16 CPU cores and 10 GB of memory
Estimated resource usage:
- Memory: 4 GB + 4 GB + 2 GB
- CPU: 16 CPU cores
Estimated performance: 200,000 to 250,000 RPS
The following table describes the parameter settings.
| Parameter | Value |
|---|---|
| binlog_convert_ring_buffer_size | 1024 |
| binlog_convert_thread_size | 13 |
| binlog_convert_number_of_concurrences | 12 |
| binlog_serialize_ring_buffer_size | 1024 |
| binlog_serialize_thread_size | 10 |
| binlog_serialize_parallel_size | 6 |
| binlog_release_ring_buffer_size | 1024 |
| binlog_release_thread_size | 4 |
| binlog_release_parallel_size | 2 |
Performance tuning
Binlog format conversion is compute-intensive. We recommend that the number of threads not exceed the number of CPU cores. Take note of the following considerations:
- Number of threads ≈ Number of CPU cores: We recommend that you set the number of threads for compute-intensive tasks to be slightly less than or equal to the number of CPU cores available. This way, CPU resources can be fully utilized, without increasing context switching overheads due to excessive threads.
- Hyper-threading: If hyper-threading is supported, two logical CPU cores can be simulated for each physical CPU core. In this case, you can set the number of threads to be twice the number of physical CPU cores. For example, if 8 physical CPU cores are available and hyper-threading is enabled, you can configure 16 threads.
- Proper parallelism: Excessive threads can cause frequent context switching, which degrades the performance. We recommend that you keep the number of threads consistent with the number of CPU cores.
High CPU utilization in kernel mode is usually caused by frequent context switching. In this case, you can reduce the number of concurrent threads.
obbinlog V4.2.0 and later return the following metrics in the running logs of binlog instances. You can determine the performance status of a binlog instance based on the corresponding metrics. Here are some sample logs:
[2024-12-23 20:44:02.897710] [info] counter.cpp(69): Counter:[Span:2000ms][Delay:123190330,123460630][RCNT:574119][RRPS:287059][RIOS:0][WCNT:2489781][WRPS:1244890][WIOS:93568488,AVG:75][XWIOS:0,AVG:0][RFETCH:1412113][ROFFER:266127][SPOLL:0][SENCODE:0][SSEND:0][ConvertRingBufferQ:1024][NDUMPER:0][NEventQ:49518][RecordQueueSize:44771][ReleaseRingBufferQ:38][SerializeRingBufferQ:8]
The following table describes the metrics.
| Metric | Description |
|---|---|
| ConvertRingBufferQ | The actual size of the ring buffer for the conversion queue. |
| NEventQ | The size of the binlog event queue to be serialized and stored to the disk. |
| RecordQueueSize | The size of the queue to read records from OceanBase Database. |
| ReleaseRingBufferQ | The actual size of the ring buffer for the release queue. |
| SerializeRingBufferQ | The size of the ring buffer for the binlog events that are to be serialized and stored to the disk. |
Scenarios
Notice
In the following scenarios, parameters are modified at the instance level. If you want to modify parameters at the tenant level, use the ALTER BINLOG <cluster_name>.<tenant_name> SET statement, for example, ALTER BINLOG `obcluster`.`test` SET binlog_convert_number_of_concurrences = '16';.
Scenario 1: ConvertRingBufferQ indicates that accumulation occurs, and SerializeRingBufferQ and NEventQ indicate that no accumulation occurs
This is usually caused by slow conversion. You can increase the values of the binlog_convert_thread_size and binlog_convert_number_of_concurrences parameters. In addition, we recommend that you set the value of the binlog_convert_ring_buffer_size parameter to be no less than that of the binlog_convert_number_of_concurrences parameter. If the number of records is large, we recommend that you decrease the values of the read_wait_num and storage_wait_num parameters in the MetaDB.
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_number_of_concurrences = '4';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_convert_ring_buffer_size = '16';
Scenario 2: ConvertRingBufferQ indicates that accumulation occurs, and SerializeRingBufferQ and NEventQ indicate that serious accumulation occurs
This is usually caused by slow serialization. We recommend that you increase the values of the binlog_serialize_thread_size and binlog_serialize_parallel_size parameters. Here is an example:
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_serialize_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_serialize_parallel_size = '4';
Scenario 3: SerializeRingBufferQ indicates that accumulation occurs, and ReleaseRingBufferQ indicates that serious accumulation occurs
This is usually caused by poor performance of asynchronous release. We recommend that you increase the values of the binlog_release_thread_size and binlog_release_parallel_size parameters. Here is an example:
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_release_thread_size = '8';
ALTER BINLOG INSTANCE `eq56dl14x5` SET binlog_release_parallel_size = '4';
Scenario 4: ConvertRingBufferQ indicates that no accumulation occurs, and RecordQueueSize indicates that no serious accumulation occurs
This is usually caused by poor performance of data reads by obcdc. Check the libobcdc.log file for further diagnosis.