Throttling

In most cases, throttling in obcdc can keep the amount of memory used by obcdc during its runtime within your expectation. You can use the memory_limit parameter to adjust your expected memory usage of obcdc.

obcdc has multiple internal modules, between which transaction data flows. You can understand the overall production-consumption model as multiple interconnected production-consumption models. Throttling involves memory usage control for the following two modules: the data consumption module and the log pulling module.

Throttling for the data consumption module

If data consumption is slow, obcdc decreases the task distribution speed of the data consumption module to prevent excessive memory usage caused by substantial unconsumed data. obcdc determines the downstream consumption speed based on the task backlog of the consumption queue and the resource recycling queue. If the backlog ratio exceeds pause_redo_dispatch_task_count_threshold / 100 and the memory usage reaches the alert threshold memory_limit * memory_usage_warn_threshold / 100, or if the amount of redo logs being processed reaches the specified threshold, obcdc deems the downstream consumption speed slow.

You can search the libobcdc.log file for the NEED_PAUSE_REDO_DISPATCH=1 keyword to query the throttling logs of the data consumption module. The REASON field shows the specific reason for throttling.

Throttling for the log pulling module

Throttling in obcdc is triggered for the log pulling module in the following cases:

• The memory usage reaches the alert threshold memory_limit * memory_usage_warn_threshold / 100.

• The number of active transactions in the memory exceeds the specified threshold, and throttling for the data consumption module is triggered.

• A backlog of tasks occurs in the sequence module.

• A backlog of data persistence tasks occurs.

You can search the libobcdc.log file for the NEED_SLOW_DOWN=1 keyword to query the throttling logs of the log puling module. The REASON field shows the specific reason for throttling.

The following table describes some of the preceding parameters:

obcdc has more fine-grained throttling algorithms and parameters that automatically adapt based on the specified value of memory_limit. In rare cases, you may need to fine-tune these parameters listed below. However, in most cases, no adjustment is needed.

The adaptive throttling parameters of obcdc are listed below. For more information, see obcdc parameters.

• redo_dispatcher_memory_limit

• extra_redo_dispatch_memory_size

• redo_dispatched_memory_limit_exceed_ratio

• part_trans_task_active_count_upper_bound

• part_trans_task_reusable_count_upper_bound

• ready_to_seq_task_upper_bound

• storager_task_count_upper_bound

• storager_mem_percentage