Tenant memory full

If the tenant memory (MemStore) is full, it is usually caused by sudden batch data import operations. This topic provides information on how to handle this issue in emergency situations.

Emergency handling procedure

If tenant memory is filled up, it is usually manifested as write failures with error messages like No memory or reach tenant memory limit or Over tenant memory limits. The emergency handling procedure is as follows:

1. Increase the tenant memory configuration. It is recommended to directly expand tenant memory in OCP by referring to OCP User Guide. If you use the command line, perform the following sub-steps:

   1. Execute the following SQL statement to confirm the unit_config used by the current tenant.

      Notice

      Multiple replicas may use different resource pools, so you need to modify the configurations of each replica separately.

      SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
      FROM
      OCEANBASE.DBA_OB_TENANTS a,
      OCEANBASE.DBA_OB_UNIT_CONFIGS b,
      OCEANBASE.DBA_OB_RESOURCE_POOLS c
      WHERE a.tenant_id=c.tenant_id
      AND b.unit_config_id = c.unit_config_id
      ORDER BY a.tenant_id desc;
      

   2. Execute the following SQL statement based on the obtained configurations.

      Notice

      As of OceanBase Database V3.2, you can modify only the CPU and memory configurations of a resource unit. The modifications to other I/O configurations, such as disk and IOPS, are temporarily ineffective and can be left as default.

      ALTER resource unit unit_config_name min_cpu = 2,max_cpu = 2,MEMORY_SIZE = '2G',max_iops = 10000,min_iops = 10000;
      

2. Increase the number of threads for minor compaction.

   Data occupies a large amount of memory before a minor compaction. The compaction_mid_thread_score parameter specifies the number of threads for minor compaction. Increasing the value of this parameter can speed up the minor compaction and the release of the occupied memory. The default value is 0, which indicates that only one thread is used for minor compaction. The modification to this parameter takes effect immediately without the need to restart the OBServer node.

   Notice

   Increasing the value of this parameter results in higher CPU usage. Decide whether to make a modification according to your specific situation.

   To change the value of this parameter, log in to the sys tenant in the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET compaction_mid_thread_score=x;
   

   After modifying the parameter, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'compaction_mid_thread_score';
   

   Alternatively, you can increase the number of threads for minor compaction by modifying the minor compaction strategy in the OCP console. For more information, see Modify minor compaction settings.

3. If the cluster can no longer expand the total tenant memory, you can temporarily increase the tenant's MemStore ratio to expand writable memory, and lower the dump threshold to speed up the minor compaction process.

   In OceanBase Database, when a tenant's active MemStore memory usage reaches freeze_trigger_percentage * memstore_limit (where memstore_limit = tenant memory * memstore_limit_percentage), the system automatically triggers a freeze (a pre-minor compaction action) and then schedules a minor compaction. After minor compaction, the system releases the MemStore memory.

   Another relevant parameter is memstore_limit_percentage, which is the proportion of tenant memory available for MemStore writes. By default, the value is 0, indicating that the system adaptively adjusts the percentage of total available memory used by MemStore. This parameter ranges from [0, 100) and can be changed without restarting OBServer, taking effect immediately.

   In emergency situations, you can increase memstore_limit_percentage while decreasing freeze_trigger_percentage to temporarily expand memory and expedite dumps.

   To implement this, log on to the sys tenant of the cluster as the root user.

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Enable write throttling to limit client import speed when memory usage reaches a certain threshold.

   writing_throttling_trigger_percentage is a tenant-level parameter that sets the threshold for adjusting write speed. When the MemStore memory usage reaches this threshold, write throttling is triggered. The default value is 100, indicating that the write throttling mechanism is off. The range is [0, 100], and changes take effect immediately without restarting OBServer. Another parameter, writing_throttling_maximum_duration, indicates how long the remaining memory can support writing after throttling is triggered, defaulting to 2 hours, and typically doesn't need modification.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '2h';
   

5. Degrade batch processing tasks on the application side.

   When user memory is full, consider degrading the batch processing tasks for the corresponding tenant on the application side.