Tenant memory full

If the tenant memory (MemStore) is full, it is usually caused by sudden batch data import operations. This topic provides information on how to handle this issue in emergency situations.

Emergency handling procedure

If tenant memory is filled up, it is usually manifested as write failures with error messages like No memory or reach tenant memory limit or Over tenant memory limits. The emergency handling procedure is as follows:

1. Increase the tenant memory configuration. It is recommended to directly expand tenant memory in OCP by referring to OCP User Guide. If you use the command line, perform the following sub-steps:

   1. Execute the following SQL statement to confirm the unit_config used by the current tenant.

      Notice

      Multiple replicas may use different resource pools, so you need to modify the configurations of each replica separately.

      ```sql
      SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
      FROM
      OCEANBASE.DBA_OB_TENANTS a,
      OCEANBASE.DBA_OB_UNIT_CONFIGS b,
      OCEANBASE.DBA_OB_RESOURCE_POOLS c
      WHERE a.tenant_id=c.tenant_id
      AND b.unit_config_id = c.unit_config_id
      ORDER BY a.tenant_id desc;
      ```
      

   2. Execute the following SQL statement based on the obtained configurations.

      Notice

      As of OceanBase Database V3.2, you can modify only the CPU and memory configurations of a resource unit. The modifications to other I/O configurations, such as disk and IOPS, are temporarily ineffective and can be left as default.

      ```sql
      ALTER resource unit unit_config_name min_cpu = 2,max_cpu = 2,MEMORY_SIZE = '2G',max_iops = 10000,min_iops = 10000;
      ```
      

2. Increase the number of threads for minor compaction.

   Data occupies a large amount of memory before a minor compaction. The compaction_mid_thread_score parameter specifies the number of threads for minor compaction. Increasing the value of this parameter can speed up the minor compaction and the release of the occupied memory. The default value is 0, which indicates that only one thread is used for minor compaction. The modification to this parameter takes effect immediately without the need to restart the OBServer node.

   Notice

   Increasing the value of this parameter results in higher CPU usage. Decide whether to make a modification according to your specific situation.

   To change the value of this parameter, log in to the sys tenant in the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET compaction_mid_thread_score=x;
   

   After modifying the parameter, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'compaction_mid_thread_score';
   

   Alternatively, you can increase the number of threads for minor compaction by modifying the minor compaction strategy in the OCP console. For more information, see Modify minor compaction settings.

3. If the total memory of the tenant cannot be expanded, you can reduce the freeze_trigger_percentage while increasing the memstore_limit_percentage to speed up the minor compaction.

   The freeze_trigger_percentage parameter specifies the threshold for triggering a minor compaction. The default value of this parameter is 20, which indicates that a minor compaction is triggered if the MemStore usage exceeds 20%.

   The memstore_limit_percentage parameter specifies the percentage of writable MemStore memory in the total tenant memory. The default value is 50, which indicates that the writable MemStore memory occupies 50% of the total memory available for the tenant. For example, if the total memory of a tenant is 10 GB, the writable portion for MemStore is 5 GB. The value of this parameter ranges from 1 to 99. The modification to this parameter takes effect immediately without the need to restart the OBServer node.

   To change the values of the two parameters, log in to the sys tenant in the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Trigger write throttling.

   As a tenant-level parameter, writing_throttling_trigger_percentage specifies the threshold of write throttling. If the MemStore memory usage exceeds the threshold, write throttling is triggered. The default value of this parameter is 60, which indicates that write throttling is triggered if the MemStore usage exceeds 60%. The value of this parameter ranges from 0 to 100. The modification to this parameter takes effect immediately without the need to restart the OBServer node.

   Another related parameter is writing_throttling_maximum_duration, which specifies the maximum duration of the remaining memory to support writing after write throttling is triggered. The default value is 1 hour. Generally, you do not need to modify this parameter.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '1h';
   

5. Deprioritize batch processing tasks on the application side for the tenant.