Tenant memory full

In most cases, the full occupation of tenant memory (MemStore) is caused by sudden batch data import. To resolve this issue at the earliest opportunity, you can perform several emergency operations, such as enabling write throttling and modifying the threshold that triggers a minor compaction.

Emergency handling procedure

If the memory of a tenant is full, the data write fails, and one of the following error messages is returned: No memory or reach tenant memory limit and Over tenant memory limits. The following content describes the emergency operations by priority from high to low.

1. Increase the configurations of tenant memory. We recommend that you scale up tenant memory in the OceanBase Cloud Platform (OCP) console. For more information, see OCP documentation. Perform the following steps to modify memory configurations by using statements:

   1. Execute the following SQL statement to confirm the unit configurations of the tenant:

      Notice

      Replicas may use different resource_pool settings. You need to separately modify the setting of each replica.

      SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
      FROM
      OCEANBASE.DBA_OB_TENANTS a,
      OCEANBASE.DBA_OB_UNIT_CONFIGS b,
      OCEANBASE.DBA_OB_RESOURCE_POOLS c
      WHERE a.tenant_id=c.tenant_id
      AND b.unit_config_id = c.unit_config_id
      ORDER BY a.tenant_id desc;
      

   2. Execute the following SQL statement based on the obtained configurations:

      Notice

      As of OceanBase Database V3.2, you can modify only the CPU and memory configurations of a resource unit. The modifications to other I/O configurations, such as the disk space and input/output operations per second (IOPS), do not take effect, and the default values are used.

      ALTER resource unit unit_config_name min_cpu = 2,max_cpu = 2,MEMORY_SIZE = '2G',max_iops = 10000,min_iops = 10000;
      

2. Increase the number of threads for minor compactions.

   Data takes up a large memory space before a minor compaction. However, you can speed up minor compactions to facilitate the release of the occupied memory space. To speed up minor compactions, increase the value of the compaction_mid_thread_score parameter. This parameter determines the number of concurrent threads for a minor compaction. The default value is 0, which indicates that only one thread is used. The modification to this parameter immediately takes effect without restarting the OBServer node.

   Notice

   The increase in the parameter value results in higher CPU utilization. You need to make modifications based on your resource specifications.

   To do so, log in to the sys tenant of the cluster as the root user, and execute the following statement:

   ALTER SYSTEM SET compaction_mid_thread_score=x;
   

   After the parameter is modified, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'compaction_mid_thread_score';
   

   You can also increase the number of threads for minor compactions by modifying the minor compaction strategy in the OCP console. For more information, see Modify minor compaction settings.

3. If the total memory of a tenant cannot be scaled up in a cluster, you can increase the percentage of MemStore memory for tenants to temporarily expand the writable memory and set a lower threshold that triggers a minor compaction.

   In OceanBase Database, when the usage of the active MemTable of a tenant reaches the limit specified by freeze_trigger_percentage * memstore_limit, the system automatically triggers a freeze, which is the prerequisite step for a minor compaction, and then schedules a minor compaction. memstore_limit is equal to the memory size of the tenant * memstore_limit_percentage. After the minor compaction is completed, the system releases the occupied MemStore memory.

   You can also modify the memstore_limit_percentage parameter. This parameter specifies the percentage of the writable MemStore memory out of the total tenant memory. The default value is 0, which indicates that the percentage of the available MemStore memory of a tenant to the total available memory of the tenant is adaptively adjusted by the system. The value range of this parameter is [0, 100). The modification to this parameter immediately takes effect without restarting the OBServer node.

   To temporarily scale up the memory and facilitate the release of memory space, you can increase the value of the memstore_limit_percentage parameter and lower the value of the freeze_trigger_percentage parameter.

   To do so, log in to the sys tenant of the cluster as the root user, and execute the following statements:

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Enable write throttling. OceanBase Database throttles the data import from the client when the memory usage reaches a specific threshold.

   The writing_throttling_trigger_percentage parameter is a tenant-level parameter that specifies the threshold percentage for write throttling. If the MemStore memory usage exceeds the threshold, write throttling is triggered. The default value of this parameter is 100, which indicates that write throttling is disabled. The value range of this parameter is [0, 100]. The modification to this parameter immediately takes effect without restarting the OBServer node. Another related parameter is writing_throttling_maximum_duration. The parameter specifies the period for which the remaining MemStore memory can sustain after write throttling is triggered. The default value is 2h. In most cases, you do not need to modify this parameter.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '2h';
   

5. Downgrade the batch tasks of the application.

   If the memory of a tenant is fully occupied, you can downgrade the batch tasks of the tenant application.