Full occupation of tenant memory

In most cases, the full occupation of tenant memory (MemStore) is caused by sudden batch data import. To resolve this issue at the earliest opportunity, you can perform several emergency operations, such as enabling write throttling and modifying the threshold that triggers a minor compaction.

Emergency procedure

If the memory of a tenant is full, the data write fails, and one of the following error messages is returned: No memory or reach tenant memory limit and Over tenant memory limits. The following content describes the emergency operations by priority from high to low.

1. Increase the configurations of tenant memory. We recommend that you scale up tenant memory in the OceanBase Cloud Platform (OCP) console. For more information, see OCP User Guide. Perform the following steps to modify memory configurations by using commands:

   1. Execute the following SQL statement to confirm the unit configurations of the tenant:

      Notice

      Replicas may use different resource pools. You need to separately modify the configurations of each replica.``

      SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu,
      round((b.max_memory/1024/1024/1024)) max_memory,round((b.min_memory/1024/1024/1024)) min_memory
      FROM
      __all_tenant a,
      __all_unit_config b,
      __all_resource_pool c
      WHERE a.tenant_id=c.tenant_id
      AND b.unit_config_id = c.unit_config_id
      ORDER BY a.tenant_id desc;
      

   2. Execute the following SQL statement based on the obtained configurations:

      Notice

      As of OceanBase Database V3.2, you can modify only the CPU and memory configurations of a resource unit. The modifications to other I/O configurations, such as the disk and input/output operations per second (IOPS), do not take effect, and the default values are used.

      ALTER resource unit unit_config_name min_cpu=2,max_cpu=2, max_memory='4G',min_memory='4G', max_disk_size='500G',max_iops=10000,min_iops=10000,max_session_num=10000;
      

2. Increase the number of threads for minor compactions.

   Data takes up a large memory space before a minor compaction. However, you can speed up the minor compaction to facilitate the release of occupied memory space. To speed up the minor compaction, increase the value of the minor_merge_concurrency parameter. This parameter determines the number of concurrent threads for a minor compaction. The default value is 0, which indicates that only one thread is used. The modification to this parameter immediately takes effect without restarting the OBServer node.

   Notice

   The increase in the parameter value results in higher CPU utilization. You need to make modifications based on your resource specifications.

   To modify the parameter, log on to the sys tenant as the root user and execute the following statement:

   ALTER SYSTEM SET minor_merge_concurrency=x;
   

   After the parameters are modified, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'minor_merge_concurrency';
   

   You can also increase the number of threads for minor compactions by modifying the minor compaction strategy in the OCP console. For more information, see Modify minor compaction settings.

3. If the total memory of a tenant cannot be scaled up in a cluster, you can increase the percentage of MemStore memory for tenants to temporarily expand the writable memory and set a lower threshold that triggers a minor compaction.

   If the MemStore memory usage of a tenant in OceanBase Database reaches the value of the freeze_trigger_percentage parameter, a minor compaction is automatically triggered to release the memory space. The default value of this parameter is 70, which indicates that a minor compaction is triggered if the MemStore memory usage reaches 70%.

   You can also modify the memstore_limit_percentage parameter. This parameter specifies the percentage of the writable MemStore memory out of the total tenant memory. The default value is 50, which indicates that the MemStore memory that can be used by a tenant is at most 50% of the total tenant memory. For example, if the total memory of a tenant is 10 GB, at most 5 GB of data can be written to the MemStore. The value range of this parameter is 1 to 99. The modification to this parameter immediately takes effect without restarting the OBServer node.

   To temporarily scale up the memory and facilitate the release of memory space, you can increase the value of the memstore_limit_percentage parameter and lower the value of the freeze_trigger_percentage parameter.

   To modify the parameter, log on to the sys tenant as the root user and execute the following statement:

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Enable write throttling. OceanBase Database throttles the data import from the client when the memory write reaches a specific threshold.

   The writing_throttling_trigger_percentage parameter is a tenant-level parameter that specifies the threshold percentage for write throttling. If the MemStore memory usage exceeds the threshold, write throttling is triggered. The default value of this parameter is 100, which indicates that write throttling is disabled. The value range of this parameter is 0 to 100. The modification to this parameter immediately takes effect without restarting the OBServer node. Another related parameter is writing_throttling_maximum_duration. The parameter specifies the period in which the remaining MemStore memory is allocated after write throttling is triggered. The default period is 1 hour. In most cases, you do not need to modify this parameter.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '1h';
   

5. Downgrade the batch tasks of the application.

   If the memory of a tenant is fully occupied, you can downgrade the batch tasks of the tenant application.