Tenant memory full

If the tenant memory (memstore) is full, it is usually caused by sudden batch data import operations. In this case, you can enable write throttling and adjust the dump threshold.

Emergency handling procedure

If tenant memory is filled up, it is generally manifested as write failures with error messages like No memory or reach tenant memory limit or Over tenant memory limits. The emergency handling procedure is as follows:

1. Increase the tenant memory configuration. We recommend that you directly expand tenant memory in OCP by referring to the OCP User Guide. If you use the command line, follow these steps:

   1. Execute the following SQL statement to confirm the unit_config used by the current tenant.

  ```sql
  SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
  FROM
  OCEANBASE.DBA_OB_TENANTS a,
  OCEANBASE.DBA_OB_UNIT_CONFIGS b,
  OCEANBASE.DBA_OB_RESOURCE_POOLS c
  WHERE a.tenant_id=c.tenant_id
  AND b.unit_config_id = c.unit_config_id
  ORDER BY a.tenant_id desc;
  ```

2. Execute the generated SQL statement.

  ```sql
  ALTER resource unit unit_config_name min_cpu = 2,max_cpu = 2,MEMORY_SIZE = '2G',max_iops = 10000,min_iops = 10000;
  ```

2. Increase the number of threads for minor compaction.

   Data occupies a large amount of memory before a minor compaction. The compaction_high_thread_score parameter specifies the number of threads for parallel minor compaction. Increasing the value of this parameter can speed up the minor compaction and the release of the occupied memory. The default value is 0, which indicates that only one thread is used for minor compaction. The modification to this parameter takes effect immediately without the need to restart the OBServer node.

To implement this, log in to the sys tenant of the cluster as the root user and execute the following statement:

ALTER SYSTEM SET compaction_high_thread_score=x;

After modifying the parameter, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

SHOW PARAMETERS LIKE 'compaction_high_thread_score';

3. If the cluster can no longer expand the total tenant memory, you can temporarily increase the tenant's memory usage ratio of memstore to expand writable memory, and lower the dump threshold to speed up the minor compaction process.

   In OceanBase Database, when the memory usage of MemTables in a tenant reaches the percentage specified in the freeze_trigger_percentage parameter, an automatic minor compaction is triggered to release the occupied memory. This parameter is set to 70 by default, which means that a minor compaction is triggered when the memstore usage exceeds 70%.

   Another relevant parameter is memstore_limit_percentage, which is the proportion of memory available for memstore writes. For example, if this parameter is set to 50% for a tenant with 10 GB of memory, the tenant has 5 GB of memory available for memstore writes. The default value is 50%, and the range is [1, 99]. This parameter can be changed without restarting OBServer, and the modification takes effect immediately.

   In an emergency situation, you can increase memstore_limit_percentage while decreasing freeze_trigger_percentage to temporarily expand memory and expedite dumps.

   To implement this, log in to the sys tenant of the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Enable write throttling to limit client import speed when memory usage reaches a certain threshold.

   writing_throttling_trigger_percentage is a tenant-level parameter that sets the threshold for adjusting write speed. When the memstore memory usage reaches this threshold, write throttling is triggered. The default value is 100, which means that the write throttling mechanism is off. The range is [0, 100], and changes take effect immediately without restarting OBServer. Another parameter, writing_throttling_maximum_duration, indicates how long the remaining memory can support writing after throttling is triggered, defaulting to 2 hours, and typically doesn't need modification.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '2h';
   

5. Degrade batch processing tasks on the application side.

   When user memory is full, consider degrading the batch processing tasks for the corresponding tenant on the application side.