Tenant memory full

If the tenant memory (memstore) is full, it is usually caused by sudden batch data import operations. In this case, you can enable write throttling and adjust the dump threshold.

Emergency handling procedure

If the tenant memory is full, it is generally manifested as write failures with error messages like No memory or reach tenant memory limit or Over tenant memory limits. The emergency handling procedure is as follows:

1. Increase the tenant memory configuration. We recommend that you expand the tenant memory in OCP by referring to OCP documentation. If you use the command line, follow these steps:

   1. Execute the following SQL statement to confirm the unit_config used by the current tenant.

  ```sql
  SELECT a.tenant_name,a.tenant_id,b.name unit_config,c.name pool_name,b.max_cpu,b.min_cpu
  FROM
  OCEANBASE.DBA_OB_TENANTS a,
  OCEANBASE.DBA_OB_UNIT_CONFIGS b,
  OCEANBASE.DBA_OB_RESOURCE_POOLS c
  WHERE a.tenant_id=c.tenant_id
  AND b.unit_config_id = c.unit_config_id
  ORDER BY a.tenant_id desc;
  ```

2. Execute the following SQL statement.

  ```sql
  ALTER resource unit unit_config_name min_cpu = 2,max_cpu = 2,MEMORY_SIZE = '2G',max_iops = 10000,min_iops = 10000;
  ```

2. Increase the number of threads for minor compaction.

   Data occupies a large amount of memory before a minor compaction. The compaction_high_thread_score parameter specifies the number of threads for minor compaction. Increasing the value of this parameter can expedite the minor compaction and the release of the occupied memory. The default value is 0, which indicates that six threads work concurrently. This parameter can be modified without restarting OBServer, and the modification takes effect immediately.

   Notice

   Increasing the value of this parameter results in higher CPU usage. Decide whether to make a modification according to your specific situation.

   To modify this parameter, log in to the sys tenant of the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET compaction_high_thread_score=x;
   

   After modifying the parameter, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'compaction_high_thread_score';
   

3. If the cluster can no longer expand the total tenant memory, you can temporarily increase the tenant's memory for writes by increasing the tenant's memstore ratio, and lower the dump threshold to expedite the minor compaction process.

   In OceanBase Database, when a tenant's active memstore memory usage reaches freeze_trigger_percentage * memstore_limit (where memstore_limit = tenant memory * memstore_limit_percentage), the system automatically triggers a freeze (a pre-minor compaction action) and then schedules a minor compaction. After the minor compaction, the system releases the memstore memory.

   Another relevant parameter is memstore_limit_percentage, which is the proportion of tenant memory available for memstore writes. By default, 50% of the total memory is available for memstore writes. This means that 50% of the total available memory is used by memstores. For example, 5 GB of memory (half of the total 10 GB of memory) is available for memstore writes for a tenant whose memory configuration is 10 GB. The value range is [1, 99]. The modification of this parameter takes effect immediately without restarting OBServer.

   In an emergency situation, you can increase memstore_limit_percentage while decreasing freeze_trigger_percentage to temporarily expand memory and expedite dumps.

   To modify these parameters, log in to the sys tenant of the cluster as the root user and execute the following statement:

   ALTER SYSTEM SET freeze_trigger_percentage=40;
   ALTER SYSTEM SET memstore_limit_percentage=70;
   

4. Enable write throttling to limit client import speed when memory usage reaches a certain threshold.

   writing_throttling_trigger_percentage is a tenant-level parameter that sets the threshold for adjusting write speed. When the memstore memory usage reaches this threshold, write throttling is triggered. The default value is 100, which means the write throttling mechanism is off. The value range is [0, 100]. The modification of this parameter takes effect immediately without restarting OBServer. Another parameter, writing_throttling_maximum_duration, indicates how long the remaining memory can support writing after throttling is triggered, defaulting to 2 hours, and typically doesn't need modification.

   ALTER SYSTEM SET writing_throttling_trigger_percentage = 80;
   ALTER SYSTEM SET writing_throttling_maximum_duration = '2h';
   

5. Degrade batch processing tasks on the application side.

   When user memory is full, consider degrading the batch processing tasks for the corresponding tenant on the application side.