Memory-related issues

The following memory-related errors may occur.

Memory issues related to error code 4013

Error code 4013 indicates insufficient memory in various modules, for example, insufficient working area. A possible cause is insufficient memory in the compilation and execution modules. In this case, MemStore is not full; the error is caused only by insufficient memory in various modules.

OceanBase Database's memory management prohibits any module from allocating and releasing memory through the malloc and free interfaces in the glibc library. Except for node reserved memory (system_memory) and the memory of the sys tenant, most of OceanBase Database's memory is allocated to user tenants. The memory limit of user tenants can be adjusted. However, the memory limit of user tenants restricts the CTX memory of user tenants, and CTX memory is allocated and used by multiple MODs. Therefore, you can identify the operation type that failed to allocate memory by finding the corresponding MOD or CTX.

Memory issues related to error code 4030

Error code 4030 indicates insufficient MemStore memory. This type of issue typically occurs in scenarios involving MemStore operations, such as INSERT, UPDATE, DELETE statements, and TABLE_SCAN operations.

When MemStore memory exceeds the limit, check whether data writes are excessive or not rate-limited. This error occurs when there are heavy writes and minor compaction cannot keep up with the write speed. You can execute the following statement to view the memory status.

obclient(root@sys)[oceanbase]> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID, SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE TENANT_ID >1000 OR TENANT_ID=1
ORDER BY TENANT_ID, TOTAL_GB DESC;

For more memory-related query operations, see View memory usage.

Error code 4030 may occur for the following reasons:

• Module memory exceeds the limit.

  The emergency measure for this issue is to increase tenant memory.

  After the issue is resolved, analyze the cause. If the issue was caused by the absence of rate limiting, add appropriate measures and then roll back the previous tenant memory modification. If the tenant memory is insufficient to support the business due to business scale growth, set an appropriate tenant memory size based on the dump frequency. If MemStore memory has not exceeded the limit, you can run the following statement to determine which memory module has exceeded the limit.

  obclient(root@sys)[oceanbase]> SELECT TENANT_ID, SVR_IP, SUM(HOLD) MODULE_SUM
  FROM oceanbase.GV$OB_MEMORY
  WHERE TENANT_ID > 1000 AND HOLD<>0 AND CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
  GROUP BY TENANT_ID, SVR_IP;
  

  The criterion for determining module memory overflow is: MODULE_SUM > tenant min_memory - tenant MemStore. If a module's memory exceeds the limit, you may need to adjust the memory of individual modules first, for example, by modifying the ob_sql_work_area_percentage parameter to adjust workspace memory. If tenant memory is too small, you also need to increase tenant memory.

• PLANCACHE hit rate is lower than 90%.

  The PLANCACHE hit rate of an OLTP system should be no lower than 90%. Execute the following statements to view the PLANCACHE hit rate.

  • View PLANCACHE with a hit rate lower than 90%.

    obclient(root@sys)[oceanbase]> SELECT PLAN_ID, HIT_COUNT, EXECUTIONS, (HIT_COUNT/EXECUTIONS) AS HIT_RATIO
    FROM oceanbase.V$OB_PLAN_CACHE_PLAN_STAT
    WHERE (HIT_COUNT/EXECUTIONS) < 0.9;
    

  • View PLANCACHE with more than 1000 executions and a hit rate lower than 90%.

    obclient(root@sys)[oceanbase]> SELECT PLAN_ID, HIT_COUNT, EXECUTIONS, (HIT_COUNT/EXECUTIONS) AS HIT_RATIO
    FROM oceanbase.V$OB_PLAN_CACHE_PLAN_STAT
    WHERE (HIT_COUNT/EXECUTIONS) < 0.9 AND EXECUTIONS > 1000;
    

    If the PLANCACHE hit rate is lower than 90%, confirm whether there are similar statements, for example, statements where the number of parameters after in or not in is random, which leads to significant waste. If not, the issue may be caused by insufficient memory due to a surge in business volume or sessions. In this case, adjust the tenant memory size.

• The log contains error messages such as fail to alloc memory or allocate memory fail.

  The log contains tenant_id (tenant ID) and context (CTX name of the memory) information. You can query the specific memory module information by using the following statement.

  obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME = xxx and tenant_id = xxx;
  

  Based on the query result, if a module's memory exceeds the limit, you may need to adjust the memory of individual modules first. If tenant memory is too small, you also need to increase tenant memory.