Memory-related errors

The following table describes two memory-related errors.

Memory issues related to error code 4013

The error code 4013 indicates that the memory of various modules is insufficient, for example, the memory of the working area is insufficient. This error is usually caused by insufficient memory for the compilation and execution modules. In this case, the MemStore is not fully occupied. Instead, the error is reported due to insufficient memory of the modules.

OceanBase Database prohibits any module from requesting and releasing memory by using the malloc and free interfaces in the glibc library. In OceanBase Database, in addition to the reserved memory of the node (system_memory) and the memory of the sys tenant, most of the memory is allocated to regular tenants, and the memory limit of regular tenants can be adjusted. However, the memory limit of regular tenants applies to the CTX memory of regular tenants, and the CTX memory is allocated and occupied by multiple MODs. Therefore, you can identify the operation type that fails to request memory by finding the corresponding MOD or CTX.

Memory issues related to error code 4030

The error code 4030 indicates that the MemStore memory is insufficient. This error usually occurs in scenarios that involve MemStore operations, such as INSERT, UPDATE, and DELETE statements and TABLE_SCAN operations.

When the MemStore memory usage exceeds the limit, you need to check whether the data writes are excessive or are not rate-limited. This error is reported when a large number of writes are performed and the data dumping speed cannot keep up with the write speed. You can execute the following statement to view the memory status.

obclient(root@sys)[oceanbase]> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID, SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE TENANT_ID >1000 OR TENANT_ID=1
ORDER BY TENANT_ID, TOTAL_GB DESC;

For more queries related to memory, see View memory usage.

The error code 4030 may be reported for the following reasons:

• The memory of a module exceeds the limit.

  The emergency response to this issue is to increase the memory of the tenant.

  After the issue is resolved, you need to analyze the cause. If the issue is caused by the absence of rate limiting, you need to add corresponding measures and then roll back the previous change to the tenant memory. If the tenant memory is insufficient to support the business due to business growth, you need to set a reasonable tenant memory size based on the dumping frequency. If the MemStore memory does not exceed the limit, you can run the following statement to identify which memory module exceeds the limit.

  obclient(root@sys)[oceanbase]> SELECT TENANT_ID, SVR_IP, SUM(HOLD) MODULE_SUM
  FROM oceanbase.GV$OB_MEMORY
  WHERE TENANT_ID > 1000 AND HOLD<>0 AND CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
  GROUP BY TENANT_ID, SVR_IP;
  

  The criterion for determining whether the memory of a module exceeds the limit is: MODULE_SUM > Tenant min_memory - Tenant MemStore. If the memory of a module exceeds the limit, you may need to adjust the memory of the module first. For example, you can modify the ob_sql_work_area_percentage parameter to adjust the workspace memory. If the tenant memory is too small, you also need to increase the tenant memory.

• The hit rate of PLANCACHE is lower than 90%.

  The PLANCACHE hit rate of an OLTP system must not be lower than 90%. Execute the following statements to view the PLANCACHE hit rate.

  • View the PLANCACHE whose hit rate is lower than 90%.

    obclient(root@sys)[oceanbase]> SELECT PLAN_ID, HIT_COUNT, EXECUTIONS, (HIT_COUNT/EXECUTIONS) AS HIT_RATIO
    FROM oceanbase.V$OB_PLAN_CACHE_PLAN_STAT
    WHERE (HIT_COUNT/EXECUTIONS) < 0.9;
    

  • View the PLANCACHE that has been executed more than 1000 times and whose hit rate is lower than 90%.

    obclient(root@sys)[oceanbase]> SELECT PLAN_ID, HIT_COUNT, EXECUTIONS, (HIT_COUNT/EXECUTIONS) AS HIT_RATIO
    FROM oceanbase.V$OB_PLAN_CACHE_PLAN_STAT
    WHERE (HIT_COUNT/EXECUTIONS) < 0.9 AND EXECUTIONS > 1000;
    

    If the PLANCACHE hit rate is lower than 90%, you need to check whether there are similar statements. For example, if the number of parameters following in or not in in a statement is random, a large amount of memory is wasted. If not, the insufficient memory may be caused by a surge in the business volume or sessions, and you need to adjust the tenant memory size.

• The logs contain error messages such as fail to alloc memory or allocate memory fail.

  The logs contain the tenant_id and context (the name of the CTX to which the memory belongs) information. You can execute the following statement to query the specific memory module information.

  obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME = xxx and tenant_id = xxx;
  

  Based on the query result, if the memory of a module exceeds the limit, you may need to adjust the memory of the module first. If the tenant memory is too small, you also need to increase the tenant memory.