Common memory issues

What do I do when the tenant memory limit is exceeded?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits.

When you encounter the preceding error, first determine whether the MemStore memory limit is exceeded. If the MemStore memory limit is exceeded, check whether data is written in excess or throttling is not performed. This error is reported when a large amount of data is written and the data cannot be minor-compacted in time. Run the following statement to view the current memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

The emergency handling procedure is to increase the tenant memory. Contact technical support for assistance. After the issue is resolved, analyze the cause. If the cause is the lack of throttling, take corresponding measures and then roll back the tenant memory increase. If the tenant memory is insufficient for the business scale, set a reasonable tenant memory size based on the minor compaction frequency. If the MemStore memory limit is not exceeded, run the following statement to check whether any memory module is exceeded:

obclient> SELECT tenant_id, svr_ip, hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;

The standard for determining whether a memory module is exceeded is module_sum > (tenant memory_size - tenant MemStore). If a module memory limit is exceeded, you may need to adjust the memory of the corresponding module first, for example, set the ob_sql_work_area_percentage variable. If the tenant memory is too small, you can also increase the tenant memory.

What do I do when the plan cache hit rate is lower than 90% in an OLTP system?

The plan cache hit rate should be no less than 90% in an OLTP system. Run the following statement to view the plan cache hit rate:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check whether similar statements exist. For example, the number of parameters in the IN or NOT IN clause varies randomly, which causes a large amount of memory to be wasted. If this is not the case, an increase in business volume or the number of sessions may cause insufficient memory. In this case, you can adjust the tenant memory size.

What do I do when the log contains information such as fail to allocate memory or allocate memory fail?

The log contains tenant_id (tenant ID) and context (memory module) information. You can run the following statement to view the specific memory module information:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

As mentioned in Q1, if a memory module is exceeded, you can first adjust the memory of the corresponding module. If the tenant memory is insufficient, you can increase the tenant memory. Contact technical support for assistance.

What do I do when tenant 500 exceeds the memory limit?

The internal tenant with the tenant_id of 500 is called the 500 tenant. This issue may occur repeatedly. It is likely that the system memory is exhausted or all the remaining memory of the server is preallocated, and the user cannot expand the memory any more.

You can check the memory usage. The memory usage of the 500 tenant is not statistics in the V$OB_MEMORY view, but in the oceanbase.GV$OB_MEMORY view. You can also check the memory usage in the system. If the system memory is exhausted, determine whether it is an occasional system issue. If it is and does not affect business, wait for the system to release the memory. If there is a significant impact on business or the system processes due to memory usage and preallocation, you can consider increasing the memory. Contact technical support for assistance.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
//Memory usage check command