Memory issues

What do I do when the memory usage exceeds the tenant limit?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits.

If you encounter the preceding error, first check whether the MemStore memory limit is exceeded. If so, check whether data is written in excess or throttling is not performed. This error is reported when a large amount of data is written and the data cannot be minor-compacted in time. Execute the following statement to query the current memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

The emergency handling procedure is to increase the tenant memory. Contact technical support for assistance. After the issue is resolved, analyze the cause. If the cause is lack of throttling, take corresponding measures and then roll back the tenant memory increase. If the tenant memory is insufficient for business operations due to business scale growth, set a reasonable tenant memory size based on the minor compaction frequency. If the MemStore memory limit is not exceeded, execute the following statement to check which memory module is exceeded:

obclient> SELECT tenant_id, svr_ip, hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;

The memory module limit is module_sum > (tenant memory_size - tenant MemStore memory). If a module memory limit is exceeded, you may need to adjust the memory of the corresponding module, for example, set ob_sql_work_area_percentage to a proper value. If the tenant memory is small, you can also increase the tenant memory.

What do I do when the plan cache hit rate of an OLTP system is lower than 90%?

For an OLTP system, the plan cache hit rate should be no less than 90%. Execute the following statement to query the plan cache hit rate:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check whether similar statements, for example, IN or NOT IN statements, have random parameter counts. This leads to waste of memory. If this is not the case, the memory shortage is likely caused by a surge in business volume or concurrent sessions. You can adjust the tenant memory size.

What do I do when the log contains fail to allocate memory or allocate memory fail information?

The log contains tenant_id (tenant number) and context (memory module) information. You can execute the following statement to query the specific memory module information:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

As mentioned in Question 1, if a memory module limit is exceeded, you can adjust the memory of the corresponding module. If the tenant memory is small, you can increase the tenant memory. Contact technical support for assistance.

What do I do when tenant 500 exceeds the memory limit?

The tenant with tenant_id = 500 is an internal OB tenant, known as the 500 tenant. This issue may occur repeatedly. It is likely that the system memory is exhausted or the remaining memory of the current server is preallocated, and the user cannot expand the memory any more.

To resolve this issue, check the memory usage. The memory usage of the 500 tenant is not statistics by V$OB_MEMORY. You can query the oceanbase.GV$OB_MEMORY table. At the same time, check the memory usage of the system. If the system memory is exhausted, determine whether it is an occasional system issue. If so, and if it does not affect businesses, wait for the system to release the memory. If large business impact or normal system process activities cause memory usage and preallocation, you can consider increasing memory. Contact technical support for assistance.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
//Memory usage check command