Common memory issues

Issue 1: What do I do when the tenant memory limit is exceeded?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits.

When you encounter the preceding error, first determine whether the MemStore memory limit is exceeded. If the MemStore memory limit is exceeded, check whether data is being written excessively or throttling is not enabled. This error is reported when a large amount of data is written, but the data cannot be minor-compacted in time. Run the following statement to view the current memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

The emergency handling for this issue is to increase the tenant memory. Contact technical support for assistance. After the issue is resolved, analyze the cause. If the cause is the absence of throttling, take corresponding measures and then roll back the tenant memory increase. If the tenant memory is insufficient to support the business due to business growth, set a reasonable tenant memory size based on the minor compaction frequency. If the MemStore memory limit is not exceeded, run the following statement to check which memory module is exceeded:

obclient> SELECT tenant_id, svr_ip, hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;

The standard for judging that a memory module is exceeded is module_sum > (tenant memory_size - tenant MemStore). If a memory module is exceeded, you may need to adjust the memory of the module separately, for example, change the value of ob_sql_work_area_percentage. If the tenant memory is small, you can also increase the tenant memory.

Issue 2: What do I do when the PLANCACHE hit rate is lower than 90%?

For OLTP systems, the PLANCACHE hit rate should be no less than 90%. Run the following statement to view the PLANCACHE hit rate:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check whether similar statements, such as in or not in, have random parameter counts, which causes a large amount of memory waste. If not, the shortage of memory may be caused by the surge in business volume or the number of sessions. In this case, you need to adjust the tenant memory size.

Issue 3: What do I do when the log contains fail to allocate memory or allocate memory fail?

The log contains tenant_id (tenant number) and context (memory module) information. You can run the following statement to view the specific memory module information:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

As mentioned in Issue 1, if a memory module is exceeded, you need to adjust the memory of the module separately. If the tenant memory is small, you need to increase the tenant memory. Contact technical support for assistance.

Issue 4: What do I do when tenant memory limit is exceeded for tenant 500?

tenant_id =500 indicates an internal OB tenant, which is briefly referred to as a 500 tenant. Normally, this error frequently occurs because the system memory is exhausted or the remaining memory of the current server is fully preallocated. In this case, you cannot expand the memory.

To handle this issue, view the memory usage. The memory usage of a 500 tenant is not statistics in the V$OB_MEMORY view, but in the oceanbase.GV$OB_MEMORY view. You also need to check the memory usage of the system. If the system memory is exhausted, determine whether the memory shortage is caused by an occasional system issue. If not, and if the business is not affected, the system can release the memory by itself. If the system, major business activities, or memory preallocation occupy the memory, you can consider increasing the memory. Contact technical support for assistance.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
//Command for checking system memory usage