Common memory issues

What do I do when the tenant memory limit is exceeded?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits.

When you encounter the preceding error, first determine whether the MemStore memory limit is exceeded. If the MemStore memory limit is exceeded, check whether data is written in excess or throttling is not performed. This error is reported when a large amount of data is written and the data cannot be minor-compacted in time. Run the following command to view the current memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

The emergency handling procedure is to increase the tenant memory. Contact technical support for assistance. After the issue is resolved, analyze the cause. If the cause is the absence of throttling, take corresponding measures and then roll back the tenant memory increase. If the tenant memory is insufficient to support the business due to the growth of business scale, set a reasonable tenant memory size based on the minor compaction frequency. If the MemStore memory limit is not exceeded, run the following command to check which memory module is exceeded:

obclient> SELECT tenant_id, svr_ip, hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;

The standard for determining that a memory module is exceeded is module_sum > (tenant memory_size - tenant MemStore memory). If a memory module is exceeded, you may need to adjust the memory of the corresponding module, for example, set the ob_sql_work_area_percentage parameter. If the tenant memory is insufficient, you can increase it. For more information about system variables, see the following table.

What do I do when the plan cache hit rate of an OLTP system is below 90%?

The plan cache hit rate of an OLTP system should be no less than 90%. Run the following command to view the plan cache hit rate:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check whether similar statements, such as in or not in, have random parameter counts, which leads to a large waste of memory. If not, an increase in business volume or the number of sessions may cause memory insufficiency. In this case, you can adjust the tenant memory size.

What do I do when the log contains information such as fail to allocate memory or allocate memory fail?

The log contains tenant_id (tenant number) and context (memory module) information. You can run the following command to view the specific memory module information:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

As mentioned in Question 1, if a memory module is exceeded, you can adjust the memory of the corresponding module. If the tenant memory is insufficient, you can increase it. Contact technical support for assistance.

What do I do when tenant 500 exceeds the memory limit?

The tenant with tenant_id = 500 is an internal OB tenant, commonly known as Tenant 500. This issue usually occurs repeatedly, indicating that the system memory is exhausted or all the remaining memory of the server is preallocated and cannot be expanded any more.

To resolve this issue, check the memory usage. The memory usage of Tenant 500 is not statistics-collected by using the V$OB_MEMORY view, but you can query the oceanbase.GV$OB_MEMORY table. At the same time, check the memory usage of the entire system. If the system memory is exhausted, determine whether it is an occasional system issue. If so, and if it does not affect businesses, wait for the system to release the memory. If there is a significant impact on businesses or if the system processes and activities occupy and preallocate memory, you can consider increasing the memory. Contact technical support for assistance.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
//Memory usage check command