Common memory issues

Issue 1: What do I do when the tenant memory limit is exceeded?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits.

When you encounter the above error, first determine whether the MemStore memory limit is exceeded. If the MemStore memory limit is exceeded, check whether data is written excessively or throttling is not performed. This error is reported when a large amount of data is written and the data minor compaction cannot keep up with the write speed. Run the following command to view the current memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB 
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

The emergency handling for this issue is to increase the tenant memory. For this, contact technical support. After the issue is resolved, analyze the cause. If the cause is lack of throttling, take corresponding measures and then roll back the tenant memory increase. If the tenant memory is insufficient for business operations due to business scale growth, set a reasonable tenant memory size based on the data minor compaction frequency. If the MemStore memory limit is not exceeded, run the following command to check which memory module is exceeded:

obclient> SELECT tenant_id, svr_ip, hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;

The standard for judging that a memory module is exceeded is module_sum > (tenant memory_size - tenant MemStore memory). If the memory of a memory module is insufficient, you can adjust the memory of the module. For example, you can change the value of ob_sql_work_area_percentage. If the tenant memory is small, you can also increase the tenant memory. For more information, see the following table.

Issue 2: What do I do when the PLAN CACHE hit rate is lower than 90%?

For OLTP systems, the PLAN CACHE hit rate should be no less than 90%. Run the following command to view the PLAN CACHE hit rate:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check whether similar statements, such as in or not in, have random parameter counts, which leads to a large waste of memory. If this is not the case, an increase in business volume or session count may cause memory insufficiency. In this case, you can adjust the tenant memory size.

Issue 3: What do I do when the log contains information such as fail to allocate memory or allocate memory fail?

The log contains tenant_id (tenant number) and context (memory module) information. You can run the following command to view the specific memory module information:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

As mentioned in Issue 1, if a memory module is exceeded, you can first adjust the memory of the memory module. If the tenant memory is insufficient, you can increase the tenant memory. For this, contact technical support.

Issue 4: What do I do when tenant 500 encounters a memory limit exceeded error?

The tenant with tenant_id =500 is an internal OB tenant, commonly known as tenant 500. Generally, this error frequently occurs, indicating that the system memory is exhausted or the remaining memory of the current server is fully preallocated, and the user cannot expand the memory any more.

You need to check the memory usage. The memory usage of tenant 500 is not statistics by V$OB_MEMORY. You need to query the oceanbase.GV$OB_MEMORY table. At the same time, check the memory usage in the system. If the system memory is exhausted, you need to determine whether it is a sporadic system issue. If so, and without business impact, you can wait for the system to release the memory by itself. If there is a significant business impact or the system process is occupied or the memory is preallocated due to system normal process activity, you can consider increasing the memory. For this, contact technical support.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
//Command for checking system memory usage