FAQ about memory

Question 1: What do I do if the memory usage of a tenant exceeds the upper limit?

ERROR 4030 (HY000): OB-4030:Over tenant memory limits

If you receive the preceding error message, check whether the MemStore usage exceeds the upper limit. If yes, check whether an excessive amount of data was written to the MemStore and whether throttling is disabled. This error message is returned if a large amount of data is written into the MemStore and the minor compaction speed does not keep up with the data write speed. Execute the following statement to view the memory usage:

obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
           round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
           round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
           round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
           round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
           round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;

Increase the tenant memory as an emergency measure. For more information, contact OceanBase Technical Support for assistance. After the issue is temporarily resolved, analyze the root cause. If the issue is caused because throttling is not implemented, implement throttling and roll back the operation that increases the tenant memory. If the issue is caused because the tenant memory fails to support business growth, set a proper memory size for the tenant based on the minor compaction frequency. If the MemStore memory does not exceed the upper limit, execute the following statement to check which memory module exceeds the upper limit:

obclient> SELECT tenant_id,svr_ip,hold module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
             CTX_NAME NOT IN ( 'OB_KVSTORE_CACHE','OB_MEMSTORE')
GROUP BY tenant_id,svr_ip;

If the value of module_sum of a memory module is greater than the difference between the value of memory_size of the tenant and the size of the MemStore of the tenant, it is considered an overrun has occurred on the memory module. If an overrun has occurred on a memory module, the memory for the individual memory modules needs to be adjusted, such as modifying the ob_sql_work_area_percentage parameter. If the memory size of the tenant is too small, increase the memory size.

Question 2: What do I do if the hit rate of the plan cache is lower than 90%?

The hit rate of the plan cache must reach at least 90% in an online transaction processing (OLTP) system. You can execute the following statements to view the hit rate of the plan cache:

obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;

obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;

Check for similar statements, such as statements with a random number of parameters following in or not in, that consume a lot of memory unnecessarily. If such statements do not exist, the issue may occur when a sudden increase in traffic or sessions results in insufficient memory. In this case, you need to increase the memory of the tenant.

Question 3: What do I do if logs contain "fail to allocate memory" or "allocate memory fail"?

The error message contains the tenant_id (tenant ID) and context (memory module) information, with which you can obtain the information about the specific memory module by executing the following statement:

obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME=xxx AND tenant_id = xxx

If a memory overrun occurs in the memory module, you must adjust the memory size for the module. If the memory of the tenant is insufficient, increase the memory of the tenant. Contact OceanBase Technical Support for assistance.

Question 4: What do I do if the memory of the sys500 tenant exceeds the upper limit?

The tenant whose ID is 500 is an internal tenant of OceanBase Database, also known as the sys500 tenant. In normal cases, this error may be reported multiple times. The most possible reason is that the server has used up its system memory or you cannot increase the memory of the sys500 tenant because the remaining memory of the server is already pre-allocated.

To resolve this issue, you must check the memory usage of the sys500 tenant and that of the server. The memory usage of the sys500 tenant is not recorded in the V$OB_MEMORY view. Therefore, you must query the oceanbase.GV$OB_MEMORY view and check the memory usage in the system. If the server has used up its system memory, check whether this is an occasional situation. If yes and it has no impact on your business, you can wait for the system to release memory. If your business is affected, or the normal activities of system processes lead to memory occupation or pre-allocation, you can increase the memory. For more information, contact OceanBase Technical Support for assistance.

SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
// Check the system memory usage