This topic describes how to query virtual tables and logs for the memory usage information.
Query virtual tables
Query the overall memory usage of an OBServer node
The following sample code shows how to query the memory usage of a specific OBServer node under the sys tenant. For more information about the view, see gv$sysstat.
obclient> SELECT * FROM oceanbase.gv$sysstat WHERE svr_ip = '10.0.0.0' AND svr_port=2882 AND con_id=1 AND name like '%observer memory%';
+--------+-------------+----------+------------+-------------+---------+---------------------------+-------+
| CON_ID | SVR_IP | SVR_PORT | STATISTIC# | VALUE | STAT_ID | NAME | CLASS |
+--------+-------------+----------+------------+-------------+---------+---------------------------+-------+
| 1 |10.0.0.0 | 2882 | 557 | 19377684480 | 140008 | observer memory used size | 64 |
| 1 |10.0.0.0 | 2882 | 558 | 64887980032 | 140009 | observer memory free size | 64 |
| 1 |10.0.0.0 | 2882 | 560 | 84265664512 | 140011 | observer memory hold size | 64 |
+--------+----------------+----------+------------+-------------+---------+---------------------------+-------+
3 rows in set
View the used memory of each tenant
obclient> SELECT tenant_id, sum(hold) FROM oceanbase.__all_virtual_tenant_ctx_memory_info WHERE svr_ip = '10.0.0.0' AND svr_port=2882 GROUP BY tenant_id;
+-----------+-------------+
| tenant_id | sum(hold) |
+-----------+-------------+
| 1 | 5158993920 |
| 21 | 4194304 |
| 500 | 11119099904 |
| 501 | 27262976 |
| 502 | 10485760 |
| 503 | 14680064 |
| 504 | 48234496 |
| 505 | 79691776 |
| 506 | 2097152 |
| 507 | 2097152 |
| 508 | 41943040 |
| 509 | 16777216 |
| 510 | 2097152 |
| 512 | 10485760 |
| 999 | 188743680 |
| 1001 | 864026624 |
| 1002 | 775946240 |
| 1003 | 125829120 |
| 1004 | 731906048 |
+-----------+-------------+
19 rows in set
View the memory usage details of a specified tenant
obclient> SELECT ctx_name, sum(hold) FROM oceanbase.__all_virtual_tenant_ctx_memory_info WHERE svr_ip = '10.0.0.0' AND svr_port=2882 AND tenant_id = 1002 GROUP BY ctx_name;
+--------------------------------+-----------+
| ctx_name | sum(hold) |
+--------------------------------+-----------+
| DEFAULT_CTX_ID | 337641472 |
| MEMSTORE_CTX_ID | 211812352 |
| COMPILE_CTX_ID | 0 |
| PHY_PLAN_CTX_ID | 0 |
| EXECUTE_CTX_ID | 6291456 |
| HASHJOIN_CTX_ID | 0 |
| SORT_CTX_ID | 0 |
| TRANS_CTX_MGR_ID | 10485760 |
| ELECTION_CTX_ID | 0 |
| MEMSTORE_OBJECT_CTX_ID | 0 |
| PARTITION_CTX_ID | 0 |
| SSSTORE_CTX_ID | 0 |
| SSTABLE_CTX_ID | 0 |
| PARTITION_STORAGE_CTX_ID | 0 |
| PARTITION_CTX_MGR_CTX_ID | 0 |
| STORAGE_LONG_TERM_META_CTX_ID | 0 |
| STORAGE_SHORT_TERM_META_CTX_ID | 0 |
| PARTITION_LOG_SERVICE_CTX_ID | 0 |
| REPLAY_STATUS_CTX_ID | 10485760 |
| TRANS_PART_CTX_ID | 0 |
| TRANS_COORD_CTX_ID | 0 |
| TRANS_SCHE_CTX_ID | 0 |
| OB_TRANS_LOCAL_TASK_ID | 0 |
| PLAN_CACHE_CTX_ID | 14680064 |
| REQ_MANAGER_CTX_ID | 0 |
| WORK_AREA | 6291456 |
| GLIBC | 0 |
| LOG_INDEX_CTX_ID | 0 |
| CO_STACK | 0 |
| LIBEASY | 0 |
| QUERY_EXEC_CTX_ID | 0 |
| LOGGER_CTX_ID | 0 |
| KVSTORE_CACHE_ID | 178257920 |
+--------------------------------+-----------+
33 rows in set
In OceanBase Database V3.x, the common fields and their meanings for CTX_NAME are as follows:
| CTX_NAME | Description |
|---|---|
| DEFAULT_CTX_ID | The ID of the unallocated memory pool context. |
| MEMSTORE_CTX_ID | The ID of the user MemTable data context. |
| COMPILE_CTX_ID | The ID of the compile context. |
| PHY_PLAN_CTX_ID | The ID of the context for physical execution plans. |
| EXECUTE_CTX_ID | The ID of the execution context. |
| HASHJOIN_CTX_ID | The ID of the context for Hash Join. |
| SORT_CTX_ID | The ID of the sorting context. |
| TRANS_CTX_MGR_ID | The ID of the transaction context. |
| ELECTION_CTX_ID | The ID of the election context. |
| MEMSTORE_OBJECT_CTX_ID | The ID of the context for MemTable objects. |
| PARTITION_CTX_ID | The ID of the partition context. |
| SSSTORE_CTX_ID | The ID of the SSStore context. |
| SSTABLE_CTX_ID | The ID of the SSTable context. |
| PARTITION_STORAGE_CTX_ID | The ID of the partition storage context. |
| PARTITION_CTX_MGR_CTX_ID | The ID of the context for the partition context manager. |
| STORAGE_LONG_TERM_META_CTX_ID | The ID of the context for long-term storage metadata. |
| STORAGE_SHORT_TERM_META_CTX_ID | The ID of the context for short-term storage metadata. |
| PARTITION_LOG_SERVICE_CTX_ID | The ID of the context for the partition log service. |
| REPLAY_STATUS_CTX_ID | The ID of the context for replay status. |
| TRANS_PART_CTX_ID | The ID of the transaction partition context. |
| TRANS_COORD_CTX_ID | The ID of the context for transaction coordinator. |
| TRANS_SCHE_CTX_ID | The ID of the context for transaction scheduler. |
| OB_TRANS_LOCAL_TASK_ID | The ID of the local task for OceanBase Database transactions. |
| PLAN_CACHE_CTX_ID | The ID of the context for execution plan cache. |
| REQ_MANAGER_CTX_ID | The ID of the request manager context. |
| WORK_AREA | The SQL work area memory. |
| GLIBC | The memory allocated using the Glibc Malloc interface. |
| LOG_INDEX_CTX_ID | The ID of the context for log indexing. |
| CO_STACK | The thread execution stack memory. |
| LIBEASY | The memory used in the network framework (EASY). |
| QUERY_EXEC_CTX_ID | The ID of the context for query execution. |
| LOGGER_CTX_ID | The ID of the context for program logging memory. |
| KVSTORE_CACHE_ID | The ID of the auto-scaling cache. |
View the memory usage details of a specific ctx_id
You can view the memory usage details of a specific OBServer node or tenant by querying the __all_virtual_memory_info table under the sys tenant.
obclient> SELECT * FROM oceanbase.__all_virtual_memory_info WHERE svr_ip = '10.0.0.0' AND svr_port=2882 AND tenant_id = 1002 ORDER BY hold desc limit 10;
+-----------+----------------+----------+--------+---------------------+-------------------+----------+--------+---------------------+-------+-----------+-----------+-------+-------------+------------+
| tenant_id | svr_ip | svr_port | ctx_id | label | ctx_name | mod_type | mod_id | mod_name | zone | hold | used | count | alloc_count | free_count |
+-----------+----------------+----------+--------+---------------------+-------------------+----------+--------+---------------------+-------+-----------+-----------+-------+-------------+------------+
| 1002 | 10.0.0.0 | 2882 | 1 | OB_MEMSTORE | MEMSTORE_CTX_ID | user | 0 | OB_MEMSTORE | zone1 | 209664000 | 209620000 | 100 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 32 | OB_KVSTORE_CACHE_MB | KVSTORE_CACHE_ID | user | 0 | OB_KVSTORE_CACHE_MB | zone1 | 178257920 | 0 | 85 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | TransAudit | DEFAULT_CTX_ID | user | 0 | TransAudit | zone1 | 104857600 | 104855040 | 1 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | MysqlRequesReco | DEFAULT_CTX_ID | user | 0 | MysqlRequesReco | zone1 | 96465408 | 94622720 | 8 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | OB_KVSTORE_CACHE | DEFAULT_CTX_ID | user | 0 | OB_KVSTORE_CACHE | zone1 | 14676480 | 14672896 | 7 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | SqlPlanMon | DEFAULT_CTX_ID | user | 0 | SqlPlanMon | zone1 | 11290240 | 11244800 | 6 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 23 | OB_SQL_PHY_PLAN | PLAN_CACHE_CTX_ID | user | 0 | OB_SQL_PHY_PLAN | zone1 | 2896192 | 2846256 | 283 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | SqlDtl | DEFAULT_CTX_ID | user | 0 | SqlDtl | zone1 | 2862080 | 2827981 | 43 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | PartitLogServic | DEFAULT_CTX_ID | user | 0 | PartitLogServic | zone1 | 2096640 | 2096128 | 1 | 0 | 0 |
| 1002 | 10.0.0.0 | 2882 | 0 | Election | DEFAULT_CTX_ID | user | 0 | Election | zone1 | 2096640 | 2096128 | 1 | 0 | 0 |
+-----------+----------------+----------+--------+---------------------+-------------------+----------+--------+---------------------+-------+-----------+-----------+-------+-------------+------------+
10 rows in set
Query logs
Query the overall memory usage of an OBServer node
grep CHUNK_MGR observer.log
[2021-10-29 16:48:56.459839] INFO [COMMON] ob_tenant_mgr.cpp:610 [66667][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] [CHUNK_MGR] free=1 pushes=28515 pops=28514 limit= 68,719,476,736 hold= 10,819,207,168 used= 10,817,110,016 freelist_hold= 2,097,152 maps= 7,493 unmaps= 6,555 large_maps= 2,374 large_unmaps= 2,056 memalign=0 virtual_memory_used= 18,991,874,048
The log information is described as follows.
| Parameter | Description |
|---|---|
| limit | The maximum memory of the OBServer node. |
| hold | The total memory that the OBServer node obtains from the operating system. |
| used | The memory that is actually used by the OBServer node, including the cache memory. |
| freelist_hold | The memory that is not used by the OBServer node. |
View the used memory of each tenant
grep ob_malloc_allocator.cpp observer.log
[2021-12-17 10:46:51.560130] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] [MEMORY] tenant: 1, limit: 16,106,127,360 hold: 3,789,553,664 rpc_hold: 0 cache_hold: 2,631,925,760 cache_used: 2,631,925,760 cache_item_count: 1,255
[2021-12-17 10:46:51.560770] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 8,896,118,784 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[2021-12-17 10:46:51.561333] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] [MEMORY] tenant: 501, limit: 1,073,741,824 hold: 2,097,152 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[2021-12-17 10:46:51.561505] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] [MEMORY] tenant: 502, limit: 1,073,741,824 hold: 10,485,760 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[2021-12-17 10:46:51.561699] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] [MEMORY] tenant: 503, limit: 1,073,741,824 hold: 10,485,760 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[2021-12-17 10:46:51.561917] INFO [LIB] ob_malloc_allocator.cpp:516 [4652][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] [MEMORY] tenant: 504, limit: 1,073,741,824 hold: 23,068,672 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
View the memory usage details of a specified tenant
View the memory usage of Tenant 1002:
Use Vim to open the
observer.logfile.Search for
ob_malloc_allocator.cpp.*1002.[MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 312,475,648 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 41,943,040 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 8,388,608 [MEMORY] ctx_id= REPLAY_STATUS_CTX_ID hold_bytes= 8,388,608 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 6,291,456 [MEMORY] ctx_id= WORK_AREA hold_bytes= 2,097,152
View the memory usage details of a specific ctx_id
View the memory usage of Tenant 1002 (DEFAULT_CTX_ID):
Use Vim to open the
observer.logfile.Search for
MEMORY.*1002.*DEFAULT_CTX_ID.[2021-10-29 16:37:44.143628] INFO [LIB] ob_tenant_ctx_allocator.cpp:151 [66667][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 312,475,648 used= 207,305,208 [MEMORY] hold= 104,857,600 used= 104,855,040 count= 1 avg_used= 104,855,040 mod=TransAudit [MEMORY] hold= 81,788,928 used= 80,000,000 count= 1 avg_used= 80,000,000 mod=MysqlRequesReco [MEMORY] hold= 8,386,560 used= 8,384,512 count= 4 avg_used= 2,096,128 mod=Election [MEMORY] hold= 4,193,280 used= 4,192,256 count= 2 avg_used= 2,096,128 mod=ElectionGroup [MEMORY] hold= 2,096,640 used= 2,096,128 count= 1 avg_used= 2,096,128 mod=PartitLogServic [MEMORY] hold= 2,096,640 used= 2,096,128 count= 1 avg_used= 2,096,128 mod=OB_KVSTORE_CACHE [MEMORY] hold= 1,580,800 used= 1,573,024 count= 2 avg_used= 786,512 mod=HashBuckPlanSta