Diagnose memory errors

Dynamic memory leak diagnosis

You can query the GV$OB_MEMORY view for information about memory blocks allocated by the memory allocation function ob_malloc on an OBServer node. Each block of memory has a label or mod_id, indicating the module to which the memory belongs. You can view the statistics of the modules to analyze the system memory usage and locate the modules that are suspected to have a memory leak.

To help you troubleshoot memory leaks, OceanBase Database supports dynamic memory leak diagnosis. You can enable a checker to track each memory allocation request of a module and record the pointer and the corresponding call stack of the module. A record is cleared when the requested memory is released. The call stacks for the normally requested and released memory can be cleared right away, whereas those for the leaked memory are permanently recorded. Then, the call stacks are grouped and the existing call stack records are displayed in the __all_virtual_mem_leak_checker_info table. Therefore, a frequently recorded call stack may indicate a memory leak. However, you need to analyze the cause on a case by case basis because caches may also lead to frequent call stacks.

Enable memory_leak

obclient> alter system set leak_mod_to_check='OB_COMMON_ARRAY';

The preceding command sets OB_COMMON_ARRAY as the module to be checked. Then, the checker monitors the __all_virtual_mem_leak_checker_info table for you to view the possible problematic call stacks.

Query

obclient> select * from __all_virtual_mem_leak_checker_info order by alloc_count desc;

Generally, the number of records of call stacks that involve a memory leak is large and continues to grow. You can run the addr2line command to print such call stacks, as shown in the following example:

obclient> addr2line -pCfe bin/observer  0xe77df5 0xedf5a4 0xee14b0 0xee0923 1 0xeea558 0x4a05c3 0x1485cd9 0x1485223 0x1483c4b 0x1526f2a 0x1401359 0x1403075 0x140325a 0x1406416 0x14a51bb 0x14a5130 0x140x48f3aa7db8 0x14a4cc8 0x14a50f4 0x14a5cb1 0x130166e 0xd08bc9 0xd069d2 0xd01e97 0xeff033 0xefe6
$oceanbase_root/src/lib/utility/utility.cpp:58
$oceanbase_root/src/lib/../../src/lib/allocator/ob_mem_leak_checker.h:125
$oceanbase_root/src/lib/allocator/ob_tc_malloc.cpp:399
$oceanbase_root/src/lib/allocator/ob_tc_malloc.cpp:218
$oceanbase_root/src/observer/../../src/lib/allocator/ob_malloc.h:38
$oceanbase_root/src/lib/allocator/ob_malloc.cpp:121
$oceanbase_root/src/lib/../../src/lib/allocator/ob_malloc.h:116
$oceanbase_root/src/sql/../../src/lib/container/ob_array.h:295
$oceanbase_root/src/sql/../../src/lib/container/ob_array.h:291
$oceanbase_root/src/sql/../../src/lib/container/ob_array.h:468
$oceanbase_root/src/sql/optimizer/ob_log_table_scan.cpp:85
$oceanbase_root/src/sql/optimizer/ob_log_plan.cpp:1183
$oceanbase_root/src/sql/optimizer/ob_log_plan.cpp:1484
$oceanbase_root/src/sql/optimizer/ob_log_plan.cpp:1557
$oceanbase_root/src/sql/optimizer/ob_log_plan.cpp:2092
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:467
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:455
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:890
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:378
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:450
$oceanbase_root/src/sql/optimizer/ob_select_log_plan.cpp:568
$oceanbase_root/src/sql/optimizer/ob_optimizer.cpp:23
$oceanbase_root/src/sql/ob_sql.cpp:1160
$oceanbase_root/src/sql/ob_sql.cpp:887
$oceanbase_root/src/sql/ob_sql.cpp:152
$oceanbase_root/src/observer/mysql/obmp_query.cpp:263

Disable memory_leak

The checker reduces the execution performance. We recommend that you run the following command to disable the checker after the error is located:

obclient> alter system set leak_mod_to_check='';

Use memory_context to dynamically diagnose memory leaks

For the management of long-lived memory, memory_leak provides an excellent diagnostic capability to locate the problematic call stacks. However, memory_leak cannot deal with short-lived memory due to the quick release of the memory. Therefore, a new feature is required to solve this problem.

OceanBase Database uses memory_context to manage memory based on the memory lifecycle. memory_context and the memory allocated by it share the same lifecycle. Therefore, you can use memory_context to diagnose the leaks of short-lived memory. Note that memory_context ensures that the memory is fully released despite memory leaks. OceanBase Database can help you dynamically diagnose the leaks of short-lived memory.

Find the static_id field

When a HAS UNFREE error is printed in the log, find the static_id field as shown in the following example:

[2020-10-22 15:30:57.923806] ERROR has_unfree_callback (object_set.cpp:27) [67779][462][Y40DC64589025-0005B23D7165EB9F] [lt=24] [dc=0] HAS UNFREE PTR!!!label: mytest1,static_id: 12,static_info: {filename_:"obmp_query.cpp", line_:475, function_:"process_single_stmt"}, dynamic_info: {tid_:67779, cid_:462, create_time_:1603351857840041}

Enable memory_leak

obclient> alter system set leak_mod_to_check='mytest1@12';

The preceding command tracks the memory requests of the mytest1 module in the memory_context with a static_id of 12. You can also use a wildcard, which is not recommended for online operations because it decreases the performance and increases the space cost.

obclient> alter system set leak_mod_to_check='*@12';

The preceding command tracks the memory requests of all modules in the memory_context with a static_id of 12.

Wait for the reoccurrence of the HAS UNFREE error

When the HAS UNFREE error reoccurs, the corresponding call stack is displayed, as shown in the following example:

[2020-10-22 15:41:02.530881] INFO  object_set.cpp:523 [67784][472][Y40DC64589025-0005B23D7135EBC2] [lt=15] [dc=0] CONTEXT MEMORY LEAK. ptr: 0x7f8f8a4145b0, size: 200, label: mytest1, lbt: 0xb979d8b 0xb72d239 0xb71ad7a 0x3a88b97 0x7fd735c 0x7fcee89 0x7fcd41d 0xbac5fad 0x8044c7a 0x80443b5 0x804001a 0x803f2ff 0x32e2cd7 0x32e2b7c 0x2d49d6d 0xb76f163 0xb76cd74 0xb76bdae

You can also run the addr2line command to continue the analysis.

Disable memory_leak

The checker reduces the execution performance. We recommend that you run the following command to disable the checker after the error is located:

obclient> alter system set leak_mod_to_check='';

Dump the memory metadata

You can write all the memory metadata to the disk.

Procedure

1. Open the deployment directory of OceanBase Database.

2. Create the dump.config file in the etc/ directory and write commands into the file as needed. Valid commands are described in the Commands section.

3. Run the kill -62 $pid command to view the memory_meta file in the log/ directory.

Commands

• Run the following command to export all memory information:

  dump chunk all

• Run the following command to export the memory information of the specified tenant and context:

  dump chunk tenant_id=$tenant_id,$ctx_id=ctx_id. Note that the tenant ID and context ID can contain only digits.

View the memory requested by a glibc library

The memory requests by using libraries such as glibc malloc can be transmitted to memory_context of OceanBase Database. The module name of glibc malloc is glibc_malloc. The following example shows how to view the memory requested by glibc malloc:

obclient> select *from GV$OB_MEMORY where mod_name = 'glibc_malloc' and hold > 0;
+-----------+----------------+----------+--------+--------------+----------+----------+--------+--------------+------+-----------+-----------+-------+-------------+------------+
| tenant_id | svr_ip         | svr_port | ctx_id | label        | ctx_name | mod_type | mod_id | mod_name     | zone | hold      | used      | count | alloc_count | free_count |
+-----------+----------------+----------+--------+--------------+----------+----------+--------+--------------+------+-----------+-----------+-------+-------------+------------+
|       500 | xx.xx.xx.xx    |    46824 |     26 | glibc_malloc | GLIBC    | user     |      0 | glibc_malloc | z1   | 249800208 | 219156754 | 41247 |           0 |          0 |
+-----------+----------------+----------+--------+--------------+----------+----------+--------+--------------+------+-----------+-----------+-------+-------------+------------+