Memleak-based regular memory leak diagnostics mechanism

OceanBase Database uses Memleak to diagnose memory leaks. Memleak records the memory allocation stacks of the tracing module. If the number of times that a stack is recorded increases continuously within a period, the memory allocation context corresponding to this stack may have encountered a memory leak. However, Memleak is generally enabled after a memory leak. If no memory leak occurs after Memleak is enabled, it cannot track the module with the memory leak and has to wait for the occurrence of another memory leak. Therefore, we want a Memleak-based regular memory leak diagnostics mechanism that can sample memory allocation information in real time and analyze modules with possible memory leak issues based on the sampling result.

The _min_malloc_sample_interval and _max_malloc_sample_interval parameters are used to control the sampling rate for memory allocation information.

Three sampling modes are available.

• Zero sampling mode: The system does not record memory allocation information.

  obclient> ALTER SYSTEM SET _min_malloc_sample_interval=10000;
  Query OK, 0 rows affected
  
  obclient> ALTER SYSTEM SET _max_malloc_sample_interval=10000;
  Query OK, 0 rows affected
  

• Full sampling mode: The system records all memory allocation information. In this mode, the performance of OceanBase Database significantly deteriorates.

  obclient> ALTER SYSTEM SET _min_malloc_sample_interval=1;
  Query OK, 0 rows affected
  
  obclient> ALTER SYSTEM SET _max_malloc_sample_interval=1;
  Query OK, 0 rows affected
  

• Regular mode: The system records memory allocation information based on the specified sampling interval.

  obclient> ALTER SYSTEM SET _min_malloc_sample_interval=16;
  Query OK, 0 rows affected
  
  obclient> ALTER SYSTEM SET _max_malloc_sample_interval=256;
  Query OK, 0 rows affected
  

Use Memleak for memory leak diagnostics in regular mode

1. Log on to the sys tenant of the cluster as the root user.

2. Query the __all_virtual_malloc_sample_info virtual table for memory allocation information about the module that is suspected of memory leaks.

   The following example shows how to query the memory allocation information about the glibc_malloc module.

   obclient [oceanbase]> SELECT * FROM __all_virtual_malloc_sample_info WHERE mod_name='glibc_malloc' ORDER BY alloc_bytes DESC limit 10;
   +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+
   | svr_ip      | svr_port | tenant_id | ctx_id | mod_name     | back_trace                                                                                                                                                                  | ctx_name | alloc_count | alloc_bytes |
   +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+
   | xx.xx.xx.xx |    25224 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC    |           3 |     9437232 |
   | xx.xx.xx.xx |    25225 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC    |           3 |     9437232 |
   | xx.xx.xx.xx |    25226 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC    |           3 |     9437232 |
   | xx.xx.xx.xx |    25224 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0                                   | GLIBC    |           1 |     4194320 |
   | xx.xx.xx.xx |    25225 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0                                   | GLIBC    |           1 |     4194320 |
   | xx.xx.xx.xx |    25226 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0                                   | GLIBC    |           1 |     4194320 |
   | xx.xx.xx.xx |    25224 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0               | GLIBC    |           1 |     3145744 |
   | xx.xx.xx.xx |    25225 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0               | GLIBC    |           1 |     3145744 |
   | xx.xx.xx.xx |    25226 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0               | GLIBC    |           1 |     3145744 |
   | xx.xx.xx.xx |    25224 |       500 |     22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e28d9 0x161a7a6b 0x161a7148 0x161aa227 0x64f4472 0x64f003c 0x64b9e68 0x64b1d45 0x6520720 0x651e8db       | GLIBC    |           2 |      131120 |
   +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+
   10 rows in set (0.012 sec)
   

   The query result shows that the 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a stack has the largest alloc_bytes value and is suspected of memory leaks.

   Columns in the query result are described as follows:

   Column	Type	Description
   svr_ip	varchar:MAX_IP_ADDR_LENGTH	The IP address.
   svr_port	bigint(20)	The port number of the server.
   tenant_id	bigint(20)	The ID of the tenant.
   ctx_id	bigint(20)	The CTX ID.
   mod_name	varchar:OB_MAX_CHAR_LENGTH	The name of the module.
   back_trace	varchar:DEFAULT_BUF_LENGTH	The stack for memory allocation.
   ctx_name	varchar:OB_MAX_CHAR_LENGTH	The CTX name.
   alloc_count	bigint(20)	The number of memory allocation times.
   alloc_bytes	bigint(20)	The total size of allocated memory.

3. Based on the virtual table query result, analyze the back_trace stack that has a large alloc_bytes value.

   $addr2line -pCfe bin/observer 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a
   void* oceanbase::lib::ObTenantCtxAllocator::common_alloc<oceanbase::lib::ObjectMgr>(long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:414 (discriminator 2)
   oceanbase::lib::ObTenantCtxAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:35
   oceanbase::lib::ObMallocAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at ??:?
   oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at ./build_sanity/deps/oblib/src/lib/../../../../../deps/oblib/src/lib/allocator/ob_malloc.h:39
   ob_malloc_hook(unsigned long, void const*) at ??:?
   operator new(unsigned long) at ??:?
   operator new[](unsigned long, std::nothrow_t const&) at ??:?
   oceanbase::share::schema::ObSchemaMgrCache::init(long, oceanbase::share::schema::ObSchemaMgrCache::Mode) at ./build_sanity/src/share/./src/share/schema/ob_schema_mgr_cache.cpp:153
   oceanbase::share::schema::ObSchemaStore::init(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:29
   oceanbase::share::schema::ObSchemaStoreMap::create(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:131 (discriminator 2)
   oceanbase::share::schema::ObMultiVersionSchemaService::init_multi_version_schema_struct(unsigned long) at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:218 (discriminator 52)
   oceanbase::share::schema::ObServerSchemaService::update_schema_mgr(oceanbase::common::ObISQLClient&, oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::share::schema::ObSchemaMgr&, long, oceanbase::share::schema::ObServerSchemaService::AllSchemaKeys&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:4232 (discriminator 54)
   oceanbase::share::schema::ObServerSchemaService::refresh_increment_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5663 (discriminator 4)
   oceanbase::share::schema::ObServerSchemaService::refresh_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5288
   oceanbase::share::schema::ObMultiVersionSchemaService::refresh_tenant_schema(unsigned long) at ??:?
   operator() at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:2328 (discriminator 4)
   

   Use the addr2line command to parse the stack that has the largest alloc_bytes value. Contact OceanBase Technical Support to analyze whether the stack has encountered memory leaks.