1. What are the names of tenant worker threads of an OBServer node?
OceanBase Database supports multitenancy. Each tenant has a unique ID that is assigned to the tenant when you create the tenant. Each tenant has an independent worker thread. If the ID of a tenant is 1001, the name of its worker thread is TNT_L0_1001.
TNT: indicates a tenant thread.L0: indicates that the thread is used to process requests whose nested level is 0.1001: indicates the ID of the tenant.
The worker threads of other tenants are named in the preceding format.
2. How does OceanBase Database determine the number of worker threads?
OceanBase Database creates required threads when it is started. After that, the number of threads does not change unless you modify related parameters or tenant specifications. For a tenant, the number of its worker threads on an OBServer node depends on the unit configs of the tenant.
3. How does OceanBase Database isolate CPU cores in a multi-tenant architecture?
OceanBase Database manages the consumption of CPU cores by controlling the number of active threads that consume CPU cores. When you create a tenant in OceanBase Database, you need to specify a resource pool for the tenant. The CPU specifications defined in the resource pool can limit the CPU utilization of a tenant. This way, CPU utilization is isolated among tenants. The latest version of OceanBase Database implements cgroup isolation to control and limit the consumption of CPU cores. However, you must set related parameters at the operating system level to enable this feature.
4. How do I check the number of worker threads in OceanBase Database? Does OceanBase Database start new threads for execution if the load is high?
In observer.log, the snippet that contains dump tenant info shows the number of active worker threads and the maximum number of worker threads. Here are the details:
token_count = cpu_count × cpu_quota_concurrency. The value of cpu_count indicates the number of CPU cores allocated to the tenant, which is greater than the value of min_cpu but smaller than the value max_cpu.
If the load is high, the OBServer node allocates new threads for execution. Assume that the T1 tenant on an OBServer node is configured with unit_min_cpu = 2 and unit_max_cpu=8 and CPU cores configured for the OBServer node are oversold. In this case, if 5 CPU cores are allocated to the T1 tenant, the value of the token_count parameter equals 5 plus the value of the cpu_quota_concurrency parameter.
5. How does OceanBase Database control the number of active tenant threads and the total number of tenant threads?
OceanBase Database calculates the number of active threads by multiplying the value of cpu_count by the value of cpu_quota_concurrency. These threads are created when a tenant is created. For example, the active Thread A is suspended when it executes a large query. In this case, the tenant creates another active thread. However, the total number of threads that can be created by a tenant cannot exceed the value of cpu_count multiplied by the value of workers_per_cpu_quota. If the number of threads of a tenant reaches the upper limit, new threads cannot be created and Thread A will not be suspended. This keeps the number of active threads unchanged. Assume that a tenant is configured with 16 CPU cores and the cpu_quota_concurrency parameter is set to 2. In this case, the tenant can have up to 32 active SQL threads. The number of active threads is controlled by the user-mode thread scheduler of OceanBase Database.
6. What is the CPU resource allocation principle for large queries? Can there be contention for CPU resources when OLAP and OLTP exist at the same time?
OceanBase Database allows you to define what is a large query by setting the large_query_threshold parameter. If the execution time of a query exceeds the specified threshold, the query is a large query. If large and small queries run at the same time, OceanBase Database allocates a part of CPU cores to large queries and uses the large_query_worker_percentage parameter to limit the maximum number of active worker threads that execute large queries. The default value of the large_query_worker_percentage parameter is 30, in percentage. This way, OceanBase Database limits the maximum CPU cores that can be used for large queries. This ensures that the system has sufficient CPU cores to process the load of online transaction processing (OLTP) business, such as small transactions used for trading business. This way, OLTP loads, which require fast response, can be executed with sufficient CPU cores at the earliest opportunity. Although OceanBase Database balances the allocation of CPU cores for large queries and OLTP loads, you must set the large_query_threshold parameter within a reasonable value range. If you specify a large value, threads that execute large queries may seize CPU cores, which leads to slow OLTP response or a queue with accumulated requests.
7. What is the cause of the situation where the CPU utilization of an OBServer node is high or an OBServer node is hung?
If the CPU utilization of an OBServer node reaches 80% to 90% or even higher, or the OBServer node demonstrates poor performance or is hung, you can use the OBStack tool to analyze the cause. OceanBase Database V2.2.3x advanced edition, V2.2.7x, and V3.x support the OBStack tool. You can run the following command to start OBStack, obtain the thread stack, and locate the issue in the obstack.trc file:
[root@hostname /]#obstack -p ${pid of observer} > obstack.trc