Tenant management |V2.2.77|OceanBase Database| docs|Distributed Database

Tenant management

Last Updated：2023-08-18 09:26:34 Updated

This topic describes the precautions for tenant management in the primary/standby cluster configuration.

In OceanBase, system tenants can manage common tenants. The system tenant of the primary cluster is independent of the system tenant of a standby cluster, and the system tenants are not physically synchronized. However, the standby cluster can perceive the changes of common tenants in the primary cluster. This ensures data synchronization between all common tenants. The tenant management operations logically synchronized from the primary cluster to the standby clusters include creating, deleting, and renaming tenants.

Standby clusters also can independently perform some tenant management operations. You can independently manage the following tenant configurations for each of the primary and standby clusters:

Resource configurations
Resource pools
Locality configurations
Primary Zone configurations

In the primary/standby configuration, the primary and standby clusters do not support physical restoration of tenants.

Automatically create a tenant in a standby cluster

When a standby cluster is configured, the process of creating a tenant in the primary cluster still include following three steps:

CREATE RESOURCE UNIT: Prepare resources for the tenant.
CREATE RESOURCE POOL: Create a resource pool for the tenant.
CREATE TENANT: Execute the DDL statement to create the tenant.

By default, after you create a tenant in the primary cluster, each standby cluster can detect the DDL statement for creating the tenant and automatically creates a tenant. The process of creating a tenant in the standby cluster is the same as that in the primary cluster. However, because the primary cluster and the standby cluster may have different hardware configurations, the standby cluster dynamically creates a resource pool for the tenant based on its own configurations.

The following example describes the process of automatically creating a tenant in a standby cluster.

Assume that the standby cluster has 4 zones, namely z1, z2, z3, and z4. z1has an OBServer, z2 and z3has two OBServers, abd z4 is empty. A tenant tenant_test is created in the primary cluster. The tenant ID is 1001.

To create a tenant in the standby cluster, perform the following actions:

Create resource units for the tenant.

The unit configuration describes the resource specifications for each unit. By default, the standby cluster has the same specifications as the primary cluster. If the primary cluster uses multiple resource pools, the resource pool for the primary zone of the primary cluster is used as the template.

You can also customize the unit resource specifications for the standby cluster. When a unit configuration named standby_unit_config_template exists in the standby cluster, the system uses the configurations defined in the template to create unit configurations.

The standby cluster automatically creates a unit configuration named in the format of the tenant ID prefixed with __unit_config_. In this example, the tenant ID for the primary cluster is 1001, so the standby cluster automatically creates a unit configuration named __unit_config_1001.
Create a resource pool for the tenant.

The creation of a resource pool depends on the unit configuration, the zone list, and the number of units in each zone.

Where:
- The standby cluster selects valid zones and put them into the zone list, which determines the maximum number of replicas.
  
  A zone is valid if it has an OBServer deployed. In this example, the valid zones are z1, z2, and z3. z4is an invalid zone as it does not have an OBServer.
- By default, the number of units for each zone in the standby cluster is consistent with that of the primary cluster. However, it depends on the actual number of OBservers in each zone of the standby cluster.
  
  In this example, z1 has only one OBserver. Therefore, only one unit is allocated to each zone, even though the primary cluster has two units configured. If z1 has two OBservers, which is consistent with the primary cluster, two units can be allocated to each zone of the standby cluster.
The standby cluster automatically creates a resource pool named in the format of the tenant ID prefixed with __resource_pool_. In this example, the standby cluster automatically creates a tenant resource pool named __resource_pool_1001.
Create the tenant.

The standby cluster will use custom locality and primary zone parameters, and keep other parameters consistent with those of the primary cluster.

Where:
- Locality determines the replica type and distribution of all partitions under the tenant. By default, the standby cluster creates full-featured (F) replicas and encrypted voting (E) replicas, and distributes them across all zones of the resource pool.
  
  In this example, the resource pool has a zone list of z1,z2,z3, and the locality of the primary cluster is FULL{1}@z1,FULL{1}@z2,FULL{1}@z3. Assume that z3is encrypted. Then, the locality of the standby cluster is FULL{1}@z1,FULL{1}@z2,ENCRYPTION_LOGONLY{1}@z3.
- The primary zone determines the distribution of the leader replica. By default, the primary zone of the standby cluster is the same as that of the primary cluster.
  
  The system verifies the validity of the primary zone of the standby cluster:
  - When the primary zone of the primary cluster is random, no zone in the zone list can cros regions. Otherwise, the standby cluster automatically selects another appropriate primary zone.
  - When the primary zone of the primary cluster is not random, all zones must be valid and included in the zone list. When more than one zone is available, at least two non-encrypted zones are required in the region where the primary zone of the standby cluster is located.

Precautions for tenant creation

To ensure proper creation of a tenant in the standby cluster, observe the following notes when you create a tenant in the primary cluster:

When you configure resource units for the tenant in the primary cluster, consider the resource usage of the standby cluster.

By default, the resource unit configuration of the primary cluster is consistent with that of the standby cluster. If the standby cluster has insufficient resources, the tenant may fail to be created in the standby cluster. We recommend that you use the same tenant resource configurations for the primary and standby clusters. Therefore, when you configure resource units for the primary cluster, consider the resource usage of the standby cluster, to ensure that a resource pool can be created in the standby cluster.

To use different tenant resource configurations for the primary and standby clusters, configure a resource unit configuration template in the standby cluster.

If the hardware resources of the standby cluster are fewer than those of the primary cluster and you want to use different tenant resource configurations for the primary and standby clusters, you can create a resource unit configuration template standby_unit_config_template in the standby cluster in advance, or modify the template definition to customize tenant resource configurations for the standby cluster.

You can create, delete, or modify standby_unit_config_template in the standby cluster.

Sample commands for creating, viewing, and deleting a resource unit configuration template are as follows:

Create a resource unit configuration template for the standby cluster

obclient> CREATE RESOURCE UNIT standby_unit_config_template
     MAX_CPU 1, MIN_CPU=1, MAX_MEMORY 1073741824, MIN_MEMORY=1073741824, 
     MAX_IOPS 128, MAX_DISK_SIZE 536870912, MAX_SESSION_NUM 64, MIN_IOPS=128;
Query OK, 0 rows affected

View the resource unit configuration template

obclient> SELECT NAME, MAX_CPU, MAX_MEMORY, MAX_DISK_SIZE
     FROM __ALL_UNIT_CONFIG
     WHERE NAME LIKE 'standby_unit_config_template';
+------------------------------+---------+------------+---------------+
|               NAME                         | MAX_CPU | MAX_MEMORY | MAX_DISK_SIZE |
+------------------------------+---------+------------+---------------+
| standby_unit_config_template |               1 |      1073741824 |        536870912 |
+------------------------------+---------+------------+---------------+
1 row in set

Change the resource unit configuration template

obclient> ALTER RESOURCE UNIT  standby_unit_config_template max_cpu = 2;
Query OK, 0 rows affected (0.01 sec)

Delete the resource unit configuration template

obclient> DROP RESOURCE UNIT standby_unit_config_template;
Query OK, 0 rows affected

Check the zone configuration of the standby cluster.

When the standby cluster has multiple zones, at least two non-encrypted zones must be in the same region. This ensures that the standby cluster can generate proper primary zone configurations. Otherwise, the creation task will fail.
After a tenant is created in the primary cluster, monitor whether a tenant is created in each standby cluster.

The process of creating a tenant in the standby cluster may get stuck. The most common cause is the failure of creating a resource pool for the tenant due to insufficient resources in the standby cluster. In this case, you need to constantly monitor whether the tenant is created.

To check whether the tenant creation process gets stuck, perform the following operations:
1. Check the SYNCHRONIZATION_STATUS column in the V$OB_STANDBY_STATUS view of the primary cluster.
  
  If SYS SCHEMA NOT SYNC is displayed, the schemas of the system tenants are not synchronized, and the tenant creation process in the standby cluster may have got stuck.
2. Check the schema and user data synchronization status of each tenant in the V$OB_CLUSTER_STATS view of the standby cluster.
  
  tenant_id of the system tenant is 1. You can compare REFRESHED_SCHEMA_VERSION of the primary cluster with that of the standby cluster to check the DDL synchronization progress of the system tenant of the standby cluster. If the progress is out-of-sync with the primary cluster, the DDL statement for synchronously creating a tenant in the standby cluster may have got stuck.
To troubleshoot this issue, see Tenant creation for a standby cluster got stuck.

Precautions for routine tenant maintenance

After you adjust the tenant resource configurations of the primary cluster, we recommend that you also manually adjust the tenant resource configurations of the standby cluster.

We recommend that you use the same resource configurations for the primary and standby clusters. The standby cluster does not automatically synchronize tenant resource configuration modifications from the primary cluster. Therefore, after you adjust the tenant resource configurations of the primary cluster, we recommend that you also manually adjust the tenant resource configurations of the standby cluster.
Adjust the multiversion data retention period in the primary cluster, to enable the standby cluster to perform a failover provided the data in the standby cluster lags behind that in the primary cluster for a time less than the specified period.

For example, you can configure the tenant-specific system variable undo_retention to adjust the multiversion data retention period, which is in the unit of seconds. The value range is [0, 2^32^). For more information about the system variable undo_retention, see undo_retention.

Example:
```
obclient> SET GLOBAL undo_retention = 1800;
```
In this example, the multiversion data can be retained for 1800s, indicating that the standby cluster can succeed in performing a failover provided that the data in the standby cluster is out-of-sync with that in the primary cluster by less than 1800s.
After you modify tenant-specific configuration items in the primary cluster, manually modify the configuration items in the standby cluster.

Tenant-specific configurations are not automatically synchronized to the standby cluster. Therefore, after you modify tenant-specific configuration items in the primary cluster, we recommend that you also manually modify the configuration items in the standby cluster.