Tenant management

This topic describes the precautions for tenant management in the primary/standby cluster configuration.

In OceanBase Database, system tenants can manage normal tenants. The system tenant of the primary cluster is independent of the system tenant of a standby cluster, and the system tenants are not physically synchronized. However, the standby cluster can perceive the changes of normal tenants in the primary cluster. This ensures data synchronization between all normal tenants. The tenant management operations logically synchronized from the primary cluster to the standby clusters include creating, deleting, and renaming tenants.

Standby clusters also can independently perform some tenant management operations. You can independently manage the following tenant configurations for each of the primary and standby clusters:

• Resource configurations

• Resource pools

• Locality configurations

• Primary zone configurations

In the primary/standby configuration, the primary and standby clusters do not the support physical restore of tenants.

Automatically create a tenant in a standby cluster

After a standby cluster is configured, the procedure of creating a tenant in the primary cluster includes the following three steps:

1. CREATE RESOURCE UNIT: Create resource units for the tenants.

2. CREATE RESOURCE POOL: Create a resource pool for the tenant.

3. CREATE TENANT: Execute the DDL statement to create the tenant.

By default, after you create a tenant in the primary cluster, each standby cluster can detect the DDL statement for creating the tenant and automatically creates a tenant. The process of creating a tenant in the standby cluster is the same as that in the primary cluster. However, because the primary cluster and the standby cluster may have different hardware configurations, the standby cluster dynamically creates a resource pool for the tenant based on its own cluster configurations.

The following example describes the process of automatically creating a tenant in a standby cluster.

Assume that the standby cluster has 4 zones, which are z1, z2, z3, and z4. z1 has one OBServer node, each of z2 and z3 has two OBServer nodes, and z4 is empty. A tenant named tenant_test is created in the primary cluster. The tenant ID is 1001.

To create a tenant in the standby cluster, perform the following steps:

1. Create resource units for the tenant.

   Resource unit configurations describe the specifications of each resource unit. By default, the standby cluster has the same specifications as the primary cluster. If the primary cluster uses multiple resource pools, the resource pool for the primary zone of the primary cluster is used as the template.

   You can also customize the resource unit specifications for the standby cluster. If a resource unit configuration template named standby_unit_config_template exists in the standby cluster, the system uses the template to create resource unit configurations.

   The standby cluster automatically creates a resource unit configuration named in the format of the tenant ID prefixed with __unit_config_. In this example, the tenant ID for the primary cluster is 1001, so the standby cluster automatically creates a resource unit configuration named __unit_config_ 1001.

2. Create a resource pool for the tenant.

   The creation of a resource pool depends on the resource unit configurations, the zone list, and the number of resource units in each zone.

   Note:

   • The standby cluster selects valid zones and put them into the zone list, which determines the maximum number of replicas.

     A zone is valid if it has an OBServer node deployed. In this example, the valid zones are z1, z2, and z3. z4 is an invalid zone because it does not have an OBServer node.

   • By default, the number of resource units for each zone in the standby cluster is consistent with that in the primary cluster. However, the setting depends on the actual number of OBServer nodes in each zone of the standby cluster.

     In this example, z1 has only one OBServer node. Therefore, only one resource unit is allocated to each zone, even though the primary cluster has two resource units configured. If z1 has two OBServer nodes, which is consistent with the primary cluster, two resource units can be allocated to each zone of the standby cluster.

   The standby cluster automatically creates a resource pool named in the format of the tenant ID prefixed with __resource_pool_. In this example, the standby cluster automatically creates a tenant resource pool named __resource_pool_1001.

3. Create the tenant.

   The standby cluster uses custom locality and primary zone configurations, and keeps other configurations consistent with those of the primary cluster.

   Note:

   • Locality determines the replica type and distribution of all partitions under the tenant. By default, the standby cluster creates full-featured (F) replicas and encrypted voting (E) replicas, and distributes them across all zones of the resource pool.

     In this example, the resource pool has a zone list of z1,z2,z3, and the locality of the primary cluster is FULL{1}@z1,FULL{1}@z2,FULL{1}@z3. Assume that z3 is encrypted. Then, the locality of the standby cluster is FULL{1}@z1,FULL{1}@z2,ENCRYPTION_LOGONLY{1}@z3.

   • The primary zone determines the distribution of the leader replica. By default, the primary zone of the standby cluster is the same as that of the primary cluster.

     The system verifies the validity of the primary zone of the standby cluster:

     • If the primary zone of the primary cluster is random, no zone in the zone list can cross regions. Otherwise, the standby cluster automatically selects another appropriate primary zone.

     • If the primary zone of the primary cluster is not random, all zones must be valid and included in the zone list. If more than one zone is available, at least two non-encrypted zones are required in the region where the primary zone of the standby cluster is located.

Precautions for tenant creation

To ensure proper creation of a tenant in the standby cluster, take note of the following rules when you create a tenant in the primary cluster:

• When you configure resource units for the tenant in the primary cluster, consider the resource usage of the standby cluster.

  By default, the resource unit configuration of the primary cluster is consistent with that of the standby cluster. If the standby cluster has insufficient resources, the tenant may fail to be created in the standby cluster. We recommend that you use the same resource unit configurations for the primary and standby clusters. Therefore, when you configure resource units for the primary cluster, consider the resource usage of the standby cluster, to ensure that a resource pool can be created in the standby cluster.

• To use different resource unit configurations for the primary and standby clusters, configure a resource unit configuration template in the standby cluster.

  If the hardware resources of the standby cluster are fewer than those of the primary cluster and you want to use different resource unit configurations for the primary and standby clusters, you can create a resource unit configuration template named standby_unit_config_template in the standby cluster in advance. Alternatively, you can modify the template definition to customize resource unit configurations when a tenant is automatically created in the standby cluster.

  Examples of creating, querying, modifying, and deleting a resource unit configuration template:

  • Create a resource unit configuration template for the standby cluster

    obclient> CREATE RESOURCE UNIT standby_unit_config_template
         MAX_CPU 1, MIN_CPU=1, MAX_MEMORY 1073741824, MIN_MEMORY=1073741824,
         MAX_IOPS 128, MAX_DISK_SIZE 536870912, MAX_SESSION_NUM 64, MIN_IOPS=128;
    Query OK, 0 rows affected
    

  • Query a resource unit configuration template

    obclient> SELECT NAME, MAX_CPU, MAX_MEMORY, MAX_DISK_SIZE
         FROM oceanbase.__ALL_UNIT_CONFIG
         WHERE NAME LIKE 'standby_unit_config_template';
    +------------------------------+---------+------------+---------------+
    |               NAME           | MAX_CPU | MAX_MEMORY | MAX_DISK_SIZE |
    +------------------------------+---------+------------+---------------+
    | standby_unit_config_template |       1 | 1073741824 |     536870912 |
    +------------------------------+---------+------------+---------------+
    1 row in set
    

  • Modify the resource unit configuration template

    obclient> ALTER RESOURCE UNIT standby_unit_config_template max_cpu = 2;
    Query OK, 0 rows affected
    

  • Delete the resource unit configuration template

    obclient> DROP RESOURCE UNIT standby_unit_config_template;
    Query OK, 0 rows affected
    

• You need to check the zone configurations of the standby cluster.

  If the standby cluster has multiple zones, at least two non-encrypted zones must be in the same region. This ensures that the standby cluster can generate proper primary zone configurations. Otherwise, the creation task fails.

• After a tenant is created in the primary cluster, check whether a tenant is created in each standby cluster.

  The process of creating a tenant in the standby cluster may get stuck. The most common cause is the failure of creating a resource pool for the tenant due to insufficient resources in the standby cluster. In this case, you need to constantly check whether the tenant is created.

  To check whether the tenant creation process gets stuck, perform the following steps:

  1. Check the SYNCHRONIZATION_STATUS column in the V$OB_STANDBY_STATUS view of the primary cluster.

     If SYS SCHEMA NOT SYNC is displayed, the schemas of the system tenants are not synchronized, and the tenant creation process in the standby cluster may get stuck.

  2. Check the schema and user data synchronization status of each tenant in the V$OB_CLUSTER_STATS view of the standby cluster.

     tenant_id of the system tenant is 1. You can compare REFRESHED_SCHEMA_VERSION of the primary cluster with that of the standby cluster to check the DDL synchronization progress of the system tenant of the standby cluster. If the progress is out-of-sync with the primary cluster, the DDL statement for synchronously creating a tenant in the standby cluster may get stuck.

  For information about how to troubleshoot this issue, see Tenant creation for a standby cluster got stuck.

Recommendations for routine tenant maintenance

• After you adjust the resource unit configuration of the primary cluster, we recommend that you also manually adjust the resource unit configuration of the standby cluster.

  We recommend that you use the same resource unit configurations for the primary and standby clusters. The standby cluster does not automatically synchronize resource unit configuration modifications from the primary cluster. Therefore, after you adjust the resource unit configuration of the primary cluster, we recommend that you also manually adjust the resource unit configuration of the standby cluster.

• Adjust the multiversion data retention period in the primary cluster, to allow the standby cluster to perform a failover provided the data in the standby cluster lags behind that in the primary cluster for a period less than the specified period.

  For example, you can set the tenant-specific system variable undo_retention to adjust the multiversion data retention period, which is in the unit of seconds. The value range is [0, 2³²). For more information about the system variable undo_retention, see undo_retention.

  Example:

  obclient> SET GLOBAL undo_retention = 1800;
  

  In this example, the multiversion data can be retained for 1800s, indicating that the standby cluster can succeed in performing a failover provided that the data in the standby cluster is out-of-sync with that in the primary cluster by less than 1800s.

• After you modify tenant-specific variables in the primary cluster, manually modify the variables in the standby cluster.

  Tenant-specific configurations are not automatically synchronized to the standby cluster. Therefore, after you modify tenant-specific variables in the primary cluster, we recommend that you also manually modify the variables in the standby cluster.