Modify locality

This topic describes how to modify the locality of a tenant to adjust the number and distribution of replicas.

Scenarios

• Increase the number of replicas

  You can increase the number of replicas for a tenant in a cluster. Only one replica can be added at a time. For example, if you want to increase the number of replicas from 1 to 3, you must change the locality from F@z1 to F@z1,F@z2, and then from F@z1,F@z2 to F@z1,F@z2,F@z3. For more information, see Add replicas.

  Notice

  You can directly increase the number of replicas for a tenant from 3 to 5.

• Reduce the number of replicas

  You can reduce the number of replicas for a tenant in a cluster. Only one replica can be removed at a time. For example, if you want to reduce the number of replicas from 3 to 1, you must change the locality from F@z1,F@z2,F@z3 to F@z1,F@z2, and then from F@z1,F@z2 to F@z1. For more information, see Remove replicas.

  Notice

  You can directly reduce the number of replicas for a tenant from 5 to 3.

• Adjust the distribution of replicas

  You can modify the locality of a tenant several times to adjust the distribution of replicas across zones in a cluster. For example, the original locality of a tenant is F@z1,F@z2,F@z3. Assuming that Zone z3 becomes unavailable. You need to migrate the replica to Zone z4 and change the locality to F@z1,F@z2,F@z4. Specifically, you need to change the locality from F@z1,F@z2,F@z3 to F@z1,F@z2,F@z3,F@z4, and then from F@z1,F@z2,F@z3,F@z4 to F@z1,F@z2,F@z4. For more information, see Adjust the distribution of replicas.

Prerequisites

Before you modify the locality of a tenant, check the status of the resource units and the resource pool in the target zone. If OBServer nodes in the target zone do not have sufficient resources to accommodate the new resource units of the tenant, the locality cannot be changed.

For more information about how to check the status of the resource units and resource pool in a zone, see View resource information of a cluster.

Considerations

• The locality of a tenant must be specified.

• Modifying the locality of a tenant also changes the distribution of table replicas for the tenant.

• If a tenant locality modification is not finished, the next tenant locality modification cannot be performed.

• You can perform only one action at a time when you modify the locality. For example, if you want to replace zone3 with zone4, you must add zone4 first and then remove zone3.

• We recommend that you modify the primary zone before you modify the locality.

  Example:

  1. Assume that zone3 is a primary zone with the highest priority. When you remove zone3, read and write services of the tenant are affected during locality modification.

  2. Assume that the primary zone is set to RANDOM when you add zone4. After zone4 is added, zone4 also provides read and write services.

• Modifying a tenant's locality involves changing Paxos members and copying data. You must evaluate the impact on system stability and performance. Regarding changes of Paxos members, evaluate the impact on the tolerance to OBServer node failures. Regarding changes of replica distribution, evaluate the impact on the service level agreement (SLA) design for IDC and city-wide failures in the deployment architecture. Regarding data copying, observe the impact on other tenants on the same OBServer node. We recommend that you copy data during off-peak hours and prepare a contingency plan in advance.

• Locality modification must not reduce the expected tolerance to OBServer node failures. For example, if you want to replace an IDC in a cluster with three replicas, you must add a replica and then remove a replica to avoid reducing the expected tolerance to OBServer node failures during locality modification. Theoretically, a single-replica failure during locality modification does not affect service continuity. If service continuity is affected, immediately recover the availability of the database. We recommend that you try to rectify hardware faults first.

Procedure

The following example describes how to modify the locality of tenant mq_t1 from FULL{1}@sa128_obv4_1,FULL{1}@sa128_obv4_2 to F{1}@sa128_obv4_1,F{1}@sa128_obv4_2,F{1}@sa128_obv4_3.

1. Log on to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys -pxxxx -A
   

2. Access the database named oceanbase.

   obclient>use oceanbase;
   

3. View the locality.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE                           | LOCALITY                                                         |
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+
   |         1 | sys         | SYS         | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 |
   |      1001 | META$1002   | META        | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 |
   |      1002 | midas0_3790 | USER        | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 |
   |      1003 | META$1004   | META        | sa128_obv4_1,sa128_obv4_2              | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2                       |
   |      1004 | mq_t1       | USER        | sa128_obv4_1,sa128_obv4_2              | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2                       |
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+
   

4. Modify the locality of the mq_t1 tenant.

   obclient>ALTER TENANT mq_t1 locality="F{1}@sa128_obv4_1,F{1}@sa128_obv4_2,F{1}@sa128_obv4_3";
   

5. View the execution status of the locality change job.

    obclient>SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2023-01-05 19:33:46.730319 | 2023-01-05 19:33:46.920111 |      1002 | ALTER TENANT mq_t1 locality="F{1}@sa128_obv4_1,F{1}@sa128_obv4_2,F{1}@sa128_obv4_3"                | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2 | xx.xx.xx.xx |        2882 |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   

   If the value of JOB_STATUS is SUCCESS, the locality modification job is executed successfully.

6. View the modified locality.

   obclient>SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+--------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE                           | LOCALITY                                                         | PREVIOUS_LOCALITY  |
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+--------------------+
   |         1 | sys         | SYS         | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 | null               |
   |      1001 | META$1002   | META        | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 | null               |
   |      1002 | midas0_3790 | USER        | sa128_obv4_2;sa128_obv4_1,sa128_obv4_3 | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 | null               |
   |      1003 | META$1004   | META        | sa128_obv4_1,sa128_obv4_2              | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 | null               |
   |      1004 | mq_t1       | USER        | sa128_obv4_1,sa128_obv4_2              | FULL{1}@sa128_obv4_1, FULL{1}@sa128_obv4_2, FULL{1}@sa128_obv4_3 | null               |
   +-----------+-------------+-------------+----------------------------------------+------------------------------------------------------------------+--------------------+
   

   The query results show that the locality of tenant mq_t1 is changed to F{1}@sa128_obv4_1,F{1}@sa128_obv4_2,F{1}@sa128_obv4_3. If the PREVIOUS_LOCALITY parameter is empty, the modification is successful. If the PREVIOUS_LOCALITY parameter is not empty, the modification is in progress.

More information

• Locality

• View locality