Reduce the number of replicas

This topic describes how to reduce the number of replicas of log streams in a tenant.

Reduce the number of replicas by modifying the locality

You can modify the locality of a tenant in a cluster to reduce the number of replicas of log streams in the tenant. For more information, see Modify the locality.

Prerequisites

Before you change the locality of a tenant, verify the resource allocation status of each node in the target zone. If the zone lacks sufficient resources to store the required resource units of the tenant, the locality of the tenant cannot be changed.

For more information about the resource allocation status of each node in the zone, see View tenant and resource information.

Considerations

• The locality of a tenant cannot be set to an empty value.

• When the locality of a tenant changes, the replica distribution of all tables in the tenant will be updated accordingly.

• A new locality change request for a tenant is not allowed until the previous locality change request is completed.

• If you reduce the number of replicas, make sure to shrink the resource pool of the tenant.

  Taking z1,z2,z3 as an example, if z3 is deleted, there are two scenarios:

  1. The tenant has a dedicated resource pool in z3, for example: the tenant has resource_pool1 and resource_pool2, and resource_pool2 is located in z3. In this case, after the number of replicas is reduced, remove resource_pool2 from the resource pool list of the tenant, and then delete the resource pool to release the resources. For more information, see Delete a resource pool.

  2. z3 does not have a dedicated resource pool. For example, the tenant's resource pool resource_pool1 is located in z2 and z3. In this case, after the number of replicas is reduced, remove z3 from the zone list of resource_pool1 to release the resources. For more information, see Modify the properties of a resource pool.

Procedure

Example: Change the locality of the mysql001 tenant from FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 to FULL{1}@zone1, FULL{1}@zone2 to reduce the number of replicas of the tenant.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Switch to the oceanbase database.

   obclient>use oceanbase;
   

3. View the locality of the mysql001 tenant before the modification.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   

   The query result shows that the mysql001 tenant has a full-featured replica in zone1, a full-featured replica in zone2, and a full-featured replica in zone3.

4. Modify the locality of the mysql001 tenant.

   obclient>ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2";
   

5. View the status of the locality change task.

    obclient>SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2023-01-05 19:33:46.730319 | 2023-01-05 19:33:46.920111 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2'                | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | xx.xx.xx.xx |        2882 |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   

   When JOB_STATUS changes to SUCCESS, the locality change task has succeeded.

6. View the locality of the mysql001 tenant after the modification.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   

   Compared with the result of the query in step 3, you can see that the locality of the mysql001 tenant has been changed from FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 to FULL{1}@zone1, FULL{1}@zone2, and the number of replicas of the tenant has been reduced.

Manually delete a replica

When you reduce the number of replicas by modifying the locality, the system normally automatically deletes the redundant replicas. However, due to some exceptions, if the system fails to work properly, you can manually delete the redundant log stream replicas by using the ALTER SYSTEM REMOVE REPLICA command.

Limitations

• The sys tenant can manage the log streams of all tenants, whereas a user tenant can only manage the log streams of the current tenant.

• A log stream can only perform one disaster recovery task at a time. The disaster recovery tasks are adding a replica, deleting a replica, converting the type of a replica, modifying the quorum size of a log stream, and replica migration.

  The sys tenant can query the CDB_OB_LS_REPLICA_TASKS view to check whether a disaster recovery task is in progress; a user tenant can query the DBA_OB_LS_REPLICA_TASKS view to check whether a disaster recovery task is in progress.

Prerequisites

• Before you delete a replica, make sure that the current user has the ALTER SYSTEM privilege. Otherwise, the ALTER SYSTEM REMOVE REPLICA command cannot be executed.

• Before you query the views, make sure that the current user has the SELECT privilege on the views. Otherwise, the query cannot be executed.

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

Procedure

Assume that a tenant named tenant1 exists, with the locality of F@zone1,F@zone3. In addition, tenant1 has three F-type replicas of log stream 1001 in zones 1, 2, and 3. Now, you need to manually delete the replica in zone 2.

1. Log in to the target tenant of the cluster.

   Here is an example of a connection string:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information about how to connect to a database, see Overview (MySQL mode) and Overview (Oracle mode).

2. Obtain the tenant ID and locality of the target tenant.

   • System tenant

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM SYS.DBA_OB_TENANTS;
     

     Here is an example of the query result:

     +-------------+-----------+------------------------------+
     | TENANT_NAME | TENANT_ID | LOCALITY                     |
     +-------------+-----------+------------------------------+
     | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone3 |
     +-------------+-----------+------------------------------+
     1 row in set
     

     As shown in the query result, the tenant ID of the target tenant is 1002.

     For more information about the columns of the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Obtain information about all log streams of the target tenant.

   • System tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the columns of the CDB_OB_LS view, see CDB_OB_LS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

     Here is an example of the query result:

     +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     | LS_ID | STATUS | PRIMARY_ZONE | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG |
     +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     |     1 | NORMAL | zone1;zone3  |             0 |           0 |                NULL |     NULL | 1712136316868815003 | 1712136316868815003 |      |
     |  1001 | NORMAL | zone1;zone3  |          1001 |        1001 | 1711951923021132001 |     NULL | 1712136316868815003 | 1712136316868815003 |      |
     +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     2 rows in set
     

     As shown in the query result, tenant1 has log streams 1 and 1001.

     For more information about the columns of the DBA_OB_LS view, see DBA_OB_LS.

4. View the distribution of replicas of log stream 1001 based on the previously obtained information.

   • System tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the columns of the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Here is an example of the query result:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                      | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:23:15.934957 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                             |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.588944 | 2024-04-03 17:23:14.904070 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                             |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:23:14.080074 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

     As shown in the query result, tenant1 has F-type replicas of log stream 1001 on 172.xx.xxx.192:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.226:2882. Combining this with the locality information, you need to delete the replica on 172.xx.xxx.212:2882 (zone 2).

     For more information about the columns of the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

5. Execute the delete replica statement.

   The statement is as follows:

   ALTER SYSTEM REMOVE REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   Here is a breakdown of the parameters in the statement:

   • ls_id: the ID of the log stream from which you want to delete a replica.

   • svr_ip:svr_port: the IP address and port number of the server where the replica to be deleted resides. For example, 172.xx.xxx.212:2882.

   • paxos_replica_num: the new quorum size, namely, the number of full-featured (F) replicas in the locality of the log stream. The maximum number of F-type replicas in a log stream is 7.

     When you specify this parameter, it must meet the following conditions:

     1. After the modification, paxos_replica_num must be a part of a quorum.

     2. paxos_replica_num must be greater than or equal to the number of replicas in the MEMBER_LIST field of the current leader replica.

     3. The absolute value of the difference between the new and old values of paxos_replica_num must be 1. For example, in this example, the current value of paxos_replica_num is 3, and it can be modified to 2.

     If the specified value does not meet the preceding conditions, an error will be returned. If this parameter is not specified, the system will use the current default value of paxos_replica_num.

   • tenant_name: the name of the tenant from which you want to delete a replica. In a system tenant, you can specify another tenant, but in a user tenant, you can specify only the current tenant. If this parameter is not specified, the default tenant name is the name of the current tenant. This statement does not support the use of all, all_user, or all_meta to specify all tenants, user tenants, or all meta tenants, respectively.

   • This statement can delete only one replica at a time.

   In this example, the replica of log stream 1001 on 172.xx.xxx.212:2882 in zone 2 is deleted, and the value of PAXOS_REPLICA_NUM is set to 2:

   obclient> ALTER SYSTEM REMOVE REPLICA LS = 1001 SERVER = '172.xx.xxx.212:2882' PAXOS_REPLICA_NUM = 2;
   

6. After the execution succeeds, query the information about log streams again. You will find that a replica has been deleted.

   • System tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Here is an example of the query result:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     As shown in the query result, tenant1 has F-type replicas of log stream 1001 on 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882. The replica that was deleted on 172.xx.xxx.212:2882 in zone 2 is not shown.

References

• Overview of locality

• Modify locality

• Query locality

• Add a replica

• Change the replica type

• Modify the quorum size of a log stream

• Migrate a replica