Reduce replicas

This topic describes how to reduce the number of replicas in log streams of a tenant.

Reduce replicas by modifying locality

You can reduce the number of replicas in log streams of a tenant by modifying the locality of the tenant in the cluster. For more information about modifying locality, see Modify locality.

Prerequisites

Before modifying locality, verify the resource allocation on each node in the zone to be modified. If the nodes in the zone do not have sufficient resources to accommodate the resource units required by the tenant, the locality modification cannot proceed.

For more information about the resource allocation of each node in a zone, see View tenant and resource information.

Considerations

• The locality of the tenant must be specified.

• When the locality of a tenant changes, the distribution of table replicas in the tenant also changes.

• When a tenant locality modification is not finished, the next tenant locality modification cannot be performed.

• After reducing replicas, if you want to release resources, also pay attention to scaling down the resource pools of the target tenant.

  For example, with z1, z2, and z3, when removing z3, there are two scenarios:

  1. The tenant has an independent resource pool in z3. For example, the tenant has resource_pool1 and resource_pool2, and resource_pool2 is in z3. After reducing the replica, you must remove resource_pool2 from the tenant's resource pool list, then you can optionally delete the resource pool to release resources. For more information, see Delete resource pool.

  2. There is no independent resource pool in z3. For example, the tenant's resource pool resource_pool1 is in z2 and z3. After reducing the replica, you must adjust the zone list of resource_pool1 to remove z3 to release resources. For more information, see Modify resource pool properties.

Procedure

The following example describes how to modify the locality of the mysql001 tenant from FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 to FULL{1}@zone1, FULL{1}@zone2, and reduce the number of replicas for the tenant.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Access the database named oceanbase.

   obclient(root@sys)[(none)]> use oceanbase;
   

3. View the locality of the mysql001 tenant before modification.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   You can see that the mysql001 tenant has one full-featured replica in each of zone1, zone2, and zone3.

4. Modify the locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2";
   

5. View the execution status of the locality modification job.

   obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
   

   The query result is as follows:

   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2023-01-05 19:33:46.730319 | 2023-01-05 19:33:46.920111 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2'                | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | xx.xx.xx.xx    |        2882 |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   1 row in set
   

   If the value of JOB_STATUS is SUCCESS, the locality modification job is successful.

6. View the locality of the mysql001 tenant after modification.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   By comparing the query results of steps 3 and 6, the locality of the mysql001 tenant has changed from FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 to FULL{1}@zone1, FULL{1}@zone2, and the number of replicas for the mysql001 tenant has been reduced.

Manually remove a replica

When you remove replicas from a tenant by modifying the locality of the tenant, the system automatically removes the redundant replicas. However, if the system does not properly remove replicas due to exceptions, you can execute the ALTER SYSTEM REMOVE REPLICA statement to remove the redundant log stream replicas.

Limitations

• You can manage log streams of all tenants in the sys tenant but manage only log streams of the current tenant in a user tenant.

• For the same log stream, multiple disaster recovery tasks (such as replica migration) can be performed in parallel, but other tasks (such as adding a replica, deleting a replica, converting the type of a replica, and modifying the quorum size of a log stream) can be performed only one at a time.

  You can query ongoing replica tasks from the CDB_OB_LS_REPLICA_TASKS view in the sys tenant or from the DBA_OB_LS_REPLICA_TASKS view in a user tenant.

Prerequisites

• Before executing the delete replica operation, ensure that the current user has the ALTER SYSTEM privilege; otherwise, the ALTER SYSTEM REMOVE REPLICA statement cannot be executed.

• Before querying the views, ensure that the current user has the SELECT privilege on the following views; otherwise, the relevant information cannot be queried.

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

Procedure

Assume that the locality of a tenant named tenant1 is F@zone1,F@zone3, and log stream 1001 of the tenant1 tenant has a full-featured replica in each of zone1, zone2, and zone3. To remove the replica in zone2, perform the following steps:

1. Log in to the cluster as the corresponding tenant.

   Here is an example:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information about how to connect to a database, see Database connection (MySQL-compatible mode) and Database connection (Oracle-compatible mode).

2. Obtain the tenant ID and locality of the tenant to be operated.

   • sys tenant

     obclient(root@sys)[oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM SYS.DBA_OB_TENANTS;
     

   A sample query result is as follows:

   +-------------+-----------+------------------------------+
   | TENANT_NAME | TENANT_ID | LOCALITY                     |
   +-------------+-----------+------------------------------+
   | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone3 |
   +-------------+-----------+------------------------------+
   1 row in set
   

   The query result shows that the tenant ID is 1002.

   For more information about the fields in the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Obtain all log stream information of the tenant to be operated.

   • sys tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS view, see CDB_OB_LS.

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

   A sample query result is as follows:

   +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
   | LS_ID | STATUS | PRIMARY_ZONE | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG | UNIT_LIST      |
   +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
   |     1 | NORMAL | zone1;zone3  |             0 |           0 |                NULL |     NULL | 1712136316868815003 | 1712136316868815003 |      |                |
   |  1001 | NORMAL | zone1;zone3  |          1001 |        1001 | 1711951923021132001 |     NULL | 1712136316868815003 | 1712136316868815003 |      | 1001,1002,1003 |
   +-------+--------+--------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
   2 rows in set
   

   The query result shows that the tenant1 tenant has log stream 1 and log stream 1001.

   For more information about the fields in the DBA_OB_LS view, see DBA_OB_LS.

4. Query the replica distribution of log stream 1001 based on the obtained log stream information.

   • sys tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                      | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:23:15.934957 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                             |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.588944 | 2024-04-03 17:23:14.904070 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                             |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:23:14.080074 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+----------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

     The query result shows that log stream 1001 has a full-featured replica on each of 172.xx.xxx.192:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.226:2882. Based on the locality information, you must remove the replica of the log stream on 172.xx.xxx.212:2882 (zone2).

     For more information about the fields in the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

5. Execute the delete replica command.

   The syntax is as follows:

   ALTER SYSTEM REMOVE REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   The parameters in the syntax are described as follows:

   • ls_id: the ID of the log stream for which you want to remove the replica.

   • svr_ip:svr_port: the IP address and port number of the OBServer node from which you want to remove the replica, for example, 172.xx.xxx.212:2882.

   • paxos_replica_num: the modified number of Paxos replicas of the log stream, which is the number of full-featured replicas specified in the locality of the tenant. A log stream supports a maximum of seven full-featured replicas.

     The value of the paxos_replica_num parameter must meet the following conditions:

     1. After the value of the paxos_replica_num parameter is modified, the available replicas are the majority.

     2. The value of the paxos_replica_num parameter must be greater than or equal to the number of members in the MEMBER_LIST of the leader.

     3. The absolute value of the variation of the paxos_replica_num value is not greater than 1. In this example, the value of the paxos_replica_num parameter is 3, and you can change it only to 2.

     If any one of the preceding conditions is not met, the system will return an error. If this parameter is not explicitly specified, the default value is used.

   • tenant_name: the name of the target tenant. You can specify another tenant in the sys tenant and specify only the current tenant in a user tenant. If this parameter is not explicitly specified, the name of the current tenant is used. You cannot use the all, all_user, or all_meta option in this statement to specify all tenants, all user tenants, or all meta tenants.

   • This statement removes only one replica at a time.

   Example: Remove the replica of log stream 1001 on 172.xx.xxx.212:2882 and set PAXOS_REPLICA_NUM to 2 after removal:

   obclient> ALTER SYSTEM REMOVE REPLICA LS = 1001 SERVER = '172.xx.xxx.212:2882' PAXOS_REPLICA_NUM = 2;
   

6. After the command is executed successfully, query the log stream information again. You can see that one replica of the log stream has been removed.

   • sys tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     The query result shows that log stream 1001 has a full-featured replica on each of 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882. The replica on 172.xx.xxx.212:2882 has been removed.

References

• Locality overview

• Modify locality

• View locality

• Add replica

• Change replica type

• Modify the quorum of a log stream

• Unit migration