Add a replica

This topic describes how to add a replica for a log stream of a tenant.

Add replicas by modifying locality

You can add replicas for a tenant by modifying the locality of the tenant in the cluster. For more information about how to modify the locality, see Modify the locality.

Prerequisites

Before you add replicas, make sure that the tenant has a resource pool in the target zone. If not, you need to modify the zone list of the existing resource pool or add a new resource pool for the tenant. If you add a new resource pool, the number of resource units in the new resource pool must be the same as that in the existing resource pool. For more information about how to modify the resource pool or add a new resource pool, see Modify the properties of a resource pool or Create a resource pool.

You also need to make sure that the nodes in the target zone have enough resources to accommodate the resource units required by the tenant. If the nodes in the target zone do not have enough resources, you cannot modify the locality of the tenant.

For information about the resource allocation of the nodes in the zone, see View tenant and resource information.

Considerations

• The locality of a tenant cannot be empty.

• When the locality of a tenant changes, the replica distribution of all tables in the tenant will also change.

• If the previous locality modification task is not completed, the new locality modification task cannot be executed.

• You can only execute one action at a time when you modify the locality. For example, you cannot directly replace Zone3 with Zone4. You can first add a new zone, Zone4, and then remove Zone3.

• You need to pay attention to the impact on the primary zone when you modify the locality. We recommend that you adjust the primary zone in advance.

  For example:

  1. If you need to reduce the number of replicas in Zone3 and Zone3 is the primary zone, the adjustment process will affect the read and write services of the tenant.

  2. If you add Zone4 and the primary zone is set to RANDOM, Zone4 will also provide read and write services.

Procedure

Example: Modify the locality of the mysql001 tenant from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, adding a replica in zone3.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Switch to the oceanbase database.

   obclient(root@sys)[(none)]> USE oceanbase;
   

3. Query the current locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   You can see that the mysql001 tenant has one full-featured replica in zone1 and one full-featured replica in zone2.

4. Modify the locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3";
   

5. Query the execution status of the locality modification task.

   obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
   

   The query result is as follows:

   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |        NULL |        0 | 2023-01-05 19:38:38.416011 | 2023-01-05 19:38:38.416011 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | FULL{1}@zone1, FULL{1}@zone2                | xx.xx.xx.237 |        2882 |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   1 row in set
   

   If JOB_STATUS is SUCCESS, the locality modification task is executed successfully.

6. Query the modified locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   By comparing the query results of step 3 and step 6, you can see that the locality of the mysql001 tenant has changed from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3. The mysql001 tenant has one full-featured replica in zone1, one full-featured replica in zone2, and one full-featured replica in zone3.

If the added zone is involved in a zone switch, you need to modify the primary zone. For more information about how to modify the primary zone, see Adjust the primary zone. If the added zone is not involved in a zone switch, you do not need to modify the primary zone.

Manually add a replica

When you modify the locality to add a replica, the system usually automatically adds the missing replica. However, if the system does not work properly due to some exceptions, you can manually add the missing log stream replica by using the ALTER SYSTEM ADD REPLICA command.

Limitations

• The sys tenant can add replicas to the log streams of all tenants. A user tenant can add replicas only to its own log streams.

• Except for replica migration tasks, only one disaster recovery task (such as adding a replica, deleting a replica, changing the replica type, or modifying the number of required members of a log stream) can be executed at a time for the same log stream of a tenant.

  The sys tenant can query the CDB_OB_LS_REPLICA_TASKS view to check whether a disaster recovery task is in progress. A user tenant can query the DBA_OB_LS_REPLICA_TASKS view to check whether a disaster recovery task is in progress.

Prerequisites

• Before you add a replica, make sure that you have the ALTER SYSTEM privilege. Otherwise, you cannot execute the ALTER SYSTEM ADD REPLICA statement.

• Before you query a view, make sure that you have the SELECT privilege on the view. Otherwise, you cannot query the view.

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • GV$OB_UNITS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

• Before you add a replica, make sure that the target server has available resources, such as a unit, and that the unit does not have a replica of the log stream.

Procedure

Assume that there is a tenant named tenant1 with a Locality of F@zone1,F@zone2,F@zone3. The tenant1 has only F-type replicas for log stream 1001 in zone1 and zone3, but no replicas in zone2. You need to manually add an F-type replica for log stream 1001 in zone2.

1. Log in to the cluster as the tenant.

   Here is an example of the connection command:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information about how to connect to the database, see Overview of connection methods (MySQL mode) and Overview of connection methods (Oracle mode).

2. Obtain the tenant ID and Locality of the tenant to be operated on.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

      obclient(root@tenant1)[oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Execute the following statement in Oracle mode:

     obclient(SYS@tenant1)[SYS]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM SYS.DBA_OB_TENANTS;
     

     Here is an example of the query result:

     +-------------+-----------+---------------------------------------------+
     | TENANT_NAME | TENANT_ID | LOCALITY                                    |
     +-------------+-----------+---------------------------------------------+
     | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
     +-------------+-----------+---------------------------------------------+
     1 row in set
     

     The tenant ID is 1002.

     For more information about the fields in the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Obtain the information about all log streams of the tenant to be operated on.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS view, see CDB_OB_LS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

      obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Execute the following statement in Oracle mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

     Here is an example of the query result:

     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     | LS_ID | STATUS | PRIMARY_ZONE      | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG | UNIT_LIST      |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     |     1 | NORMAL | zone1;zone3;zone2 |             0 |           0 |                NULL |     NULL | 1712137184525565000 | 1712137184525565000 |      |                |
     |  1001 | NORMAL | zone1;zone3;zone2 |          1001 |        1001 | 1711951923021132001 |     NULL | 1712137184292359002 | 1712137184292359002 |      | 1001,1002,1003 |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     2 rows in set
     

     The tenant has log stream 1 and log stream 1001.

     For more information about the fields in the DBA_OB_LS view, see DBA_OB_LS.

4. Obtain the unit resources of the tenant to be operated on.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM oceanbase.GV$OB_UNITS WHERE TENANT_ID=1002;
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient(root@tenant1)[oceanbase]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM oceanbase.GV$OB_UNITS;
     

     Execute the following statement in Oracle mode:

     obclient(SYS@tenant1)[SYS]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM SYS.GV$OB_UNITS;
     

     Here is an example of the query result:

     +----------------+----------+---------+-----------+-------+-----------+--------+
     | SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | STATUS |
     +----------------+----------+---------+-----------+-------+-----------+--------+
     | 172.xx.xxx.226 |     2882 |    1001 |      1002 | zone1 | ReadWrite | NORMAL |
     | 172.xx.xxx.212 |     2882 |    1002 |      1002 | zone2 | ReadWrite | NORMAL |
     | 172.xx.xxx.192 |     2882 |    1003 |      1002 | zone2 | ReadWrite | NORMAL |
     +----------------+----------+---------+-----------+-------+-----------+--------+
     3 rows in set
     

     The tenant has available unit resources on the three servers 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882.

     For more information about the fields in the GV$OB_UNITS view, see GV$OB_UNITS.

5. Based on the information about the log streams of the tenant obtained in the previous step, view the replica distribution of log stream 1001.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Here is an example of the query result:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     Log stream 1001 has F-type replicas on the servers 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882.

     Based on the query result in step 4, the tenant has available unit resources on the three servers 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882. However, log stream 1001 has no replicas on the server 172.xx.xxx.212:2882. Therefore, you need to add an F-type replica for log stream 1001 on the server 172.xx.xxx.212:2882.

     For more information about the fields in the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

6. Execute the add replica command.

   The SQL statement is as follows:

   ALTER SYSTEM ADD REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' REPLICA_TYPE [=] 'replica_type' [DATA_SOURCE [=] 'data_source'] [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   The statement is used as follows:

   • ls_id: specifies the log stream ID of the replica to be added.

   • svr_ip:svr_port: specifies the IP address and port number of the target server for the replica to be added. Example: 172.xx.xxx.212:2882.

   • replica_type: specifies the type of the replica to be added. The following replica types are supported:

     • Full-featured replica: specified by FULL or F.

     • Read-only replica: specified by READONLY or R.

     • Columnstore replica: specified by COLUMNSTORE or C.

   • data_source: specifies the data source address of the replica to be added. Example: 172.xx.xxx.192:2882.

     If the specified data source is unavailable, an error is returned.

     If this parameter is not explicitly specified, the system automatically selects an available data source.

   • paxos_replica_num: specifies the number of required members in the log stream after the modification, which is the number of full-featured (F) replicas in the tenant locality. A log stream can have at most 7 F replicas.

     The value of this parameter must meet the following conditions:

     1. After the modification, paxos_replica_num must be a majority.

     2. The value of paxos_replica_num must be greater than or equal to the number of members in the MEMBER_LIST of the current leader replica.

     3. The absolute value of the change in paxos_replica_num must not exceed 1. For example, if the current value of paxos_replica_num is 2, it can only be modified to 3.

     If any of the above conditions are not met, an error is returned. If this parameter is not explicitly specified, the system uses the current default value of paxos_replica_num.

   • tenant_name: specifies the tenant to be operated on. The system tenant can specify other tenants, while the user tenant can only specify itself. If the tenant to be operated on is not explicitly specified, the default tenant name is the current tenant. This statement does not support specifying all tenants, user tenants, or all Meta tenants using all, all_user, or all_meta.

   • This statement can only add one replica at a time. If multiple replicas need to be added, the statement must be executed multiple times.

   Here is an example:

   obclient> ALTER SYSTEM ADD REPLICA LS= 1001 SERVER='172.xx.xxx.212:2882' REPLICA_TYPE='F' DATA_SOURCE='172.xx.xxx.192:2882' PAXOS_REPLICA_NUM=3;
   

7. Query the log stream information again. You can see that an F replica has been added to the log stream.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Here is an example of the query result:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                                     | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:48:08.982791 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-03 17:48:05.812041 | 2024-04-03 17:48:09.985574 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:48:09.030020 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1712137685774460,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

References

• Locality Overview

• Modify Locality

• View Locality

• Reduce Replica

• Change Replica Type

• Modify the Quorum of a Paxos Log Stream

• Replica Migration

• Cancel a Replica Task