Add replicas

This topic describes how to add replicas to log streams in a tenant.

Add replicas by modifying locality

You can add replicas to log streams of a tenant by modifying the locality of the tenant in the cluster. For more information about modifying locality, see Modify locality.

Prerequisites

Before you add replicas, make sure that the tenant has a resource pool in the target zone. If not, you need to adjust the zone list of the existing resource pool or add a new resource pool for the tenant. If you add a new resource pool, the number of units in the new resource pool must be the same as that in the existing resource pool. For more information about how to modify the resource pool or add a new resource pool, see Modify the resource pool or Create a resource pool.

You also need to check the resource allocation of the nodes in the target zone. If the resources of the nodes in the target zone are insufficient to store the required resource units of the tenant, you cannot modify the locality.

For more information about the resource allocation of the nodes in the target zone, see View the tenant and resource information.

Considerations

• The locality of the tenant cannot be empty.

• When the locality of the tenant changes, the replica distribution of all tables in the tenant will also change.

• If the previous round of locality changes is not completed, the new round of locality changes cannot be executed.

• Only one action can be executed at a time when you modify the locality. For example, you cannot directly replace Zone3 with Zone4. Instead, you can first add a Zone4 and then remove a Zone3.

• When you modify the locality, you need to pay attention to the impact on the primary zone. We recommend that you adjust the primary zone in advance.

  For example:

  1. If you need to reduce the number of replicas in Zone3 and Zone3 is the primary zone, the adjustment of the locality will affect the read and write services of the tenant.

  2. If the primary zone is set to RANDOM, the new zone will also provide read and write services.

Procedure

Example: Modify the locality of the mysql001 tenant from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 to add a replica in zone3.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Enter the oceanbase database.

   obclient(root@sys)[(none)]> use oceanbase;
   

3. Query the current locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   You can see that the mysql001 tenant has one full-featured replica in each of zone1 and zone2.

4. Modify the locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3";
   

5. Query the execution status of the locality change task.

   obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
   

   The query result is as follows:

   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |        NULL |        0 | 2023-01-05 19:38:38.416011 | 2023-01-05 19:38:38.416011 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | FULL{1}@zone1, FULL{1}@zone2                | xx.xx.xx.237   |        2882 |
   +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   1 row in set
   

   If JOB_STATUS is SUCCESS, the locality change task is executed successfully.

6. Query the new locality of the mysql001 tenant.

   obclient(root@sys)[oceanbase]> SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   

   The query result is as follows:

   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   3 rows in set
   

   By comparing the query results of steps 3 and 6, the locality of the mysql001 tenant has changed from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, and the mysql001 tenant has one full-featured replica in each of zone1, zone2, and zone3.

Adding replicas may cause a mismatch between the tenant's locality and primary zone. If the added zone participates in leader election, you need to modify the primary zone. For more information, see Adjust the primary zone. If the added zone does not participate in leader election, you do not need to modify the primary zone.

Manually add a replica

When you modify the locality to increase the number of replicas, the system automatically adds the missing replicas. However, if the system fails to do so due to some exceptions, you can manually add the missing log stream replicas by using the ALTER SYSTEM ADD REPLICA command.

Limitations

• The sys tenant can add replicas to the log streams of all tenants, while a user tenant can only add replicas to its own log streams.

• For the same log stream of a tenant, except for replica migration tasks, only one disaster recovery task (such as adding, deleting, or changing the type of replicas, or modifying the number of mandatory members of a log stream) can be executed at a time.

  The sys tenant can query the CDB_OB_LS_REPLICA_TASKS view to check if a disaster recovery task is in progress. A user tenant can query the DBA_OB_LS_REPLICA_TASKS view to check if a disaster recovery task is in progress.

Prerequisites

• Before you execute the ALTER SYSTEM ADD REPLICA command, make sure that the current user has the ALTER SYSTEM privilege.

• Before you query a view, make sure that the current user has the SELECT privilege on the view.

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • GV$OB_UNITS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

• Before you execute the ALTER SYSTEM ADD REPLICA command, make sure that the target server has available resources (units) and that the unit does not contain a replica of the log stream.

Procedure

Assume that you have a tenant named tenant1 with a locality of F@zone1,F@zone2,F@zone3. The tenant has one F-type replica of log stream 1001 in each of zone1 and zone3, but no replica in zone2. Now, you need to manually add an F-type replica of log stream 1001 in zone2.

1. Log in to the cluster as the tenant.

   Here is an example of the connection command:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information about how to connect to the database, see Overview of database connections (MySQL-compatible mode) and Overview of database connections (Oracle-compatible mode).

2. Obtain the tenant ID and locality of the target tenant.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT TENANT_NAME, TENANT_ID, LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT TENANT_NAME, TENANT_ID, LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT TENANT_NAME, TENANT_ID, LOCALITY FROM SYS.DBA_OB_TENANTS;
     

     Here is an example of the query result:

     +-------------+-----------+---------------------------------------------+
     | TENANT_NAME | TENANT_ID | LOCALITY                                    |
     +-------------+-----------+---------------------------------------------+
     | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
     +-------------+-----------+---------------------------------------------+
     1 row in set
     

     The tenant ID is 1002.

     For more information about the fields in the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Obtain the information about all log streams of the target tenant.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS view, see CDB_OB_LS.

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

     Here is an example of the query result:

     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     | LS_ID | STATUS | PRIMARY_ZONE      | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG | UNIT_LIST      |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     |     1 | NORMAL | zone1;zone3;zone2 |             0 |           0 |                NULL |     NULL | 1712137184525565000 | 1712137184525565000 |      |                |
     |  1001 | NORMAL | zone1;zone3;zone2 |          1001 |        1001 | 1711951923021132001 |     NULL | 1712137184292359002 | 1712137184292359002 |      | 1001,1002,1003 |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+----------------+
     2 rows in set
     

     The tenant has one log stream 1 and one log stream 1001.

     For more information about the fields in the DBA_OB_LS view, see DBA_OB_LS.

4. Obtain the unit resources of the target tenant.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM oceanbase.GV$OB_UNITS WHERE TENANT_ID=1002;
     

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM oceanbase.GV$OB_UNITS;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT SVR_IP, SVR_PORT, UNIT_ID, TENANT_ID, ZONE, ZONE_TYPE, STATUS FROM SYS.GV$OB_UNITS;
     

     Here is an example of the query result:

     +----------------+----------+---------+-----------+-------+-----------+--------+
     | SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | STATUS |
     +----------------+----------+---------+-----------+-------+-----------+--------+
     | 172.xx.xxx.226 |     2882 |    1001 |      1002 | zone1 | ReadWrite | NORMAL |
     | 172.xx.xxx.212 |     2882 |    1002 |      1002 | zone2 | ReadWrite | NORMAL |
     | 172.xx.xxx.192 |     2882 |    1003 |      1002 | zone2 | ReadWrite | NORMAL |
     +----------------+----------+---------+-----------+-------+-----------+--------+
     3 rows in set
     

     The tenant has available unit resources on three servers: 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882.

     For more information about the fields in the GV$OB_UNITS view, see GV$OB_UNITS.

5. Based on the log stream information of the tenant obtained in the preceding steps, view the distribution of replicas of log stream 1001.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Here is an example of the query result:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     The tenant has an F-type replica of log stream 1001 on each of 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882.

     Based on the query result in the previous step, the tenant has available unit resources on the servers 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882. However, the server 172.xx.xxx.212:2882 does not have a replica of the log stream 1001. Therefore, you need to add an F-type replica of log stream 1001 to the server 172.xx.xxx.212:2882.

     For more information about the fields in the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

6. Execute the statement to add the replica.

   The SQL statement is as follows:

   ALTER SYSTEM ADD REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' REPLICA_TYPE [=] 'replica_type' [DATA_SOURCE [=] 'data_source'] [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   The statement uses the following parameters:

   • ls_id: specifies the ID of the log stream to which you want to add a replica.

   • svr_ip:svr_port: specifies the IP address and port number of the server to which you want to add a replica. Example: 172.xx.xxx.212:2882.

   • replica_type: specifies the type of the replica to be added. The following replica types are supported:

     • Full-featured replica: specified by FULL or F.

     • Read-only replica: specified by READONLY or R.

     • Columnstore replica: specified by COLUMNSTORE or C.

   • data_source: specifies the data source address of the replica to be added. Example: 172.xx.xxx.192:2882.

     If the specified data source is unavailable, an error is returned.

     If you do not explicitly specify this parameter, the system automatically selects an available data source.

   • paxos_replica_num: specifies the number of full-featured replicas in the locality of the log stream after the replica is added. The number of full-featured replicas in a log stream cannot exceed 7.

     The value of this parameter must meet the following conditions:

     1. After the replica is added, the value of paxos_replica_num must be a majority.

     2. The value of paxos_replica_num must be greater than or equal to the number of members in the MEMBER_LIST of the current leader replica.

     3. The absolute value of the change in the value of paxos_replica_num must not exceed 1. For example, if the current value of paxos_replica_num is 2, you can only change it to 3.

     If the specified value does not meet any of the above conditions, an error is returned. If you do not explicitly specify this parameter, the system uses the current default value of paxos_replica_num.

   • tenant_name: specifies the tenant to which the replica belongs. The system tenant can specify other tenants, and the user tenant can only specify itself. If you do not explicitly specify the tenant, the current tenant is used by default. This statement does not support specifying all tenants, user tenants, or all Meta tenants using all, all_user, or all_meta.

   • This statement can add only one replica at a time. If you need to add multiple replicas, you must execute this statement multiple times.

   Example:

   obclient> ALTER SYSTEM ADD REPLICA LS= 1001 SERVER='172.xx.xxx.212:2882' REPLICA_TYPE='F' DATA_SOURCE='172.xx.xxx.192:2882' PAXOS_REPLICA_NUM=3;
   

7. Query the log stream information again. You can see that one F-type replica has been added to the log stream.

   • System tenant

     obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • User tenant

     MySQL-compatible mode

     Oracle-compatible mode

     Execute the following statement in MySQL-compatible mode:

     obclient(root@tenant1)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle-compatible mode:

     obclient(SYS@tenant1)[SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     The query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                                     | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:48:08.982791 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-03 17:48:05.812041 | 2024-04-03 17:48:09.985574 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:48:09.030020 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1712137685774460,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

References

• Locality overview

• Modify locality

• View locality

• Reduce the number of replicas

• Change replica type

• Modify the quorum of a log stream

• Migrate a replica

• Cancel a replica task