Add replicas

This topic describes how to add replicas to a log stream under a tenant.

Increase the number of replicas by modifying the locality

You can increase the number of replicas of log streams in a tenant by modifying the locality of the tenant. For more information, see Modify the locality.

Prerequisites

Before you add a replica, make sure that the tenant has a resource pool in the target zone. If not, modify the zone list of an existing resource pool or create a new resource pool for the tenant. If you create a new resource pool, the number of units in the pool must be the same as that in the existing resource pools. For more information, see Modify resource pool properties or Create a resource pool.

Additionally, check the resource allocation on each node in the zone to be changed. If the zone has insufficient resources to store the required resource units for the tenant, the locality of the tenant cannot be changed.

For more information about the resource allocation on each node in the zone, see View tenant and resource information.

Considerations

• The locality of a tenant cannot be empty.

• When the locality of a tenant changes, the replica distribution of all tables under the tenant will also change.

• A new locality change cannot be executed until the previous one is completed.

• You can change the locality of a tenant only once at a time. For example, you can replace Zone3 with Zone4, but you cannot directly replace Zone1 with Zone2. Instead, you can add Zone4 first and then remove Zone1.

• When you change the locality of a tenant, pay attention to the impact on the primary zone. We recommend that you reassign the primary zone in advance.

  For example:

  1. If you want to reduce zone3 and zone3 is the primary zone with the highest priority, the locality change will affect the read/write services of the tenant during the process.

  2. When you add zone4 and the primary zone is set to RANDOM, zone4 will provide read/write services for the tenant after zone4 is added.

Procedure

Example: Add a zone to the mysql001 tenant by modifying its locality from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, thereby adding a replica for the tenant in zone3.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Enter the oceanbase database.

   obclient>use oceanbase;
   

3. View the locality of the mysql001 tenant before the modification.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   
   As we can see, the `mysql001` tenant has full-featured replicas in zones 1 and 2.
   
   

4. Modify the locality of the mysql001 tenant.

   obclient>ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3";
   

5. View the status of the locality modification task.

    obclient>SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |        NULL |        0 | 2023-01-05 19:38:38.416011 | 2023-01-05 19:38:38.416011 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | FULL{1}@zone1, FULL{1}@zone2                | xx.xx.xx.237 |        2882 |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   

   When JOB_STATUS is set to SUCCESS, the locality modification task is completed.

6. View the locality of the mysql001 tenant after the modification.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   

   As we can see, the locality of the mysql001 tenant has been modified from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, and the mysql001 tenant now has full-featured replicas in zones 1, 2, and 3.

The addition of a replica causes the locality of the tenant to be inconsistent with its primary zone. If the added zone participates in primary zone switchover, you need to modify the primary zone. For more information, see Modify the primary zone. If the added zone does not participate in primary zone switchover, no action is required.

Manually add a replica

When you add a replica by modifying the locality, the system automatically adds the required replicas to zones where they are missing. However, due to some exceptions, the system fails to work properly and you can manually add a replica by executing the ALTER SYSTEM ADD REPLICA statement.

Limitations

• The sys tenant (sys tenant) can add replicas to all tenants, and a user tenant can add replicas only to the current tenant.

• A tenant can have only one disaster recovery task on a log stream. The disaster recovery tasks include adding a replica, deleting a replica, converting the replica type, modifying the quorum size of the log stream, and migrating a replica.

  The sys tenant can check whether a disaster recovery task is running by querying the CDB_OB_LS_REPLICA_TASKS view; a user tenant can check whether a disaster recovery task is running by querying the DBA_OB_LS_REPLICA_TASKS view.

Prerequisites

• Before you add a replica, make sure that you have the ALTER SYSTEM privilege. Otherwise, you cannot execute the ALTER SYSTEM ADD REPLICA statement.

• Before you query the views, make sure that you have the SELECT privilege on the views. Otherwise, you cannot query the relevant information.

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • GV$OB_UNITS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

• Before you add a replica, make sure that the target server has a user unit, which is a resource of the OB unit, and that the server does not have a replica of the specified log stream.

Procedure

Assume that there is a tenant1 tenant whose locality is F@zone1,F@zone2,F@zone3. The 1001 log stream of the tenant1 tenant has a full-featured replica only in zone1 and zone3, and no replica in zone2. Now, you need to manually add a full-featured replica to zone2.

1. Log in to the tenant corresponding to the cluster.

   Here is an example of a connection statement:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information, see Overview (MySQL mode) and Overview (Oracle mode).

2. Obtain the tenant ID and locality of the tenant to be operated on.

   • For a sys tenant, execute the following statement:

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • For a user tenant, execute the following statement in MySQL mode or the following statement in Oracle mode:

     MySQL mode

     Oracle mode

     MySQL mode: execute the following statement:

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Oracle mode: execute the following statement:

     obclient [SYS]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM SYS.DBA_OB_TENANTS;
     

     A sample query result is as follows:

     +-------------+-----------+---------------------------------------------+
     | TENANT_NAME | TENANT_ID | LOCALITY                                    |
     +-------------+-----------+---------------------------------------------+
     | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
     +-------------+-----------+---------------------------------------------+
     1 row in set
     

     According to the query result, the tenant ID of the tenant1 tenant is 1002.

     For more information about the fields in the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Obtain information about all log streams of the tenant to be operated on.

   • For a sys tenant, execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS view, see CDB_OB_LS.

   • For a user tenant, execute the following statement in MySQL mode or the following statement in Oracle mode:

     MySQL mode

     Oracle mode

     MySQL mode: execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Oracle mode: execute the following statement:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

     A sample query result is as follows:

     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     | LS_ID | STATUS | PRIMARY_ZONE      | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     |     1 | NORMAL | zone1;zone3;zone2 |             0 |           0 |                NULL |     NULL | 1712137184525565000 | 1712137184525565000 |      |
     |  1001 | NORMAL | zone1;zone3;zone2 |          1001 |        1001 | 1711951923021132001 |     NULL | 1712137184292359002 | 1712137184292359002 |      |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     2 rows in set
     

     According to the query result, the tenant1 tenant has a log stream 1 and a log stream 1001.

     For more information about the fields in the DBA_OB_LS view, see DBA_OB_LS.

4. Obtain the units of the tenant.

   • For a sys tenant, execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_UNITS WHERE TENANT_ID=1002;
     

   • For a user tenant, execute the following statement in MySQL mode or the following statement in Oracle mode:

     MySQL mode

     Oracle mode

     MySQL mode: execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_UNITS;
     

     Oracle mode: execute the following statement:

     obclient [SYS]> SELECT * FROM SYS.GV$OB_UNITS;
     

     A sample query result is as follows:

     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     | SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | REGION         | MAX_CPU | MIN_CPU | MEMORY_SIZE | MAX_IOPS            | MIN_IOPS            | IOPS_WEIGHT | LOG_DISK_SIZE | LOG_DISK_IN_USE | DATA_DISK_IN_USE | STATUS | CREATE_TIME                |
     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     | 172.xx.xxx.226 |     2882 |    1001 |      1002 | zone1 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       665975868 |         17629184 | NORMAL | 2024-04-01 14:11:32.208968 |
     | 172.xx.xxx.212 |     2882 |    1002 |      1002 | zone2 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       373696561 |         19714048 | NORMAL | 2024-04-01 14:11:32.209358 |
     | 172.xx.xxx.192 |     2882 |    1003 |      1002 | zone3 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       263322684 |         19726336 | NORMAL | 2024-04-01 14:11:32.209473 |
     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     3 rows in set
     

     According to the query result, the servers 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882 all have available units for the tenant.

     For more information about the fields in the GV$OB_UNITS view, see GV$OB_UNITS.

5. View the distribution of replicas of log stream 1001 based on the previously obtained information.

   • For a sys tenant, execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • For a user tenant, execute the following statement in MySQL mode or the following statement in Oracle mode:

     MySQL mode

     Oracle mode

     MySQL mode: execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Oracle mode: execute the following statement:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     According to the query result, the log stream 1001 of the tenant has full-featured replicas on the servers 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882.

     Combining the result of the previous step, the servers 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882 all have available units for the tenant, and the server 172.xx.xxx.212:2882 does not have a replica of the log stream 1001. You need to manually add a full-featured replica to the server 172.xx.xxx.212:2882 for the log stream 1001 of the tenant.

     For more information about the fields in the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

6. Execute the statement to add a replica.

   The sample statement is as follows:

   ALTER SYSTEM ADD REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' REPLICA_TYPE [=] 'replica_type' [DATA_SOURCE [=] 'data_source'] [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   Here is a breakdown of the parameters in the statement:

   • ls_id: the ID of the log stream to which the replica belongs.

   • svr_ip:svr_port: the IP address and port number of the server where the replica is to be added. Example: 172.xx.xxx.212:2882.

   • replica_type: the type of the replica. Currently, the following replica types are supported:

     • Full-featured replica: FULL or F

     • Read-only replica: READONLY or R

     • Columnar replica: COLUMNSTORE or C

   • data_source: the data source address of the replica. Example: 172.xx.xxx.192:2882.

     If the specified data source is unavailable, the system returns an error.

     If this parameter is not specified, the system automatically selects an available data source.

   • paxos_replica_num: the new quorum size of the log stream, which is the number of full-featured replicas in the tenant locality. The maximum number of full-featured replicas in a log stream is 7.

     The value of this parameter must meet the following conditions:

     1. After the modification, the value of paxos_replica_num must ensure the majority.

     2. The value of paxos_replica_num must be greater than or equal to the number of members in MEMBER_LIST of the current leader replica.

     3. The absolute value of the difference between the new and old values of paxos_replica_num cannot exceed 1. For example, in this example, the current value of paxos_replica_num is 2, and you can change it only to 3.

     If the specified value does not meet the preceding conditions, the system returns an error. If this parameter is not specified, the system uses the default value of paxos_replica_num.

   • tenant_name: the name of the tenant. The sys tenant can specify other tenants, and a user tenant can specify only the current tenant. If the name of the tenant is not specified in the statement, the default tenant name is the name of the current tenant. The ALL, ALL_USER, and ALL_META keywords are not supported in this statement.

   • This statement can add only one replica at a time. If multiple replicas need to be added, you must execute this statement multiple times.

   Here is an example:

   obclient> ALTER SYSTEM ADD REPLICA LS= 1001 SERVER='172.xx.xxx.212:2882' REPLICA_TYPE='F' DATA_SOURCE='172.xx.xxx.192:2882' PAXOS_REPLICA_NUM=3;
   

7. Query the information about log streams again. You will find that the log stream has a new full-featured replica.

   • For a sys tenant, execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • For a user tenant, execute the following statement in MySQL mode or the following statement in Oracle mode:

     MySQL mode

     Oracle mode

     MySQL mode: execute the following statement:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Oracle mode: execute the following statement:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                                     | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:48:08.982791 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-03 17:48:05.812041 | 2024-04-03 17:48:09.985574 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:48:09.030020 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1712137685774460,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

References

• Locality overview

• Modify locality

• View locality

• Reduce the number of replicas

• Change the replica type

• Modify the quorum size

• Migrate a replica

• Cancel a replica task