Add replicas

This topic describes how to add log stream replicas to a tenant.

Add replicas by modifying the locality

You can add log stream replicas to a tenant in a cluster by modifying the locality of the tenant. For more information about how to modify the locality, see Modify locality.

Prerequisites

Before you add a replica, check whether the tenant has a resource pool in the target zone. If not, modify the zone list of the existing resource pool of the tenant or add a resource pool to the tenant. When you add a resource pool, the number of units in the resource pool must be the same as that in the existing resource pool of the tenant. For more information, see Modify attributes of a resource pool or the "Step 2: Create a resource pool" section in Create a tenant.

You must also check the resource information of each node in the target zone. If OBServer nodes in the target zone do not have sufficient resources to accommodate the new resource units of the tenant, the locality cannot be changed.

For more information about how to view the resource information of each node in a zone, see View tenant and resource information.

Considerations

• The locality of the tenant must be specified.

• When the locality of a tenant changes, the distribution of table replicas in the tenant also changes.

• When a tenant locality modification is not finished, the next tenant locality modification cannot be performed.

• You can perform only one action at a time when you modify the locality. For example, if you want to replace zone3 with zone4, you must add zone4 and then remove zone3.

• We recommend that you modify the primary zone before you modify the locality.

  Here are some examples:

  1. Assume that zone3 is a primary zone with the highest priority. When you remove zone3, read and write services of the tenant are affected during locality modification.

  2. Assume that the primary zone is set to RANDOM when you add zone4. After zone4 is added, it also provides read and write services.

Procedure

The following example describes how to modify the locality of the mysql001 tenant from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, and add a replica to the tenant in zone3.

1. Log in to the sys tenant of the cluster as the root user.

   obclient -h172.30.xxx.xxx -P2883 -uroot@sys#obdemo -pxxxx -A
   

2. Access the database named oceanbase.

   obclient>use oceanbase;
   

3. View the locality of the mysql001 tenant.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2                |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   

   The query result shows that the mysql001 tenant has a full-featured replica in both zone1 and zone2.

4. Modify the locality of the mysql001 tenant.

   obclient>ALTER TENANT mysql001 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3";
   

5. View the execution status of the locality modification job.

    obclient>SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    | JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                     | EXTRA_INFO                                  | RS_SVR_IP      | RS_SVR_PORT |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
    |      1 | ALTER_TENANT_LOCALITY | SUCCESS    |        NULL |        0 | 2023-01-05 19:38:38.416011 | 2023-01-05 19:38:38.416011 |      1002 | ALTER TENANT mysql001 locality='FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | FULL{1}@zone1, FULL{1}@zone2                | xx.xx.xx.237 |        2882 |
    +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+------------------------------------------------------------------------------+---------------------------------------------+----------------+-------------+
   

   If the value of JOB_STATUS is SUCCESS, the locality modification job is successful.

6. View the modified locality of the mysql001 tenant.

   obclient> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   | TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   |         1 | sys         | SYS         | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1001 | META$1002   | META        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   |      1002 | mysql001    | USER        | zone1;zone2       | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
   +-----------+-------------+-------------+-------------------+---------------------------------------------+
   

   The query results in step 3 and step 6 show that the locality of the mysql001 tenant is changed from FULL{1}@zone1, FULL{1}@zone2 to FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, and the mysql001 tenant has a full-featured replica in each of zone1, zone2, and zone3.

After you add a replica to a tenant, the locality of the tenant does not match the primary zone of the tenant. If the new zone is involved in leader switchover, you must modify the primary zone. For more information about how to modify the primary zone, see Modify the primary zone. If the new zone is not involved in leader switchover, you do not need to modify the primary zone.

Manually add replicas

When you add replicas to a tenant by modifying the locality of the tenant, the system automatically adds the missing replicas. However, if the system does not properly add replicas due to exceptions, you can execute the ALTER SYSTEM ADD REPLICA statement to add the missing log stream replicas.

Limitations

• You can add log stream replicas to all tenants in the sys tenant but add log stream replicas only to the current tenant in a user tenant.

• One log stream of a tenant supports only one replica task at a time, such as a task to add replicas, remove replicas, convert the type of a replica, modify the number of Paxos replicas of a log stream, or migrate replicas.

  You can query ongoing replica tasks from the CDB_OB_LS_REPLICA_TASKS view in the sys tenant or from the DBA_OB_LS_REPLICA_TASKS view in a user tenant.

Prerequisites

• You have the ALTER SYSTEM privilege.

• You have the SELECT privilege on the following views:

  • DBA_OB_TENANTS

  • DBA_OB_LS/CDB_OB_LS

  • GV$OB_UNITS

  • DBA_OB_LS_LOCATIONS/CDB_OB_LS_LOCATIONS

• The target OBServer node has an available unit that does not contain replicas of the log stream.

Procedure

Assume that the locality of a tenant named tenant1 is F@zone1,F@zone2,F@zone3, and log stream 1001 of the tenant1 tenant has a full-featured replica in each of zone1 and zone3 and has no replica in zone2. To add a full-featured replica to zone2, perform the following steps:

1. Log in to the target tenant.

   Here is an example:

   obclient -h172.30.xxx.xxx -P2883 -uroot@tenant1#obdemo -pxxxx -A
   

   For more information about how to connect to a database, see Overview (MySQL mode) or Overview (Oracle mode).

2. Query the tenant ID and locality of the target tenant.

   • sys tenant

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='tenant1';
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT TENANT_NAME,TENANT_ID,LOCALITY FROM SYS.DBA_OB_TENANTS;
     

     A sample query result is as follows:

     +-------------+-----------+---------------------------------------------+
     | TENANT_NAME | TENANT_ID | LOCALITY                                    |
     +-------------+-----------+---------------------------------------------+
     | tenant1     |      1002 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
     +-------------+-----------+---------------------------------------------+
     1 row in set
     

     The query result shows that the tenant ID is 1002.

     For more information about the fields in the DBA_OB_TENANTS view, see DBA_OB_TENANTS.

3. Query all log streams of the target tenant.

   • sys tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS WHERE TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS view, see CDB_OB_LS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS;
     

     A sample query result is as follows:

     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     | LS_ID | STATUS | PRIMARY_ZONE      | UNIT_GROUP_ID | LS_GROUP_ID | CREATE_SCN          | DROP_SCN | SYNC_SCN            | READABLE_SCN        | FLAG |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     |     1 | NORMAL | zone1;zone3;zone2 |             0 |           0 |                NULL |     NULL | 1712137184525565000 | 1712137184525565000 |      |
     |  1001 | NORMAL | zone1;zone3;zone2 |          1001 |        1001 | 1711951923021132001 |     NULL | 1712137184292359002 | 1712137184292359002 |      |
     +-------+--------+-------------------+---------------+-------------+---------------------+----------+---------------------+---------------------+------+
     2 rows in set
     

     The query result shows that the tenant1 tenant has log streams 1 and 1001.

     For more information about the fields in the DBA_OB_LS view, see DBA_OB_LS.

4. Query units of the target tenant.

   • sys tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_UNITS WHERE TENANT_ID=1002;
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_UNITS;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.GV$OB_UNITS;
     

     A sample query result is as follows:

     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     | SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | REGION         | MAX_CPU | MIN_CPU | MEMORY_SIZE | MAX_IOPS            | MIN_IOPS            | IOPS_WEIGHT | LOG_DISK_SIZE | LOG_DISK_IN_USE | DATA_DISK_IN_USE | STATUS | CREATE_TIME                |
     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     | 172.xx.xxx.226 |     2882 |    1001 |      1002 | zone1 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       665975868 |         17629184 | NORMAL | 2024-04-01 14:11:32.208968 |
     | 172.xx.xxx.212 |     2882 |    1002 |      1002 | zone2 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       373696561 |         19714048 | NORMAL | 2024-04-01 14:11:32.209358 |
     | 172.xx.xxx.192 |     2882 |    1003 |      1002 | zone3 | ReadWrite | default_region |       2 |       2 |  4294967296 | 9223372036854775807 | 9223372036854775807 |           2 |   14495514624 |       263322684 |         19726336 | NORMAL | 2024-04-01 14:11:32.209473 |
     +----------------+----------+---------+-----------+-------+-----------+----------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
     3 rows in set
     

     The query result shows that the tenant has available units on OBServer nodes 172.xx.xxx.226:2882, 172.xx.xxx.212:2882, and 172.xx.xxx.192:2882.

     For more information about the fields in the GV$OB_UNITS view, see GV$OB_UNITS.

5. Query the replica distribution of log stream 1001 based on the obtained log stream information.

   • sys tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

     For more information about the fields in the CDB_OB_LS_LOCATIONS view, see CDB_OB_LS_LOCATIONS.

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:32:49.079886 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                       |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:32:49.124409 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.226:2882:1 |                    2 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+------------------------------------------------------------+----------------------+--------------+--------------+---------+
     2 rows in set
     

     The query result shows that log stream 1001 has a full-featured replica on each of OBServer nodes 172.xx.xxx.192:2882 and 172.xx.xxx.226:2882, and

     has no replica on OBServer node 172.xx.xxx.212:2882. However, the query result of the previous step shows that the tenant has available units on all the three OBServer nodes. Therefore, you must add a full-featured replica to OBServer node 172.xx.xxx.212:2882 for log stream 1001 of the tenant.

     For more information about the fields in the DBA_OB_LS_LOCATIONS view, see DBA_OB_LS_LOCATIONS.

6. Execute the ALTER SYSTEM ADD REPLICA statement to add a replica.

   The SQL syntax is as follows:

   ALTER SYSTEM ADD REPLICA LS [=] ls_id SERVER [=] 'svr_ip:svr_port' REPLICA_TYPE [=] 'replica_type' [DATA_SOURCE [=] 'data_source'] [PAXOS_REPLICA_NUM [=] paxos_replica_num] [TENANT [=] 'tenant_name'];
   

   The parameters in the syntax are described as follows:

   • ls_id: the ID of the log stream for which you want to add the replica.

   • svr_ip:svr_port: the IP address and port number of the OBServer node to which you want to add the replica, for example, 172.xx.xxx.212:2882.

   • replica_type: the type of the replica that you want to add. The following types of replicas are supported:

     • Full-featured replica: denoted as FULL or F.

     *Read-only replica: denoted as READONLY or R.

     • Columnstore replica: denoted as COLUMNSTORE or C.

   • data_source: the data source of the replica that you want to add, for example, 172.xx.xxx.192:2882.

     If the specified data source is unavailable, the system will return an error.

     If this parameter is not specified, the system automatically selects an available data source.

   • paxos_replica_num: the modified number of Paxos replicas of the log stream, which is the number of full-featured replicas specified in the locality of the tenant. A log stream supports a maximum of seven full-featured replicas.

     The value of the paxos_replica_num parameter must meet the following conditions:

     1. After the value of the paxos_replica_num parameter is modified, the available replicas are the majority.

     2. The value of the paxos_replica_num parameter must be greater than or equal to the number of members in the MEMBER_LIST of the leader.

     3. The absolute value of the variation of the paxos_replica_num value is not greater than 1. In this example, the value of the paxos_replica_num parameter is 2, and you can change it only to 3.

     If any one of the preceding conditions is not met, the system will return an error. If this parameter is not specified, the default value is used.

   • tenant_name: the name of the target tenant. You can specify another tenant in the sys tenant and specify only the current tenant in a user tenant. If this parameter is not specified, the name of the current tenant is used. You cannot use the all, all_user, or all_meta option in this statement to specify all tenants, all user tenants, or all meta tenants.

   • This statement adds only one replica at a time. To add multiple replicas, you must execute this statement multiple times.

   Here is an example:

   obclient> ALTER SYSTEM ADD REPLICA LS= 1001 SERVER='172.xx.xxx.212:2882' REPLICA_TYPE='F' DATA_SOURCE='172.xx.xxx.192:2882' PAXOS_REPLICA_NUM=3;
   

7. Query information about the log stream again to verify that one full-featured replica is added for the log stream.

   • sys tenant

     obclient [oceanbase]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE LS_ID=1001 AND TENANT_ID=1002;
     

   • User tenant

     MySQL mode

     Oracle mode

     Execute the following statement in MySQL mode:

     obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     Execute the following statement in Oracle mode:

     obclient [SYS]> SELECT * FROM SYS.DBA_OB_LS_LOCATIONS WHERE LS_ID=1001;
     

     A sample query result is as follows:

     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | CREATE_TIME                | MODIFY_TIME                | LS_ID | SVR_IP         | SVR_PORT | SQL_PORT | ZONE  | ROLE     | MEMBER_LIST                                                                                     | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     | 2024-04-03 17:23:06.381521 | 2024-04-03 17:48:08.982791 |  1001 | 172.xx.xxx.192 |     2882 |     2881 | zone3 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-03 17:48:05.812041 | 2024-04-03 17:48:09.985574 |  1001 | 172.xx.xxx.212 |     2882 |     2881 | zone2 | FOLLOWER | NULL                                                                                            |                 NULL | FULL         |              | FALSE   |
     | 2024-04-01 14:12:08.589084 | 2024-04-03 17:48:09.030020 |  1001 | 172.xx.xxx.226 |     2882 |     2881 | zone1 | LEADER   | 172.xx.xxx.192:2882:1712136186291857,172.xx.xxx.212:2882:1712137685774460,172.xx.xxx.226:2882:1 |                    3 | FULL         |              | FALSE   |
     +----------------------------+----------------------------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------------------------------------+----------------------+--------------+--------------+---------+
     3 rows in set
     

References

• Locality

• Modify locality

• View locality

• Remove replicas

• Convert the type of a replica

• Modify the number of Paxos replicas of a log stream

• Migrate replicas

• Cancel a replica task