Deploy an OceanBase cluster with two replicas and arbitration service using the CLI

Starting from version V4.1, OceanBase Database supports the arbitration service (AS), which addresses the issue of increased response time (RT) when a fault occurs in one of the two IDCs in a geo-redundant setup, reduces cross-region bandwidth usage, and keeps the costs of the third IDC low.

This topic will guide you through the deployment of an OceanBase cluster with two replicas and an arbitration service using the CLI.

Prerequisites

Before installing OceanBase Database, make sure that the following conditions are met:

• The OBServer node has been properly configured. For more information, see Server configuration, (Optional) Configure a clock source, and Initialize the OBServer node using oat-cli.

• You have obtained the RPM package of OceanBase Database. For more information, see Prepare installation packages.

Procedure

Step 1: Deploy an OceanBase cluster

1. Install the RPM packages on the OBServer servers.

   1. Install the OceanBase Database RPM package.

      Navigate to the directory where the OceanBase RPM package is stored:

      [root@xxx /]# cd $rpm_dir
      

      Install the OceanBase RPM package:

      [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
      

      Here, $rpm_dir indicates the directory where the RPM package is stored, and $rpm_name indicates the name of the RPM package.

      Note

      After installation, OceanBase Database will be installed in the /home/admin/oceanbase directory by default.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
          1:oceanbase-4.2.0.0-100000052023073################################# [100%]
      

   2. (Optional) Install OceanBase Client.

      OceanBase Client (OBClient) is a dedicated command-line client tool for OceanBase Database. With OBClient, you can connect to a MySQL tenant and an Oracle tenant of OceanBase Database. If you only want to connect to a MySQL tenant, you can also use the mysql client to connect to OceanBase Database.

      Notice

      Due to the dependency of obclient V2.2.1 and earlier on libobclient, you must install libobclient before installing obclient. Contact the technical support team to obtain the RPM packages of OBClient and libobclient.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh obclient-2.2.1-20221122151945.el7.alios7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
         1:obclient-2.2.1-20221122151945.el7################################# [100%]
      

      Check whether OBClient has been installed:

      [root@xxx /home/admin/rpm]# which obclient
      /usr/bin/obclient
      

2. Configure the observer process directory.

   1. (Optional) Clear the directory.

      When you deploy OceanBase Database for the first time on a new server, you do not need to clear the directory.

      If any of the following conditions are met, you can choose to clear the old OceanBase directory:

      • You want to clear the directory before you upgrade OceanBase Database.
      • The installation environment is messy or the installation process has generated files that will affect the next installation.

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /data/log1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/obdemo
      [root@xxx /home/admin]# rm -rf /data/log1/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

   2. Initialize the OceanBase directory.

      The data directory of OceanBase Database is usually recommended to be located on an independent disk, and then soft-linked to the home directory of the software.

      Note

      Starting from OceanBase Database V4.3.0, you can independently locate the slog from the data (data) disk. In other words, the slog and data files do not need to be on the same disk. You can configure the disks to share an SSD for slog and clog. For more information about the installation directories of OceanBase Database, see Directory structure of an OBServer node.

      Switch to the admin user:

      [root@xxx /home/admin]# su - admin
      

      Use the admin user to run the following commands to create the required directories:

      mkdir -p /data/1/$cluster_name/{etc3,sstable,slog}
      

      mkdir -p /data/log1/$cluster_name/{clog,etc2}
      

      mkdir -p /home/admin/oceanbase/store/$cluster_name
      

      Use the admin user to run the following command to create a soft link:

      for t in {etc3,sstable,slog};do ln -s /data/1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      for t in {clog,etc2};do ln -s /data/log1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# su - admin
      -bash-4.2$ mkdir -p /data/1/obdemo/{etc3,sstable,slog}
      -bash-4.2$ mkdir -p /data/log1/obdemo/{clog,etc2}
      -bash-4.2$ mkdir -p /home/admin/oceanbase/store/obdemo
      -bash-4.2$ for t in {etc3,sstable,slog};do ln -s /data/1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      -bash-4.2$ for t in {clog,etc2};do ln -s /data/log1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      

      Note

      The obdemo directory is created based on the cluster name and can be customized. It is used when the process is started.

      Check the result:

      -bash-4.2$ cd /home/admin/oceanbase
      -bash-4.2$ tree store/
      store/
      `-- obdemo
         |-- clog -> /data/log1/obdemo/clog
         |-- etc2 -> /data/log1/obdemo/etc2
         |-- etc3 -> /data/1/obdemo/etc3
         |-- slog -> /data/1/obdemo/slog
         `-- sstable -> /data/1/obdemo/sstable
      
      6 directories, 0 files
      

3. Initialize the OceanBase cluster.

   Note

   The example IP address has been desensitized and does not need to be changed during deployment.

   1. Start the observer process.

      Start the observer process on each node as the admin user.

      Notice

      The parameters for starting the observer process on each node are not exactly the same. When you start the observer process, specify the two (or more) servers where the Root Service is located. You do not need to specify all servers when you create a cluster. After the cluster is created, you can add new servers.

      cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer {-I $ip | -i $devname} -P $rpc_port -p $sql_port -z $zone_name -d /home/admin/oceanbase/store/$cluster_name -r '$ip:2882:2881' -c $cluster_id -n $cluster_name -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/$cluster_name/etc3;/data/log1/$cluster_name/etc2"
      

      The parameters are described as follows:

      Parameter	Description
      -I | -i	-I: the IP address of the server to be started. In a multi-server deployment, 127.0.0.1 cannot be used as the target IP address. We recommend that you use the specified IP address (-I 10.10.10.1) to start the server. -i: the name of the network card. You can view the available network card names by using the ifconfig command. Note You can also specify both the IP address and the network card name (-I 10.10.10.1 -i eth0) to start the server, but we recommend that you do not do this.
      -p	the service port number, which is generally set to 2881.
      -P	the RPC port number, which is generally set to 2882.
      -n	the cluster name. Custom cluster names must not be repeated.
      -z	the zone to which the observer process belongs.
      -d	the root directory of the cluster, which is the directory created when the cluster is initialized. Except for the cluster name $cluster_name, other parameters must not be changed.
      -c	the cluster ID. It is a group of digits that can be customized, but must not be repeated among clusters.
      -l	the log level.
      -r	the Root Service list in the $ip:2882:2881 format, semicolon-separated.
      -o	the list of cluster startup parameters (configurations), which is optional. Multiple configurations can be specified, and the configurations are separated by commas. Select appropriate startup parameters based on your business needs and specify suitable values for each parameter to optimize the cluster performance and resource utilization. Some common cluster startup configurations are described as follows: cpu_count: the total number of CPU cores. system_memory: the memory size reserved for tenant ID 500, namely, the memory that is internally reserved for OceanBase Database. If the server has limited memory, you can set this parameter to a smaller value. Note that the memory shortage may lead to an "out of memory" error during performance testing. memory_limit: the maximum total size of available memory. datafile_size: the size of disk space available for data files to be initialized. In other words, it is the initial size of the data file sstable of OceanBase Database. The value is assessed based on the available space in the /data/1/ directory. We recommend that you set the value to at least 100G. datafile_disk_percentage: the percentage of total disk space occupied by data files. datafile_next: the automatic expansion step length of disk data files. datafile_maxsize: the maximum automatic expansion size of disk data files. config_additional_dir: the additional local storage directories for storing configuration files for redundancy. log_disk_size: the size of the redo log disk. log_disk_percentage: the percentage of total disk space occupied by redo logs. syslog_level: the system log level. syslog_io_bandwidth_limit: the maximum disk I/O bandwidth allowed for system logs. System logs exceeding the bandwidth limit will be discarded. max_syslog_file_count: the maximum number of log files that can be stored before the log file recycling mechanism is triggered. enable_syslog_recycle: the flag to enable the log file recycling mechanism. It is used in conjunction with the max_syslog_file_count parameter to determine whether the log file recycling mechanism applies to old log files. The datafile_size, datafile_disk_percentage, datafile_next, and datafile_maxsize parameters are used to enable automatic expansion of disk data files. For more information, see Automatic expansion of disk data files. For more information about cluster configurations, see Overview of cluster-level configurations.

      Here is an example:

      Start the observer process on the OBServer servers. Take 10.10.10.1 and 10.10.10.2 as an example.

      zone1:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.1:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      zone2:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.2 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.2:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      Run the following command to check whether the observer process has been started:

      • Use the netstat -ntlp command. If the 2881 and 2882 ports are listened to, the observer process has been started.
      • Use the ps -ef|grep observer command to obtain the observer process information.

      Here is an example:

      -bash-4.2$ netstat -ntlp
      (Not all processes could be identified, non-owned process info
      will not be shown, you would have to be root to see it all.)
      Active Internet connections (only servers)
      Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
      tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      11112/observer
      tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      11112/observer
      ...        ...    ...                       ...                     ...         ...
      
      -bash-4.2$ ps -ef|grep observer
      admin     11112      0 40 16:18 ?        00:00:17 /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r 10.10.10.1:2882:2881;10.10.10.2:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
      

   2. Perform the cluster bootstrap operation.

      Use OBClient to connect to any server as the admin user. The password is empty.

      [root@xxx admin]# obclient -h127.0.0.1 -uroot -P2881 -p******
      
      obclient [(none)]> SET SESSION ob_query_timeout=1000000000;
      Query OK, 0 rows affected
      
      obclient [(none)]> ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '10.10.10.1:2882',ZONE 'zone2' SERVER '10.10.10.2:2882';
      Query OK, 0 rows affected
      

      Notice

      If an error occurs when you perform the cluster bootstrap operation, check whether the observer process startup parameters are correct, whether the permissions of the observer-related directories are correct, whether the log directory has sufficient space, whether the node time is synchronized, and whether the node has sufficient memory resources. If the problem persists, clear the OceanBase directory and deploy OceanBase Database again.

   3. Verify whether the cluster has been initialized.

      After the cluster is initialized, execute the SHOW DATABASES; statement to verify it. If the query result shows that the oceanbase database is included in the database list, the cluster has been initialized successfully.

   4. Change the password.

      The password of the root user in the sys tenant is empty by default. After the initialization is completed, you need to change the password.

      ALTER USER root IDENTIFIED BY '******';
      

Step 2: Deploy the arbitration service

1. Install the OceanBase RPM package on the arbitration server.

   Switch to the directory where the OceanBase RPM package is stored:

   [root@xxx /]# cd $rpm_dir
   

   Install the OceanBase RPM package:

   [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
   

   Here, $rpm_dir represents the directory where the RPM package is stored, and $rpm_name represents the name of the RPM package.

   Note

   By default, OceanBase Database will be installed in the /home/admin/oceanbase directory.

   Here is an example:

   [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
   Preparing...                          ################################# [100%]
   Updating / installing...
      1:oceanbase-4.2.0.0-100000052023073################################# [100%]
   

2. Configure the arbitration server process directory.

   Create a clog directory for the arbitration server process on the arbitration server.

   Note

   • When you start the arbitration server process, make sure that the clog directory has been created in the store directory specified with the -d option.
   • Do not use a disk with full space to store the data of the store directory, or the arbitration server's functionality may be abnormal.

   Switch to the admin user:

   [root@xxx /home/admin]# su - admin
   

   As the admin user, run the following command to create the required directories:

   mkdir -p /home/admin/oceanbase/store/clog
   

3. Start the arbitration server process.

   As the admin user on the arbitration server, start the arbitration server process in arbitration mode and specify the necessary parameters. The sample statement is as follows:

   cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P $rpc_port -d $obdir/store [{-I $ip | -i $devname}] [-l $log_level]  [-o "parameters_list"]
   

   Here is a breakdown of the parameters:

   Parameter	Description
   -m	Specifies the mode of the arbitration process. The value is arbitration.
   -P	Specifies the RPC port number of the arbitration server process. The default value is 2882.
   -d	Specifies the storage directory path. The default value is /home/admin/oceanbase/store. You must create the clog subdirectory in this directory before you start the process.
   -I | -i	Optional. -I: Specifies the IP address of the node to be started. -i: Specifies the network card name. You can run the ifconfig command to view the network card names. Note You can specify both the IP address and the network card name (for example, eth0) to start a node, but it is not recommended.
   -l	Specifies the log level. The default value is WDIAG. The value range is: {DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR}. For more information, see Log levels.
   -o	Specifies the parameter list. The value is a comma-separated list of parameter names. You can set values for the specified parameters. The following parameters are commonly used when you start the arbitration server process: cpu_count system_memory memory_limit __easy_memory_limit log_disk_size log_disk_percentage syslog_level syslog_io_bandwidth_limit max_syslog_file_count enable_syslog_recycle Notice To ensure automatic cleanup of the log directory of an arbitration node and prevent the disk from filling up, set enable_syslog_recycle to True and max_syslog_file_count to an effective value. Otherwise, log files will grow continuously and may occupy all disk space.

   Here is an example:

   Start the arbitration server process on the arbitration server. For example, start the arbitration server process with 10.10.10.3 as the IP address.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3 -o "enable_syslog_recycle=True,max_syslog_file_count=100"
   

   Run the following command to check whether the observer process has started:

   • The netstat -ntlp command. If it listens on port 2882, the observer process has started.
   • The ps -ef|grep observer command. It will return information about the observer process.

   Here is an example:

   -bash-4.2$ netstat -ntlp
   (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
   tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      18231/observer
   ...
   
   -bash-4.2$ ps -ef|grep observer
   admin     18231      0  3 10:33 ?        00:00:00 /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3
   ...
   

Step 3: Add an arbitration service to the cluster

Run the following command to add an arbitration service to the cluster:

ALTER SYSTEM ADD ARBITRATION SERVICE "$arb_server_ip:$arb_server_port";

Here is an example:

obclient [(none)]> ALTER SYSTEM ADD ARBITRATION SERVICE "10.10.10.3:2882";
Query OK, 0 rows affected

obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| 2023-03-06 17:29:36 | 2023-03-06 17:29:36 | default                 | 10.10.10.3:2882     | NULL                         | ADDR |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
1 row in set

Step 4: Enable the arbitration service for the sys tenant

You can enable the arbitration service for a tenant when you create the tenant. For more information, see CREATE TENANT.

If you do not enable the arbitration service when you create the tenant, you can enable it after the tenant is created.

Run the following statement to enable the arbitration service for an existing tenant.

ALTER TENANT $tenant_name [SET] enable_arbitration_service = true;

Here is an example:

Enable the arbitration service for the sys tenant in the cluster.

obclient [(none)]> ALTER TENANT sys SET enable_arbitration_service = true;
Query OK, 0 rows affected

Step 5: Query the status of the arbitration service for a tenant

1. Check whether the arbitration service is enabled for the tenant.

   Query the information of the tenant from the DBA_OB_TENANTS view and check the value of the arbitration_service_status column for the status of the arbitration service.

   The possible values of arbitration_service_status are as follows:

   • ENABLED: the arbitration service is enabled for the tenant.
   • ENABLING: the arbitration service is being enabled for the tenant.
   • DISABLED: the arbitration service is disabled for the tenant.
   • DISABLING: the arbitration service is being disabled for the tenant.

   Here is an example:

   obclient [(none)]> SELECT TENANT_ID,TENANT_NAME,ARBITRATION_SERVICE_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE tenant_name = 'sys';
   +-----------+-------------+----------------------------+
   | TENANT_ID | TENANT_NAME | ARBITRATION_SERVICE_STATUS |
   +-----------+-------------+----------------------------+
   |         1 | sys         | ENABLED                    |
   +-----------+-------------+----------------------------+
   1 row in set
   

2. Check whether the arbitration service is available for the tenant.

   If the arbitration service is enabled for the tenant, the arbitration service is available for the log streams created at the time when it is enabled. However, for log streams created after it is enabled, it may be unavailable because creating a log stream is performed in non-strict mode, and there is no guarantee that the arbitration service is created successfully. In this case, you can run the following SQL statement to check whether the arbitration service is available for all log streams of the tenant.

   ##Query the list of log streams without an arbitration replica in the tenant##
   (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx and role = 'LEADER' and arbitration_member = "$arb_server_ip:$arb_server_port");
   

   Here is an example:

   Query the list of log streams without an arbitration replica in the sys tenant. If the query result set is an empty set, the arbitration service is available for the current tenant.

   obclient [oceanbase]> (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1 and role = 'LEADER' and arbitration_member = "10.10.10.3:2882");
   Empty set
   

Next steps

After the cluster is created, you can create user tenants as needed.

For more information about how to create a user tenant, see Create a tenant.

References

• Connect to an OceanBase tenant by using OBClient
• Overview of system configuration items
• ALTER USER
• ALTER TENANT
• Introduction to user tenants
• CREATE TENANT
• Enable the arbitration service
• Disable the arbitration service
• Modify the degradation timeout period
• Replace the arbitration service
• Remove the arbitration service