Deploy an OceanBase cluster with two replicas and arbitration service using the CLI

Starting from V4.1, OceanBase Database supports the arbitration service (AS), which addresses the issue of increased response time (RT) when a fault occurs in one of the two IDCs in a three IDCs scenario, reduces cross-region bandwidth usage, and keeps the costs of the third IDC at a low level.

This topic will guide you through the deployment of an OceanBase cluster with two replicas and an arbitration service using the CLI.

Prerequisites

Before installing OceanBase Database, make sure that the following conditions are met:

• The OBServer node has been configured. For more information, see Server configuration, (Optional) Configure a clock source, and Initialize the OBServer node using oat-cli.

• You have obtained the RPM package of OceanBase Database. For more information, see Prepare installation packages.

Procedure

Step 1: Deploy an OceanBase cluster

1. Install the RPM packages on the two OBServer servers.

   1. Install the OceanBase Database RPM package.

      Navigate to the directory where the OceanBase RPM package is stored:

      [root@xxx /]# cd $rpm_dir
      

      Install the OceanBase RPM package:

      [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
      

      Here, $rpm_dir indicates the directory where the RPM package is stored, and $rpm_name indicates the name of the RPM package.

      Note

      After installation, OceanBase Database will be installed in the /home/admin/oceanbase directory by default.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
          1:oceanbase-4.2.0.0-100000052023073################################# [100%]
      

   2. (Optional) Install OceanBase Client.

      OceanBase Client (OBClient) is a dedicated CLI tool for OceanBase Database. With OBClient, you can connect to MySQL and Oracle tenants in OceanBase Database. If you only need to connect to a MySQL tenant in OceanBase Database, you can also use the mysql client to connect to OceanBase Database.

      Notice

      Since obclient V2.2.1 depends on libobclient, you must install libobclient first. Contact the technical support team to obtain the RPM packages of OBClient and libobclient.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh obclient-2.2.1-20221122151945.el7.alios7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
         1:obclient-2.2.1-20221122151945.el7################################# [100%]
      

      Check whether OBClient has been installed:

      [root@xxx /home/admin/rpm]# which obclient
      /usr/bin/obclient
      

2. Configure the observer process directory.

   1. (Optional) Clear the directory.

      When you deploy OceanBase Database for the first time on a new server, you do not need to clear the directory.

      If any of the following conditions are met, you can choose to clear the old OceanBase directory:

      • You want to clear the directory before you upgrade OceanBase Database.
      • The installation environment is chaotic or the installation process has generated files that will affect the next installation.

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /data/log1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/obdemo
      [root@xxx /home/admin]# rm -rf /data/log1/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

   2. Initialize the OceanBase directory.

      The data directory of OceanBase Database is usually stored on an independent disk. Then, it is soft-linked to the home directory of the software.

      Note

      Starting from V4.3.0, OceanBase Database allows you to independently store slog from the data (data) disk. In other words, slog does not need to be stored on the same disk as data files. You can configure the disks to share an SSD for slog and clog. For more information about the installation directories of OceanBase Database, see Directory structure on an OBServer node.

      Switch to the admin user:

      [root@xxx /home/admin]# su - admin
      

      As the admin user, run the following commands to create the required directories:

      mkdir -p /data/1/$cluster_name/{etc3,sstable,slog}
      

      mkdir -p /data/log1/$cluster_name/{clog,etc2}
      

      mkdir -p /home/admin/oceanbase/store/$cluster_name
      

      As the admin user, run the following commands to create a soft link:

      for t in {etc3,sstable,slog};do ln -s /data/1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      for t in {clog,etc2};do ln -s /data/log1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# su - admin
      -bash-4.2$ mkdir -p /data/1/obdemo/{etc3,sstable,slog}
      -bash-4.2$ mkdir -p /data/log1/obdemo/{clog,etc2}
      -bash-4.2$ mkdir -p /home/admin/oceanbase/store/obdemo
      -bash-4.2$ for t in {etc3,sstable,slog};do ln -s /data/1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      -bash-4.2$ for t in {clog,etc2};do ln -s /data/log1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      

      Note

      The obdemo directory is created based on the cluster name and can be customized. It is used when the processes are started.

      Check the results:

      -bash-4.2$ cd /home/admin/oceanbase
      -bash-4.2$ tree store/
      store/
      `-- obdemo
         |-- clog -> /data/log1/obdemo/clog
         |-- etc2 -> /data/log1/obdemo/etc2
         |-- etc3 -> /data/1/obdemo/etc3
         |-- slog -> /data/1/obdemo/slog
         `-- sstable -> /data/1/obdemo/sstable
      
      6 directories, 0 files
      

3. Initialize the OceanBase cluster.

   Note

   • The example IP addresses are desensitized. This does not affect the actual installation. When you deploy OceanBase Database, you need to fill in the IP addresses of your servers.
   • OceanBase Database can start the observer process on a server with a specified IPv4 or IPv6 address. The example below shows how to start the observer process on an IPv4 address. If you want to learn how to start the observer process on an IPv6 address, see Deploy an OceanBase cluster using the command line.

   1. Start the observer process.

      Start the observer process on each node as the admin user.

      Notice

      The start parameters of the observer process are different on the two replicas. When you start the observer process, specify the IP address and port number of the two root service nodes. You do not need to specify all the servers when you create a cluster, but you can add new servers after the cluster is created.

      cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer {-I $ip | -i $devname} -P $rpc_port -p $sql_port -z $zone_name -d /home/admin/oceanbase/store/$cluster_name -r '$ip:2882:2881' -c $cluster_id -n $cluster_name -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/$cluster_name/etc3;/data/log1/$cluster_name/etc2"
      

      The parameters are described as follows:

      Parameter	Description
      -I | -i	-I: the IP address of the server where the observer process is to be started. In a multi-server deployment, 127.0.0.1 cannot be used as the target IP address. We recommend that you use the specified IP address (-I 10.10.10.1) to start the observer process. -i: the name of the network card. You can view the available network card names by using the ifconfig command. Note You can also specify both the IP address and the network card name (-I 10.10.10.1 -i eth0) to start the observer process, but we do not recommend that you do so.
      -p	the service port number, which is usually set to 2881.
      -P	the RPC port number, which is usually set to 2882.
      -n	the cluster name. Custom cluster names must be unique.
      -z	the zone to which the observer process belongs.
      -d	the cluster's main directory, which is the directory created during the initialization. Apart from the cluster name $cluster_name, other parameters in the path should not be changed.
      -c	the cluster ID. A sequence of digits. Custom cluster IDs must be unique.
      -l	the log level.
      -r	the RS list. The information of the root service is separated by semicolons ($ip:2882:2881).
      -o	the list of cluster startup parameters (configurations). It is an optional parameter. You can specify multiple configurations, and separate the values of the configurations with commas. You can set the startup parameters as needed to optimize the performance and resource efficiency of the cluster. Here are some commonly used cluster startup configurations: cpu_count: the total number of system CPU cores. system_memory: the memory size reserved for tenant ID 500. In other words, it specifies the internal memory of OceanBase Database. If the server has limited memory, you can set this value to a smaller one. Note that during performance tests, the memory might be insufficient. memory_limit: the total size of available memory. datafile_size: the size of disk space available for data files to be initialized. In other words, it specifies the size of the sstable file in OceanBase Database. Based on the evaluation of the available space in /data/1/, we recommend that you set the value to at least 100G. datafile_disk_percentage: the percentage of total disk space occupied by data files. datafile_next: the automatic expansion step length of data files. datafile_maxsize: the maximum size of data files during automatic expansion. config_additional_dir: the additional local storage directories for storing configuration files to ensure redundancy. log_disk_size: the size of the redo log disk. log_disk_percentage: the percentage of total disk space occupied by redo logs. syslog_level: the system log level. syslog_io_bandwidth_limit: the upper limit on disk I/O bandwidth available for system logs. System logs exceeding the bandwidth limit will be discarded. max_syslog_file_count: the maximum number of log files that can be stored before the system starts to recycle logs. enable_syslog_recycle: the switch to enable log recycling. It is used in conjunction with max_syslog_file_count to determine whether to recycle old log files. The datafile_size, datafile_disk_percentage, datafile_next, and datafile_maxsize parameters are used together to enable automatic expansion of data files. For more information, see Configure automatic expansion of data files. For more information about cluster configurations, see Overview of configurations - Cluster-level configurations.

      Here is an example:

      Start the observer process on the OBServer servers. Take 10.10.10.1 and 10.10.10.2 as an example.

      zone1:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.1:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      zone2:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.2 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.2:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      Run the following command to check whether the observer process has been started:

      • The netstat -ntlp command. If the process has listened on ports 2881 and 2882, the process has been started.
      • The ps -ef|grep observer command. It will return information about the observer process.

      Here is an example:

      -bash-4.2$ netstat -ntlp
      (Not all processes could be identified, non-owned process info
      will not be shown, you would have to be root to see it all.)
      Active Internet connections (only servers)
      Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
      tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      11112/observer
      tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      11112/observer
      ...        ...    ...                       ...                     ...         ...
      
      -bash-4.2$ ps -ef|grep observer
      admin     11112      0 40 16:18 ?        00:00:17 /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r 10.10.10.1:2882:2881;10.10.10.2:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
      

   2. Perform the bootstrap operation on the cluster.

      Use OBClient to connect to any of the nodes. The password is empty.

      [root@xxx admin]# obclient -h127.0.0.1 -uroot -P2881 -p******
      
      obclient [(none)]> SET SESSION ob_query_timeout=1000000000;
      Query OK, 0 rows affected
      
      obclient [(none)]> ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '10.10.10.1:2882',ZONE 'zone2' SERVER '10.10.10.2:2882';
      Query OK, 0 rows affected
      

      Notice

      If an error occurs in this step, check whether the start parameters of the observer process are correct, whether the directory permissions of the observer process are correct, whether the log directory has sufficient space, whether the node time is synchronized, and whether the node has sufficient memory resources. If any of these conditions are not met, correct the problem and then clear the OceanBase directory to deploy OceanBase Database again.

   3. Verify that the cluster has been initialized.

      After the cluster is initialized, execute the SHOW DATABASES; command to verify it. If the query result shows that the oceanbase database is included in the database list, the cluster has been initialized successfully.

   4. Change the password.

      The password of the root user in the sys tenant is empty by default. After the initialization, you need to change the password.

      ALTER USER root IDENTIFIED BY '******';
      

Step 2: Deploy the arbitration service

1. Install the OceanBase RPM package on the arbitration server.

   Switch to the directory where the OceanBase RPM package is stored:

   code-placeholder/6f3a8117-b589-4595-abc7-bad3337e118

   Install the OceanBase RPM package:

   [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
   

   Here, $rpm_dir represents the directory where the RPM package is stored, and $rpm_name represents the name of the RPM package.

   Note

   By default, OceanBase Database will be installed in the /home/admin/oceanbase directory.

   Here is an example:

   [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
   Preparing...                          ################################# [100%]
   Updating / installing...
      1:oceanbase-4.2.0.0-100000052023073################################# [100%]
   

2. Configure the arbitration server process directory.

   Create a clog directory for the arbitration server process on the arbitration server.

   Note

   • When you start the arbitration server process, make sure that the clog directory has been created in the store directory specified with the -d option.
   • Do not use up all the disk space of the disk where the store directory is located, otherwise the arbitration server functionality may be compromised.

   Switch to the admin user:

   [root@xxx /home/admin]# su - admin
   

   As the admin user, run the following command to create the required directories:

   mkdir -p /home/admin/oceanbase/store/clog
   

3. Start the arbitration server process.

   As the admin user on the arbitration server, start the arbitration server process in arbitration mode and specify the necessary parameters. The sample statement is as follows:

   cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P $rpc_port -d $obdir/store [{-I $ip | -i $devname}] [-l $log_level]  [-o "parameters_list"]
   

   Here is a breakdown of the parameters:

   Parameter	Description
   -m	Specifies the mode of the arbitration process. The value is arbitration.
   -P	Specifies the RPC port number of the arbitration server process. The default value is 2882.
   -d	Specifies the storage directory path. The default value is /home/admin/oceanbase/store. You must create the clog subdirectory in this directory before you start the process.
   -I | -i	Optional. -I: the IP address of the node to be started. -i: the name of the network card. You can use the ifconfig command to view the network card names. Note You can start a node by specifying both the IP address and the network card name (for example: -I 10.10.10.1 -i eth0). However, we recommend that you do not specify both parameters.
   -l	Specifies the log level. The default value is WDIAG. The value range is: {DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR}. For more information, see Log levels.
   -o	Specifies the configuration parameter list. The parameters can be specified multiple times, separated by commas. The following configuration parameters are commonly used when you start the arbitration server process: cpu_count system_memory memory_limit __easy_memory_limit log_disk_size log_disk_percentage syslog_level syslog_io_bandwidth_limit max_syslog_file_count enable_syslog_recycle Notice To ensure automatic cleanup of the log directory of the arbitration node and prevent the disk from being filled up, set enable_syslog_recycle to True and specify an effective value for max_syslog_file_count. Otherwise, log files will grow indefinitely, which may lead to a disk full.

   Here is an example:

   Start the arbitration server process on the arbitration server. For example, start the arbitration server process with 10.10.10.3 as the IP address.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3 -o "enable_syslog_recycle=True,max_syslog_file_count=100"
   

   You can use the following command to check whether the observer process has started successfully:

   • Run the netstat -ntlp command. If the 2882 port is monitored, the process has started successfully.
   • Run the ps -ef|grep observer command to return the observer process information.

   Here is an example:

   -bash-4.2$ netstat -ntlp
   (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
   tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      18231/observer
   ...
   
   -bash-4.2$ ps -ef|grep observer
   admin     18231      0  3 10:33 ?        00:00:00 /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3
   ...
   

Step 3: Add an arbitration service to the cluster

Run the following command to add an arbitration service to the cluster:

ALTER SYSTEM ADD ARBITRATION SERVICE "$arb_server_ip:$arb_server_port";

Here is an example:

obclient [(none)]> ALTER SYSTEM ADD ARBITRATION SERVICE "10.10.10.3:2882";
Query OK, 0 rows affected

obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| 2023-03-06 17:29:36 | 2023-03-06 17:29:36 | default                 | 10.10.10.3:2882     | NULL                         | ADDR |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
1 row in set

Step 4: Enable the arbitration service for the sys tenant

You can enable the arbitration service for a tenant when you create the tenant. For more information, see CREATE TENANT.

If you do not enable the arbitration service when you create the tenant, you can enable it after the tenant is created.

Run the following statement to enable the arbitration service for an existing tenant.

ALTER TENANT $tenant_name [SET] enable_arbitration_service = true;

Here is an example:

Enable the arbitration service for the sys tenant of a cluster.

obclient [(none)]> ALTER TENANT sys SET enable_arbitration_service = true;
Query OK, 0 rows affected

Step 5: Query the status of the arbitration service for a tenant

1. Check whether the arbitration service is enabled for the tenant.

   You can query the DBA_OB_TENANTS view for tenant information and check the value of the arbitration_service_status column for the status of the arbitration service.

   The possible values of arbitration_service_status are as follows:

   • ENABLED: The arbitration service is enabled for the tenant.
   • ENABLING: The arbitration service is being enabled for the tenant.
   • DISABLED: The arbitration service is disabled for the tenant.
   • DISABLING: The arbitration service is being disabled for the tenant.

   Here is an example:

   obclient [(none)]> SELECT TENANT_ID,TENANT_NAME,ARBITRATION_SERVICE_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE tenant_name = 'sys';
   +-----------+-------------+----------------------------+
   | TENANT_ID | TENANT_NAME | ARBITRATION_SERVICE_STATUS |
   +-----------+-------------+----------------------------+
   |         1 | sys         | ENABLED                    |
   +-----------+-------------+----------------------------+
   1 row in set
   

2. Check whether the arbitration service is available for the tenant.

   If the arbitration service is enabled for a tenant, the arbitration service is available for the log streams created at the time of its enablement. However, for log streams created after it is enabled, the arbitration service might be unavailable, because creating a log stream is not a strict process and does not guarantee the creation of the arbitration service. In this case, you can run the following SQL statement to check whether the arbitration service is available for all log streams of the tenant.

   ##Query the list of log streams without an arbitration replica in the tenant##
   (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx and role = 'LEADER' and arbitration_member = "$arb_server_ip:$arb_server_port");
   

   Here is an example:

   Query the list of log streams without an arbitration replica in the sys tenant. If the query result set is an empty set, the arbitration service is available in the current tenant.

   obclient [oceanbase]> (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1 and role = 'LEADER' and arbitration_member = "10.10.10.3:2882");
   Empty set
   

Next steps

After the cluster is created, you can create user tenants as needed.

For more information about how to create a user tenant, see Create a tenant.

References

• Connect to an OceanBase tenant by using OBClient
• Overview of system configuration items
• ALTER USER
• ALTER TENANT
• Introduction to user tenants
• CREATE TENANT
• Enable the arbitration service
• Disable the arbitration service
• Modify the degradation timeout period
• Replace the arbitration service
• Remove the arbitration service