Deploy an OceanBase cluster with two replicas and arbitration service using the CLI

Starting from version V4.1, OceanBase Database supports the arbitration service (AS), which addresses the issue of increased round-trip time (RT) when a fault occurs in one of the two IDCs in a geo-redundant setup, reduces cross-region bandwidth usage, and keeps the costs of the third IDC low.

This topic will guide you through the deployment of an OceanBase cluster with two replicas and an arbitration service using the CLI.

Prerequisites

Before installing OceanBase Database, make sure that the following conditions are met:

• The OBServer node has been configured. For more information, see Server configuration, (Optional) Configure a clock source, and Initialize the OBServer node using oat-cli.

• You have obtained the RPM package of OceanBase Database. For more information, see Prepare installation packages.

Procedure

Step 1: Deploy an OceanBase cluster

1. Install the RPM packages on the two OBServer servers.

   1. Install the OceanBase Database RPM package.

      Navigate to the directory where the OceanBase Database RPM package is stored:

      [root@xxx /]# cd $rpm_dir
      

      Install the OceanBase Database RPM package:

      [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
      

      Here, $rpm_dir indicates the directory where the RPM package is stored, and $rpm_name indicates the name of the RPM package.

      Note

      After the installation, OceanBase Database will be installed in the /home/admin/oceanbase directory by default.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
          1:oceanbase-4.2.0.0-100000052023073################################# [100%]
      

   2. (Optional) Install OceanBase Client.

      OceanBase Client (OBClient) is a dedicated CLI tool for OceanBase Database. With OBClient, you can connect to MySQL and Oracle tenants in OceanBase Database. If you only need to connect to a MySQL tenant in OceanBase Database, you can also use the mysql client to connect to OceanBase Database.

      Notice

      Due to the dependency of obclient V2.2.1 and earlier on libobclient, you must install libobclient before installing obclient. Contact the technical support team to obtain the RPM packages of OBClient and libobclient.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh obclient-2.2.1-20221122151945.el7.alios7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
         1:obclient-2.2.1-20221122151945.el7################################# [100%]
      

      Check whether OBClient has been installed:

      [root@xxx /home/admin/rpm]# which obclient
      /usr/bin/obclient
      

2. Configure the observer process directory.

   1. (Optional) Clear the directory.

      When you deploy OceanBase Database for the first time on a new server, you do not need to clear the directory.

      If any of the following conditions are met, you can choose to clear the old OceanBase directory:

      • You want to clear the directory before you upgrade OceanBase Database.
      • The installation environment is chaotic or the installation process generates files that will affect the next installation.

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /data/log1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/obdemo
      [root@xxx /home/admin]# rm -rf /data/log1/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

   2. Initialize the OceanBase directory.

      Generally, it is recommended to install OceanBase Database data directory on an independent disk and then use a soft link to connect the data directory to the home directory of OceanBase Database.

      Note

      Starting from OceanBase Database V4.3.0, you can independently install the slog from the data (data) disk. In other words, the slog file does not need to be on the same disk as the data file. You can configure the disks to share an SSD for the slog and clog files. For more information about the installation directories of OceanBase Database, see OBServer node installation directory structure.

      Switch to the admin user:

      [root@xxx /home/admin]# su - admin
      

      Use the admin user to run the following commands to create the required directories:

      mkdir -p /data/1/$cluster_name/{etc3,sstable,slog}
      

      mkdir -p /data/log1/$cluster_name/{clog,etc2}
      

      mkdir -p /home/admin/oceanbase/store/$cluster_name
      

      Use the admin user to run the following commands to create a soft link:

      for t in {etc3,sstable,slog};do ln -s /data/1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      for t in {clog,etc2};do ln -s /data/log1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# su - admin
      -bash-4.2$ mkdir -p /data/1/obdemo/{etc3,sstable,slog}
      -bash-4.2$ mkdir -p /data/log1/obdemo/{clog,etc2}
      -bash-4.2$ mkdir -p /home/admin/oceanbase/store/obdemo
      -bash-4.2$ for t in {etc3,sstable,slog};do ln -s /data/1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      -bash-4.2$ for t in {clog,etc2};do ln -s /data/log1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      

      Note

      The obdemo directory is created based on the cluster name and can be customized. It is used by the process when it is started.

      Check the result:

      -bash-4.2$ cd /home/admin/oceanbase
      -bash-4.2$ tree store/
      store/
      `-- obdemo
         |-- clog -> /data/log1/obdemo/clog
         |-- etc2 -> /data/log1/obdemo/etc2
         |-- etc3 -> /data/1/obdemo/etc3
         |-- slog -> /data/1/obdemo/slog
         `-- sstable -> /data/1/obdemo/sstable
      
      6 directories, 0 files
      

3. Initialize the OceanBase cluster.

   Note

   • The example IP addresses are desensitized. This is not a requirement. When you deploy OceanBase Database, you should fill in the IP addresses of your servers.
   • OceanBase Database can start the observer process on a server with a specified IPv4 or IPv6 address. The example below shows how to start the observer process on an IPv4 address. For information about how to start the observer process on an IPv6 address, see Deploy an OceanBase cluster using the CLI.

   1. Start the observer process.

      Start the observer process on the admin user on each node.

      Notice

      The start parameters of the observer process in one replica are different from those in the other replica. When you start the observer process, you only need to specify the two (or more) servers where the Root Service is located, without specifying all servers when you create a cluster. After the cluster is created, you can add new servers.

      cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer {-I $ip | -i $devname} -P $rpc_port -p $sql_port -z $zone_name -d /home/admin/oceanbase/store/$cluster_name -r '$ip:2882:2881' -c $cluster_id -n $cluster_name -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/$cluster_name/etc3;/data/log1/$cluster_name/etc2"
      

      The parameters are described as follows:

      Parameter	Description
      -I | -i	-I: the IP address of the server where you want to start the observer process. In a multi-server deployment, 127.0.0.1 cannot be used as the target IP address. We recommend that you use the specified IP address (-I 10.10.10.1) to start the server. -i: the name of the network card. You can view the available network card names by using the ifconfig command. Note You can also specify both the IP address and the network card name (-I 10.10.10.1 -i eth0) to start the server, but we recommend that you do not do this.
      -p	the port number of the service. Generally, set this parameter to 2881.
      -P	the port number of RPC. Generally, set this parameter to 2882.
      -n	the cluster name. Custom cluster names must not be repeated.
      -z	the zone to which the observer process belongs.
      -d	the directory where the cluster is located. This parameter specifies the directory that is created when you initialize the cluster. Except for the cluster name $cluster_name, other parameters must not be changed.
      -c	the cluster ID. This parameter specifies a numeric value that must be unique among clusters.
      -l	the log level.
      -r	the Root Service list. The format of each Root Service is $ip:2882:2881, and the Root Services are separated by semicolons.
      -o	the list of cluster startup parameters (configurations). This parameter is optional. You can specify multiple configurations, and separate the values of the configurations with commas. Choose the startup parameters and specify appropriate values based on your needs to optimize the cluster performance and resource utilization. Some commonly used startup configurations are described as follows: cpu_count: the total number of CPU cores. system_memory: the memory size reserved for Tenant ID 500, namely, the memory reserved for OceanBase Database. If the server has limited memory, you can set this parameter to a smaller value. Note that during performance tests, the memory might be insufficient. memory_limit: the total size of available memory. datafile_size: the size of disk space available for data files to be initialized. The data file sstable is specified to have this size (initialized) in the /data/1/ directory. We recommend that you set the value of this parameter based on the available space in the /data/1/ directory. We recommend that you set the value to at least 100G. datafile_disk_percentage: the percentage of total disk space occupied by data files. datafile_next: the automatic expansion step length of data files. datafile_maxsize: the maximum size of data files after automatic expansion. config_additional_dir: the additional local storage directory for storing configuration files. This parameter is used to store multiple copies of configuration files for redundancy. log_disk_size: the size of the redo log disk. log_disk_percentage: the percentage of total disk space occupied by the redo log. syslog_level: the system log level. syslog_io_bandwidth_limit: the upper limit on disk I/O bandwidth available for system logs. System logs exceeding the bandwidth upper limit will be discarded. max_syslog_file_count: the maximum number of system log files that can be stored before the log file recycling mechanism is triggered. enable_syslog_recycle: the switch to enable log file recycling. It is used in conjunction with the max_syslog_file_count parameter to determine whether to recycle old log files. The datafile_size, datafile_disk_percentage, datafile_next, and datafile_maxsize parameters are used together to enable automatic expansion of data files. For more information, see Configure automatic expansion of data files. For more information about cluster configurations, see Overview of cluster-level configurations.

      Here is an example:

      Start the observer process on the OBServer servers. Take 10.10.10.1 and 10.10.10.2 as an example.

      zone1:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.1:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      zone2:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.2 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.2:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      Run the following command to check whether the observer process has been started:

      • Use the netstat -ntlp command. If the 2881 and 2882 ports are being listened to, the observer process has been started.
      • Use the ps -ef|grep observer command to return the observer process information.

      Here is an example:

      -bash-4.2$ netstat -ntlp
      (Not all processes could be identified, non-owned process info
      will not be shown, you would have to be root to see it all.)
      Active Internet connections (only servers)
      Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
      tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      11112/observer
      tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      11112/observer
      ...        ...    ...                       ...                     ...         ...
      
      -bash-4.2$ ps -ef|grep observer
      admin     11112      0 40 16:18 ?        00:00:17 /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r 10.10.10.1:2882:2881;10.10.10.2:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
      

   2. Perform the bootstrap operation on the cluster.

      Use OBClient to connect to any of the nodes. The password is empty.

      [root@xxx admin]# obclient -h127.0.0.1 -uroot -P2881 -p******
      
      obclient [(none)]> SET SESSION ob_query_timeout=1000000000;
      Query OK, 0 rows affected
      
      obclient [(none)]> ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '10.10.10.1:2882',ZONE 'zone2' SERVER '10.10.10.2:2882';
      Query OK, 0 rows affected
      

      Notice

      If an error occurs in this step, the possible reasons are incorrect start parameters of the observer process, incorrect permissions of the observer-related directories, insufficient space in the log directory (if it is shared with the data directory, space in the data directory is also insufficient), out-of-synchronization of the node clock, and insufficient node memory resources. Check these points and then clear the OceanBase directory and deploy OceanBase Database from the very beginning.

   3. Verify whether the cluster has been initialized.

      After the cluster is initialized, execute the SHOW DATABASES; command to verify it. If the query result shows a list of databases that includes the oceanbase database, the cluster has been initialized successfully.

   4. Change the password.

      The password of the root user in the sys tenant is empty by default. After the initialization, you need to change the password.

      ALTER USER root IDENTIFIED BY '******';
      

Step 2: Deploy the arbitration service

1. Install the OceanBase RPM package on the arbitration server.

   Switch to the directory where the OceanBase RPM package is stored:

   [root@xxx /]# cd $rpm_dir
   

   Install the OceanBase RPM package:

   [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
   

   Here, $rpm_dir represents the directory where the RPM package is stored, and $rpm_name represents the name of the RPM package.

   Note

   OceanBase Database will be installed by default in the /home/admin/oceanbase directory.

   Here is an example:

   [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
   Preparing...                          ################################# [100%]
   Updating / installing...
      1:oceanbase-4.2.0.0-100000052023073################################# [100%]
   

2. Configure the arbitration server process directory.

   Create a clog directory for the arbitration server process on the arbitration server.

   Note

   • When you start the arbitration server process, make sure that the clog directory has been created in the store directory specified with the -d option.
   • Do not use up all the disk space of the disk where the store directory is located. Otherwise, the arbitration server may experience abnormal performance.

   Use the following command to switch to the admin user:

   [root@xxx /home/admin]# su - admin
   

   Use the admin user to execute the following command to create the required directories:

   mkdir -p /home/admin/oceanbase/store/clog
   

3. Start the arbitration server process.

   Log in to the arbitration server as the admin user and start the arbitration server process in arbitration mode, making sure that the necessary parameters are specified in the startup command. The sample startup command is as follows:

   cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P $rpc_port -d $obdir/store [{-I $ip | -i $devname}] [-l $log_level]  [-o "parameters_list"]
   

   The parameters are described as follows:

   Parameter	Description
   -m	Specifies the arbitration mode. The value is arbitration.
   -P	Specifies the RPC port number of the arbitration server process. The default value is 2882.
   -d	Specifies the storage directory path. The default value is /home/admin/oceanbase/store. You must create the clog subdirectory in this directory before you start the process.
   -I | -i	Optional. -I: the IP address of the node to be started. -i: the name of the network card. You can use the ifconfig command to view the network card names. Note You can start a node by specifying both the IP address and the network card name (for example: -I 10.10.10.1 -i eth0). However, we recommend that you do not specify both parameters.
   -l	Specifies the log level. The default value is WDIAG. The value range is: {DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR}. For more information, see Log levels.
   -o	Specifies the configuration parameter list. The value is a comma-separated list of configuration parameters. You can specify values for the parameters. The following parameters are commonly used when you start the arbitration server process: cpu_count system_memory memory_limit __easy_memory_limit log_disk_size log_disk_percentage syslog_level syslog_io_bandwidth_limit max_syslog_file_count enable_syslog_recycle Notice To ensure automatic cleanup of the log directory of an arbitration node and prevent the disk from being filled up, set enable_syslog_recycle to True and specify an effective value for max_syslog_file_count. Otherwise, log files will grow indefinitely, which may lead to a disk full.

   Here is an example:

   Start the arbitration server process on the arbitration server. For example, start the arbitration server process with 10.10.10.3 as the IP address.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3 -o "enable_syslog_recycle=True,max_syslog_file_count=100"
   

   You can use the following command to check whether the observer process has started successfully:

   • Run the netstat -ntlp command. If the 2882 port is listened to, the process has started successfully.
   • Run the ps -ef|grep observer command to return the observer process information.

   Here is an example:

   -bash-4.2$ netstat -ntlp
   (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
   tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      18231/observer
   ...
   
   -bash-4.2$ ps -ef|grep observer
   admin     18231      0  3 10:33 ?        00:00:00 /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3
   ...
   

Step 3: Add an arbitration service to the cluster

Run the following command to add an arbitration service to the cluster:

ALTER SYSTEM ADD ARBITRATION SERVICE "$arb_server_ip:$arb_server_port";

Here is an example:

obclient [(none)]> ALTER SYSTEM ADD ARBITRATION SERVICE "10.10.10.3:2882";
Query OK, 0 rows affected

obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| 2023-03-06 17:29:36 | 2023-03-06 17:29:36 | default                 | 10.10.10.3:2882     | NULL                         | ADDR |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
1 row in set

Step 4: Enable the arbitration service for the sys tenant

You can enable the arbitration service for a tenant when you create the tenant. For more information, see CREATE TENANT.

If you do not enable the arbitration service when you create a tenant, you can enable it for the tenant after the tenant is created.

Execute the following statement to enable the arbitration service for an existing tenant.

ALTER TENANT $tenant_name [SET] enable_arbitration_service = true;

Here is an example:

Enable the arbitration service for the sys tenant of a cluster.

obclient [(none)]> ALTER TENANT sys SET enable_arbitration_service = true;
Query OK, 0 rows affected

Step 5: Query the status of the arbitration service for a tenant

1. Check whether the arbitration service is enabled for the tenant.

   You can query the DBA_OB_TENANTS view for tenant information and check the value of the arbitration_service_status column for the status of the arbitration service.

   The possible values of arbitration_service_status are as follows:

   • ENABLED: the arbitration service is enabled for the tenant.
   • ENABLING: the arbitration service is being enabled for the tenant.
   • DISABLED: the arbitration service is disabled for the tenant.
   • DISABLING: the arbitration service is being disabled for the tenant.

   Here is an example:

   obclient [(none)]> SELECT TENANT_ID,TENANT_NAME,ARBITRATION_SERVICE_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE tenant_name = 'sys';
   +-----------+-------------+----------------------------+
   | TENANT_ID | TENANT_NAME | ARBITRATION_SERVICE_STATUS |
   +-----------+-------------+----------------------------+
   |         1 | sys         | ENABLED                    |
   +-----------+-------------+----------------------------+
   1 row in set
   

2. Check whether the arbitration service is available for the tenant.

   If the arbitration service is enabled for a tenant, the arbitration service is available for all log streams created by the tenant at the time when the service is enabled. However, for log streams created after the service is enabled, there is no guarantee that the arbitration service is available, because creating log streams is not a strict process and the arbitration service may not be created successfully. In this case, you can execute the following SQL statement to check whether the arbitration service is available for all log streams of the tenant.

   ##Query the list of log streams without an arbitration replica in the tenant##
   (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx and role = 'LEADER' and arbitration_member = "$arb_server_ip:$arb_server_port");
   

   Here is an example:

   Query the list of log streams without an arbitration replica in the sys tenant. If the query result set is an empty set, the arbitration service is available in the current tenant.

   obclient [oceanbase]> (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1 and role = 'LEADER' and arbitration_member = "10.10.10.3:2882");
   Empty set
   

Next steps

After the cluster is created, you can create user tenants as needed.

For more information about how to create a user tenant, see Create a tenant.

References

• Connect to an OceanBase tenant by using OBClient
• Overview of system configuration items
• ALTER USER
• ALTER TENANT
• Introduction to user tenants
• CREATE TENANT
• Enable the arbitration service
• Disable the arbitration service
• Modify the degradation timeout period
• Replace the arbitration service
• Remove the arbitration service