Deploy and use obconfigserver

You can choose to deploy obconfigserver separately or deploy it together with OceanBase Database and OceanBase Database Proxy (ODP). This topic describes how to use obd to deploy these items together and how to use obconfigserver.

About the components

• ODP

  ODP is a high-performance reverse proxy service designed for OceanBase Database. For more information, see ODP Documentation.

• obconfigserver

  obconfigserver allows you to register, store, and query the metadata of OceanBase Database. For more information, see ob-configserver.

Deploy obconfigserver

This section describes how to deploy obconfigserver together with OceanBase Database and ODP. After OceanBase Database is started, it is registered with obconfigserver.

Step 1: Modify the configuration file

If you want to deploy OceanBase Database, ODP, and obconfigserver together, you only need to add the ob-configserver component information to the configuration file of OceanBase Database and ODP and configure the ob-configserver dependency for OceanBase Database and ODP components.

Create a configuration file named config.yaml. The parameters for the components in the configuration file are described as follows:

1. User configuration

   ## Only need to configure when remote login is required
   user:
     username: admin
   #   password: Optional. The password.
     key_file: /home/admin/.ssh/id_rsa
   #   port: your ssh port, default 22
   #   timeout: ssh connection timeout (second), default 30
   

   where username specifies the username used to log on to the target server. Make sure that this user has the write privilege on home_path. Both password and key_file are used for user authentication, and generally only one of them needs to be specified.

   Notice

   After specifying the path of the key, if your key does not require a password, comment out or delete the password parameter to prevent it from being regarded as the password of the key for logon, resulting in verification failure.

2. OceanBase Database

   oceanbase-ce:
     depends:
       - ob-configserver
     servers:
       - name: server1
         # Do not use the hostname. Only IP addresses are supported.
         ip: 10.10.10.2
       - name: server2
         ip: 10.10.10.3
       - name: server3
         ip: 10.10.10.4
     global:
       # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
       home_path: /home/admin/observer
       # The directory for data storage. The default value is $home_path/store.
       # data_dir: /data
       # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
       # redo_dir: /redo
       # Set devname as the name of the network adapter whose IP address is the same as the one specified for the servers parameter.
       # If the IP address specified for the servers parameter is "127.0.0.1", set devname as "lo".
       # If the current IP address is 192.168.1.10, and the name of the network adapter with this IP address is "eth0", set devname as "eth0".
       devname: eth0
       cluster_id: 1
       # please set memory limit to a suitable value which is matching resource.
       memory_limit: 6G # The maximum running memory for an observer
       system_memory: 1G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
       datafile_size: 20G # Size of the data file.
       log_disk_size: 15G # The size of disk space used by the clog files.
       cpu_count: 16
       scenario: htap
       mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
       rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
       production_mode: false
       enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
       enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
       max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
       # Cluster name for OceanBase Database. The default value is obcluster. When you deploy OceanBase Database and obproxy, this value must be the same as the cluster_name for obproxy.
       appname: obcluster
       root_password: ****** # root user password
       proxyro_password: ****** # proxyro user password, consistent with obproxy's observer_sys_password, can be empty
     server1:
       zone: zone1
     server2:
       zone: zone2
     server3:
       zone: zone3
   

   For the description of the parameters in the configuration file, see Configuration file. You need to pay attention to the following parameters:

   • depends: You must configure the oceanbase-ce component to depend on the ob-configserver component. Otherwise, OceanBase Database cannot be registered with obconfigserver.

   • appname: Specifies the name of your cluster. If you do not specify this parameter, OceanBase Database cannot be registered with obconfigserver.

   • If you do not specify the scenario option in the configuration file when you deploy OceanBase Database V4.3.0 or later, obd provides interactive options for you to select a load type.

3. ODP

   obproxy-ce:
     # Set dependent components for the component.
     # When the associated configurations are not done, obd will automatically get these configurations from the dependent components.
     depends:
       - oceanbase-ce
       - ob-configserver
     servers:
       - 10.10.10.5
     global:
       listen_port: 2883 # External port. The default value is 2883.
       prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
       rpc_listen_port: 2885
       home_path: /home/admin/obproxy
       # The OceanBase root server list
       # format: ip:mysql_port;ip:mysql_port. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
       # rs_list: 10.10.10.1:2881;10.10.10.2:2881;10.10.10.3:2881
       enable_cluster_checkout: false
       # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
       # cluster_name: obcluster
       skip_proxy_sys_private_check: true
       enable_strict_kernel_release: false
       # obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
       # observer_sys_password: # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
   

   For the description of the parameters in the configuration file, see Configuration file. Do not configure the rs_list parameter. Otherwise, ODP cannot obtain the cluster information registered in obconfigserver.

4. obconfigserver

   ob-configserver:
     servers:
       - 10.10.10.1
     global:
       listen_port: 8080 # The port of ob-configserver web
       server_ip: 0.0.0.0 # Listen to the ob-configserver server IP. When you want to listen to the specified IP address,use it.
       home_path: /home/admin/ob-configserver  # The working directory for prometheus. ob-configserver is started under this directory. This is a required field.
       ## log config
       # log_level: info # Log printing level of ob-configserver. The default value is `info`
       # log_maxsize: 30 # The total size of manager ob-configserver.Log size is measured in Megabytes.The default value is 30
       # log_maxage:7 # The days of manager expired ob-configserver.Log retention days. The default value is 7
       # log_maxbackups: 10  #The number of manager expired ob-configserver.Log. The default value is 10
       # log_localtime: true #  Switch of ob-configserver.Log naming with localtime. The default value is true
       # log_compress: true # Compress ob-configserver.Log switch. The default value is true
   
       ## vip config, configserver will generate url with vip_address and port and return it to the client
       ## do not use some random value that can't be connected
       # vip_address: "10.10.10.1"
       # vip_port:8080
       ## storage config
       # storage:
         ## database type, support sqlite3 or mysql
         # sqlite3:
         # database_type: sqlite3
         # connection_url: "/home/admin/ob-configserver/.data.db?cache=shared&_fk=1"
   
         # mysql:
         # database_type: mysql
         # connection_url: "$user:$password@tcp($IP:$PORT)/$metadb?parseTime=true"
   

   The parameters are described as follows:

   • server_ip: the allowlist for accessing ob-configserver. The default value is 0.0.0.0, which indicates that all IP addresses on the server where ob-configserver is located can access it.

   • vip_address and vip_port: the configuration parameters for load balancing. If you are using load balancing, configure these parameters after setting it up.

   • database_type: the type of the database. Valid values are sqlite3 and mysql. The default value is sqlite3, but it is recommended to use mysql.

   • connection_url: the URL for connecting to the database. If the database type is sqlite3, the default value is $home_path/.data.db?cache=shared&_fk=1. If the database type is mysql, the connection_url can be either an OceanBase database or a MySQL database. The specified database user must have the privileges to perform Data Definition Language (DDL) and Data Manipulation Language (DML) operations on the corresponding database in the URL. Please note that when using an OceanBase database, do not specify the database name as oceanbase, as it will result in an error message indicating lack of privileges.

Step 2: Start obconfigserver

1. Deploy obconfigserver.

   [admin@obtest ~]$ obd cluster deploy test -c config.yaml
   

2. Start obconfigserver.

   [admin@obtest ~]$ obd cluster start test
   

   As shown in the returned results, you can see the access URL of obconfigserver.

   +-----------------------------------------------------------+
   |                      ob-configserver                      |
   +---------+------+---------------+----------+--------+------+
   | server  | port | vip_address   | vip_port | status | pid  |
   +---------+------+---------------+----------+--------+------+
   | 0.0.0.0 | 8080 | 10.10.10.1    | 8080     | active | 6270 |
   +---------+------+---------------+----------+--------+------+
   curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .
   Wait for observer init ok
   +-------------------------------------------------+
   |                     observer                    |
   +---------------+---------+------+-------+--------+
   | ip            | version | port | zone  | status |
   +---------------+---------+------+-------+--------+
   | 10.10.10.2    | 4.2.0.0 | 2881 | zone1 | ACTIVE |
   +---------------+---------+------+-------+--------+
   | 10.10.10.3    | 4.2.0.0 | 2881 | zone2 | ACTIVE |
   +---------------+---------+------+-------+--------+
   | 10.10.10.4    | 4.2.0.0 | 2881 | zone3 | ACTIVE |
   +---------------+---------+------+-------+--------+
   obclient -h10.10.10.2 -P2881 -uroot -p'********' -Doceanbase -A
   
   +-------------------------------------------------+
   |                     obproxy                     |
   +---------------+------+-----------------+--------+
   | ip            | port | prometheus_port | status |
   +---------------+------+-----------------+--------+
   | 10.10.10.5    | 2883 | 2884            | active |
   +---------------+------+-----------------+--------+
   obclient -h10.10.10.5 -P2883 -uroot -p'********' -Doceanbase -A
   

3. Copy the access information in the ob-configserver section.

   Note

   The jq command in the access information is used to format the JSON output returned by the shell command. You can run sudo yum install jq to install the jq command tool.

   {
     "Code": 200,
     "Message": "successful",
     "Success": true,
     "Data": {
       "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
       "ObProxyDatabaseInfo": {
         "DataBase": "***",
         "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
         "Password": "***",
         "User": "***"
       },
       "ObRootServiceInfoUrlList": [
         {
           "ObRegion": "obcluster",
           "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
         }
       ],
       "Version": "4660b4b1f237893ba1da50a302e4c3e6"
     },
     "Trace": "07a7cac129713d00",
     "Server": "10.10.10.1",
     "Cost": 0
   }
   

   In the returned results, ObRootServiceInfoUrlList contains the information of the OceanBase cluster (obcluster) deployed together with obconfigserver, indicating that the cluster has been successfully registered with obconfigserver. You can verify whether the obconfigserver can parse the information of the OceanBase cluster by making another request to ObRootServiceInfoUrl. Here is the command:

   curl -s 'http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster' |jq .
   

   As shown in the returned results, the IP addresses and ports of the OBServer nodes in the cluster are successfully parsed.

   {
     "Code": 200,
     "Message": "successful",
     "Success": true,
     "Data": {
       "ObClusterId": 1,
       "ObRegionId": 1,
       "ObCluster": "obcluster",
       "ObRegion": "obcluster",
       "ReadonlyRsList": [],
       "RsList": [
         {
           "address": "10.10.10.4:2882",
           "role": "LEADER",
           "sql_port": 2881
         },
         {
           "address": "10.10.10.3:2882",
           "role": "FOLLOWER",
           "sql_port": 2881
         },
         {
           "address": "10.10.10.2:2882",
           "role": "FOLLOWER",
           "sql_port": 2881
         }
       ],
       "Type": "PRIMARY",
       "timestamp": 1694084002271443
     },
     "Trace": "2b2ee036276b068e",
     "Server": "10.10.10.1",
     "Cost": 3
   }
   

Use obconfigserver

After deploying obconfigserver, you can register the OceanBase cluster with obconfigserver by modifying the configuration file of the OceanBase cluster or executing SQL statements. The ODP started with the specified obconfigserver address can access all the OceanBase clusters registered with obconfigserver.

This section describes how to register an OceanBase cluster with obconfigserver and how to start ODP by specifying the obconfigserver address.

Register an OceanBase cluster with obconfigserver

There are two methods to register an OceanBase cluster with obconfigserver.

Start ODP

You can use obd to deploy a new ODP and configure the startup option obproxy_config_server_url for it. A sample configuration file is as follows:

##Only need to configure when remote login is required
user:
  username: admin
#   password: Optional. The password.
  key_file: /home/admin/.ssh/id_rsa
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
obproxy-ce:
  servers:
    - 10.10.10.2
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/obproxy
    obproxy_config_server_url: http://10.10.10.1:8080/services?Action=GetObProxyConfig
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    # obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
    observer_sys_password: ******** # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends.
    obproxy_sys_password: ********

For more information about ODP parameters, see ODP parameters.

After ODP is started through the startup option obproxy_config_server_url, you can use ODP to connect to the clusters registered with obconfigserver. You can run the following command to view the information about the OceanBase clusters registered with obconfigserver:

curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .

As shown in the returned results, two clusters named obcluster and test1 are registered with obconfigserver and can be successfully connected through ODP.

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": {
    "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
    "ObProxyDatabaseInfo": {
      "DataBase": "***",
      "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
      "Password": "***",
      "User": "***"
    },
    "ObRootServiceInfoUrlList": [
      {
        "ObRegion": "obcluster",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
      }
      {
        "ObRegion": "test1",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=test1"
      }
    ],
    "Version": "4840ade8a158753aa5b9ea69ba014fc1"
  },
  "Trace": "d4b4b27fae24e7b4",
  "Server": "10.10.10.1",
  "Cost": 1
}

Delete the information about a registered cluster

You can run the following command to delete the information about a registered cluster from obconfigserver. The test1 cluster is used as an example here.

curl -X DELETE 'http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=test1&ObClusterId=1&version=2' |jq .

The returned results are as follows:

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": "success",
  "Trace": "caba1209b3a00c56",
  "Server": "10.10.10.1",
  "Cost": 4
}

View the information about the clusters registered with obconfigserver.

curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .

As shown in the returned results, the information about the test1 cluster no longer exists.

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": {
    "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
    "ObProxyDatabaseInfo": {
      "DataBase": "***",
      "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
      "Password": "***",
      "User": "***"
    },
    "ObRootServiceInfoUrlList": [
      {
        "ObRegion": "obcluster",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
      }
    ],
    "Version": "4840ade8a158753aa5b9ea69ba014fc1"
  },
  "Trace": "d4b4b27fae24e7b4",
  "Server": "10.10.10.1",
  "Cost": 1
}

FAQ

The error prepared statement not supported is returned when obconfigserver is started

Symptom

Assume that obconfigserver uses OceanBase Database as its MetaDB. The error prepared statement not supported is returned when obconfigserver is started. The log is as follows:

2022-11-16T09:52:14.47799+08:00 ERROR [12258,] caller=cmd/main.go:36:func1: start configserver failed: start config server: create configserver schema: sql/schema: reading schema information Error 1235: while parameter _ob_enable_prepared_statement is disabled, prepared statement not supported
[10.10.10.1:2882] [2022-11-16 09:52:14.459102] [YB42AC1EC731-0005EB5F1D83A0A7] fields: args:="[]"

Possible cause

When obconfigserver is started, the Prepared Statement (PS) feature is not enabled for OceanBase Database.

Solution

• Method 1: Add interpolateParams=true to the connection_url parameter for obconfigserver in the configuration file. A sample configuration is as follows:

  connection_url: "user:password@tcp(IP:PORT)/test?parseTime=true&interpolateParams=true"
  

• Method 2: Connect to the OceanBase cluster and enable the _ob_enable_prepared_statement parameter.

  • Check whether the _ob_enable_prepared_statement parameter is enabled.

    obclient> select name,value,svr_ip,svr_port from oceanbase.__all_virtual_sys_parameter_stat where name='_ob_enable_prepared_statement';
    +-------------------------------+-------+---------------+----------+
    | name                          | value | svr_ip        | svr_port |
    +-------------------------------+-------+---------------+----------+
    | _ob_enable_prepared_statement | False | 10.10.10.1    |     2882 |
    +-------------------------------+-------+---------------+----------+
    1 row in set
    

  • Enable the _ob_enable_prepared_statement parameter.

    obclient> alter system set _ob_enable_prepared_statement='True';
    

An error is returned when ODP is used to connect to a cluster registered with obconfigserver

Symptom

An error is returned when ODP is used to connect to an OceanBase cluster registered with obconfigserver. Here is an example:

$ obclient -h10.10.10.2 -P2883 -uroot@sys#cluster -p -Doceanbase -A
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 11

Possible cause

ODP cannot use the configured username and password to access the OceanBase cluster.

Solution

• Method 1: Run the obd cluster edit-config command to view the configuration file. Check whether the value of the proxyro_password parameter in the configuration file of the OceanBase cluster is the same as that of the observer_sys_password parameter in the configuration file of ODP. If not, set the two parameters to the same value and run the restart command according to the returned results. After the restart, you can use ODP to connect to the cluster.

• Method 2: Check whether the proxyro user exists in the OceanBase cluster.

  • Log on to the sys tenant of the OceanBase cluster as the root user.

    obclient -h10.10.10.2 -P2881 -uroot@sys -p -Doceanbase -A
    

  • Check whether the proxyro user exists in the cluster.

    select user,password from mysql.user;
    

  • Take the corresponding actions based on the returned result:

    • If the proxyro user does not exist in the cluster, run the following command to create the proxyro user and set the password to the same as the value of the observer_sys_password parameter in the configuration file of ODP:

      create user 'proxyro' identified by '*****';
      

    • If the proxyro user exists in the cluster, run the following command to change the password of the proxyro user to the same as the value of the observer_sys_password parameter in the configuration file of ODP:

      alter user 'proxyro' identified by '*****';