This topic describes how to deploy the binlog service for OceanBase Database Community Edition. For more information about how to deploy the binlog service for OceanBase Database Enterprise Edition, see the Product introduction section in Overview.
Prerequisites
Before you deploy the binlog service, make sure that:
A MySQL database or an OceanBase Database in MySQL mode is available as the MetaDB, and a database for storing the binlog cluster metadata has been created in the MetaDB.
OceanBase Database Proxy (ODP) is deployed in the OceanBase cluster where binlog files are to be generated, and both the OceanBase Database and ODP are compatible with the binlog service. For more information, see Release notes.
The
obconfig_urlparameter is specified for the OceanBase cluster where binlog files are to be generated. You can check the parameter value by logging in to the OceanBase cluster and executing theSHOW PARAMETERS LIKE 'obconfig_url';statement.If the OceanBase cluster is installed by using OceanBase Cloud Platform (OCP), OCP automatically configures the
obconfig_urlparameter.If the OceanBase cluster is installed by using other methods and the
obconfig_urlparameter is not configured, you need to manually configure it. For more information about how to install the config server and configure theobconfig_urlparameter, see OceanBase Deployer > User Guide > Deploy through CLI > Deploy and use obconfigserver.
Procedure
To perform the following operations, you must have the privileges of the ADMIN user.
Step 1: Install the program package
You can install the program package of the binlog service online or offline, depending on whether your server is connected to the Internet.
If your server can access the Internet, add the OceanBase Database software repository and directly run the yum command to install the program package of the binlog service to the directory /home/ds/oblogproxy.
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y obbinlog-ce
Note
The yum command automatically installs the latest version. You can declare a version number to install the specified version. If you use the yum install -y obbinlog-ce-4.0.1 command to install the binlog service, we recommend that you install the latest version.
If your server cannot access the Internet, download the binlog service program package for OceanBase Database Community Edition from OceanBase Download Center by searching for OceanBase Binlog Service, or from Alibaba Cloud YUM Repository. Then, upload the downloaded package to your server. For more information about the release notes of the program package, see Release notes. We recommend that you download the package of the latest version.
wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
After downloading the RPM package, install it to the directory /home/ds/oblogproxyby by running the following command in the directory where the installation package is located:
sudo yum install ./obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
Alternatively, install the package to the specified directory by running the following command:
sudo rpm -ivh --prefix=/usr/local/oblogproxy ./obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
Step 2: Configure and start the binlog server
OceanBase Database allows you to configure and start the binlog server through quick deployment or custom configurations as needed.
To simplify the binlog server configurations, a deployment script env/deploy.sh is provided in the installation directory to allow you to quickly start the binlog server with default configurations.
Step 1: Modify the env/deploy.conf.json file
Go to the working directory of the program package and populate the prepared MetaDB information in the env/deploy.conf.json file.
{
"host": "",
"port": 0,
"user": "",
"password": "",
"database": "",
"sys_user": "",
"sys_password": "",
"supervise_start": "",
"init_schema": "",
"node_ip": ""
}
The following table describes the parameters.
| Parameter | Description |
|---|---|
| host | The IP address or domain name of the MetaDB. |
| port | The SQL port of the MetaDB. |
| user | The username for accessing the MetaDB. |
| password | The password for accessing the MetaDB. |
| database | The name of the database in the MetaDB. Make sure that the specified database already exists in the MetaDB. If this parameter is not specified, the default value binlog_cluster is used. |
| sys_user | The username of the sys tenant of OceanBase Database where binlog files are to be generated.
NoticeThe username should not include the tenant name or cluster name, such as |
| sys_password | The password for the user specified by sys_user in the sys tenant. |
| supervise_start | Specifies whether to run the supervisord command on the current node to start the binlog service. If this parameter is not specified, the default value false is used.
NoticeIf the installation directory of the binlog service is customized, |
| init_schema | Specifies whether to initialize the MetaDB. If this parameter is not specified, the default value true is used. Set this parameter to true when you start the first node in the cluster, and false in other scenarios. |
| node_ip | The IP address of the current node. If this parameter is not specified, the return value obtained after running the hostname -i command is used.
NoteIf the server has multiple network interface cards (NICs), the IP address returned after running the |
Note
sys_user and sys_password in the env/deploy.conf.json file are optional parameters used to authenticate the database where binlog files are to be generated. If the two parameters are not configured in the `env/deploy.conf.json` file, you must specify them in the command for creating a binlog task later.
Step 2: Start the binlog server
Run the following command in the binlog installation directory to start the binlog server:
sudo sh env/deploy.sh -m deploy -f env/deploy.conf.json
The following operations are performed by running the deploy.sh script:
Install dependencies such as
jq,libstdc++.so.6,mysql, anddiff.Read attributes from the
env/deploy.conf.jsonfile. The default value is used for parameters that are not specified.Create metadata tables if
init_schemais set totrue.Write metadata connection configurations to the
conf/conf.jsonfile and setbinlog_modetotrue.Check whether the value of the
supervise_startparameter istrueorfalse.If
supervise_startis set totrue, call theenv/install.shscript, which installs supervisord and starts the program with it.If
supervise_startis set tofalse, the processing logic is as follows:If both the
sys_userandsys_passwordparameters are specified, thedeploy.shscript calls theconfig_sysfunction in therun.shscript to replace theob_sys_usernameandob_sys_passwordparameters in theconf/conf.jsonfile, and then runs therun.sh startcommand to start the program.If either the
sys_userorsys_passwordparameter is empty, thedeploy.shscript directly runs therun.sh startcommand to start the program.
Once the program is started, you can use the MySQL client or OceanBase Client (OBClient) to connect to the binlog server and run related commands.
mysql -h127.0.0.1 -P2983
Step 1: Install runtime dependencies
Configure the GNU Compiler Collection (GCC) environment.
cd ${deploy_path} sudo cp ./deps/lib/libstdc++.so.6.0.28 /usr/lib64/ cd /usr/lib64 sudo ln -sf libstdc++.so.6.0.28 libstdc++.so.6Install other dependency tools.
sudo yum install -y jq mysql diffutils
Step 2: Configure the conf.json file
Modify the conf/conf.json file. For more information about the parameters in the file, see Parameters. The following parameters must be configured based on the actual situation:
- binlog_mode: specifies whether to start the binlog service. You must set this parameter to
true. - database_ip: the IP address of the MetaDB.
- database_port: the port number of the MetaDB.
- database_name: the name of the database in the MetaDB.
- user: the username for accessing the MetaDB.
- password: the password for accessing the MetaDB.
- node_ip: the IP address of the current node.
To set a default account and password for creating binlog tasks later, run the following command to write the account and password of the sys tenant to the configuration file:
sudo ./run.sh config_sys <sys_user> <sys_password>
Notice
You must use sys_user and sys_password to specify the username and password for the `sys` tenant of OceanBase Database under which binlog files will be generated. The username should not include the tenant or cluster name, such as root.
Step 3: Initialize the MetaDB
Before starting the first binlog server, create required tables in the MetaDB. After connecting to the corresponding database in the MetaDB, run the conf/schema.sql file to initialize the MetaDB. The sample command is as follows:
SOURCE conf/schema.sql
For more information about metadata tables, see Parameters.
Step 4: Start the binlog server
Run the following command to start the binlog server:
sudo ./run.sh start
Once the program is started, you can use the MySQL client or OBClient to connect to the binlog server and run related commands.
mysql -h127.0.0.1 -P2983
(Optional) Step 3: Add a binlog server to an existing binlog cluster
The binlog service treats all binlog servers that use the same database within the same MetaDB as part of the same binlog cluster. Therefore, to deploy multiple binlog servers within a single binlog service, you only need to repeat the steps in [Step 2: Configure and start the binlog server] on different physical servers by using the same MetaDB configurations.
Notice
You need to initialize the metadata tables only when you start the first binlog server node.
After the binlog server node is deployed, you can obtain the information about the current binlog server node from the metadata table nodes. For more information, see the MetaDB configurations section in Parameters.
Step 4: Create a binlog task
Confirm the relevant information about the OceanBase Database tenant for which you want to create a binlog task. After connecting to the tenant, execute the following SQL statements to obtain the information.
Obtain the name of the cluster.
SHOW PARAMETERS LIKE 'cluster';The cluster name of the connected binlog service of OceanBase Database is displayed as
demoin thevaluecolumn.+-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault | +-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone1 | observer | 10.10.10.1 | 2882 | cluster | NULL | demo | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | obcluster | 0 | +-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+Obtain the name of the tenant.
SHOW TENANT;The tenant name of the connected binlog service of OceanBase Database is displayed as
obmysql.+---------------------+ | Current_tenant_name | +---------------------+ | obmysql | +---------------------+Obtain the Config URL.
SHOW PARAMETERS LIKE 'obconfig_url';The Config URL of the connected binlog service of OceanBase Database is displayed as the value in the
valuecolumn.+-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault | +-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone1 | observer | 10.10.10.1 | 2882 | obconfig_url | NULL | http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=demo | URL for OBConfig service | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | 0 | +-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+
Once the preceding information is confirmed, connect to the binlog server and execute the CREATE BINLOG statement to create a binlog task for the OceanBase Database tenant. For more information about the CREATE BINLOG statement, see the Create a binlog task section in Task management.
The example runs the following command to create a binlog task for the obmysql tenant in the demo cluster. The CLUSTER URL parameter specifies the Config URL, and the value 2 of the REPLICATE NUM parameter indicates that two binlog instances will be created for this tenant.
CREATE BINLOG FOR TENANT `demo`.`obmysql`
TO USER `root` PASSWORD `********`
WITH CLUSTER URL `http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=demo`
, REPLICATE NUM 2;
Notice
USERandPASSWORDin the command specify the username and password of thesystenant, rather than theobmysqltenant for which the binlog task is to be created. If the username and password of the `sys` tenant are configured when you start the binlog server, removeTO USER `root` PASSWORD `********`in the command.- At present, you cannot create binlog tasks for the `sys` tenant.
Once the binlog task is created, execute the SHOW BINLOG INSTANCES; statement to obtain the status of all binlog instances. The output is as follows:
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
| name | ob_cluster | ob_tenant | ip | port | zone | region | group | running | state | obcdc_running | obcdc_state | service_mode | convert_running | convert_delay | convert_rps | convert_eps | convert_iops | dumpers | version | odp_addr |
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
| bl09vjd9tl | demo | obmysql | 10.10.10.1 | 8100 | | | | Yes | Running | Yes | Running | enabled | Yes | 676 | 1 | 0 | 77 | 0 | 4.0.1-92748c6c59534426d187977b13b61b7cdf4fa200 | NULL |
| wi7emnxpat | demo | obmysql | 10.10.10.1 | 8101 | | | | Yes | Running | Yes | Running | enabled | Yes | 570 | 1 | 0 | 77 | 0 | 4.0.1-92748c6c59534426d187977b13b61b7cdf4fa200 | NULL |
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
The columns running, obcdc_running, and convert_running indicate the status of the main process of the binlog instance, the obcdc process, and the binlog converter (BC) process, respectively. If all three parameters are set to Yes, the binlog instance runs properly. If any of these three parameters is set to No, the corresponding processes run abnormally. To troubleshoot the process that encounters an error, see the Troubleshoot logs section in FAQ and perform troubleshooting in the working directory of the binlog instance.
For more information about the binlog statements, see Overview.
Step 5: Configure ODP
Notice
If you need to use multiple ODP, please perform the following operations on each ODP.
Log in to the
systenant of OceanBase Database as therootuser by using ODP, or connect to ODP as theroot@proxysysuser.Configure the IP address and port number of the binlog server for the
binlog_service_ipparameter in theip:portformat. The default port number is2983. You can configure multiple IP addresses forbinlog_service_ipin ODP V4.3.1 and later. Separate them with semicolons (;).ALTER proxyconfig SET binlog_service_ip='10.10.10.1:2983';Enable the binlog service.
Note
The
enable_binlog_serviceparameter has been removed in ODP Community Edition V4.2.3 and later. Therefore, you can ignore the error indicating that this parameter does not exist.ALTER proxyconfig SET enable_binlog_service='True';(Optional) To ensure that the execution result of the
SHOW CREATE TABLEstatement is fully compatible with MySQL syntax, excluding the extended syntax of OceanBase Database, execute the following statement:ALTER proxyconfig SET init_sql='set _show_ddl_in_compat_mode = 1;';Execute the
SHOW PROXYCONFIG LIKE 'config_name'statement to verify the parameters, such asbinlog_service_ip.SHOW PROXYCONFIG LIKE 'binlog_service_ip';The output is as follows:
+-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+ | name | value | info | need_reboot | visible_level | range | config_level | +-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+ | binlog_service_ip | 10.10.10.1:2983 | binlog service ip/hostname list, format ip1:sql_port1;hostname2:sql_port2, separate with ';'. prefer to use the forward | false | SYS | | LEVEL_VIP | +-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+
Step 6: Verify the result
Use ODP to connect to the tenant where the binlog task is generated and check the task status by executing statements such as SHOW MASTER STATUS;. For more information about the binlog statements, see Compatibility.
Here is an example:
Use ODP to connect to the tenant where the binlog task is generated.
obclient -h10.10.10.1 -P2883 -uroot@obmysql -p -Doceanbase -ANotice
The binlog statements in this section can be executed only when you use ODP to connect to the tenant.
Query the binlog file.
SHOW MASTER STATUS;The output is as follows:
+------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+-------------------+ | mysql-bin.000001 | 155 | | | | +------------------+----------+--------------+------------------+-------------------+Query all binlog events.
SHOW BINLOG EVENTS;The output is as follows:
+------------------+------+----------------+-----------+-------------+-----------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+------+----------------+-----------+-------------+-----------------------------------+ | mysql-bin.000001 | 4 | Format_desc | 96754989 | 124 | Server ver: 5.7.38, Binlog ver: 4 | | mysql-bin.000001 | 124 | Previous_gtids | 96754989 | 155 | | +------------------+------+----------------+-----------+-------------+-----------------------------------+Perform a data change to verify whether the binlog service runs properly.
CREATE database test1;Query the binlog file again.
SHOW MASTER STATUS;The following new binlog file is generated by the binlog service:
+------------------+----------+--------------+------------------+----------------------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+----------------------------------------+ | mysql-bin.000001 | 297 | | | 0f5addad-9be6-11ef-9c9d-00163e0808cc:1 | +------------------+----------+--------------+------------------+----------------------------------------+Query all binlog events.
SHOW BINLOG EVENTS;The following new binlog events are generated, indicating that the binlog service runs properly:
+------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+ | mysql-bin.000001 | 4 | Format_desc | 96754989 | 124 | Server ver: 5.7.38, Binlog ver: 4 | | mysql-bin.000001 | 124 | Previous_gtids | 96754989 | 155 | | | mysql-bin.000001 | 155 | Gtid | 96754989 | 220 | SET @@SESSION.GTID_NEXT= '0f5addad-9be6-11ef-9c9d-00163e0808cc:1' | | mysql-bin.000001 | 220 | Query | 96754989 | 297 | use obmysql;CREATE database test1 | +------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+
Related operations
Release a binlog task
To stop all binlog instances for a tenant and delete the generated binlog files, perform the following steps to release the binlog task:
Log in to the binlog server.
mysql -h127.0.0.1 -P2983In this example, the IP address is
127.0.0.1and the port number is2983. You must change them based on the actual situation.Release the binlog task.
The binlog task for the
obmysqltenant in thedemocluster is released in the following example.DROP BINLOG FOR TENANT demo.obmysql;Query the status of all binlog instances
SHOW BINLOG INSTANCES;If the values
demoandobmysqlare not displayed in theob_clusterandob_tenantcolumns respectively, the binlog task for the corresponding tenant is released.
Stop the binlog server
You can stop the binlog server by executing the following statement in the binlog installation directory. The default installation directory is /home/ds/oblogproxy.
[admin@test001 oblogproxy]$./run.sh stop
Note
The binlog instances that are not stopped continue to run even after the binlog server is stopped.