This topic describes how to deploy binlog service Community Edition. For information about the differences between the Enterprise Edition and Community Edition of the binlog service, see the Product introduction section in Overview.
Prerequisites
Before you deploy the binlog service, make sure that:
A MySQL database or an OceanBase Database in MySQL mode is available as the MetaDB, and a database for storing the binlog cluster metadata has been created in the MetaDB.
OceanBase Database Proxy (ODP) is deployed in the OceanBase cluster where binlog files are to be generated, and both OceanBase Database and ODP are compatible with the binlog service. For more information, see Release notes.
The
obconfig_urlparameter is specified for the OceanBase cluster where binlog files are to be generated. You can check the parameter value by logging in to the OceanBase cluster and executing theSHOW PARAMETERS LIKE 'obconfig_url';statement.If the OceanBase cluster is installed by using OceanBase Cloud Platform (OCP), OCP automatically configures the
obconfig_urlparameter.If the OceanBase cluster is installed by using other methods and the
obconfig_urlparameter is not configured, you need to manually configure it. For more information about how to install obconfigserver and configure theobconfig_urlparameter, see Deploy and use obconfigserver.
The network connections between the Binlog server and the MetaDB, ODP, and OceanBase Database are normal.
ODP uses
binlog_service_ipas the endpoint of the binlog server. Make sure that the server where ODP resides can access the endpoint.ODP needs to access the binlog instances of business tenants to obtain binlog information. Make sure that the server where ODP resides can access the binlog instances. By default, the binlog server allocates 256 ports starting from port 8100 to binlog instances. Make sure that the server where ODP resides can access ports 8100 to 8355 of the servers where the binlog instances reside. You can modify the
start_tcp_portandreserved_ports_numparameters in the local configuration file to change the port range.The Binlog Server can access the IP address, SQL port, and remote procedure call (RPC) port of the OceanBase Database instance, as well as the IP address and port of the MetaDB.
Procedure
To perform the following operations, you must have the privileges of the ADMIN user.
Note
This section describes how to manually deploy binlog service Community Edition. You can use OceanBase Deployer (obd) V3.1.0 or later to deploy and manage binlog service Community Edition. For more information, see Deploy the binlog service.
Step 1: Install the program package
You can install the program package online or offline, depending on whether your server is connected to the Internet.
Notice
This step describes how to install obbinlog Community Edition. To obtain the installation package of obbinlog Enterprise Edition, contact OceanBase Technical Support.
If your server can access the Internet, add the OceanBase Database software repository and directly run the yum command to install obbinlog to the /home/ds/oblogproxy directory.
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y obbinlog-ce
Note
The yum command automatically installs the latest version. You can declare a version number to install the specified version. For example, you can use the yum install -y obbinlog-ce-4.0.1 command to install obbinlog V4.0.1. However, we recommend that you install the latest version.
If your server cannot access the Internet, download the required obbinlog-ce package from OceanBase Download Center by searching for OceanBase Binlog Service, or from Alibaba Cloud YUM Repository. Then, upload the downloaded package to your server. For more information about the release notes of the program package, see Release notes. We recommend that you download the package of the latest version.
wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
After downloading the RPM package, install it to the directory /home/ds/oblogproxy by running the following command in the directory where the installation package is located:
sudo yum install ./obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
Alternatively, install the package to the specified directory by running the following command:
sudo rpm -ivh --prefix=/usr/local/oblogproxy ./obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm
Step 2: Configure and start a binlog server
OceanBase Database allows you to configure and start a binlog server through quick deployment or custom configurations as needed.
To simplify the binlog server configurations, a deployment script env/deploy.sh is provided in the installation directory to allow you to quickly start the binlog server with default configurations.
Step 1: Modify the env/deploy.conf.json file
Go to the working directory of the program package and populate the prepared MetaDB information in the env/deploy.conf.json file.
{
"host": "",
"port": 0,
"user": "",
"password": "",
"database": "",
"sys_user": "",
"sys_password": "",
"supervise_start": "",
"init_schema": "",
"node_ip": ""
}
The following table describes the parameters.
| Parameter | Description |
|---|---|
| host | The IP address or domain name of the MetaDB. |
| port | The SQL port of the MetaDB. |
| user | The username for accessing the MetaDB. |
| password | The password for accessing the MetaDB. |
| database | The name of the database in the MetaDB. Make sure that the specified database already exists in the MetaDB. If this parameter is not specified, the default value binlog_cluster is used. |
| sys_user | The username of a user in the sys tenant of the OceanBase cluster where binlog files are to be generated.
NoticeThe username cannot include the tenant name or cluster name. For example, the username is |
| sys_password | The password for the user specified by sys_user in the sys tenant. |
| supervise_start | Specifies whether to run the supervisord command on the current node to start the binlog service. If this parameter is not specified, the default value false is used.
NoticeIf the installation directory of the binlog service is customized, |
| init_schema | Specifies whether to initialize the MetaDB. If this parameter is not specified, the default value true is used. Set this parameter to true when you start the first node in the cluster, and false in other scenarios. |
| node_ip | The IP address of the current node. You can specify only one IPv4 address. If this parameter is not specified, the value returned by the hostname -i command is used.
NoteIf the server has multiple network interface cards (NICs), multiple IP addresses may be returned by the |
Note
sys_user and sys_password in the env/deploy.conf.json file are optional parameters used to authenticate the database where binlog files are to be generated. If the two parameters are not configured in the file, you must specify them in the command for creating a binlog task later.
Step 2: Start the binlog server
Run the following command in the obbinlog installation directory to start the binlog server:
sudo sh env/deploy.sh -m deploy -f env/deploy.conf.json
Notice
The deploy.sh script modifies the conf/conf.json file. This script can be executed only once, and modifications made by this script are irreversible. If the first execution fails, perform the follow-up operations by referring to Configure custom settings to start a binlog server.
The following operations are performed by running the deploy.sh script:
Install dependencies such as
jq,libstdc++.so.6,mysql, anddiff.Read attributes from the
env/deploy.conf.jsonfile. The default value is used for parameters that are not specified.Create metadata tables if
init_schemais set totrue.Write metadata connection configurations to the
conf/conf.jsonfile and setbinlog_modetotrue.Check whether the value of the
supervise_startparameter istrueorfalse.If
supervise_startis set totrue, call theenv/install.shscript, which installs supervisord and starts the program with it.If
supervise_startis set tofalse, the processing logic is as follows:If both the
sys_userandsys_passwordparameters are specified, thedeploy.shscript calls theconfig_sysfunction in therun.shscript to replace theob_sys_usernameandob_sys_passwordparameters in theconf/conf.jsonfile, and then runs therun.sh startcommand to start the program.If either the
sys_userorsys_passwordparameter is empty, thedeploy.shscript directly runs therun.sh startcommand to start the program.
Once the program is started, you can use the MySQL client or OceanBase Client (OBClient) to connect to the binlog server and run related commands.
mysql -h127.0.0.1 -P2983
Step 1: Install runtime dependencies
Configure the GNU Compiler Collection (GCC) environment.
cd ${deploy_path} sudo cp ./deps/lib/libstdc++.so.6.0.28 /usr/lib64/ cd /usr/lib64 sudo ln -sf libstdc++.so.6.0.28 libstdc++.so.6Install other dependency tools.
sudo yum install -y jq mysql diffutils
Step 2: Configure the conf.json file
Modify the conf/conf.json file. For more information about the parameters in the file, see Local configurations. The following parameters must be configured based on the actual situation:
binlog_mode: specifies whether to start the binlog service. You must set this parameter totrue.database_ip: the IP address of the MetaDB.database_port: the port number of the MetaDB.database_name: the name of the database in the MetaDB.user: the username for accessing the MetaDB.password: the password for accessing the MetaDB.node_ip: the IP address of the current node.
To set a default account for creating binlog tasks later, run the following command to write the username and password of the account under the sys tenant to the configuration file:
sudo ./run.sh config_sys <sys_user> <sys_password>
Notice
You must use sys_user and sys_password to specify the username and password for the sys tenant of OceanBase Database under which binlog files will be generated. The username cannot include the tenant or cluster name. For example, the username is root.
Step 3: Initialize the MetaDB
Before starting the first binlog server, create required tables in the MetaDB. After connecting to the corresponding database in the MetaDB, run the conf/schema.sql file to initialize the MetaDB. The sample command is as follows:
SOURCE conf/schema.sql
For more information about metadata tables, see Metadata.
Step 4: Start the binlog server
Run the following command to start the binlog server:
sudo ./run.sh start
Once the program is started, you can use the MySQL client or OBClient to connect to the binlog server and run related commands.
mysql -h127.0.0.1 -P2983
(Optional) Step 3: Add a binlog server to an existing binlog cluster
The binlog service treats all binlog servers that use the same database within the same MetaDB as part of the same binlog cluster. Therefore, to deploy multiple binlog servers within a single binlog service, you only need to repeat the steps in Step 2: Configure and start a binlog server on different physical servers by using the same MetaDB configurations.
Notice
You need to initialize the metadata tables only when you start the first binlog server.
After the binlog server is deployed, you can execute the SHOW NODES statement to query information about it. For more information about the statement, see Node management.
Step 4: Create a binlog task
Confirm the relevant information about the OceanBase Database tenant for which you want to create a binlog task. After connecting to the tenant, execute the following SQL statements to obtain the information.
Obtain the name of the cluster.
SHOW PARAMETERS LIKE 'cluster';The name of the cluster to which the connected OceanBase Database tenant belongs is displayed as
demoin thevaluecolumn.+-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault | +-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone1 | observer | 10.10.10.1 | 2882 | cluster | NULL | demo | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | obcluster | 0 | +-------+----------+------------+----------+---------+-----------+-------+---------------------+----------+---------+---------+-------------------+---------------+-----------+Obtain the name of the tenant.
SHOW TENANT;The name of the connected OceanBase Database tenant is displayed as
obmysql.+---------------------+ | Current_tenant_name | +---------------------+ | obmysql | +---------------------+Obtain the ConfigUrl.
SHOW PARAMETERS LIKE 'obconfig_url';The ConfigUrl of the cluster to which the connected OceanBase Database tenant belongs is displayed in the
valuecolumn.+-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault | +-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+ | zone1 | observer | 10.10.10.1 | 2882 | obconfig_url | NULL | http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=demo | URL for OBConfig service | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | 0 | +-------+----------+------------+----------+--------------+-----------+----------------------------------------------------------------------------+--------------------------+----------+---------+---------+-------------------+---------------+-----------+
Once the preceding information is confirmed, connect to the binlog server and execute the CREATE BINLOG statement to create a binlog task for the OceanBase Database tenant. For more information about the statement, see the Create a binlog task section in Task management.
The example runs the following command to create a binlog task for the obmysql tenant in the demo cluster. The CLUSTER URL parameter specifies the ConfigUrl, and the value 2 of the REPLICATE NUM parameter indicates that two binlog instances will be created for this tenant.
CREATE BINLOG FOR TENANT `demo`.`obmysql`
TO USER `root` PASSWORD `********`
WITH CLUSTER URL `http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=demo`
, REPLICATE NUM 2;
Notice
USERandPASSWORDin the command specify the username and password of thesystenant, rather than theobmysqltenant for which the binlog task is to be created. If the username and password of thesystenant are configured when you start the binlog server, removeTO USER `root` PASSWORD `********`in the command.- At present, you cannot create binlog tasks for the
systenant.
Once the binlog task is created, execute the SHOW BINLOG INSTANCES; statement to obtain the status of all binlog instances. The output is as follows:
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
| name | ob_cluster | ob_tenant | ip | port | zone | region | group | running | state | obcdc_running | obcdc_state | service_mode | convert_running | convert_delay | convert_rps | convert_eps | convert_iops | dumpers | version | odp_addr |
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
| bl09vjd9tl | demo | obmysql | 10.10.10.1 | 8100 | | | | Yes | Running | Yes | Running | enabled | Yes | 676 | 1 | 0 | 77 | 0 | 4.0.1-92748c6c59534426d187977b13b61b7cdf4fa200 | NULL |
| wi7emnxpat | demo | obmysql | 10.10.10.1 | 8101 | | | | Yes | Running | Yes | Running | enabled | Yes | 570 | 1 | 0 | 77 | 0 | 4.0.1-92748c6c59534426d187977b13b61b7cdf4fa200 | NULL |
+------------+------------+-----------+---------------+------+------+--------+-------+---------+---------+---------------+-------------+--------------+-----------------+---------------+-------------+-------------+--------------+---------+------------------------------------------------+----------+
The running, obcdc_running, and convert_running columns indicate the status of the main process of the binlog instance, the obcdc process, and the binlog converter (BC) process, respectively. If the values of all the three columns are Yes, the binlog instance runs properly. If the value of any of the three columns is No, the corresponding process is abnormal. In this case, perform troubleshooting by referring to Troubleshooting.
For more information about the binlog commands, see Control Commands.
Step 5: Configure ODP
Notice
If you want to use multiple ODPs, perform the following steps on each ODP.
Log in to the
systenant of OceanBase Database as therootuser by using ODP, or connect to ODP as theroot@proxysysuser.Configure the IP address and port number of the binlog server for the
binlog_service_ipparameter in theip:portformat. The default port number is2983. You can configure multiple IP addresses forbinlog_service_ipin ODP V4.3.1 and later. Separate them with semicolons (;).ALTER proxyconfig SET binlog_service_ip='10.10.10.1:2983';Enable the binlog service.
Note
The
enable_binlog_serviceparameter has been removed in ODP Community Edition V4.2.3 and later. Therefore, you can ignore the error indicating that this parameter does not exist.ALTER proxyconfig SET enable_binlog_service='True';(Optional) To ensure that the execution result of the
SHOW CREATE TABLEstatement is fully compatible with MySQL syntax, excluding the extended syntax of OceanBase Database, execute the following statement:ALTER proxyconfig SET init_sql='set _show_ddl_in_compat_mode = 1;';Execute the
SHOW PROXYCONFIG LIKE 'config_name'statement to verify the parameters, such asbinlog_service_ip.SHOW PROXYCONFIG LIKE 'binlog_service_ip';The output is as follows:
+-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+ | name | value | info | need_reboot | visible_level | range | config_level | +-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+ | binlog_service_ip | 10.10.10.1:2983 | binlog service ip/hostname list, format ip1:sql_port1;hostname2:sql_port2, separate with ';'. prefer to use the forward | false | SYS | | LEVEL_VIP | +-------------------+--------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------------+-------+--------------+
Step 6: Verify the result
Use ODP to connect to the tenant where the binlog task is generated and check the binlog file generation status by executing statements such as SHOW MASTER STATUS;. For more information about the binlog statements, see Compatibility.
Here is an example:
Use ODP to connect to the tenant where the binlog task is generated.
obclient -h10.10.10.1 -P2883 -uroot@obmysql -p -Doceanbase -ANotice
The binlog statements in this section can be executed only when you use ODP to connect to the tenant.
Query the binlog file.
SHOW MASTER STATUS;The output is as follows:
+------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+-------------------+ | mysql-bin.000001 | 155 | | | | +------------------+----------+--------------+------------------+-------------------+Query all binlog events.
SHOW BINLOG EVENTS;The output is as follows:
+------------------+------+----------------+-----------+-------------+-----------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+------+----------------+-----------+-------------+-----------------------------------+ | mysql-bin.000001 | 4 | Format_desc | 96754989 | 124 | Server ver: 5.7.38, Binlog ver: 4 | | mysql-bin.000001 | 124 | Previous_gtids | 96754989 | 155 | | +------------------+------+----------------+-----------+-------------+-----------------------------------+Perform a data change to verify whether the binlog service runs properly.
CREATE database test1;Query the binlog file again.
SHOW MASTER STATUS;The following new binlog file is generated by the binlog service:
+------------------+----------+--------------+------------------+----------------------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+----------------------------------------+ | mysql-bin.000001 | 297 | | | 0f5addad-9be6-11ef-9c9d-00163e0808cc:1 | +------------------+----------+--------------+------------------+----------------------------------------+Query all binlog events.
SHOW BINLOG EVENTS;The following new binlog events are generated, indicating that the binlog service runs properly:
+------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+ | mysql-bin.000001 | 4 | Format_desc | 96754989 | 124 | Server ver: 5.7.38, Binlog ver: 4 | | mysql-bin.000001 | 124 | Previous_gtids | 96754989 | 155 | | | mysql-bin.000001 | 155 | Gtid | 96754989 | 220 | SET @@SESSION.GTID_NEXT= '0f5addad-9be6-11ef-9c9d-00163e0808cc:1' | | mysql-bin.000001 | 220 | Query | 96754989 | 297 | use obmysql;CREATE database test1 | +------------------+------+----------------+-----------+-------------+-------------------------------------------------------------------+
Related operations
Release a binlog task
To stop all binlog instances for a tenant and delete the generated binlog files, perform the following steps to release the binlog task:
Log in to the binlog server.
mysql -h127.0.0.1 -P2983In this example, the IP address is
127.0.0.1and the port number is2983. You must change them based on the actual situation.Release the binlog task.
The binlog task for the
obmysqltenant in thedemocluster is released in the following example.DROP BINLOG FOR TENANT demo.obmysql;Query the status of all binlog instances.
SHOW BINLOG INSTANCES;If the values
demoandobmysqlare not displayed in theob_clusterandob_tenantcolumns respectively, the binlog task for the corresponding tenant is released.
Stop the binlog server
You can stop the binlog server by executing the following statement in the binlog installation directory. The default installation directory is /home/ds/oblogproxy.
[admin@test001 oblogproxy]$./run.sh stop
Note
The binlog instances that are not stopped continue to run even after the binlog server is stopped.
References
- For information about how to deploy binlog service Enterprise Edition by using OCP, see Create a binlog cluster.