The binlog service collects transaction logs of OceanBase Database and converts them into MySQL binlogs. This service is designed for scenarios such as real-time data subscription.
Background information
MySQL binlogs are the basis for data subscription. You can enable the binlog service for downstream business scenarios. The binlog service has low requirements on performance and features low resource consumption.
OceanBase Database provides the binlog service for compatibility with the binlog capability of MySQL. You can reuse your existing MySQL binlog incremental parsing system to synchronize incremental data of OceanBase Database in MySQL mode without additional development work or building a new environment. The binlog service is aligned with MySQL Binlog Version 4. Specifically, the binlog service provides the same features as MySQL Binlog Version 4, such as support for multiple subscription connections and support for pulling logs based on global transaction identifiers (GTIDs), and ensures data uniqueness in binlogs. You can seamlessly migrate your MySQL-based incremental data subscription solutions to OceanBase Database.
Notice
The binlog service of OceanBase Database cannot be used for scenarios such as incremental restore and building primary/standby tenants.
Product introduction
Components
The binlog service of OceanBase Database comprises three parts:
- OceanBase Database: serves as a business database that generates binlogs.
- OceanBase Database Proxy (ODP): provides connections for clients that subscribe to binlogs. ODP is also known as OBProxy.
- obbinlog (known as oblogproxy earlier): pulls clogs and generates binlogs.
obbinlog is the core component of the binlog service.
Enterprise Edition and Community Edition
The binlog service of OceanBase Database is available in two editions: Enterprise Edition and Community Edition. The two editions have no differences in the overall architecture and functionality. Therefore, the descriptions of MySQL compatibility, command operations, and monitoring metrics in the documentation of the binlog service are applicable to both editions. The two editions apply to different OceanBase Database editions.
- The Enterprise Edition of the binlog service applies only to OceanBase Database Enterprise Edition, and the Community Edition of the binlog service applies only to OceanBase Database Community Edition.
- The Enterprise Edition of the binlog service is provided as a cloud service for ApsaraDB for OceanBase customers. For deployments outside of ApsaraDB for OceanBase, contact OceanBase Technical Support to obtain the installation package of this edition. The Community Edition of the binlog service fully relies on open-source components. You can download the software package of this edition for installation and deployment.
If you are using OceanBase Database Enterprise Edition, obtain support based on your system environment.
- ApsaraDB for OceanBase: Enable and use the binlog service by referring to Binlog service.
- Other environments such as Apsara Stack: Contact OceanBase Technical Support. If you have the installation package of obbinlog Enterprise Edition, you can deploy the binlog service by referring to Create a binlog cluster.
Version mapping information
For the version mapping information of obbinlog, see Release notes.
Architecture
Concepts
Binlog cluster
A binlog cluster is an external service that comprises one or more binlog servers. When the binlog service is started for a tenant in a binlog cluster that includes multiple binlog servers, a binlog server is selected based on the load balancing logic. When any binlog server fails, a functioning binlog server in the cluster will take over its services.
Binlog server
A binlog server is a binlog service node. The main process (referred to as OBM in the preceding figure) of a binlog service node processes SQL requests and manages binlog instances on the node. Multiple binlog servers share information through the MetaDB.
Binlog instance
A binlog instance (referred to as OBI in the preceding figure) is a tenant-level instance that generates and sends binlogs, as well as monitors and processes SQL requests through TCP based on the MySQL protocol. An OceanBase Database binlog instance is equivalent to a MySQL instance. Multiple OceanBase Database binlog instances in the same tenant are equivalent to a group of primary and standby MySQL instances. An OceanBase Database binlog instance can process SQL statements, protocols, and variables related to MySQL binlogs, such as the SHOW MASTER STATUS statement.
Cluster high availability
When you create a binlog instance, the binlog service selects a binlog server based on the loads of available binlog servers and the configuration in the creation statement. The creation task is then executed on the selected binlog server. The following figure shows the selection logic.
This selection strategy aims to ensure that the OBI instances of a tenant are distributed to different nodes. The details are described as follows:
If an IP address, zone, or group is specified in the statement for creating an OBI instance, the binlog cluster preferentially distributes the OBI instance to a node within the specified constraints.
OBI instances are preferentially distributed to zones and nodes without OBI instances.
The resource usage of a node where an instance is to be distributed is checked to make sure that the CPU utilization, memory usage, and disk usage on the node are below the thresholds and that the node meets the minimum memory and disk requirements.
To shorten the fault recovery time, we recommend that you set the number of instance replicas to a value greater than or equal to 2 when you create an OBI instance. This way, when you create a binlog task, the binlog cluster will start at least one redundant OBI instance apart from the primary OBI instance. According to the load balancing strategy, these OBI instances will be preferentially distributed to different nodes. When an OBI instance fails, the system automatically tries to recover this OBI instance. When the primary OBI instance fails, the system selects a redundant OBI instance as the new primary instance while recovering the faulty instance, and switches subsequent sessions to the new primary OBI instance.
Description
The binlog protocol of OceanBase Database in MySQL mode is basically consistent with MySQL Binlog Version 4. Therefore, its basic features are almost the same as those of the MySQL binlog protocol. The basic features of the OceanBase binlog protocol are described as follows:
It generates binlogs only in the row-based replication (RBR) format.
It generates binlogs for MySQL tenants of OceanBase Database based on the tenant-level clogs.
It generates only binlog events that record changes, namely, events of the Format_desc, Previous_gtids, Rotate, Gtid, Query, Table_map, Write_rows, Update_rows, and Delete_rows types.
It supports pulling binlogs based on GTIDs.
It supports resumable transmission for binlog events, ensuring the uniqueness and integrity of binlogs.
It supports converting DDL statements for incremental synchronization in OceanBase Database into the native syntax of MySQL.
Notice
The upstream cannot use the following OceanBase Database-specific data types that are incompatible with MySQL: string, array, map, and roaringbitmap.
The Binlog service supports high availability. It can be automatically recovered within minutes in the event of a failure. However, downstream subscribers need to support reconnection in the event of a failure to avoid subscription disconnection.
Limitations
Compatibility
For more information about the compatibility of the binlog service with MySQL, see Compatibility.
Performance
The binlog conversion performance is subject to resource configurations and clog parsing capabilities. Here is the test data:
Maximum clog parsing performance: 140 MiB/s
Maximum RPS performance for binlog conversion: 250,000
If the business traffic of the subscribed data source exceeds the limits, the latency of the binlog conversion service may increase.
Under controllable business pressure, the latency of the binlog conversion service can remain within 1 second. However, in special cases, a seconds-level conversion latency cannot be guaranteed if the database involves frequent DDL operations or large transactions.
Integration with downstream ecosystems
Synchronization of existing data
The binlog service can synchronize only incremental data. However, data synchronized from OceanBase Database in MySQL mode to a downstream system may involve existing data. Take note of the following considerations during the synchronization of existing data:
Only the incremental snapshot read mode is supported.
Snapshot table locking and
FLUSH TABLES WITH READ LOCKare not supported.The timeout period parameters
interactive_timeoutandwarning_countof native MySQL Database are supported.The heartbeat timeout parameter
heart.intervalcan be configured.If the MySQL synchronization component supports parallel data pulling, existing data can be pulled in parallel.
Integration with downstream ecosystem tools
The binlog service of OceanBase Database can be integrated with popular MySQL binlog parsing tools, such as Flink MySQL Change Data Capture (CDC), Canal, and Maxwell.