Binlog service overview

The binlog service collects transaction logs of OceanBase Database and converts them into MySQL binlogs. This service is designed for scenarios such as real-time data subscription. OceanBase Cloud Platform (OCP) allows you to deploy the binlog service in standalone mode or cluster mode on the console, and perform basic O&M operations on binlog clusters and instances. OCP also provides monitoring and alerting capabilities for the binlog service. You can query resource usage views of binlog clusters and associate the views with alerts to help you quickly locate and resolve issues.

Background information

In MySQL, the binlog serves as the basis for data subscription. You can enable the binlog service for downstream business scenarios with lower performance and resource consumption.

OceanBase binlog service is introduced to be compatible with MySQL binlog capabilities. You can reuse your existing MySQL binlog incremental parsing system to synchronize the incremental data of OceanBase MySQL mode, without the need for secondary development or building a new environment. The binlog service is aligned with MySQL Binlog Version 4. Specifically, OceanBase binlog service provides the same features as MySQL Binlog Version 4, such as support for multiple subscription connections and support for pulling logs based on global transaction identifiers (GTIDs), and ensures data uniqueness in binlogs. You can seamlessly migrate your MySQL-based incremental data subscription solutions to OceanBase Database.

Architecture

Binlog cluster

A binlog cluster is an external service that comprises one or more binlog servers. In a scenario where there are multiple binlog servers, when the binlog service is started for a tenant, the system will select the appropriate binlog server according to the principle of load balancing. If any binlog server fails, other functioning binlog servers will take over its service.

Binlog server

A binlog server is responsible for managing multiple binlog instances and processing SQL requests. Multiple binlog servers share information through MetaDB. A binlog server supports custom SQL APIs, mainly for managing binlog instances.

Binlog instance

A binlog instance is a tenant-level instance responsible for generating and sending binlogs, and for monitoring and processing SQL requests through TCP based on the MySQL protocol. An OceanBase binlog instance is equivalent to a MySQL instance. Multiple OceanBase binlog instances in the same tenant are equivalent to a group of primary and standby MySQL instances. An OceanBase binlog instance can process MySQL binlog-related SQL statements, protocols, and variables, such as SHOW MASTER STATUS.

Features

The binlog protocol of OceanBase MySQL mode is basically consistent with MySQL Binlog Version 4. Therefore, its basic features are nearly the same as those of the MySQL binlog protocol. The details are as follows:

• Only supports generating binlogs in row-based replication (RBR) format.
• Generates binlogs based on the levels of OceanBase MySQL tenants.
• Only supports generating binlog events related to changes, not all binlog events, namely the Format_desc, Previous_gtids, Rotate, Gtid, Query, Table_map, Write_rows, Update_rows, and Delete_rows event types.
• Supports pulling binlogs based on GTIDs.
• Supports the resumable transmission of binlog events, without duplication or loss.
• Supports converting OceanBase incremental DDL statements to MySQL native syntax.

Limitations

Performance

The binlog conversion performance is subject to resource configuration and clog parsing capabilities. Currently, the maximum performance of clog parsing supported by the binlog service is 25 MiB/s. The upper limit for the binlog conversion rate per second (RPS) is about 50,000. If the source business traffic exceeds these limits, the latency of the binlog conversion service may increase.

Under controllable business pressure, the latency of the binlog conversion service can usually be maintained within 1 second. However, in special cases, such as when the database experiences frequent data definition language (DDL) operations or large transactions, the second-level conversion latency cannot be guaranteed.

Version requirements

The following requirements apply to all service versions and tenant modes when connecting to the binlog service:

• OBServer version 3.x must be no less than V3.2.4.4, and version 4.x must be no less than V4.1.0.1.
• OBProxy version must be no less than V4.2.1.
• Binlog service version must be no less than V4.0.0.
• MetaDB version must be no less than 2.2.77 (Built Feb 14 2022 18:11:59).
• Only tenants in MySQL mode are supported.

SQL compatibility

Due to certain differences between the DDL syntax of OceanBase MySQL mode and that of MySQL, specifically the OceanBase MySQL mode has some unique extended syntax, this part of the DDL syntax may not be able to be parsed.

Data types

Due to differences in precision and implementation between OceanBase MySQL mode and MySQL regarding data types, not all data types are fully compatible. The incompatible data types are as follows:

• The binlog service does not support the extended semantics of the ENUM and SET data types in OceanBase MySQL mode. For example, the SET data type supports more than 64 definitions and duplication, and the ENUM data type supports the insertion of undefined data such as ''.

• The VARCHAR (n > 65535) data type is not supported.

• The maximum length supported for the LONGBLOB and LONGTXT data types is 48 MiB.

• The MEDIUMBLOB, LONGBLOB, BLOB, MEDIUMTEXT, LONGTEXT, TEXT, JSON, and GIS data types are all stored as the LOB type in the underlying layer. By default, if they exceed 4 KiB, they will be stored in OUTROW mode, which is incompatible with MySQL.

  • Data stored in OUTROW mode is updated to data stored in INROW mode. The beforeimage outputs NULL values and the afterimage outputs the updated data.

  • After non-LOB columns are updated for a table with a LOB column that uses the OUTROW storage mode, both the beforeimage and afterimage output NULL values.

  • After the LOB column that uses the OUTROW storage mode is deleted, the beforeimage outputs NULL values.

    Note

    A LOB column whose data is sized 4 KiB or smaller is processed as a regular column. In other words, the column data is actually recorded in a column of the primary table. For a LOB column whose data is sized more than 4 KiB, the column data is actually stored in a LOB auxiliary table, and the primary table records information about this auxiliary table. The default size of a LOB column is 4 KiB.

• OceanBase Database V4.x and later do not support virtual generated columns.

• The CREATE TABLE AS SELECT (CTAS) statement is not supported.

  OceanBase Database V4.x cannot output the accurate table name if the table is created by using the CTAS statement. For example, the create table t2 as select * from t1 statement will create a table named t2 and write data in the t1 table to the t2 table. At present, OceanBase Database can output only __ctas_xxxx as a temporary table name, instead of outputting the accurate table name t2.

Integration with the downstream ecosystem

Synchronization of existing data

The binlog service can synchronize only incremental data. However, data synchronized from OceanBase MySQL mode to a downstream system may involve existing data. Take note of the following considerations during the synchronization of existing data:

• Only the incremental snapshot read mode is supported.
• Snapshot table locking and FLUSH TABLES WITH READ LOCK are not supported.
• The timeout period parameters interactive_timeout and warning_count of native MySQL Database are supported.
• The heartbeat timeout parameter heart.interval can be configured.
• If the MySQL synchronization component supports parallel data pulling, existing data can be pulled in parallel.

Integration with downstream ecosystem tools

The binlog service of OceanBase Database can be integrated with popular MySQL binlog parsing tools, such as Flink CDC, Canal, and Maxwell.