SeaTunnel is a distributed, highly scalable data integration platform for transferring data between various data storage systems. It provides easy-to-configure data flow task plugins, allowing users to define processes for data processing and transformation. For more information about SeaTunnel, refer to SeaTunnel's GitHub page.
Prerequisites
Before you migrate data, make sure that:
- You have downloaded SeaTunnel.
- You have installed Java 1.8 or a higher version.
- You have downloaded the MySQL driver into the
libdirectory. - You have installed OBLogProxy.
This section introduces how to execute SQL queries in a MySQL tenant of OceanBase Database by using SeaTunnel's MySQL CDC Source Connector. connector-cdc-mysql is a plugin of SeaTunnel used to capture and stream real-time change data (Change Data Capture, CDC) from MySQL databases.
Procedure
Follow the steps below to synchronize data between MySQL tenants in OceanBase Database:
Create a SeaTunnel user.
CREATE USER 'username'@'%' IDENTIFIED BY 'password'; GRANT SELECT ON *.* TO 'username';Configure
config/plugin_config.Specify
connector-cdc-mysqland executesh bin/install-plugin.sh 2.3.3.Configure SeaTunnel.
Modify the configuration file
config/v2.streaming.conf.templateas follows:source { # This is a example source plugin **only for test and demonstrate the feature source plugin** MySQL-CDC { result_table_name = "fake" parallelism = 1 server-id = 5656 username = "username" password = "password" table-names = ["seatunnel.full_types"] base-url = "jdbc:mysql://xx.xxx.x.xx:2881/seatunnel" } # If you would like to get more information about how to configure SeaTunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/category/source-v2 }Start SeaTunnel.
Run the following command to start SeaTunnel:
./bin/seatunnel.sh --config ./config/v2.streaming.conf.template -e local
This section introduces how to synchronize data using SeaTunnel's JDBC OceanBase Source Connector and JDBC OceanBase Sink Connector. JDBC OceanBase Source Connector supports full reading and writing of data from a single table in OceanBase Database. For more information, see JDBC OceanBase Source Connector. JDBC OceanBase Sink Connector supports full read and write on multi-table data in OceanBase. For more information, see JDBC OceanBase Sink Connector.
Procedure
Follow the steps below to synchronize data between MySQL tenants in OceanBase Database:
Create a SeaTunnel user.
CREATE USER 'username'@'%' IDENTIFIED BY 'password'; GRANT SELECT, INSERT, UPDATE, DELETE ON *.* TO 'username';Configure
config/plugin_config.Download OceanBase JDBC driver to the
libdirectory. For more information, see OceanBase Client.Configure SeaTunnel.
Modify the configuration file
config/ob_to_obas follows:env { parallelism = 1 job.mode = "BATCH" } source { # This is a example source plugin **only for test and demonstrate the feature source plugin** Jdbc { driver = com.oceanbase.jdbc.Driver url = "jdbc:oceanbase://e2e_oceanbase_mysql:2881/seatunnel?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true&serverTimezone=UTC" user = "username" password = "password" query = "SELECT c_bit_1, c_bit_8 FROM source" compatible_mode = "mysql" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource } sink { Jdbc { driver = com.oceanbase.jdbc.Driver url = "jdbc:oceanbase://e2e_oceanbase_mysql:2881/seatunnel?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true&serverTimezone=UTC" user = "username" password = "password" query = "insert into sink(c_bit_1, c_bit_8) values (?, ?);" compatible_mode = "mysql" } # If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/connector-v2/sink }Start SeaTunnel.
Run the following command to start SeaTunnel:
./bin/seatunnel.sh --config ./config/ob_to_ob -e local
Notice
OceanBase Database does not support the SET datatype.
Caused by: org.apache.seatunnel.api.table.catalog.DataTypeConvertException: ErrorCode:[COMMON-07], ErrorDescription:[Unsupported data type] - Convert type: SET to SeaTunnel data type error.
For more information, see the SeaTunnel MySQL CDC documentation.