ChunJun, formerly known as FlinkX, is a data integration framework based on Flink, which is stable, easy to use, efficient, and integrated with streaming and batch processing APIs. It allows you to configure tasks based on JSON templates and the Flink real-time computation engine. It is compatible with Flink SQL syntax and supports distributed execution. It supports various submission methods such as flink-standalone, yarn-session, and yarn-per job, and supports features full synchronization and incremental synchronization.
This topic describes how to use ChunJun in local mode to synchronize data from a MySQL database to OceanBase Database in MySQL mode.
Scenario
Migrate data from the tbl1 table in the MySQL database to the test_tbl1 table in OceanBase Database. The database information is as follows:
| Information of the MySQL database (source database) | Example value |
|---|---|
| Host URL | xxx.xxx.xxx.xxx |
| Port number | 3306 |
| Username | root |
| Password of the user | ****** |
| Schema name | test_mysql_to_ob |
| Table name | tbl1 |
| Information of OceanBase Database (destination database) | Example value |
|---|---|
| Cluster name | test4000 |
| Host URL | 10.10.10.2 |
| Port number | 2881 |
| Tenant name (MySQL mode) | mysql001 |
| Username | root |
| Password of the user | ****** |
| Schema name | test_data |
| Table name | test_tbl1 |
Prerequisites
You have installed JDK 1.8 and configured the JAVA_HOME environment variable.
Procedure
Step 1: Prepare the ChunJun environment
Download and decompress the ChunJun package.
Download chunjun-dist.tar.gz.
wget https://github.com/DTStack/chunjun/releases/download/v1.12.6/chunjun-dist.tar.gzDecompress the package to the
/chunjun_Home/chunjundirectory.mkdir /ChunJun_Home/chunjun && tar zxvf chunjun-dist.tar.gz -C /ChunJun_Home/chunjunConfigure the ChunJun environment variable.
export ChunJun_HOME=/ChunJun_Home/chunjun-dist
Step 2: Configure the JSON file
Configure the JSON file based on the environment information to enable migration of data from the tbl1 table in the MySQL database to the test_tbl1 table in OceanBase Database.
For example:
[root@xxx /]
$cd /ChunJun_Home/chunjun-dist/chunjun-examples/json
[root@xxx /ChunJun_Home/chunjun-dist/chunjun-examples/json]
$mkdir test_data
[root@xxx /ChunJun_Home/chunjun-dist/chunjun-examples/json]
$cd mkdir test_data
[root@xxx /ChunJun_Home/chunjun-dist/chunjun-examples/json/test_data]
$vi test_mysql_to_ob.json
{
"job": {
"setting": {
"errorLimit": {
"record": 0,
"percentage": 0.02},
"speed": {"bytes": 0,
"channel": 1}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "******",
"column": [
"*"
],
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://xxx.xxx.xxx.xxx:3306/test_mysql_to_ob"
],
"table": [
"tbl1"
]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": [
"*"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://10.10.10.2:2881/test_data",
"table": [
"test_tbl1"
]
}
],
"username": "root@mysql001",
"password": "******"
}
}
}
]
}
}
Step 3: Run the JSON configuration file
Start a JVM process to execute the ChunJun task.
Go to the /ChunJun_Home/chunjun-dist directory and run the following command:
[root@xxx /]
$cd /ChunJun_Home/chunjun-dist
[admin@xxx /ChunJun_Home/chunjun-dist]
$sh bin/chunjun-local.sh -job chunjun-examples/json/test_data/test_mysql_to_ob.json
# #
# #
#
##### ###### # # # #### #### # # # ####
# # # # # ## # # # # ## #
# # # # # # # # # # # #
# # # # ## # # # # ## # #
##### # # #### # # # # #### # # #
#
Reference site: https://dtstack.github.io/chunjun
chunjun is starting ...
CHUNJUN_HOME is auto set /ChunJun_Home/chunjun-dist
FLINK_HOME is empty!
HADOOP_HOME is empty!
...
<ellipsis>
...
*********************************************
numWrite | 3
last_write_num_0 | 0
conversionErrors | 0
writeDuration | 20130
duplicateErrors | 0
numRead | 3
snapshotWrite | 0
otherErrors | 0
readDuration | 78
byteRead | 389
last_write_location_0 | 0
byteWrite | 389
nullErrors | 0
nErrors | 0
*********************************************
2023-01-06 19:18:57,306 - 50207 INFO [main] com.dtstack.chunjun.Main:program Flink_Job execution success
Step 4: View the data migration status
Log on to OceanBase Database to check whether the data has been migrated.
[admin@xxx /home/admin]
$obclient -h10.10.10.2 -P2881 -uroot@mysql001 -p -A
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 3221630190
Server version: OceanBase 4.0.0.0 (r101000022022120716-0d7927892ad6d830e28437af099f018b0ad9a322) (Built Dec 7 2022 16:22:15)
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient [(none)]> use test_data;
Database changed
obclient [test_data]> SELECT * FROM test_tbl1;
+------+-----------+------+
| COL1 | COL2 | COL3 |
+------+-----------+------+
| 1 | China | 86 |
| 2 | Taiwan | 886 |
| 3 | Hong Kong | 852 |
+------+-----------+------+
3 rows in set
More information
- For more information about features of ChunJun, see Features of ChunJun.
- For more information about how to get started with ChunJun, see Quick start.
- For more script examples, see the files in the
/chunjun-dist/chunjun-examplesfolder of the ChunJun project.