Full-Import Parameters

Coordinator

Condition

The format of the Condition module is as follows:

{
  "briefWhiteCondition": [], // Simplified allowlist configuration. If present, the filtering rules of the store are built from this parameter.
  "whiteCondition": [], // Allowlist configuration
  "blackCondition": [] // Blocklist configuration
}

For example, when synchronizing data from source_db.source_table to sink_db.sink_table, the format of the Condition module is as follows.

{
  "whiteCondition": [
    {
      "name": "source_db",
      "map": "sink_db",
      "all": false,
      "subs": [
        {
          "name": "source_table",
          "map": "sink_table"
        }
      ]
    }
  ],
  "blackCondition": [

  ]
}

The Condition module is a multi-level nested configuration. The first level is the database, the second level is the table, and the third level is the column.

The above parameters can meet the needs of ordinary migration scenarios. However, OMS also supports wildcard and regular expression matching capabilities and the attrs capability on this basis.

(Wildcard/regular) matching capability

{
  "whiteCondition": [
    {
      "name": "source_*",
      "map": "sink_db",
      "func": "fn",
      "all": false,
      "subs": [
        {
          "name": "source_*",
          "map": "sink_table",
          "func": "fn"
        }
      ]
    }
  ],
  "blackCondition": [
    {
      "name": "source_db_[0-10]",
      "map": "sink_db",
      "func": "reg",
      "all": false,
      "subs": [
        {
          "name": "source_table_[0-10]",
          "map": "sink_table",
          "func": "reg"
        }
      ]
    }
  ]
}

The func parameter specifies the matching function. The default value is string equals. The reg parameter specifies the regex match, and the fn parameter specifies the fnmatch.

The preceding example indicates that all tables that start with source in all databases need to be synchronized to sink_db.sink_table. However, the source_table_0...source_table_10 tables in the source_db_0...source_db_10 databases need to be excluded.

attrs capabilities

• Filter DML

  {
  "whiteCondition": [
      {
      "name": "source_db",
      "map": "sink_db",
      "all": false,
      "attr": {
          "dml": "i,u,d"
      },
      "subs": [
          {
          "name": "source_table",
          "map": "sink_table"
          }
      ]
      }
  ],
  "blackCondition": []
  }
  

  dml:i,u,d in attr indicates that the task only synchronizes change messages of the INSERT, DELETE, and UPDATE types. Note that full messages do not belong to the INSERT type.

  The attributes of attr are inherited by default. For example, if the attr parameter does not exist at the table level, it will attempt to obtain the attr parameter at the parent level.

• Filter DDL

  {
  "whiteCondition": [
      {
      "name": "source_db",
      "map": "sink_db",
      "all": false,
      "subs": [
          {
          "name": "source_table",
          "map": "sink_table",
          "attr": {
              "ddl": "ct,dt"
          }
          }
      ]
      }
  ],
  "blackCondition": []
  }
  

  ddl: ct,dt in attr indicates that only CREATE TABLE and DROP TABLE DDL statements are accepted.

• Data conversion by using SQL syntax

  {
  "whiteCondition": [
      {
      "name": "source_db",
      "map": "sink_db",
      "all": false,
      "subs": [
          {
          "name": "source_table",
          "map": "sink_table",
          "attr": {
              "op": "select substr(c1, 1, 2) as c3 from record where c2 = x"
          }
          }
      ]
      }
  ],
  "blackCondition": []
  }
  

  op in attr specifies the column calculation and column filtering capabilities for records. For example, the statement select substr(c1, 1, 2) as c3 from record where c2 = x in the preceding example indicates that the following operations need to be performed on the source_db.source_table table:

  1. Filter out data where c2 = x.

  2. Extract the first two characters from the c1 column, rename the extracted data to c3, and pass it down. Note that the c1 column still exists.

• Some attr parameters in the message queue scenario

  {
  "whiteCondition": [
      {
      "name": "source_db",
      "all": false,
      "subs": [
          {
          "name": "source_table",
          "map": "topic_1",
          "attr": {
              "shard_fields": "c1,c2",
              "ddl_map": "topic_2"
          }
          }
      ]
      }
  ],
  "blackCondition": []
  }
  

  • shard_fields specifies to use the c1 and c2 columns as the sharding columns of the source_table table. This is mainly used to determine which partition, shard, or queue the data is to be delivered to.

  • ddl_map specifies that when the source_table table undergoes DDL changes, the change messages need to be delivered to topic_2.

Parameters of the target MySQL compatible mode of OceanBase Database

Parameters of the target Oracle compatible mode of OceanBase Database

Parameters of the target MySQL and PostgreSQL

Parameters of the target Oracle database

Parameters of the target RocketMQ

Parameters of the target Kafka

Parameters of the target DataHub

Parameters of the source Kafka

Parameters of the source logproxy

DataFlowSource parameters

JDBC-SINK startup parameters

STORE_MULTI_SOURCE parameters