Full-Import parameters

Coordinator

Condition

The format of the Condition module is as follows:

{
  "briefWhiteCondition": [], // The brief allowlist. If an allowlist exists, the filtering rules are created for the store based on this parameter.
  "whiteCondition": [], // The allowlist configuration.
  "blackCondition": [] // The blocklist configuration.
}

The following example shows the Condition module for data synchronization from source_db.source_table to sink_db.sink_table:

{
  "whiteCondition": [
    {
      "name": "source_db",
      "map": "sink_db",
      "all": false,
      "subs": [
        {
          "name": "source_table",
          "map": "sink_table"
        }
      ]
    }
  ],
  "blackCondition": [

  ]
}

The Condition module is a multi-layer nested parameter structure. The first layer is the database layer, the second layer is the table layer, and the third layer is the column layer.

The preceding parameters can satisfy common migration requirements. OceanBase Migration Service (OMS) also provides wildcard and regular expression (regex) matching capabilities and attrs capabilities.

Wildcard and regex matching capabilities

{
  "whiteCondition": [
    {
      "name": "source_*",
      "map": "sink_db",
      "func": "fn",
      "all": false,
      "subs": [
        {
          "name": "source_*",
          "map": "sink_table",
          "func": "fn"
        }
      ]
    }
  ],
  "blackCondition": [
    {
      "name": "source_db_[0-10]",
      "map": "sink_db",
      "func": "reg",
      "all": false,
      "subs": [
        {
          "name": "source_table_[0-10]",
          "map": "sink_table",
          "func": "reg"
        }
      ]
    }
  ]
}

The func parameter specifies the matching function, which is equals by default. reg indicates regex match. fn indicates fnmatch.

The preceding example specifies to synchronize all tables starting with source in all databases starting with source to sink_db.sink_table, excluding the source_table_0...source_table_10 table in the source_db_0...source_db_10 database.

attrs capabilities

• Filter DML statements

  {
    "whiteCondition": [
      {
        "name": "source_db",
        "map": "sink_db",
        "all": false,
        "attr": {
          "dml": "i,u,d"
        },
        "subs": [
          {
            "name": "source_table",
            "map": "sink_table"
          }
        ]
      }
    ],
    "blackCondition": []
  }
  

  In attr, dml:i,u,d indicates that the project only synchronizes messages of INSERT, DELETE, and UPDATE changes. Note that INSERT changes do not involve full migration messages.

  Default attr properties are inherited. For example, if the table-level attr property does not exist, OMS tries to obtain the parent-level attr property.

• Filter DDL statements

  {
    "whiteCondition": [
      {
        "name": "source_db",
        "map": "sink_db",
        "all": false,
        "subs": [
          {
            "name": "source_table",
            "map": "sink_table",
            "attr": {
              "ddl": "ct,dt"
            }
          }
        ]
      }
    ],
    "blackCondition": []
  }
  

  In attr, ddl: ct,dt indicates that only CREATE TABLE and DROP TABLE DDL statements are accepted.

• Data conversion using SQL syntax

  {
    "whiteCondition": [
      {
        "name": "source_db",
        "map": "sink_db",
        "all": false,
        "subs": [
          {
            "name": "source_table",
            "map": "sink_table",
            "attr": {
              "op": "select substr(c1, 1, 2) as c3 from record where c2 = x"
            }
          }
        ]
      }
    ],
    "blackCondition": []
  }
  

  In attr, op uses SQL syntax to indicate the column computing and filtering capabilities of records. In the preceding example, select substr(c1, 1, 2) as c3 from record where c2 = x indicates that the following operations need to be performed on source_db.source_table:

  1. Retrieve the data that matches c2 = x.

  2. Truncate the c1 column to 2 bits, rename the new column as c3, and pass it down. Note that the c1 column still exists.

• Some attrs properties in message queue scenarios

  {
    "whiteCondition": [
      {
        "name": "source_db",
        "all": false,
        "subs": [
          {
            "name": "source_table",
            "map": "topic_1",
            "attr": {
              "shard_fields": "c1,c2",
              "ddl_map": "topic_2"
            }
          }
        ]
      }
    ],
    "blackCondition": []
  }
  

  • shard_fields indicates that the c1 and c2 columns are used as the sharding key columns of source_table. This parameter specifies the partition, shard, or queue to which the data record is delivered.

  • ddl_map specifies to deliver the data change message to topic_2 when a DDL change is performed on source_table.

Parameters of a destination MySQL tenant of OceanBase Database

Parameters of a destination Oracle tenant of OceanBase Database

Parameters of a destination MySQL or PostgreSQL instance

Parameters of a destination Oracle instance

Parameters of a destination RocketMQ instance

Parameters of a destination Kafka instance

Parameters of a destination DataHub instance

Parameters of a source Kafka instance

logproxy parameters

DataFlowSource parameters

JDBC-Sink startup parameters

STORE_MULTI_SOURCE parameters