DataFlowSource

Starter class

connector-dataflow.jar

com.oceanbase.oms.connector.source.dataflow.DataFlowSourceBooster

[Metadata] clients

• Description: specifies the data source information.

• Default value: None. This parameter is required.

• Value range:

  {
    "clientId": "clientId, unique identifier",
    "cluster": "OceanBase cluster",
    "tenant": "OceanBase Database tenant",
    "instance": "ip:port/[schema]", // The schema must exist in the PostgreSQL, DB2 LUW, or Oracle database.
    "tableWhiteLists": { // This parameter takes effect only for logical tables. Physical tables depend on the Conditions parameter. Therefore, this parameter does not take effect for physical tables.
      "db": ["tab", "tab2"]
    },
    "username": "Username",
    "password": "Password",
    "weakRead": "false", // Optional. Default value: false
    "sysInstance": "Instance of the sys tenant,ip:port/[schema]", // Optional. Default value: Null.
    "sysTenantUsername": "Username of the sys tenant", // Optional. Default value: Null.
    "sysTenantPassword": "Password of the sys tenant" // Optional. Default value: Null.
  }
  

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Metadata] dbType

• Description: specifies the type of the database to be synchronized.

• Default value: None. This parameter is required.

• Value range: The value range of DbTypeEnum.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] maxConnNum

• Description: specifies the maximum number of connections.

• Default value: 32.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] databaseMaxConnection

• Description: specifies the maximum number of connections.

• Default value: 128.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] connTimeoutSec

• Description: specifies the timeout period of connections.

• Default value: 120.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] charset

• Description: specifies the character set.

• Default value: utf8

• Value range: character set

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection parameters] timezone

• Description: specifies the timezone.

• Default value: +08:00

• Value range: timezone

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceWorkerNum

• Description: specifies the number of threads for data sharding.

• Default value: 16.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceQueueSize

• Description: specifies the maximum number of data sharding queues.

• Default value: 128.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] batchSize

• Description: specifies the number of records in a data shard for batch writing.

  For example, if the number of records in each data shard (sliceBatchSize) is 2000, the value 200 of batchSize specifies to write 200 records in each batch to the target.

• Default value: 200.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Sharding] sliceBlocks

• Description: specifies the number of blocks in each data shard.

• Default value: 128

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Sharding] sliceBatchSize

• Description: specifies the number of records in each data shard.

• Default value: 2000

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] sourceMaxRetryTimes

• Description: specifies the maximum number of retries for data retrieval.

• Default value: 3

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] retryIntervalTime

• Description: specifies the interval between retries.

• Default value: 3

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] maxSqlRetryTime

• Description: specifies the maximum retry time for data retrieval.

• Default value: 3600

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] maxSqlNoActiveTime

• Description: specifies the JDBC timeout period.

• Default value: 700

• Value range: int

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Filtering] filterTmpTable

• Description: specifies whether to filter temporary tables.

• Default value: true

• Value range: boolean

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Preprocessing] sourceJdbcTableCache

• Description: specifies whether to cache the results of preprocessing when the source is a database.

• Default value: false

• Value range: true or false

  true: caches the results and uses the tables in the global cache. false: does not cache the results and retrieves the tables based on the source configuration.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Tables without primary or unique keys] nonePkUkQueryWholeTable

• Description: specifies whether to query the entire table when there are no unique indexes.

• Default value: true

• Value range: boolean

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Tables without primary or unique keys] nonePkUkTruncateDstTable

• Description: specifies whether to truncate the destination table when there are no unique indexes.

• Default value: true

• Value range: boolean

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] sourceBatchMemorySize

• Description: specifies the maximum memory for each data shard, in bytes.

• Default value: 16777216.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] workerNum

• Description: specifies the number of worker threads for pulling data.

• Default value: 16.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nonePkUkSupportRowsMax

• Description: specifies the maximum number of records without a primary key.

• Default value: 5000000.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nullReplaceEnable

• Description: specifies the whether to enable character replacement in a scenario where the ORACLE gb18030 field value is chr(129)-chr(254), a NOT NULL constraint is present, and the value read by JDBC is NULL.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nullReplaceString

• Description: specifies whether to replace the queried value with the specified string in a scenario where the ORACLE gb18030 field value is chr(129)-chr(254), a NOT NULL constraint is present, and the value read by JDBC is NULL. when nullReplaceEnable is set to true

• Default value: " "

• Valid values: Characters and strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] ncharCharsetMap

• Description: specifies the mapping between NLS_NCHAR_CHARACTERSET and the character set in Java. When an Oracle database reads data of the nvarchar2 or nchar type, NLS_NCHAR_CHARACTERSET is required for encoding.

• Default value: {"AL16UTF16":"UTF16"}.

• Valid values: JSON strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] rtrimOracleChar

• Description: specifies whether to remove trailing spaces when the data type of the source is char. If the data is all spaces, reserve one space. The maximum length of the char type is 2000. When the character sets of the source and target are inconsistent, the data may exceed the length limit and cannot be written to the target.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] rtrimDB2Graphic

• Description: specifies whether to remove trailing spaces for data of the graphic type read from a DB2 database.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] indexHint

• Description: specifies whether to use hints when the source contains index hints.

• Default value: true.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] intervalTypeFmt

• Description: specifies whether to adjust the interval type format for Oracle in the following cases:

  • The source is an Oracle database but the target is not an Oracle database or an Oracle-compatible tenant of OceanBase Database.

  • The target is an Oracle database but the source is not an Oracle database or an Oracle-compatible tenant of OceanBase database.

    {"intervalDayDef":1,"intervalToSecondDef":1,"intervalYearDef":1,"intervalToMonthDef":2,"intervalSign":true,"intervalDayFmttime":2}

  • The source is an Oracle-compatible tenant of OceanBase Database but the target is not an Oracle database or an Oracle-compatible tenant of OceanBase Database.

  • The target is an Oracle-compatible tenant of OceanBase Database but the source is not an Oracle database or an Oracle-compatible tenant of OceanBase Database.

    {"intervalDayDef":2,"intervalToSecondDef":6,"intervalYearDef":1,"intervalToMonthDef":2,"intervalSign":true,"intervalDayFmttime":2}

• Default value: None.

• Valid values: JSON strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] timestamptzConvert

• Description: specifies whether to convert the city-based time zone to the hour-based time zone when reading data from an Oracle database or an Oracle-compatible tenant of OceanBase Database if the data type is a timestamp with a time zone. Example: Asia/Shanghai -> GMT+08:00

• Default value: true.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] consistencyMigration

• Description: specifies whether to lock full migration snapshots when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V4.1.1-CE and later.

[Data pulling] binlogOffsetDb

• Description: specifies the database for storing binlog offsets when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database.

• Default value: None.

• Value type: String.

• Supported version: OMS Community Edition V4.1.1-CE and later.

[Data pulling] binlogOffsetTable

• Description: specifies the table for storing binlog offsets when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database.

• Default value: None.

• Value type: String.

• Supported version: OMS Community Edition V4.1.1-CE and later.

[Data pulling] externalTable

• Description: specifies whether to enable external tables.

• Default value: true.

• Value type: Boolean.

• Supported version: OMS Community Edition V4.2.11-CE and later.