DataFlowSource

Starter class

connector-dataflow.jar

com.oceanbase.oms.connector.source.dataflow.DataFlowSourceBooster

[Metadata] clients

• Description: specifies the data source information.

• Default value: None. This parameter is required.

• Value range:

  {
    "clientId": "clientId, unique identifier",
    "cluster": "OceanBase cluster",
    "tenant": "OceanBase Database tenant",
    "instance": "ip:port/[schema]", // The schema must exist in the PostgreSQL, DB2 LUW, or Oracle database.
    "tableWhiteLists": { // This parameter takes effect only for logical tables. Physical tables depend on the Conditions parameter. Therefore, this parameter does not take effect for physical tables.
      "db": ["tab", "tab2"]
    },
    "username": "Username",
    "password": "Password",
    "weakRead": "false", // Optional. Default value: false
    "sysInstance": "Instance of the sys tenant,ip:port/[schema]", // Optional. Default value: Null.
    "sysTenantUsername": "Username of the sys tenant", // Optional. Default value: Null.
    "sysTenantPassword": "Password of the sys tenant" // Optional. Default value: Null.
  }
  

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Metadata] dbType

• Description: specifies the type of the database to be synchronized.

• Default value: None. This parameter is required.

• Value range: The value range of DbTypeEnum.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] maxConnNum

• Description: specifies the maximum number of connections.

• Default value: 32.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] databaseMaxConnection

• Description: specifies the maximum number of connections.

• Default value: 128.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] connTimeoutSec

• Description: specifies the timeout period of connections.

• Default value: 120.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceWorkerNum

• Description: specifies the number of threads for data sharding.

• Default value: 16.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceQueueSize

• Description: specifies the maximum number of data sharding queues.

• Default value: 128.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] batchSize

• Description: specifies the number of records in a data shard for batch writing.

  For example, if the number of records in each data shard (sliceBatchSize) is 2000, the value 200 of batchSize specifies to write 200 records in each batch to the destination.

• Default value: 200.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] sourceBatchMemorySize

• Description: specifies the maximum memory for each data shard, in bytes.

• Default value: 16777216.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] workerNum

• Description: specifies the number of worker threads for pulling data.

• Default value: 16.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nonePkUkSupportRowsMax

• Description: specifies the maximum number of records without a primary key.

• Default value: 5000000.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection parameters] timezone

• Description: specifies the time zone.

• Default value: UTC+8:00.

• Valid values: time zones.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] sourceMaxRetryTimes

• Description: specifies the maximum allowed number of retries to obtain data.

• Default value: 3.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] retryIntervalTime

• Description: specifies the retry interval.

• Default value: 3.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Filtering] filterTmpTable

• Description: specifies whether to filter temporary tables.

• Default value: true.

• Value type: Boolean.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceBlocks

• Description: specifies the number of blocks in each data shard.

• Default value: 128.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data sharding] sliceBatchSize

• Description: specifies the number of records in each data shard.

• Default value: 2000.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Connection information] charset

• Description: specifies the character set.

• Default value: utf8.

• Valid values: character sets.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] maxSqlRetryTime

• Description: specifies the maximum allowed number of retries to pull data.

• Default value: 3600.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Error handling] maxSqlNoActiveTime

• Description: specifies the JDBC timeout period.

• Default value: 700.

• Value type: int.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nullReplaceEnable

• Description: specifies the whether to enable character replacement in a scenario where the ORACLE gb18030 field value is chr(129)-chr(254), a NOT NULL constraint is present, and the value read by JDBC is NULL.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] nullReplaceString

• Description: specifies whether to replace the queried value with the specified string in a scenario where the ORACLE gb18030 field value is chr(129)-chr(254), a NOT NULL constraint is present, and the value read by JDBC is NULL. when nullReplaceEnable is set to true

• Default value: " "

• Valid values: Characters and strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] ncharCharsetMap

• Description: specifies the mapping between NLS_NCHAR_CHARACTERSET and the character set in Java. When an Oracle database reads data of the nvarchar2 or nchar type, NLS_NCHAR_CHARACTERSET is required for encoding.

• Default value: {"AL16UTF16":"UTF16"}.

• Valid values: JSON strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] rtrimOracleChar

• Description: specifies whether to remove trailing spaces when the data type of the source is char. If the data is all spaces, reserve one space. The maximum length of the char type is 2000. When the character sets of the source and destination are inconsistent, the data may exceed the length limit and cannot be written to the destination.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] rtrimDB2Graphic

• Description: specifies whether to remove trailing spaces for data of the graphic type read from a DB2 database.

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] indexHint

• Description: specifies whether to use hints when the source contains index hints.

• Default value: true.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Preprocessing] sourceJdbcTableCache

• Description: specifies whether to cache the preprocessing result when the source is a database.

• Default value: false.

• Valid values: true and false.

  The value true indicates that the preprocessing result is cached. In this case, the tables in the global cache are directly used. The value false indicates that the preprocessing result is not cached. In this case, you must obtain the tables to be migrated based on the source configuration.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Tables without a primary key] nonePkUkQueryWholeTable

• Description: specifies whether to query the whole table in full migration when the table has no unique index.

• Default value: true.

• Value type: Boolean.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Tables without a primary key] nonePkUkTruncateDstTable

• Description: specifies whether to truncate the destination table in full migration when the table has no unique index.

• Default value: true.

• Value type: Boolean.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] intervalTypeFmt

• Description: specifies whether to adjust the interval type format for Oracle in the following cases:

  • The source is an Oracle database but the destination is not an Oracle database or an Oracle tenant of OceanBase Database.

  • The destination is an Oracle database but the source is not an Oracle database or an Oracle tenant of OceanBase database.

    {"intervalDayDef":1,"intervalToSecondDef":1,"intervalYearDef":1,"intervalToMonthDef":2,"intervalSign":true,"intervalDayFmttime":2}

  • The source is an Oracle tenant of OceanBase Database but the destination is not an Oracle database or an Oracle tenant of OceanBase Database.

  • The destination is an Oracle tenant of OceanBase Database but the source is not an Oracle database or an Oracle tenant of OceanBase Database.

    {"intervalDayDef":2,"intervalToSecondDef":6,"intervalYearDef":1,"intervalToMonthDef":2,"intervalSign":true,"intervalDayFmttime":2}

• Default value: None.

• Valid values: JSON strings.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] timestamptzConvert

• Description: specifies whether to convert the city-based time zone to the hour-based time zone when reading data from an Oracle database or an Oracle tenant of OceanBase Database if the data type is a timestamp with a time zone. Example: Asia/Shanghai -> GMT+08:00

• Default value: true.

• Valid values: true and false.

• Supported version: OMS Community Edition V3.3.1-CE and later.

[Data pulling] consistencyMigration

• Description: specifies whether to lock full migration snapshots when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database

• Default value: false.

• Valid values: true and false.

• Supported version: OMS Community Edition V4.1.1-CE and later.

[Data pulling] binlogOffsetDb

• Description: specifies the database for storing binlog offsets when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database.

• Default value: None.

• Value type: String.

• Supported version: OMS Community Edition V4.1.1-CE and later.

[Data pulling] binlogOffsetTable

• Description: specifies the table for storing binlog offsets when incremental synchronization and full migration are performed for tables without a primary key in a MySQL database.

• Default value: None.

• Value type: String.

• Supported version: OMS Community Edition V4.1.1-CE and later.