Coordinator
| Parameter | Type | Default value | Description | Applicable versions |
|---|---|---|---|---|
| taskTwoActive | BOOLEAN | false | Specifies whether the task is an active-active disaster recovery task. This parameter is supported only in full migration. | OMS V3.3.1 ~ V4.3.0 |
| taskIdentity | STRING | 1 | The task identifier. | OMS V3.3.1 and later |
| checkDstTableEmpty | BOOLEAN | true | Specifies whether to check whether the target table is empty. | OMS V3.3.1 and later |
| precheckSinkSchema | BOOLEAN | true | Specifies whether to check the target schema. If the target schema is not checked, the precheck for the target schema is skipped. | OMS V3.3.1 and later |
| selectEmptyTableParallel | LONG | 16 | The number of parallel threads for checking whether tables are empty in full migration (DATAFLOW-SOURCE). | OMS V4.2.0 and later |
| timezone | STRING | +08:00 | The ETL time zone. | OMS V3.3.1 and later |
| systemDefaultTimezone | STRING | null | The time zone for reading timestamps. If not specified, the time zone for the current system is used. | OMS V3.3.1 and later |
| sourceType | STRING | null | The type of the source. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. | OMS V3.3.1 and later |
| consistencyMigration | BOOLEAN | false | Specifies whether to perform consistent migration for MySQL databases without primary keys during full and incremental migration. | OMS V3.3.1 and later |
| allowRecordTypes | STRING | null | The message types filtered at the task level, separated with commas (,). Valid values: ROW, DELETE, UPDATE, INSERT, HEARTBEAT, REPLACE, and DDL. | OMS V3.3.1 and later |
| tableFilterCacheSize | LONG | 512 | The LRU cache capacity of CreateTableIfNotExistDDLFilter. | OMS V3.3.1 and later |
| throttleMemoryBound | STRING | null | The memory limit. Unit: bytes. | OMS V3.3.1 and later |
| throttleRecordBound | STRING | null | The maximum number of records in the framework. | OMS V3.3.1 and later |
| throttleIOPS | STRING | null | The IOPS limit. Unit: bytes. | OMS V3.3.1 and later |
| throttleRps | STRING | null | The RPS limit. | OMS V3.3.1 and later |
| enablePreprocessConfig | BOOLEAN | true | Specifies whether to allow the system to automatically calculate the queue length based on the memory size. | OMS V4.0.2 and later |
| connectorJvmParam | STRING | null | The JVM parameters for starting the connector. | OMS V3.3.1 and later |
| bridgeQueueSize | LONG | 256 | The capacity of the intermediate cache queue. Note that the capacity is measured in RecordBatch. | OMS V3.3.1 and later |
| shuffleBucketSize | LONG | 128 | The number of buckets for the ShuffleRecordDispatcher. The number of buckets indicates the number of concurrent tasks. |
OMS V3.3.1 and later |
| batchAllInOneTable | BOOLEAN | false | Specifies whether all messages in a batch are sourced from the same table. The default value is true for full migration. |
OMS V3.3.1 and later |
| shuffleFlushIntervalMs | LONG | recommended value logic | The interval for batch accumulation. Unit: ms. | OMS V3.3.1 and later |
| shuffleMinBatchSize | LONG | 20 | The minimum batch size for the ShuffleRecordDispatcher.The ShuffleRecordDispatcher is a message distributor that synchronizes messages to a message queue (Kafka, RocketMQ, or DataHub). It is mainly designed to improve the concurrency efficiency of messages. |
OMS V3.3.1 and later |
| shuffleDdlToAllRoute | BOOLEAN | false | For the ShuffleRecordDispatcher, specifies whether to synchronize DDL statements to all queues when a DDL statement is encountered. |
OMS V4.0.1 and later |
| shuffleMaxBatchSize | LONG | 64 | The maximum batch size for the ShuffleRecordDispatcher. |
OMS V3.3.1 and later |
| shuffleBucketMaxInFlightBatchSize | LONG | 2147483647 | The maximum number of unacknowledged messages allowed in a bucket for the ShuffleRecordDispatcher. |
OMS V3.3.1 and later |
| printTraceMsg | BOOLEAN | recommended value logic | Specifies whether to print msg logs. Trace synchronization prints msg logs by default. Full migration does not print msg logs by default. | OMS V3.3.1 and later |
| loggerLevel | STRING | INFO | The log level, which can be DEBUG, INFO, WARN, or ERROR. | OMS V3.3.1 and later |
| destSubId | STRING | null | The sub-ID of the target. This parameter is compatible with JDBCWriter. | OMS V3.3.1 and later |
| daemonIp | STRING | null | The IP address of the connector. This parameter is compatible with JDBCWriter. | OMS V3.3.1 and later |
| destSubTopic | STRING | null | The subtopic of the target. This parameter is compatible with JDBCWriter. | OMS V3.3.1 and later |
| checkpointUseTaskName | BOOLEAN | false | Specifies whether to use the light-connector format for the checkpoint file name. | OMS V3.3.1 and later |
| daemonPort | STRING | null | The port number of the connector. If not specified, the port number 9000 is used. This parameter is compatible with JDBCWriter. | OMS V3.3.1 and later |
| enableMetricReportTask | BOOLEAN | true | Specifies whether to report checkpoints. If you set enableOmsConnectorV2Report to true, you can set this parameter to false. |
OMS V3.3.1 and later |
| listenPort | LONG | 0 | The HTTP listening port. | OMS V3.3.1 and later |
| httpBacklog | LONG | 16 | The tcpBackLog value, which indicates the length of the TCP connection queue. | OMS V3.3.1 and later |
| taskName | STRING | null | The task name. | OMS V3.3.1 and later |
| caseStrategy | STRING | FOLLOW-SOURCE | The case sensitivity strategy. Valid values: DEFAULT (match after query), FOLLOW_SOURCE (consistent with source), UPPER (convert to uppercase), and LOWER (convert to lowercase). | OMS V3.3.1 and later |
| isCaseSensitive | BOOLEAN | false | Specifies whether the system is case sensitive. | OMS V3.3.1 and later |
| dbTableCaseStrategy | STRING | FOLLOW-SOURCE | The case sensitivity strategy for databases and tables. Valid values: DEFAULT (match after query), FOLLOW_SOURCE (consistent with source), UPPER (convert to uppercase), and LOWER (convert to lowercase). | OMS V4.2.2 and later |
| columnCaseStrategy | STRING | FOLLOW-SOURCE | The case sensitivity strategy for columns. Valid values: DEFAULT (match after query), FOLLOW_SOURCE (consistent with source), UPPER (convert to uppercase), and LOWER (convert to lowercase). | OMS V4.2.2 and later |
| noneedRetryException | STRING | null | The exceptions that do not require retries during full migration precheck. | OMS V3.3.1 and later |
| runIgnoreError | BOOLEAN | false | Specifies whether to continue running when an error occurs during table synchronization. | OMS V3.3.1 and later |
| maxRecordCapacity | LONG | 16384 | The size of the dispatcher queue. | OMS V3.3.1 and later |
| dispatcherClassName | STRING | recommended value logic, depends on sinkType and hotKeyMerge |
The dispatcher implementation class. This parameter is usually left to the system to select. The default value for full migration is MetricDispatcher. The default value for incremental migration is DefaultTransactionScheduler. If the target is a message queue, the default value is ShuffleRecordDispatcher. Hot key merging and partition writing affect the selection of this parameter. |
OMS V3.3.1 and later |
| enableActiveReportTask | BOOLEAN | true | Specifies whether to report drc.active.xxx files. If you set enableOmsConnectorV2Report to true, you can set this parameter to false. |
OMS V3.3.1 and later |
| reportIntervalSec | LONG | 5 | The interval for outputting checkpoint, heartbeat, and diagnostic files. | OMS V3.3.1 and later |
| enableDmlDdlTypeStatistics | BOOLEAN | false | Specifies whether to enable DDL/DML statistics collection. | OMS V3.3.1 and later |
| enableOmsConnectorV2Report | BOOLEAN | false | Enables the new checkpoint format provided after components are integrated. The connector file name is changed to oms.connector.checkpoint. To ensure compatibility with the previous connectors, the default value is false. After most components are successfully upgraded, the default value will be changed to true. |
OMS V4.0.1 and later |
| reportMetricsIntervalSec | LONG | 10 | The interval for outputting metrics logs. | OMS V3.3.1 and later |
| castConfig | STRING | null | The configurations for serialization. bytes are written in base64, timestamp in sec.nanos, datetime in YYYY-MM-DD HH:mm:ss, and bit in base64. Details: {"bytesType": "BASE64: Convert bytes to base64 strings","timestampType": "SEC_NANO: sec.nanos with a maximum of 9 digits, in string format; SEC: seconds, in long format; MILLI: milliseconds, in long format; MILLI_STR: milliseconds, in string format;DATETIME_NORMAL: YYYY-MM-DD HH:mm:ss","dateTimeType": "ISO: YYYY-MM-DD HH:mm:ss with T in between; NORMAL: YYYY-MM-DD HH:mm:ss; TIMESTAMP_MILL: milliseconds, in long format","bitType": "NUMBER: Convert bits to longs and write them down;BASE64: Convert bits to base64 strings and write them down","appendZoneId": "Whether to add the zone ID. The default value is true"} |
OMS V3.3.1 and later |
Condition
The format of the Condition module is as follows:
{
"briefWhiteCondition": [], // The simplified allowlist configuration. If it exists, the store filter rule is constructed based on this parameter.
"whiteCondition": [], // The allowlist configuration.
"blackCondition": [] // The blocklist configuration.
}
For example, if you want to synchronize data from source_db.source_table to sink_db.sink_table, the format of the Condition module is as follows.
{
"whiteCondition": [
{
"name": "source_db",
"map": "sink_db",
"all": false,
"subs": [
{
"name": "source_table",
"map": "sink_table"
}
]
}
],
"blackCondition": [
]
}
Condition is a configuration item with multi-layer nested structures, where the first layer is database, the second layer is table, and the third layer is column.
| Parameter | Description |
|---|---|
| name | The name of the source object at the current layer. For example, at the first layer, it is the name of the database; at the second layer, it is the name of the table. |
| map | The name of the object to be written to the target. If the target is a database, the meaning is the same as that at the source. If the target is a message queue, for Kafka and RocketMQ, which only have the namespace for topics, the map value at the table layer is the name of the table. For DataHub that has both project and topic mappings, the corresponding value is the database and table. Note that the columns map parameter indicates the mechanism for renaming column names. |
| all | Specifies whether to synchronize objects at the lower layer. For example, if all is true at the database layer, it means that all tables under this database need to be synchronized, otherwise only the tables defined in subs need to be synchronized. |
With the parameters described above, you can meet the needs of most migration scenarios. However, OMS also supports wildcard and regular expression matching and attrs capabilities on this basis.
(Wildcard/Regular Expression) Matching
{
"whiteCondition": [
{
"name": "source_*",
"map": "sink_db",
"func": "fn",
"all": false,
"subs": [
{
"name": "source_*",
"map": "sink_table",
"func": "fn"
}
]
}
],
"blackCondition": [
{
"name": "source_db_[0-10]",
"map": "sink_db",
"func": "reg",
"all": false,
"subs": [
{
"name": "source_table_[0-10]",
"map": "sink_table",
"func": "reg"
}
]
}
]
}
The func parameter specifies the matching function. The default function is the string equals function. reg indicates regular expression matching, and fn indicates fnmatch matching.
In the example, all tables whose names start with source in all databases whose names start with source need to be synchronized to sink_db.sink_table. However, source_table_0 to source_table_10 in source_db_0 to source_db_10 need to be excluded.
attrs capability
Filter DML statements
{ "whiteCondition": [ { "name": "source_db", "map": "sink_db", "all": false, "attr": { "dml": "i,u,d" }, "subs": [ { "name": "source_table", "map": "sink_table" } ] } ], "blackCondition": [] }The value
dml:i,u,dinattrspecifies to synchronize only INSERT, DELETE, and UPDATE change messages. Note that full messages do not belong to INSERT change messages.The attributes of
attrinherit the default values of the parent level if they are not specified at the current level. For example, ifattris not specified at the table level, Flink SQL readsattrfrom a parent level, such as the database level.Filter DDL statements
{ "whiteCondition": [ { "name": "source_db", "map": "sink_db", "all": false, "subs": [ { "name": "source_table", "map": "sink_table", "attr": { "ddl": "ct,dt" } } ] } ], "blackCondition": [] }The value
ddl: ct,dtinattrspecifies to accept onlyCREATE TABLEandDROP TABLEDDL statements.Convert data by using SQL syntax
{ "whiteCondition": [ { "name": "source_db", "map": "sink_db", "all": false, "subs": [ { "name": "source_table", "map": "sink_table", "attr": { "op": "select substr(c1, 1, 2) as c3 from record where c2 = x" } } ] } ], "blackCondition": [] }The value of
opinattrspecifies the columns on which record computations and column filtering are performed by using SQL syntax. For example, in the preceding example,select substr(c1, 1, 2) as c3 from record where c2 = xspecifies the following operations forsource_db.source_table:Filter data whose c2 value is x.
Truncate the c1 column to 2 decimal places, then rename it as c3 and pass it downward. Note that the c1 column still exists at this point.
Attributes specific to a message queue scenario
{ "whiteCondition": [ { "name": "source_db", "all": false, "subs": [ { "name": "source_table", "map": "topic_1", "attr": { "shard_fields": "c1,c2", "ddl_map": "topic_2" } } ] } ], "blackCondition": [] }shard_fieldsspecifies the c1 and c2 columns as the sharding columns ofsource_table. The sharding columns are used to determine which partition, shard, or queue the data is delivered to.ddl_mapspecifies that whensource_tableundergoes a DDL change, a change message needs to be delivered to topic_2.
Parameter reference for the MySQL compatible mode of OceanBase Database as the target
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| enablePartitionBucket | BOOLEAN | false | Specifies whether to enable the partition write capability. | OMS V3.3.1 and later |
| sysInstance | STRING | null | The connection information of the database service for the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| maxRetryTimes | LONG | 1000 | The maximum number of retries when sink writing fails. | OMS V3.3.1 and later |
| useSSL | BOOLEAN | false | Specifies whether to use SSL. | OMS V3.3.1 and later |
| retryIntervalTime | LONG | 1000 | The interval between retries in milliseconds. | OMS V3.3.1 and later |
| transactionTableReserveDays | LONG | 7 | The number of days to retain transaction tables. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.jdbc.sink.JdbcSinkRouteComputeFunc |
The name of the class that implements the partitioning function. This parameter must be configured when you use the ShuffleDispatcher dispatching strategy. |
OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of worker threads for writing data. | OMS V3.3.1 and later |
| ignoreRedunantColumnsReplicate | BOOLEAN | false | Specifies whether to allow writing data when the source has more columns than the target. If you set the parameter to true, data writing is performed based on the columns in the source, and the excess columns are ignored. If you set the parameter to false, an error is returned when the source has more columns than the target. | OMS V3.3.1 and later |
| correctMode | LIST | null | The compatible format for writing incompatible data to the target. The OBORACLE data source type is implemented. The following enumerations are supported, and the enumerations are separated with commas (,).
|
OMS V4.2.2 and later |
| delayThresholdToCompareAllColumns | LONG | 10 | The threshold of incremental latency in seconds for switching from column comparison to primary key comparison when isCompareAllColumns is set to true. |
OMS V3.3.1 and later |
| jdbcUrl | STRING | null | The connection string of the target. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone. | OMS V3.3.1 and later |
| useTxId | BOOLEAN | false | Specifies whether to use transaction IDs. | OMS V3.3.1 and later |
| canGeometryBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether to batch write geometry data. | OMS V3.3.1 and later |
| binlogOffsetDb | STRING | null | The database where binlogOffset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| binlogOffsetTable | STRING | binlog_offset | The table where binlogOffset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| isCompareAllColumns | BOOLEAN | false | Specifies whether to enable full-column comparison, that is, whether to add all columns to the WHERE condition in incremental UPDATE and DELETE operations. | OMS V3.3.1 and later |
| password | STRING | null | The password of the JDBC data source. | OMS V3.3.1 and later |
| sysPassword | STRING | null | The password of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| disableForeignKeyCheck | BOOLEAN | true | Disables foreign key check. | OMS V3.3.1 and later |
| enableTransactionWriter | BOOLEAN | true | Specifies whether to enable transaction write capability. | OMS V3.3.1 and later |
| isUkDependSource | BOOLEAN | false | For the incremental logs of an Oracle or PostgreSQL source, the primary key may be missing. In this case, all columns in the source whose values can be determined by indexes in the target are selected. The system automatically calculates the value of this parameter based on the source. | OMS V3.3.1 and later |
| clientQueryPartitionWithInnerTable | BOOLEAN | true | Specifies whether to use an internal table to query partition metadata. | OMS V4.2.1 and later |
| txnDb | STRING | * | The database where the transaction table scheme, which is used to prevent replicated data from being cyclically replicated, is stored. | OMS V3.3.1 and later |
| dependTargetSchema | BOOLEAN | true | Specifies whether to write data to the target based on the schema of the target. The default value is true. If you set the parameter to false, data is not written to the target by querying the target. In some cases, you may need to set this parameter to false when the target is an OceanBase database. | OMS V3.3.1 and later |
| enableNoUniqueConstraintTableReplicate | BOOLEAN | false | Specifies whether to allow synchronization of tables without unique constraints. This parameter affects only the forward and reverse data transmission between a MySQL source and a pbMySQL or PG source. | null |
| sysUsername | STRING | null | The username of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. | OMS V3.3.1 and later |
| localRegionNo | STRING | null | The written regionNo. |
OMS V3.3.1 and later |
| dryRun | BOOLEAN | false | Specifies whether to perform a dry run. In a dry run, no SQL statement is executed. | OMS V3.3.1 and later |
| username | STRING | null | The username of the JDBC data source. | OMS V3.3.1 and later |
| filterHiddenPK | BOOLEAN | false | The parameter is valid only when enableNoUniqueConstraintTableReplicate is set to true. If the conditions are met, this parameter determines whether to ignore hidden primary keys of tables without unique constraints in an OceanBase database during data writing to the target. If you set the parameter to true, hidden columns are filtered out during data writing to the target. |
OMS V3.3.1 and later |
| sqlMode | STRING | null | The SQL mode. | OMS V3.3.1 and later |
| charset | STRING | depends on the values of jdbcUrl, username, password, and sinkType |
The character set of the connection. | OMS V3.3.1 and later |
| writeMode | STRING | batch | The write mode of the target. Valid values: batch and non-batch. | OMS V3.3.1 and later |
| isPreLoadAllTableSchema | BOOLEAN | false | Specifies whether to load all table schemas of the target to the memory for queries when dependTargetSchema is set to true. |
OMS V4.2.2 and later |
| boosterClass | STRING | null | The name of the class that serves as the startup class. Value: com.oceanbase.oms.connector.jdbc.sink.obmysql.OBMySQLJDBCSinkBooster. |
OMS V3.3.1 and later |
| isMysqlReplaceMode | BOOLEAN | false | Specifies whether to use the REPLACE mode for writing data to a MySQL database or the MySQL compatible mode of OceanBase Database. | OMS V3.3.1 and later |
| canLobBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether the write end can batch write LOB data. Currently, this parameter is set to false for the Oracle and OBORACLE data source types because the empty values of the LOB data type need to be written by using the empty_clob or empty_blob function. | OMS V3.3.1 and later |
| slowSqlThreshold | LONG | 15 | The threshold of slow SQL, which controls the execution of SQL statements. | OMS V3.3.1 and later |
| printSql | BOOLEAN | false | Specifies whether to print executed SQL statements. | OMS V3.3.1 and later |
| tenant | STRING | null | The tenant. | OMS V3.3.1 and later |
| obFreeMemoryMin | LONG | 20 | The minimum value of this parameter is 0. This parameter specifies to detect the free memory of the OceanBase database at the target. If the free memory of the OceanBase database is less than the specified value, the writing is suspended. The suspension continues until the next detection result meets the requirement. | OMS V3.3.1 and later |
| highWatermarkQueryIntervalSec | LONG | 1 | If obFreeCpuMin or obFreeMemoryMin is greater than 0, the system queries the free CPU or memory of the OceanBase database to the local storage every highWatermarkQueryIntervalSec seconds. | OMS V3.3.1 and later |
| obFreeCpuMin | LONG | 0 | The minimum value of this parameter is 0. This parameter specifies to detect the minimum free CPU of the OceanBase database at the target. If the free CPU of the OceanBase database is less than the specified value, the writing is suspended. The suspension continues until the next detection result meets the requirement. Note that the collection of CPU information may be inaccurate. Therefore, we recommend that you do not set this parameter to a specific value. | OMS V3.3.1 and later |
| retryErrorMessage | STRING | null | The error message that indicates that the task needs to be retried. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code that indicates that the task needs to be retried. | OMS V3.3.1 and later |
Parameter reference for the Oracle compatible mode of OceanBase Database as the target
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| enablePartitionBucket | BOOLEAN | false | Specifies whether to enable partition write capability. | OMS V3.3.1 and later |
| sysInstance | STRING | null | The connection information of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| obSupportNlsSet | BOOLEAN | true | Specifies whether to reset the NLS settings. This parameter is used only when the target is the Oracle compatible mode of OceanBase Database. | OMS V3.3.1 and later |
| maxRetryTimes | LONG | 1000 | The maximum number of retries when sink writing fails. | OMS V3.3.1 and later |
| useSSL | BOOLEAN | false | Specifies whether to use SSL. | OMS V3.3.1 and later |
| retryIntervalTime | LONG | 1000 | The interval between retries in milliseconds. | OMS V3.3.1 and later |
| transactionTableReserveDays | LONG | 7 | The number of days for which a transaction table is retained. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.jdbc.sink.JdbcSinkRouteComputeFunc |
The name of the class that implements the partitioning function. This parameter must be configured when you use the ShuffleDispatcher dispatcher. |
OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of threads for writing data. | OMS V3.3.1 and later |
| ignoreRedunantColumnsReplicate | BOOLEAN | false | Specifies whether to allow writing data when the source has more columns than the target. If you set the parameter to true, data replication is performed without writing the columns that exist only in the source. If you set the parameter to false, an error is returned when the source has more columns than the target. | OMS V3.3.1 and later |
| correctMode | LIST | null | Specifies the compatibility format for writing incompatible data to the target. The OBORACLE compatible format is implemented. The following enumerations are separated with commas (,).
|
OMS V4.2.2 and later |
| delayThresholdToCompareAllColumns | LONG | 10 | The threshold in seconds for the incremental latency when isCompareAllColumns is set to true. In this case, the primary key matching mode is activated when the incremental latency falls below this threshold. |
OMS V3.3.1 and later |
| jdbcUrl | STRING | null | The connection string of the target. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone of the target. | OMS V3.3.1 and later |
| useTxId | BOOLEAN | false | Specifies whether to use transaction IDs. | OMS V3.3.1 and later |
| canGeometryBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether to batch-write geometry data. | OMS V3.3.1 and later |
| binlogOffsetDb | STRING | null | The database for storing binlog_offset in full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| binlogOffsetTable | STRING | binlog_offset | The table for storing binlog_offset in full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| isCompareAllColumns | BOOLEAN | false | Specifies whether to enable full-column comparison. In other words, in incremental UPDATE and DELETE operations, whether to add all columns to the WHERE condition. | OMS V3.3.1 and later |
| password | STRING | null | The password for connecting to the target by using JDBC. | OMS V3.3.1 and later |
| sysPassword | STRING | null | The password of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| disableForeignKeyCheck | BOOLEAN | true | Disables foreign key checks. | OMS V3.3.1 and later |
| enableTransactionWriter | BOOLEAN | true | Specifies whether to enable transaction writing. | OMS V3.3.1 and later |
| isUkDependSource | BOOLEAN | false | For the incremental logs of an Oracle or PostgreSQL source, the data of columns that can be selected from the target based on the schema of the source are written to the target. This parameter is automatically calculated by the system. | OMS V3.3.1 and later |
| clientQueryPartitionWithInnerTable | BOOLEAN | true | Specifies whether to use an internal table to query partition metadata. | OMS V4.2.1 and later |
| txnDb | STRING | * | The database where the transaction table is located. The transaction table is used to prevent data replication loops. | OMS V3.3.1 and later |
| dependTargetSchema | BOOLEAN | true | Specifies whether to depend on the schema of the target when writing data to the target. By default, this parameter is set to true. If you set the parameter to false, data is not written to the target unless the target exists. In some cases, you may need to set this parameter to false when the target is OceanBase Database. | OMS V3.3.1 and later |
| enableNoUniqueConstraintTableReplicate | BOOLEAN | false | Specifies whether to allow the synchronization of tables without unique constraints. This parameter affects only the forward and reverse data transmission between a MySQL source and a pbMySQL or PG source. | OMS V4.2.2 and later |
| sysUsername | STRING | null | The username of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. | OMS V3.3.1 and later |
| localRegionNo | STRING | null | The written regionNo. |
OMS V3.3.1 and later |
| dryRun | BOOLEAN | false | Specifies whether to perform a dry run. In a dry run, no SQL statements are executed. | OMS V3.3.1 and later |
| username | STRING | null | The username for connecting to the target by using JDBC. | OMS V3.3.1 and later |
| filterHiddenPK | BOOLEAN | false | The parameter is effective only when enableNoUniqueConstraintTableReplicate is set to true, and specific combinations of the source and targets are met. If you set the parameter to true, hidden primary keys of tables without unique constraints in OceanBase Database are ignored, and hidden columns are filtered out when data is written to the target. |
OMS V3.3.1 and later |
| sqlMode | STRING | null | The SQL mode of the target. | OMS V3.3.1 and later |
| charset | STRING | depends on the values of jdbcUrl, username, password, and sinkType |
The character set for connecting to the target. | OMS V3.3.1 and later |
| writeMode | STRING | batch | The write mode of the target. Valid values: batch and non-batch. | OMS V3.3.1 and later |
| isPreLoadAllTableSchema | BOOLEAN | false | Specifies whether to pre-load all table schemas of the target to the memory for queries when dependTargetSchema is set to true. |
OMS V4.2.2 and later |
| boosterClass | STRING | null | The name of the class that serves as the startup class. Value: com.oceanbase.oms.connector.jdbc.sink.oboracle.OBOracleJDBCSinkBooster. |
OMS V3.3.1 and later |
| canLobBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether the write end can batch-write LOB data. Currently, the Oracle and OBORACLE compatible formats do not support batch writing of LOB data because the empty values of LOB data must be written by using the empty_clob or empty_blob function. | OMS V3.3.1 and later |
| slowSqlThreshold | LONG | 15 | The threshold in seconds for slow SQL statements. This threshold controls the execution of SQL statements. | OMS V3.3.1 and later |
| printSql | BOOLEAN | false | Specifies whether to print executed SQL statements. | OMS V3.3.1 and later |
| tenant | STRING | null | The tenant. | OMS V3.3.1 and later |
| obFreeMemoryMin | LONG | 20 | The minimum value of this parameter is 0. This parameter specifies the minimum value of free memory of OceanBase Database to be detected at the target. If the free memory of OceanBase Database is less than this value, the writing is suspended. The suspension continues until the next detection result meets the requirement. | OMS V3.3.1 and later |
| highWatermarkQueryIntervalSec | LONG | 1 | If obFreeCpuMin or obFreeMemoryMin is greater than 0, this parameter specifies the interval in seconds between queries to OceanBase Database for the idle CPU or memory resources. The queried resources are stored in the local storage of the program. | OMS V3.3.1 and later |
| obFreeCpuMin | LONG | 0 | The minimum value of this parameter is 0. This parameter specifies the minimum value of idle CPU resources of OceanBase Database to be detected at the target. If the idle CPU resources of OceanBase Database are less than this value, the writing is suspended. The suspension continues until the next detection result meets the requirement. Note that the collection of CPU resources may be inaccurate. Therefore, we recommend that you do not set this parameter to a specific value. | OMS V3.3.1 and later |
| retryErrorMessage | STRING | null | The error message that indicates that the task needs to be retried. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code that indicates that the task needs to be retried. | OMS V3.3.1 and later |
Parameters for MySQL and PostgreSQL targets
| Parameter | Type | Default value | Description | Applicable versions |
|---|---|---|---|---|
| enablePartitionBucket | BOOLEAN | false | Specifies whether to enable partition writing. | OMS V3.3.1 and later |
| sysInstance | STRING | null | The connection information of the database service for the system tenant of OceanBase Database. | OMS V3.3.1 and later |
| maxRetryTimes | LONG | 1000 | The maximum number of retries when sink writing fails. | OMS V3.3.1 and later |
| useSSL | BOOLEAN | false | Specifies whether to use SSL. | OMS V3.3.1 and later |
| retryIntervalTime | LONG | 1000 | The interval between retries in milliseconds. | OMS V3.3.1 and later |
| transactionTableReserveDays | LONG | 7 | The number of days to retain transaction tables. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.jdbc.sink.JdbcSinkRouteComputeFunc |
The name of the class that implements the partitioning function. This parameter must be used with ShuffleDispatcher. The default value is empty. |
OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of write threads. | OMS V3.3.1 and later |
| ignoreRedunantColumnsReplicate | BOOLEAN | false | Specifies whether to allow writing when the source has more columns than the target. If you set the value to true, the system will ignore the columns that exist in the source but not in the target. If you set the value to false, the system will return an error if the source has more columns than the target. | OMS V3.3.1 and later |
| correctMode | LIST | null | The compatible format for writing incompatible data to the target. Currently, only OBORACLE supports this parameter. The following enumerations are separated with commas (,).
|
OMS V4.2.2 and later |
| delayThresholdToCompareAllColumns | LONG | 10 | The threshold in seconds for the incremental latency to switch from the current mode to the primary key matching mode when isCompareAllColumns is set to true. |
OMS V3.3.1 and later |
| jdbcUrl | STRING | null | The connection string of the target. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone. | OMS V3.3.1 and later |
| useTxId | BOOLEAN | false | Specifies whether to use transaction IDs. | OMS V3.3.1 and later |
| canGeometryBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether to batch-write geometries. | OMS V3.3.1 and later |
| binlogOffsetDb | STRING | null | The database where binlog_offset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| binlogOffsetTable | STRING | binlog_offset | The table where binlog_offset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| isCompareAllColumns | BOOLEAN | false | Specifies whether to enable full-column comparison, that is, whether to add all columns to the WHERE condition for incremental UPDATE and DELETE operations. | OMS V3.3.1 and later |
| password | STRING | null | The password for connecting to the database by using JDBC. | OMS V3.3.1 and later |
| sysPassword | STRING | null | The password of the system tenant of OceanBase Database. | OMS V3.3.1 and later |
| disableForeignKeyCheck | BOOLEAN | true | Specifies whether to disable foreign key check. | OMS V3.3.1 and later |
| enableTransactionWriter | BOOLEAN | true | Specifies whether to enable transaction writing. | OMS V3.3.1 and later |
| isUkDependSource | BOOLEAN | false | For an Oracle or PostgreSQL source, the incremental logs may be incomplete and lack primary keys. In this case, all columns in the source that can be selected from the target are used as the source based on the schemas in the target. This parameter is automatically calculated in this scenario. | OMS V3.3.1 and later |
| clientQueryPartitionWithInnerTable | BOOLEAN | true | Specifies whether to use an internal table to query partition metadata. | OMS V4.2.1 and later |
| txnDb | STRING | * | The database where the transaction table solution is located. This parameter is used to prevent data replication loops. | OMS V3.3.1 and later |
| dependTargetSchema | BOOLEAN | true | Specifies whether to depend on the schema of the target for writing to the target. The default value is true. If you set the value to false, the system will not query the target. In some cases where the target is OceanBase Database, you may need to set this parameter to false. | OMS V3.3.1 and later |
| enableNoUniqueConstraintTableReplicate | BOOLEAN | false | Specifies whether to synchronize tables without unique constraints. This parameter affects only the forward and reverse data transmission between MySQL and pbMySQL, or between PG and OBOracle. | OMS V4.2.2 and later |
| sysUsername | STRING | null | The username of the system tenant of OceanBase Database. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the data source of the target. | OMS V3.3.1 and later |
| localRegionNo | STRING | null | The written regionNo. |
OMS V3.3.1 and later |
| dryRun | BOOLEAN | false | Specifies whether to perform a dry run, that is, whether to execute SQL statements. | OMS V3.3.1 and later |
| username | STRING | null | The username for connecting to the database by using JDBC. | OMS V3.3.1 and later |
| filterHiddenPK | BOOLEAN | false | The hidden primary keys of tables without unique constraints in OceanBase Database are ignored during writing to the target. This parameter only takes effect when enableNoUniqueConstraintTableReplicate is set to true. If the combination of the source and target does not meet the requirements, this parameter will not take effect. |
OMS V3.3.1 and later |
| sqlMode | STRING | null | The SQL mode. | OMS V3.3.1 and later |
| charset | STRING | depends on the values of jdbcUrl, username, password, and sinkType |
The character set of the connection. | OMS V3.3.1 and later |
| writeMode | STRING | batch | The write mode of the target, including batch write and non-batch write. | OMS V3.3.1 and later |
| isPreLoadAllTableSchema | BOOLEAN | false | Specifies whether to pre-load all schemas of tables in the target into the memory for queries when dependTargetSchema is set to true. |
OMS V4.2.2 and later |
| boosterClass | STRING | null | The name of the startup class. The startup class for a MySQL database is com.oceanbase.oms.connector.jdbc.sink.mysql.MySQLJDBCSinkBooster, and that for a PostgreSQL database is com.oceanbase.oms.connector.jdbc.sink.pg.PgJDBCSinkBooster. |
OMS V3.3.1 and later |
| isMysqlReplaceMode | BOOLEAN | false | Specifies whether to use the REPLACE mode for writing to the target for a MySQL database or the MySQL compatible mode of OceanBase Database. | OMS V3.3.1 and later |
| canLobBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether the write end can batch-write LOBs. Currently, this feature is not supported for Oracle and OBORACLE because the empty values of LOBs must be written by using the empty_clob or empty_blob function. |
OMS V3.3.1 and later |
| slowSqlThreshold | LONG | 15 | The threshold in seconds for identifying a slow SQL statement. | OMS V3.3.1 and later |
| printSql | BOOLEAN | false | Specifies whether to print executed SQL statements. | OMS V3.3.1 and later |
| tenant | STRING | null | The tenant. | OMS V3.3.1 and later |
| obFreeMemoryMin | LONG | 20 | The minimum value of this parameter is 0. This parameter specifies to detect the free memory of the OceanBase database in the target. If the free memory of the OceanBase database in the target is less than this value, the writing is suspended. The writing is resumed when the next detection finds that this value is met again. Note that the collection of the free CPU may be inaccurate. Therefore, we recommend that you do not set this parameter to a specific value. | OMS V3.3.1 and later |
| highWatermarkQueryIntervalSec | LONG | 1 | If obFreeCpuMin or obFreeMemoryMin is greater than 0, the system queries the free CPU or memory of the OceanBase database to the local storage every obFreeCpuMin or obFreeMemoryMin seconds. | OMS V3.3.1 and later |
| obFreeCpuMin | LONG | 0 | The minimum value of this parameter is 0. This parameter specifies to detect the free CPU of the OceanBase database in the target. If the free CPU of the OceanBase database in the target is less than this value, the writing is suspended. The writing is resumed when the next detection finds that this value is met again. Note that the collection of the free CPU may be inaccurate. Therefore, we recommend that you do not set this parameter to a specific value. | OMS V3.3.1 and later |
| retryErrorMessage | STRING | null | The error message indicating that the task needs to be retried. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message indicating that the task needs to be skipped. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code indicating that the task needs to be skipped. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code indicating that the task needs to be retried. | OMS V3.3.1 and later |
Oracle target parameters
| Parameter | Type | Default value | Description | Applicable versions |
|---|---|---|---|---|
| enablePartitionBucket | BOOLEAN | false | Specifies whether to enable partition writing. | OMS V3.3.1 and later |
| sysInstance | STRING | null | The database connection information of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| maxRetryTimes | LONG | 1000 | The maximum number of retries when a sink write fails. | OMS V3.3.1 and later |
| useSSL | BOOLEAN | false | Specifies whether to use SSL. | OMS V3.3.1 and later |
| retryIntervalTime | LONG | 1000 | The interval between retries in milliseconds. | OMS V3.3.1 and later |
| transactionTableReserveDays | LONG | 7 | The number of days to retain transaction tables. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.jdbc.sink.JdbcSinkRouteComputeFunc |
The name of the class that implements the partitioning function. This parameter must be specified when you use the ShuffleDispatcher dispatching strategy. |
OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of write threads. | OMS V3.3.1 and later |
| ignoreRedunantColumnsReplicate | BOOLEAN | false | Specifies whether to allow writing when the source has more columns than the target. If you set the parameter to true, the system skips the writing of columns that exist only in the source. If you set the parameter to false, the system returns an error when the source has more columns than the target. | OMS V3.3.1 and later |
| correctMode | LIST | null | The compatible format for writing incompatible data to the target. Currently, only OBORACLE supports this parameter. The following values are supported, separated by commas (,):
|
OMS V4.2.2 and later |
| delayThresholdToCompareAllColumns | LONG | 10 | The threshold in seconds for the incremental delay when isCompareAllColumns is set to true. In this case, the primary key matching mode is activated when the incremental delay falls below this threshold. |
OMS V3.3.1 and later |
| jdbcUrl | STRING | null | The connection string of the Oracle database. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone. | OMS V3.3.1 and later |
| useTxId | BOOLEAN | false | Specifies whether to use transaction IDs. | OMS V3.3.1 and later |
| canGeometryBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether to batch-write geometry data. | OMS V3.3.1 and later |
| binlogOffsetDb | STRING | null | The database where binlog_offset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| binlogOffsetTable | STRING | binlog_offset | The table where binlog_offset is stored for full and incremental MySQL migrations without primary keys. | OMS V3.3.1 and later |
| isCompareAllColumns | BOOLEAN | false | Specifies whether to enable full-column comparison, that is, whether to add all columns to the WHERE condition in incremental UPDATE and DELETE operations. | OMS V3.3.1 and later |
| password | STRING | null | The password for connecting to the Oracle database by using JDBC. | OMS V3.3.1 and later |
| sysPassword | STRING | null | The password of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| disableForeignKeyCheck | BOOLEAN | true | Disables foreign key check. | OMS V3.3.1 and later |
| enableTransactionWriter | BOOLEAN | true | Specifies whether to enable transaction writing. | OMS V3.3.1 and later |
| isUkDependSource | BOOLEAN | false | The Oracle and PostgreSQL sources may lack incremental logs. In the case of missing primary keys, all columns in the source that can be referenced by indexes in the target are selected. The value of this parameter is automatically calculated in this scenario. | OMS V3.3.1 and later |
| clientQueryPartitionWithInnerTable | BOOLEAN | true | Specifies whether to use an internal table to query partition metadata. | OMS V4.2.1 and later |
| txnDb | STRING | * | The database where the transaction table solution is located. The transaction table solution is used to prevent replicated data from being replicated back to the source. | OMS V3.3.1 and later |
| dependTargetSchema | BOOLEAN | true | Specifies whether to write data to the target based on the schema of the source. The default value is true. If you set the parameter to false, the system does not query the target. In some cases, you may need to set this parameter to false for OceanBase Database. | OMS V3.3.1 and later |
| enableNoUniqueConstraintTableReplicate | BOOLEAN | false | Specifies whether to allow the synchronization of tables without primary keys. This parameter affects only the forward and reverse data transmission between MySQL and OBMySQL, or between PostgreSQL and OBOracle. | OMS V4.2.2 and later |
| sysUsername | STRING | null | The username of the sys tenant of OceanBase Database. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the data source at the target. | OMS V3.3.1 and later |
| localRegionNo | STRING | null | The written regionNo. |
OMS V3.3.1 and later |
| dryRun | BOOLEAN | false | Specifies whether to perform a dry run. In a dry run, no SQL statements are executed. | OMS V3.3.1 and later |
| username | STRING | null | The username for connecting to the Oracle database by using JDBC. | OMS V3.3.1 and later |
| filterHiddenPK | BOOLEAN | false | The parameter is effective only when enableNoUniqueConstraintTableReplicate is set to true, and specific combinations of source and target columns are met. The parameter specifies to ignore hidden primary keys of tables without primary keys in OceanBase Database during writing to the target. Hidden columns will be filtered out during writing. |
OMS V3.3.1 and later |
| sqlMode | STRING | null | The SQL mode. | OMS V3.3.1 and later |
| charset | STRING | depends on the values of jdbcUrl, username, password, and sinkType |
The character set of the connection. | OMS V3.3.1 and later |
| writeMode | STRING | batch | The write mode of the target. Valid values: batch and non-batch. | OMS V3.3.1 and later |
| isPreLoadAllTableSchema | BOOLEAN | false | Specifies whether to pre-load all table schemas of the target into the memory for queries when dependTargetSchema is set to true. |
OMS V4.2.2 and later |
| oracleSocketTimeout | LONG | 50000 | The timeout period of the sink for an Oracle database in milliseconds. | OMS V3.3.1 and later |
| boosterClass | STRING | null | The name of the class that implements the booster. Value: com.oceanbase.oms.connector.jdbc.sink.oracle.OracleJDBCSinkBooster. |
OMS V3.3.1 and later |
| canLobBatched | BOOLEAN | depends on the value of sinkType |
Specifies whether to batch-write LOB data. Currently, this parameter is invalid for Oracle and OBOracle. If you write LOB data of a NULL value to Oracle or OBOracle, you must use the empty_clob or empty_blob function. |
OMS V3.3.1 and later |
| slowSqlThreshold | LONG | 15 | The threshold in seconds for slow SQL. This threshold controls when to print the execution SQL statement. | OMS V3.3.1 and later |
| printSql | BOOLEAN | false | Specifies whether to print the execution SQL statement. | OMS V3.3.1 and later |
| tenant | STRING | null | The tenant. | OMS V3.3.1 and later |
| obFreeMemoryMin | LONG | 20 | The minimum value of this parameter is 0. This parameter specifies to detect the free memory of the target OceanBase database during writing. If the free memory of the target OceanBase database is less than the value of this parameter, the writing is suspended. Writing is resumed when the next detection finds that the value of this parameter is satisfied. Note that the collection of the free CPU may be inaccurate. Therefore, we recommend that you do not specify this parameter. | OMS V3.3.1 and later |
| highWatermarkQueryIntervalSec | LONG | 1 | If obFreeCpuMin or obFreeMemoryMin is greater than 0, the system queries the free CPU or memory of the target OceanBase database to the local storage every highWatermarkQueryIntervalSec seconds. | OMS V3.3.1 and later |
| obFreeCpuMin | LONG | 0 | The minimum value of this parameter is 0. This parameter specifies to detect the free CPU of the target OceanBase database during writing. If the free CPU of the target OceanBase database is less than the value of this parameter, the writing is suspended. Writing is resumed when the next detection finds that the value of this parameter is satisfied. Note that the collection of the free CPU may be inaccurate. Therefore, we recommend that you do not specify this parameter. | OMS V3.3.1 and later |
| retryErrorMessage | STRING | null | The error message that indicates that the task needs to be retried. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code that indicates that the task needs to be skipped. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code that indicates that the task needs to be retried. | OMS V3.3.1 and later |
Parameters for the target RocketMQ data source
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| clientThreadCount | LONG | -1 | The size of the asynchronous sending thread pool of the RocketMQ data source. The default value is calculated based on the number of CPU cores of the server. | OMS V3.3.1 and later |
| tags | STRING | null | The tags of the messages. If the message contains a table name, the tags are set to tableName + _tag by default. Otherwise, the tags are set to default. |
OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of write threads. | OMS V3.3.1 and later |
| clientCallbackExecutorThreadsNum | LONG | -1 | The number of threads that process requests in the RocketMQ data source. The default value is calculated based on the number of CPU cores of the server. | OMS V3.3.1 and later |
| enableTrace | BOOLEAN | false | The enableTrace parameter of the RocketMQ client. This parameter specifies whether to enable message tracing. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. | OMS V3.3.1 and later |
| nameSvrAddr | STRING | null | The nameSvrAddr parameter of the RocketMQ data source. This parameter specifies the IP address of the NameServer. |
OMS V3.3.1 and later |
| boosterClass | STRING | null | The startup class: com.oceanbase.oms.connector.rocketmq.sink.RocketMQSinkBooster. |
OMS V3.3.1 and later |
| producerGroup | STRING | OMS | The producer group of the RocketMQ data source. This parameter identifies and manages a collection of message producer instances that have the same functionality or responsibility. | OMS V3.3.1 and later |
| sendMsgTimeout | LONG | 30000 | The timeout period for sending messages, in ms. | OMS V3.3.1 and later |
| nameSpace | STRING | - | The namespace of the RocketMQ data source. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.rocketmq.sink.RocketMQShardFunc |
The implementation class of the partitioning function. | OMS V3.3.1 and later |
| partitionMode | STRING | HASH | The distribution logic that determines which partition each data record is written to. Valid values: HASH, which specifies to use the primary key hash value to determine the partition, and TABLE, which specifies to use the table name hash value to determine the partition. | OMS V3.3.1 and later |
| secretKey | STRING | null | The Secret Key of the RocketMQ data source. The Secret Key is used for authentication. | OMS V3.3.1 and later |
| accessKey | STRING | null | The Access Key ID (Access Key) of the RocketMQ data source. The Access Key is a unique string that identifies your user identity. | OMS V3.3.1 and later |
| serializerType | STRING | DEFAULT | The serialization format of the messages written to the message queue. Valid values: DEFAULT, DEFAULT_WITH_SCHEMA, SHAREPLEX, CANAL, DATAWORKS_V2, and DATAWORKS_V1. | OMS V3.3.1 and later |
| metaExtend | JSON | null | The extended JSON data. This parameter takes effect only when extend is written to a DataWorks V2 data source. |
OMS V3.3.1 and later |
| serializerDataTypeMapper | JSON | null | The mapping relationship between SchemaType and STRING. This parameter is used to resolve the INT64 issue. | OMS V4.1.0 and later |
| maxRetryTimes | LONG | 1000 | The maximum number of retries. | OMS V3.3.1 and later |
| retryIntervalTime | LONG | 1000 | The interval between retries, in ms. | OMS V3.3.1 and later |
| retryErrorMessage | STRING | null | The error message that requires a retry. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message that requires skipping. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code that requires skipping. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code that requires a retry. | OMS V3.3.1 and later |
Parameters for the target Kafka
| Parameter | Type | Default value | Description | Applicable versions |
|---|---|---|---|---|
| shardFuncClassName | STRING | com.oceanbase.oms.connector.kafka.sink.KafkaShardFunc | The name of the partitioning function class. This parameter works with ShuffleDispatcher. The default value is empty. |
OMS V3.3.1 and later |
| workerNum | LONG | Recommended values exist, which depends on the configuration of partitionMode. |
The number of write threads. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. | OMS V3.3.1 and later |
| dryRun | BOOLEAN | false | Specifies not to actually write data to the Kafka. | OMS V3.3.1 and later |
| batchSize | LONG | 1048576 | The value of the batch.size parameter of the producer in Kafka. This parameter specifies the message batch size. |
OMS V3.3.1 and later |
| properties | JSON | null | The properties of Kafka, namely, the configuration file of Kafka. | OMS V3.3.1 and later |
| consumerGroup | STRING | test | This parameter is required when you perform a check on the existence of a topic. In this case, you need to create a Kafka consumer. This parameter is the value of the configuration parameter group.id of the consumer. |
OMS V3.3.1 and later |
| servers | STRING | null | The value of the bootstrap.servers parameter of the producer in Kafka. This parameter specifies the list of servers that the producer needs to connect to when it establishes an initial connection with the Kafka cluster. |
OMS V3.3.1 and later |
| boosterClass | STRING | null | The name of the startup class, which is com.oceanbase.oms.connector.kafka.sink.KafkaSinkBooster. |
OMS V3.3.1 and later |
| partition | LONG | 0 | If partitionMode is set to ONE, this parameter specifies the name of the fixed partition to which data is written. |
OMS V3.3.1 and later |
| kafkaVersion | STRING | 2.0 | The version of the Kafka client. Valid values: 0.9 and 2.0. | OMS V3.3.1 and later |
| lingerMs | LONG | 100 | The value of the linger.ms parameter of the producer in Kafka. This parameter specifies the message batch processing behavior. |
OMS V3.3.1 and later |
| enableAsync | BOOLEAN | true | Specifies whether to use asynchronous interface methods for writing. | OMS V3.3.1 and later |
| isTopicMandatory | BOOLEAN | true | Specifies whether to check the existence of a topic. | OMS V3.3.1 and later |
| partitionMode | STRING | HASH | The distribution logic that determines which partition each data record is written to. Valid values: HASH (primary key hash) and TABLE (table name hash). | OMS V3.3.1 and later |
| sslTruststoreLocation | STRING | null | The path of the SSL authentication file. | OMS V3.3.1 and later |
| saslConfig | JSON | null | The SASL configurations. {"mechanism": "the authentication mechanism, such as GSSAPI, PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","username": "the username required when the authentication mechanism is PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","password": "the password required when the authentication mechanism is PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","kerberosPrincipal": "the Kerberos principal","kerberosKeyTabFileLocation": "the path of the Kerberos keytab file. This parameter is required when the authentication mechanism is GSSAPI, and indicates the path of the authentication configuration file.","jaasConfFileLocation": "the path of the authentication configuration file when the authentication mechanism is GSSAPI","kerberosConfFileLocation": "the path of the krb conf file when the authentication mechanism is GSSAPI"} |
OMS V3.3.1 and later |
| securityProtocol | STRING | null | The authentication protocol. Valid values: SASL_PLAINTEXT, SASL_SSL, PLAINTEXT_SASL, and SSL_SASL. | OMS V3.3.1 and later |
| serializerType | STRING | DEFAULT | The serialization format for messages written to the message queue. Valid values: DEFAULT, DEFAULT_WITH_SCHEMA, SHAREPLEX, CANAL, DATAWORKS_V2, and DATAWORKS_V1. | OMS V3.3.1 and later |
| metaExtend | JSON | null | JSON extended data. This parameter is effective only for the extend field in DataWorks V2. |
OMS V3.3.1 and later |
| serializerDataTypeMapper | JSON | null | A mapping from SchemaType to STRING to resolve the INT64 issue. | OMS V4.1.0 and later |
| retryErrorMessage | STRING | null | Error messages that require retries. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | Error messages that require skips. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | Error codes that require skips. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | Error codes that require retries. | OMS V3.3.1 and later |
Parameters for the DataHub target
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| sequenceEnable | BOOLEAN | false | Specifies whether to enable the sequence number generation feature of DataWorks. | OMS V3.3.1 and later |
| requestRetrySec | LONG | 3600 | The retry interval in seconds when a write request fails. | OMS V3.3.1 and later |
| shardFuncClassName | STRING | com.oceanbase.oms.connector.datahub.sink.DatahubShardFunc |
The name of the partition function class. This parameter must be used with ShuffleDispatcher. The default value does not need to be specified. |
OMS V3.3.1 and later |
| enableBinary | BOOLEAN | true | Specifies whether to enable binary write for the DataHub client. | OMS V3.3.1 and later |
| workerNum | LONG | 16 | The number of write threads. | OMS V3.3.1 and later |
| accessKey | STRING | null | The AccessKey ID, which is a unique string used to identify your identity. This parameter is used for a DataHub target. | OMS V3.3.1 and later |
| connectionTimeoutMs | LONG | 10000 | The timeout period of the DataHub client. | OMS V3.3.1 and later |
| sinkType | STRING | null | The type of the target. This parameter is used for a DataHub target. | OMS V3.3.1 and later |
| sequenceStartTimestamp | LONG | 0 | The initial value of the sequence, which is used for server migration. Generally, you do not need to specify the value of this parameter if the timestamps on the servers are consistent. | OMS V3.3.1 and later |
| boosterClass | STRING | null | The name of the startup class, which is com.oceanbase.oms.connector.datahub.sink.DatahubSinkBooster. |
OMS V3.3.1 and later |
| accessId | STRING | null | The AccessKey ID of the DataHub instance. | OMS V3.3.1 and later |
| endpoint | STRING | null | The endpoint of the DataHub instance. | OMS V3.3.1 and later |
| sequenceIgnoreLocal | BOOLEAN | false | Specifies whether to ignore the local sequence. | OMS V3.3.1 and later |
| tooLargeRequestErrorCodes | LIST | [RequestEntityTooLarge, TooLargePayload] | Error codes indicating that the request is too large. | OMS V3.3.1 and later |
| protocol | STRING | TUPLE | The topic protocol, which can be BLOB or TUPLE. | OMS V3.3.1 and later |
| requestLimitSize | LONG | 3575104 | The maximum size of a write request, which is dynamically adjusted based on errors and measured in bytes. | OMS V3.3.1 and later |
| partitionMode | STRING | HASH | The distribution logic that determines which partition a data record is written to. Valid values: HASH (primary key hash) and TABLE (table name hash). | OMS V3.3.1 and later |
| serializerType | STRING | DEFAULT | The serialization format of messages to be written to the message queue. Valid values: DEFAULT, DEFAULT_WITH_SCHEMA, SHAREPLEX, CANAL, DATAWORKS_V2, and DATAWORKS_V1. | OMS V3.3.1 and later |
| metaExtend | JSON | null | JSON extended data. This parameter takes effect only when extend data is written in DataWorks V2. |
OMS V3.3.1 and later |
| serializerDataTypeMapper | JSON | null | The mapping from SchemaType to STRING to resolve the INT64 issue. | OMS V4.1.0 and later |
| retryErrorMessage | STRING | null | The error message indicating that a retry is needed. | OMS V3.3.1 and later |
| skipErrorMessage | STRING | null | The error message indicating that the corresponding data record needs to be skipped. | OMS V3.3.1 and later |
| skipErrorCode | STRING | null | The error code indicating that the corresponding data record needs to be skipped. | OMS V3.3.1 and later |
| retryErrorCode | STRING | null | The error code indicating that a retry is needed. | OMS V3.3.1 and later |
Source parameters of Kafka
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| kafkaAddr | STRING | null | The address of the Kafka instance from which the Kafka source reads data. This parameter is deprecated. You can use the kafkaServer parameter instead. |
OMS V4.0.0 and later |
| srcSubTopic | STRING | null | A unique identifier. | OMS V4.2.2 and later |
| kafkaTopic | STRING | null | The topic of the Kafka instance. | OMS V4.2.2 and later |
| boosterClass | STRING | null | The startup class: com.oceanbase.oms.connector.source.kafka.KafkaSourceBooster. |
OMS V3.3.1 and later |
| checkpoint | STRING | null | The checkpoint, which is of the Map<String, String> type. | OMS V4.2.2 and later |
| sourceType | STRING | null | The type of the source. | OMS V3.3.1 and later |
| kafkaServer | STRING | null | The address of the Kafka server. | OMS V4.2.2 and later |
| cdcDataFormat | STRING | tidbOpenProtocol | The data format of TiDB, which can be tidbOpenProtocol or tidbBinlog. |
OMS V4.0.0 and later |
| timestamp | LONG | null | The old checkpoint and topic timestamp in Kafka. | OMS V4.2.2 and later |
| kafkaPort | STRING | null | The port number of the Kafka server. This parameter is deprecated. You can use the kafkaServer parameter instead. |
OMS V4.0.0 and later |
| forceConsumeBegin | BOOLEAN | false | Whether to force the Kafka source to read data from the beginning offset of the topic. | OMS V4.2.2 and later |
| properties | JSON | null | The properties of the Kafka server. | OMS V3.3.1 and later |
| isSourceDataWriteToFile | BOOLEAN | false | Whether to write data to a file. This parameter is for testing purposes only. Do not use it in a production environment. | OMS V4.2.2 and later |
| sslTruststoreLocation | STRING | null | The path of the SSL credential file. | OMS V3.3.1 and later |
| saslConfig | JSON | null | The SASL parameters. {"mechanism": "the authentication mechanism, which can be GSSAPI, PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","username": "the username required when the authentication mechanism is PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","password": "the password required when the authentication mechanism is PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512","kerberosPrincipal": "the Kerberos principal. This parameter is required only when the authentication mechanism is GSSAPI.","kerberosKeyTabFileLocation": "the path of the Kerberos keytab file. This parameter is required only when the authentication mechanism is GSSAPI, which indicates the path of the authentication configuration file.","jaasConfFileLocation": "the path of the authentication configuration file. This parameter is required only when the authentication mechanism is GSSAPI.","kerberosConfFileLocation": "the path of the krb conf file when the authentication mechanism is GSSAPI."} |
OMS V3.3.1 and later |
| securityProtocol | STRING | null | The authentication protocol, which can be one of the following: SASL_PLAINTEXT, SASL_SSL, PLAINTEXTSASL, or SSLSASL. | OMS V3.3.1 and later |
Source parameters of logproxy
| Parameter | Type | Default value | Description | Applicable version |
|---|---|---|---|---|
| sys.password | STRING | null | The password of the sys tenant of OceanBase Database. | OMS V4.2.1 and later |
| wait.queue.size | LONG | null | The size of the waiting queue of logproxy. | OMS V3.3.1 and later |
| protocol.version | LONG | 1 | The protocol version of logproxy. | OMS V3.3.1 and later |
| retry.interval.ms | LONG | 2 | The interval between reconnection attempts. | OMS V3.3.1 and later |
| useSchemaCache | BOOLEAN | false | Specifies whether to use schema cache when records are processed. | OMS V3.3.1 and later |
| sourceType | STRING | null | The type of the source. | OMS V3.3.1 and later |
| logproxy.address | STRING | null | The address of logproxy. | OMS V3.3.1 and later |
| username | STRING | null | The username of OceanBase Database. | OMS V3.3.1 and later |
| splitThreshold | LONG | 128 | The maximum data size supported by a single RecordBatch. | OMS V3.3.1 and later |
| properties | STRING | null | The parameters to be directly passed to OBCDC. | OMS V4.2.1 and later |
| oceanbase.cluster.id | STRING | null | The ID of the OceanBase cluster. | OMS V3.3.1 and later |
| useBetaListener | BOOLEAN | false | Specifies whether to use the experimental parser for parsing. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone of the source. | OMS V3.3.1 and later |
| conn.timeout.ms | LONG | 60000 | The connection timeout period. | OMS V3.3.1 and later |
| sequence.start.timestamp | LONG | 0 | The timestamp when a sequence starts to be generated. | |
| ignore.unknown.record.type | BOOLEAN | true | Specifies whether to ignore unknown record types. | OMS V3.3.1 and later |
| sourceBatchMemorySize | LONG | with recommended value logic | The maximum memory size of a record batch. | OMS V3.3.1 and later |
| boosterClass | STRING | null | The startup class, which is com.oceanbase.oms.connector.source.logproxy.LogProxySourceBooster. |
OMS V3.3.1 and later |
| ignoreDdl | BOOLEAN | true | Specifies whether to ignore DDL. | OMS V3.3.1 and later |
| sys.username | STRING | null | The name of the sys tenant of OceanBase Database. | OMS V4.2.1 and later |
| checkpoint | STRING | null | The checkpoint of OceanBase Database. | OMS V3.3.1 and later |
| enabled.compression.table.white.list | BOOLEAN | false | Specifies whether to pass the compression whitelist. | OMS V3.3.1 and later |
| password | STRING | null | The password of OceanBase Database. | OMS V3.3.1 and later |
| idle.timeout.ms | LONG | 15 | The idle waiting period. | OMS V3.3.1 and later |
| oceanbase.cluster.url | STRING | null | The URL of the OceanBase cluster. | OMS V3.3.1 and later |
| reconnect.times | LONG | -1 | The reconnection period of logproxy. | OMS V3.3.1 and later |
| client.id | STRING | default | The client.id of logproxy. |
OMS V3.3.1 and later |
| read.timeout.ms | LONG | 60000 | The timeout period for reading data. | OMS V3.3.1 and later |
DataFlowSource parameters
| Parameter | Type | Default value | Description | Applicable version | |
|---|---|---|---|---|---|
| taskIdentity | STRING | 1 | The task identifier. | OMS V3.3.1 and later | |
| connTimeoutSec | LONG | 120 | The timeout period. | OMS V3.3.1 and later | |
| nonePkUkQueryWholeTable | BOOLEAN | true | Specifies whether to query the entire table for a table without a primary key or unique key. | OMS V3.3.1 and later | |
| rtrimOracleChar | BOOLEAN | false | Specifies whether to remove trailing spaces for CHAR data in the source Oracle database. If all the characters are spaces, one space is retained. The maximum length of a CHAR value is 2000. If the character sets of the source and target are inconsistent, the length of the data after conversion can exceed the maximum length supported by the target and result in a write failure. | OMS V3.3.1 and later | |
| maxSqlNoActiveTime | LONG | 700 | The JDBC timeout period. | OMS V3.3.1 and later | |
| taskSubId | STRING | 1 | The subdirectory for storing progress files of a full data read task. | OMS V3.3.1 and later | |
| stringEncodingCharset | STRING | null | The character set used to encode or decode strings. If you use the getString method and specify the stringEncodingCharset parameter for a source DB2 database, the byte stream returned by the database is decoded into a string based on the specified character set. If you specify the stringEncodingCharset parameter for a source Oracle database, the getBytes method is directly called to obtain the original bytes for CHAR, VARCHAR, and CLOB data, and the byte data is then decoded into a string based on the character set specified by this parameter. |
OMS V4.2.1 and later | |
| queryMetaInfoBySysTenant | BOOLEAN | false | Specifies whether to obtain partition information from the sys tenant in OceanBase Database V1.4.77. | OMS V4.2.1 and later | |
| timestamptzConvert | BOOLEAN | false | Specifies whether to convert the time zone of TIMESTAMP WITH TIME ZONE data when you read data from an Oracle database or an Oracle-compatible tenant of OceanBase Database. The conversion converts the city-based time zone to an offset-based time zone. For example, Asia/Shanghai is converted to +08:00. | OMS V3.3.1 and later | |
| retryIntervalTime | LONG | 2000 | The interval between retries. | OMS V3.3.1 and later | |
| workerNum | LONG | 16 | The number of threads that read data. | OMS V3.3.1 and later | |
| nonePkUkTruncateDstTable | BOOLEAN | false | Specifies whether to truncate a table without a primary key or unique key in the target. | OMS V3.3.1 and later | |
| consistencyMigration | BOOLEAN | false | Specifies whether to lock the snapshot of full migration for primary key migration. | OMS V3.3.1 and later | |
| nullReplaceEnable | BOOLEAN | false | Specifies whether to enable character replacement for an Oracle GB18030 field whose value is in the range of chr(129)-chr(254) and which has a NOT NULL constraint, where JDBC reads the value as NULL. |
OMS V3.3.1 and later | |
| sliceByPkIncrementMinMax | BOOLEAN | true | Specifies whether to use pk_increment for non-primary key tables in OceanBase Database in MySQL compatible or Oracle compatible mode. If you set the value to true, the minimum and maximum values of pk_increment are used for switching. If you set the value to false, ordinary splitting is performed and the order of indexes is specified by using the LIMIT clause. | OMS V3.3.1 and later | |
| enableSplitByPartition | BOOLEAN | true | Specifies whether to split data in partitioned tables by partition. | OMS V4.2.2 and later | |
| tablePartition | STRING | null | Specifies the subpartitions to be read from the partitioned table. The value of this parameter is in JSON format: {"fullName":"regex", ....}. Assume that the DB1 database exists. If you want to read data from partitions whose names match p1. for Tab1 and data from partitions whose names match x1. for Tab2, you can set this parameter to {"DB1.Tab1": "p1.", "DB1.Tab2": "x1."}. Note that the database name, table name, and column name specified in this parameter must be actual names, and case sensitivity must be considered. If the names do not match, an error occurs. |
OMS V3.3.1 and later | |
| sliceWorkerNum | LONG | 16 | The number of threads for splitting data. | OMS V3.3.1 and later | |
| stringOriginalCharset | STRING | null | The original character set of strings in a source Oracle database. If you specify the value of this parameter, the getBytes method is directly called for the char, varchar, and clob data in the source Oracle database, and the data is then converted into a string based on the specified character set. If you specify the stringEncodingCharset parameter, the value of this parameter is ignored. |
OMS V4.2.1 and later | |
| timezone | STRING | +08:00 | The time zone for ETL processing. | OMS V3.3.1 and later | |
| databaseMaxConnection | LONG | 128 | The maximum number of database connections. | OMS V3.3.1 and later | |
| forceReadWholeTableWithoutSlice | BOOLEAN | false | Specifies whether to read the entire table without splitting. If some tables in the source cannot be split, you can set the value of this parameter to true to read data from these tables without splitting. Note that this parameter takes effect only for MySQL databases. | OMS V3.3.1 and later | |
| sliceQueueSize | LONG | 128 | The maximum number of splitting queues. | OMS V3.3.1 and later | |
| binlogOffsetDb | STRING | test | The database for storing binlog_offset in full and incremental MySQL primary key migration. | OMS V3.3.1 and later | |
| sourceBatchMemorySize | LONG | 16777216 | The maximum memory size, in bytes, for a record batch. | OMS V3.3.1 and later | |
| sliceIndex | JSON | null | The index used for splitting data. The value of this parameter is in the Map<String, String> format. The key is in the database.table format. |
OMS V4.1.0 and later | |
| sliceBlocks | LONG | 128 | The number of blocks in each splitting block for CHAR, VARCHAR, and CLOB data in a source Oracle database. | OMS V3.3.1 and later | |
| binlogOffsetTable | STRING | binlog_offset | The table for storing binlog_offset in full and incremental MySQL primary key migration. | OMS V3.3.1 and later | |
| intervalTypeFmt | STRING | None | Configure the formatting parameters for Oracle Interval type adjustments.
|
OMS V3.3.1 and later versions | |
| controlFlag | STRING | NOOP | The message sending mode. Valid values: NOOP (do not send), SYNC (send in a synchronous blocking manner), and ASYNC (send in an asynchronous manner). | OMS V4.2.2 and later | |
| clientQueryPartitionWithInnerTable | BOOLEAN | true | Specifies whether to query the partition information of an OceanBase database by using an internal table. | OMS V4.2.1 and later | |
| indexHint | BOOLEAN | true | Specifies whether to use a hint during splitting. | OMS V3.3.1 and later | |
| sliceBatchSize | LONG | 2000 | The number of data rows in each splitting batch. | OMS V3.3.1 and later | |
| enableNoUniqueConstraintTableReplicate | BOOLEAN | false | Specifies whether to replicate a table without a primary key or unique key. This parameter affects only the forward and reverse data transmission between a MySQL/PG database and a PBMySQ database. | OMS V3.3.1 and later | |
| fetchSize | LONG | 1000 | The fetch size. It indicates the number of result set rows that the JDBC driver fetches from the database and stores in the memory at one time during a query. | OMS V3.3.1 and later | |
| oracleRowidSliceHint | STRING | None | Whether to use hint when querying block slices. | OMS V3.3.1 and later versions | |
| sliceTableBatchSize | LONG | 100 | The number of tables read in one batch during full data read. | OMS V3.3.1 and later | |
| nullReplaceString | STRING | None | When the ORACLE gb18030 field value is chr(129)-chr(254) with a NOT NULL constraint, it is read as NULL by JDBC. In this case, when nullReplaceEnable=true, the configured character will be used as a replacement. |
OMS V3.3.1 and later versions | |
| stringDecodingCharset | STRING | null | The original character set of strings in a source DB2 database. If you specify the value of this parameter, the getBytes method is directly called for the char, varchar, and clob data in the source DB2 database, and the data is then converted into a string based on the specified character set. If you specify the stringDecodingCharset parameter, you must specify the stringEncodingCharset parameter. |
OMS V4.2.1 and later | |
| sourceMaxRetryTimes | LONG | 1000 | The maximum number of retries for a source. | OMS V3.3.1 and later | |
| sourceType | STRING | null | The type of the source. | OMS V3.3.1 and later | |
| maxSqlRetryTime | LONG | 3600 | The maximum number of retries for data pull. | OMS V3.3.1 and later | |
| batchSize | LONG | 1000 | The maximum number of record batches. | OMS V3.3.1 and later | |
| rtrimDB2Graphic | BOOLEAN | false | Specifies whether to remove trailing spaces for GRAPHIC data in a source DB2 database. | OMS V3.3.1 and later | |
| oracleRowidSliceRowsMin | LONG | -1 | The minimum number of rows in each splitting block for CHAR, VARCHAR, and CLOB data in a source Oracle database. | OMS V3.3.1 and later | |
| oracleLobFetchSize | LONG | null | The maximum number of rows that can be read at a time for LOB data in a source Oracle database. | OMS V3.3.1 and later | |
| extConfig | JSON | null | The configuration information of the data source. The value of this parameter is in the Map<String, String> format. |
OMS V3.3.1 and later | |
| charset | STRING | utf8 | The connection character set. | OMS V3.3.1 and later | |
| clients | JSON | null | The connection information of the source. | OMS V3.3.1 and later | |
| boosterClass | STRING | null | The startup class: com.oceanbase.oms.connector.source.dataflow.DataFlowSourceBooster. |
OMS V3.3.1 and later | |
| userIntervalTypeFmt | BOOLEAN | true | Specifies whether to specify the interval type format. | OMS V3.3.1 and later | |
| taskType | STRING | migrate | The task type. | OMS V3.3.1 and later | |
| taskResume | BOOLEAN | true | Specifies whether to restore from a checkpoint. | OMS V3.3.1 and later | |
| sliceOracleOptimizerFeatures | STRING | None | Oracle block slicing feature, which sets alter session set OPTIMIZER_FEATURES_ENABLE=''. In some scenarios, setting it to 10.2.0.3 can optimize performance. |
OMS V3.3.1 and later versions |
Parameters of the JDBC-SINK connector
| DBType | Plugin Jar | BoosterClass |
|---|---|---|
| MYSQL | jdbc-sink-mysql.jar | com.oceanbase.oms.connector.jdbc.sink.mysql.MySQLJDBCSinkBooster |
| DB2 | jdbc-sink-db2.jar | com.oceanbase.oms.connector.jdbc.sink.db2.DB2JDBCSinkBooster |
| OB_MYSQL | jdbc-sink-ob-mysql.jar | com.oceanbase.oms.connector.jdbc.sink.obmysql.OBMySQLJDBCSinkBooster |
| OB_ORACLE | jdbc-sink-ob-oracle.jar | com.oceanbase.oms.connector.jdbc.sink.oboracle.OBOracleJDBCSinkBooster |
| ORACLE | jdbc-sink-oracle.jar | com.oceanbase.oms.connector.jdbc.sink.oracle.OracleJDBCSinkBooster |
| POSTGRESQL | jdbc-sink-pg.jar | com.oceanbase.oms.connector.jdbc.sink.pg.PgJDBCSinkBooster |
STORE_MULTI_SOURCE parameters
| Parameter | Type | Default value | Description | Applicable versions |
|---|---|---|---|---|
| useDRCNet | BOOLEAN | true | Specifies whether to use the drc protocol. | OMS V3.3.1 and later |
| blackRegionNo | STRING | null | Specifies the unnecessary regionNo to prevent circular replication. | OMS V3.3.1 and later |
| drcMarkDb | STRING | null | The database in which the table used for preventing cyclic replication is located. This parameter is used for transactional tables. | OMS V3.3.1 and later |
| useBetaListener | BOOLEAN | false | Specifies whether the DRC uses the experimental parser for parsing. | OMS V3.3.1 and later |
| timezone | STRING | +08:00 | The time zone of the source. | OMS V3.3.1 and later |
| clients | JSON | null | The connection information of the Store component.{"clientId": "A unique identifier, which is also the source_id filled in the Condition field. It is used to identify the heartbeat transmitted by this client. We recommend that you set the value of this field to Subtopic.","subtopic": "store subtopic","clusterManagerUrl": "store url/cm url","filterConditions": "store allowlist"} |
OMS V3.3.1 and later |
| sourceBatchMemorySize | LONG | 0 | The memory size for assembling record groups. | OMS V3.3.1 and later |
| boosterClass | STRING | null | The startup class: com.oceanbase.oms.connector.source.store.MultiStoreSourceBooster. |
OMS V3.3.1 and later |
| ignoreDdl | BOOLEAN | true | Specifies whether to ignore DDL statements. | OMS V3.3.1 and later |
| askSelfUnit | BOOLEAN | false | Specifies whether to request only data generated by DRC. | OMS V3.3.1 and later |
| checkpoint | JSON | null | The startup checkpoint of the Store component. | OMS V3.3.1 and later |
| connectionTimeoutSec | LONG | 120 | The timeout period for connecting to the Store component. | OMS V3.3.1 and later |
| maxRetries | LONG | 100 | The maximum number of retries for connecting to the Store component. | OMS V3.3.1 and later |
| useSchemaCache | BOOLEAN | false | Specifies whether to use the schema cache when records are processed. | OMS V3.3.1 and later |
| password | STRING | null | The account password required for connecting to the Store component. | OMS V3.3.1 and later |
| sourceType | STRING | null | The type of the source. | OMS V3.3.1 and later |
| isSourceDataWriteToFile | BOOLEAN | false | Specifies whether to write source data to a file. This parameter is for testing purposes only. | OMS V3.3.1 and later |
| buildRecordConcurrent | BOOLEAN | false | Specifies whether to concurrently assemble records when records are assembled. If you set this parameter to true, four threads concurrently assemble records. | OMS V3.3.1 and later |
| username | STRING | null | The username. | OMS V3.3.1 and later |
| filterOpts | LIST | null | The filtering options for files. This parameter is effective only when data is simulated and written to a file. | OMS V3.3.1 and later |
| drcProperties | JSON | null | The parameters of the store client. | OMS V3.3.1 and later |
| splitThreshold | LONG | 128 | The transaction splitting threshold. | OMS V3.3.1 and later |
| drcMarkTable | STRING | null | The table used for preventing cyclic replication in transactional tables. | OMS V3.3.1 and later |