| Parameter | Required | Default value | Description |
|---|---|---|---|
| datasource.binarypk.permit | No | true | Specifies whether the primary key can be a binary value. The value true indicates that the primary key can be a binary value. The value false indicates that the primary key cannot be a binary value. |
| datasource.image.address | Yes | None | The IP address of the destination database. The IP address is in the format of ip:port for MySQL Database and OceanBase Community Edition. |
| datasource.image.charset.map | No | {"gb18030":"gbk","gbk":"gbk","utf16":"utf16","default":"utf8"} |
The character set mapped from the destination OceanBase database. |
| datasource.image.index.ignore | No | false | Specifies whether to directly pull the index with the lowest score in the source table. If you set this parameter to true, the index with the lowest score in the source table will be directly pulled if no new index can be matched. In this case, duplicate data may exist if the destination table has no primary key or unique key. |
| datasource.image.insert.error.ignore | No | false | Specifies whether to ignore data insertion errors to avoid interruption. You can handle all the errors after the task is completed. The error information is recorded in the insertErrorIgnoreerror.log file. This log file is stored in the logs directory. |
| datasource.image.password | Yes | None | The password for accessing the destination database. |
| datasource.image.table.notexists.ignore | No | false | Specifies whether to ignore error messages indicating that the destination table is not found. The value false indicates not to ignore such error messages. The value true indicates to ignore such error messages. |
| datasource.image.table.empty.check | No | true | Specifies whether to check whether the destination table is empty. This parameter applies only to data migration scenarios.
|
| datasource.image.type | Yes | None | The type of the destination database. |
| datasource.image.username | Yes | None | The password for accessing the destination database. |
| datasource.master.address | Yes | None | The address of the source database. The address format varies with the data sources. |
| datasource.master.password | Yes | None | The username for accessing the source database. |
| datasource.master.systenant.password | No | Same as that of datasource.master.password |
The password for accessing the sys tenant. |
| datasource.master.systenant.username | No | Same as that of datasource.master.username |
The username for accessing the sys tenant, for example, root@sys#ob_1008810671.admin. |
| datasource.master.type | Yes | None | The type of the source database. |
| datasource.master.username | Yes | None | The password for accessing the source database. |
| datasource.ob.splitor.bymarcroinfo | No | false | The sharding strategy used by OceanBase Migration Service (OMS). This parameter is required when the source database is an OceanBase database and tables have primary keys.
|
| datasource.read.mod | No | stream | The mode for reading data for migration and verification.
|
| datasource.sybase.metadata.uppercase | No | true | Specifies whether table names are case sensitive. The default value is true. Set this parameter to false if the destination database is a MySQL tenant of OceanBase Database. Retain the default value in other cases. By default, table names in MySQL tenants of OceanBase Database are in lowercase. |
| datasource.timezone | No | +00:00 | The time zone. |
| filter.master.blacklist | Yes | None (nullable) | The blacklists. Multiple blacklists are separated by vertical lines ( |
| filter.master.whitelist | Yes | None (nullable) | The whitelists. Multiple whitelists are separated by vertical lines ( |
| filter.verify.inmod.keys | No | 100 | The maximum number of data records that can be queried by using a primary key or a unique key in a batch in the destination database. |
| filter.verify.inmod.tables | No | "" | The tables that need to be verified by using the IN mode. If this parameter is not specified, the prefix indexes are verified by using the IN mode by default. If this parameter is specified, only matched tables are verified by using the IN mode. The value is in the same format as a blacklist or whitelist, for example, ^sqltest$;^prefix_index_test_bigdata$;.*. In IN-mode verification, data is queried from the source database by using the sharding column, and the destination database parses the primary key/unique key from the data queried from the source database, queries data in itself by using the key in(...) statement and the primary key/unique key, and compares the data.Notice: The IN-mode verification method has lower efficiency than the default comparison method and is inapplicable if the destination database has a large amount of data. |
| filter.verify.inmod.workers | No | 1 | The number of concurrent IN queries in the destination database. |
| filter.verify.rectify.type | No | No | Specifies whether to correct data in the destination database.
|
| force.split.by.rowid | No | false | If the source database has a hidden primary key, you can set this parameter to true so that the hidden column is forcibly used as the primary key during data migration and verification. For example, __pk_increment in OceanBase Database is a hidden primary key. If you set this parameter to false, the primary key or unique key of the table is used as the primary key during data migration and verification. This parameter can be set to true only for the OB10 source database type, and must be set to false for other source database types. |
| limitator.datasource.connections.max | No | 50 | The maximum size of the database connection pool. The setting of this parameter applies to both the source and destination databases. For example, if you set this parameter to 100, the maximum number of connections is 100 for both the source and destination databases. The value must be greater than 0 and greater than the maximum number of concurrent requests. The specific value is subject to the relationship between the number of concurrent requests and the number of connections. |
| limitator.datasource.image.ob10freecpu.min | No | 30 | This is a new parameter used to prevent CPU resource exhaustion for OceanBase Database. The default value is 30, indicating that links will no longer be obtained when the CPU usage reaches 30%. If you set this parameter to 0, CPU resource exhaustion prevention is not activated. The priority of the limitator.datasource.image.ob10freecpu.min parameter is lower than that of the limitator.datasource.image.ob10freememory.min parameter, where the latter is used to prevent memory resource exhaustion. This parameter is deactivated when data write suspension has been triggered based on the setting of the limitator.datasource.image.ob10freememory.min parameter. This parameter takes effect only when the threshold specified by the limitator.datasource.image.ob10freememory.min parameter is not reached or the database is the source database, for which the limitator.datasource.image.ob10freememory.min parameter does not take effect. |
| limitator.datasource.image.ob10freememory.min | No | 20 | The memory protection threshold for OceanBase Database. Value range: 10 to 100. This value is the percentage of idle memory space of OceanBase Database that triggers data write suspension. For example, the value 30 indicates that when the percentage of idle memory space is less than 30%, data writes are suspended, and the program keeps waiting until the percentage of idle memory space exceeds 30%. This parameter takes effect only on an OceanBase database that serves as the destination of a data migration link. |
| limitator.image.insert.batch.max | No | 500 | The maximum number of records inserted into the destination database that triggers a commit. For example, the value 200 indicates that a maximum of 200 records can be inserted before a commit must be performed. |
| limitator.java.opt | No | None | The runtime Java virtual machine (JVM) parameter. This parameter is checked when the checker script /home/ds/bin/checker_new.sh is booted. If this parameter is specified, the setting of this parameter is to boot the checker script. |
| limitator.noneed.retry.exception | No | None | Specifies whether to directly exit SQL statement execution when an exception that does not allow retries, for example, a "table not found" exception, is encountered. |
| limitator.null.replace.string | No | Space | The string used to replace null values. This parameter is used together with the limitator.null.replace.enable parameter. |
| limitator.oceanbase.index.useuk | No | true | Specifies whether to use a unique key when no primary key is available and the source database is OceanBase Database. |
| limitator.oom.avoid | No | false | Specifies whether to enable out-of-memory (OOM) prevention. After OOM prevention is enabled, the system measures and records the actual memory size. This may affect the system performance. To enable OOM prevention, you must set useCursorFetch to true for setFetchSize to take effect.
useCursorFetch and useServerPrepStmts. Once OOM prevention is enabled, ensure that the invalid time value 0000-00-00 00:00:00 and FLOAT values beyond the precision range do not exist. |
| limitator.platform.split.threads.number | No | limitator.platform.threads.number/8<8?8:limitator.platform.threads.number/8 | The number of threads in a thread pool that trigger splitting of the thread pool. The minimum value is 8. The minimum value of limitator.datasource.connections.max must be the sum of the values of limitator.platform.threads.number and limitator.platform.split.threads.number. |
| limitator.platform.threads.number | No | 3 | The maximum size of the worker thread pool during migration and verification. This parameter is used together with limitator.datasource.connections.max. Generally, the number of connections must be greater than the maximum size of the worker thread pool. Otherwise, some worker threads must wait for connections. |
| limitator.prefix.index.action | No | 1 | The handling method that is used when the primary key is a prefix index in a link for migrating data from a MySQL database to a MySQL tenant in OceanBase Database. Default value: 2.
|
| limitator.prepared.splitors | No | 1000 | Task splitting suspends when the number of split tasks minus the number of running tasks is greater than the specified value of this parameter, which means that many split tasks are waiting in the queue. |
| limitator.queue.size | No | limitator.select.batch.max*4 | Migration: The size of the cache queue in which the data read from the source database is stored. Verification: The size of the cache queue and JOIN cache queue in which the data read from the source and destination databases is stored. The actual queue size is the value of this parameter multiplied by 2. Default value: limitator.select.batch.max*4. |
| limitator.query.withorder | No | true | Specifies whether to sort queries. The default value is true, which means that the queries are to be sorted. This parameter is implemented only for MySQL databases and MySQL tenants in OceanBase Database. |
| limitator.resume.verify.fromkeys | No | false | Specifies whether to reverify only the records that are found inconsistent in the last verification. This parameter takes effect only when the following parameters are set to the specified values:
|
| limitator.reviewer.period | No | 3 | The review interval, in seconds. |
| limitator.reviewer.review.batch.max | No | 100 | The number of keys queried in a review. |
| limitator.reviewer.rounds.max | No | 20 | The maximum number of reviews allowed in a verification process. For inconsistent data found in verification, the review process queries these keys in the source and destination databases and performs comparison multiple times. This parameter specifies the maximum number of times of comparison allowed. |
| limitator.reviewer.time.max | No | 60 | The maximum review time, in seconds. |
| limitator.splitor.block.number.max | No | Long.MAX_VALUE | The maximum number of blocks of a shard. When this value is exceeded, data is split based on data files. |
| limitator.splitor.compare.threads.number | No | 1 | The number of comparison threads during the verification of a single shard. This parameter is valid only for the verification process. |
| limitator.split.usecondition | No | false | Specifies whether to use conditions in the SQL statements for querying data by using the sharding column. |
| limitator.splitor.writer.number | No | 1 | The number of tasks written to the destination database by using each sharding column. This parameter is valid only for data migration projects. |
| limitator.sql.exec.max.last.time | No | 3600 | The maximum SQL statement retry time, in seconds. |
| limitator.table.diff.max | No | 1000000 | The maximum number of inconsistent records found during verification. If this value is exceeded, the verification ends and the review is not performed. |
| limitator.table.nonunique.max | No | 10000 | The maximum number of records that can be migrated without indexes. |
| limitator.verify.many2one | No | false | Specifies whether to enable the many-to-one table verification mode. In this mode, the verification is successful if the data in the source table is found in the destination table. The value true specifies to enable the many-to-one table verification mode. |
| mapper.from_master_to_image.list | No | None | The mapping of schemas from the source database to the destination database. The value is usually in the format of sourceSchema;*;*=destSchema;*;***, with a maximum of four sections. Multiple mappings are separated with vertical lines ( |
| rectifier.image.enable | No | false | Specifies whether to automatically correct data in the destination table. This parameter is used in the rectification process. Generally, we recommend that you do not set the value to true. |
| rectifier.image.operator.delete | No | false | Specifies whether to correct deleted data. This parameter is used in the rectification process. Generally, we recommend that you do not set the value to true. |
| rectifier.image.operator.insert | No | false | Specifies whether to correct inserted data. This parameter is used in the rectification process. Generally, we recommend that you do not set the value to true. |
| rectifier.image.operator.update | No | false | Specifies whether to correct updated data. This parameter is used in the rectification process. Generally, we recommend that you do not set the value to true. |
| sampler.verify.ratio | No | 100 | The percentage of records sampled for comparison. The value must be greater than 0 and less than 100. |
| src.record.filter.mapping | No | None | A GroovyRule configuration, which is required when task.split.mode is set to true. |
| task.resume | No | false | Set the value to false if the task is run for the first time and to true if the task is run for recheck. |
| src.table.whitelist | No | None | A GroovyRule configuration, which is required when task.split.mode is set to true. |
| task.active.active | No | false | The active-active flag. The value true indicates an active-active link. |
| task.id | Yes | None | The unique ID of the checker, which is related to the runtime directory. |
| task.subId | Yes | None | The initial value is 1 and the value increments by 1 each time a recheck is performed. A directory is created for each sub ID in the task directory to record the corresponding running result files: /home/ds/run/{taskname}/{task.type}/{task.subId} |
| task.type | Yes | None | The task type. Valid values: migrate and verify. |
| weak.consistency.read | No | false | Specifies whether to enable weak consistency read. This parameter is valid in OceanBase Database. The value "true" specifies to enable weak consistency read. set @@ob_read_consistency='weak' |
Checker parameters
share
Previous topic