This topic describes the CreateProject API provided by OceanBase Migration Service (OMS): how to create a migration project.
Purpose
This interface is used to create a data migration or synchronization task.
Call description
Limitations
This interface applies only to OMS V4.1.0 and later.
Request path
POST /api/v2?Action=CreateProject
Request parameters
| Parameter | Type | Description |
|---|---|---|
| name | string | The name of the task, which cannot contain spaces. |
| type | string | The type of the task. Valid values: MIGRATION (indicating a data migration task) and SYNC (indicating a data synchronization task). |
| labelIds | array | A collection of tag IDs. At most five tag IDs are supported. |
| sourceEndpointId | string | The ID of the source data source. To obtain the ID, call the ListDataSource interface. The ID is in the format of e_****. |
| sinkEndpointId | string | The ID of the destination data source. To obtain the ID, call the ListDataSource interface. The ID is in the format of e_****. |
| transferMapping | object | The mapping information about objects to be transferred. |
| ├─ mode | string | The object mapping mode. Valid values: SPECIFIC (indicating specific objects) and WILDCARD (indicating matching rules). |
| ├─ databases | object [] | A list of database objects. For more information about the specific fields, see "Parameters in databases/databasesBlack". |
| ├─ databasesBlack | object [] | A list of blacklisted database objects in the matching rules mode. For more information about the specific fields, see "Parameters in databases/databasesBlack". |
| ├─ tableAndViewWhiteList | array | The whitelist of tables and views in the matching rules mode. |
| ├─ tableAndViewBlackList | array | The blacklist of tables and views in the matching rules mode. |
| commonTransferConfig | object | General transfer configurations. |
| ├─ tableCategory | string | The supported table type. Valid values: ALL (all tables), WITH_UNIQUE_ROW_ID (tables with unique row identifiers), and WITHOUT_UNIQUE_ROW_ID (tables without unique row identifiers). Default value: ALL. |
| ├─ activeActive | boolean | Specifies whether the scenario is a disaster recovery dual-active scenario. |
| ├─ mqSerializerType | string | The JSON serialization type when data is delivered to a message queue. Valid values: DEFAULT, CANAL, DATAWORKS_V2, SHAREPLEX, DEFAULT_WITH_SCHEMA, DEBEZIUM, DEBEZIUMFLATTEN, and DEBEZIUMSMT. |
| ├─ mqPartitionMode | string | The partition routing mode when data is delivered to a message queue. Valid values: ONE (specify the partition to deliver data to), HASH (hash partition data based on primary keys or shard column values), and TABLE (hash partition data based on database and table names). |
| ├─ mqPartition | integer | When partitionMode is ONE, this parameter specifies the position of the partition in the current topic. |
| ├─ datahubTopicType | string | The type of the topic to which data is delivered to a DataHub instance. Valid values: TUPLE and BLOB. |
| ├─ rocketMqProducerGroup | string | The name of the producer group. This parameter is effective only when the destination data source is a RocketMQ data source. Default value: OMS. |
| ├─ rocketMqMsgTags | string | The message tags. This parameter is effective only when the destination data source is a RocketMQ data source. Default value: default->default_tag. If the value of this parameter is default, _tag is appended as the suffix. |
| ├─ rocketMqEnableMsgTrace | boolean | Specifies whether to enable message tracing. This parameter is effective only when the destination data source is a RocketMQ data source. |
| ├─ rocketMqSendMsgTimeout | long | The timeout period for sending messages. This parameter is effective only when the destination data source is a RocketMQ data source. |
| ├─ dataWorksBusinessName | string | The identifier of the business system. This parameter specifies the source of the business system for data. This parameter is supported only when the serializer is set to DataWorks. The value of this parameter must be 1 to 20 characters in length. |
| ├─ sinkStoreFormat | string | The object storage type of the destination table. Valid values: Default, Row storage, Column storage, and Hybrid columnar storage. This parameter is effective only when the destination OceanBase Database is V4.3.0 or later and the migration type is Schema Migration or Incremental Synchronization > DDL synchronization. |
| ├─ sourceStoreFormat | string | The object storage type of the source table. Valid values: Default, Row storage, Column storage, and Hybrid columnar storage. This parameter is effective only when the source OceanBase Database is V4.3.0 or later and the migration type is Reverse Increment > DDL synchronization. |
| enableStructTransfer | boolean | Specifies whether to allow schema transfer. |
| structTransferConfig | object | The configuration of schema transfer. |
| ├─ byteCharConvertStrategy | string | The byte/char conversion strategy. Valid values: IGNORE_BYTE_IF_BYTE_USED, FORCE_CHAR_IF_BYTE_USED, EXPAND_LEN_IF_BYTE_USED, and DO_NOTHING_IF_BYTE_USED (default). |
| ├─ deferIndexCreation | boolean | Specifies whether to allow index creation to be postponed. The default value is false. |
| enableFullTransfer | boolean | Specifies whether to enable full data transfer. |
| enableFullVerify | boolean | Specifies whether to enable full data verification. |
| fullTransferConfig | object | The configuration of full data transfer. |
| ├─ nonePkUkTruncateDstTable | boolean | Specifies whether to truncate the destination table when performing full data migration for a source table without a unique index. |
| ├─ allowDestTableNotEmpty | boolean | The processing strategy when the destination table contains records. Valid values: Ignore and Stop Migration.
|
| ├─ throttleRps | integer | The maximum allowed transfer rate in terms of the number of records per second during full data transfer. Set this parameter to null to indicate no restriction on RPS. |
| ├─ throttleIOPS | integer | The maximum allowed transfer rate in terms of bytes per second during full data transfer. Set this parameter to null to indicate no restriction on IOPS. |
| enableIncrTransfer | boolean | Specifies whether to enable incremental synchronization. |
| enableReverseIncrTransfer | boolean | Specifies whether to enable reverse incremental synchronization. |
| incrTransferConfig | object | The configuration of incremental synchronization. |
| ├─ startTimestamp | string | The starting timestamp for incremental synchronization. This parameter is supported only when incremental synchronization is enabled but full data transfer is not enabled. |
| ├─ recordTypeWhiteList | array | The data types of incremental data to be synchronized in incremental synchronization mode. Valid values: HEARTBEAT, INSERT, UPDATE, DELETE, BEGIN, COMMIT, ROLLBACK, DDL, and ROW. |
| ├─ storeLogKeptHour | integer | The log retention period in hours when the incremental log puller is a Store. The default value is 7, and the maximum value is 365. |
| ├─ enableSequencingWithinTxn | boolean | Specifies whether to enable sequence number arrangement within a transaction when the incremental log puller is a Store. |
| ├─ enableIncrSyncStatistics | boolean | Specifies whether to enable statistics collection for incremental records (DML and DDL). |
| ├─ incrOnlineDdlConfig | array | The value of this parameter indicates the name of an Online DDL tool. Valid values: GH_OST, PT_OSC, and ALIYUN_DMS, which indicate gh-ost, pt-osc, and Alibaba Cloud DMS, respectively.
|
| ├─ incrSyncConcurrency | integer | The number of threads for incremental synchronization. The default value is 64. |
| ├─ incrLogPullSpeedMode | string | The incremental log pulling speed. Valid values: STEADY (low), NORMAL (medium), FAST (high), and CUSTOM (custom). |
| ├─ incrLogPullSpeedConfig | object | The incremental log pulling resource configuration. This parameter is effective only when incrLogPullSpeedMode is set to CUSTOM. |
| ├─├─ memory | integer | The maximum memory size allowed for incremental log pulling. |
| ├─ incrLogSyncSpeedMode | string | The incremental data writing speed. Valid values: STEADY (low), NORMAL (medium), FAST (high), and CUSTOM (custom). |
| ├─ incrLogSyncSpeedConfig | object | The incremental data writing resource configuration. This parameter is effective only when incrLogSyncSpeedMode is set to CUSTOM. |
| ├─├─ memory | integer | The maximum memory size allowed for incremental log writing. |
| ├─├─ writerWorkerNum | integer | The number of concurrent incremental log writers. |
| ├─ throttleRps | integer | The maximum allowed transfer rate in terms of the number of records per second during incremental synchronization. Set this parameter to null to indicate no restriction on RPS. |
| ├─ throttleIOPS | integer | The maximum allowed transfer rate in terms of bytes per second during incremental synchronization. Set this parameter to null to indicate no restriction on IOPS. |
| ├─ supportDDLTypes | array | The supported DDL types.
|
| reverseIncrTransferConfig | object | The configuration of reverse incremental synchronization. |
| ├─ startTimestamp | string | The starting timestamp for reverse incremental synchronization. This parameter is supported only when incremental synchronization is enabled but full data transfer is not enabled. |
| ├─ recordTypeWhiteList | array | The data types of incremental data to be synchronized in reverse incremental synchronization mode. Valid values: HEARTBEAT, INSERT, UPDATE, DELETE, BEGIN, COMMIT, ROLLBACK, DDL, and ROW. |
| ├─ storeLogKeptHour | integer | The log retention period in hours when the incremental log puller is a Store for reverse incremental synchronization. The default value is 7, and the maximum value is 365. |
| ├─ enableSequencingWithinTxn | boolean | Specifies whether to enable sequence number arrangement within a transaction when the incremental log puller is a Store for reverse incremental synchronization. |
| ├─ enableIncrSyncStatistics | boolean | Specifies whether to enable statistics collection for incremental records (DML and DDL) in reverse incremental synchronization. |
| ├─ incrOnlineDdlConfig | array | The value of this parameter indicates the name of an Online DDL tool. Valid values: GH_OST, PT_OSC, and ALIYUN_DMS, which indicate gh-ost, pt-osc, and Alibaba Cloud DMS, respectively.
|
| ├─ incrSyncConcurrency | integer | The number of threads for reverse incremental synchronization. The default value is 64. |
| ├─ incrLogPullSpeedMode | string | The incremental log pulling speed. Valid values: STEADY (low), NORMAL (medium), FAST (high), and CUSTOM (custom). |
| ├─ incrLogPullSpeedConfig | object | The incremental log pulling resource configuration. This parameter is effective only when incrLogPullSpeedMode is set to CUSTOM. |
| ├─├─ memory | integer | The maximum memory size allowed for incremental log pulling. |
| ├─ incrLogSyncSpeedMode | string | The incremental data writing speed. Valid values: STEADY (low), NORMAL (medium), FAST (high), and CUSTOM (custom). |
| ├─ incrLogSyncSpeedConfig | object | The incremental data writing resource configuration. This parameter is effective only when incrLogSyncSpeedMode is set to CUSTOM. |
| ├─├─ memory | integer | The maximum memory size allowed for incremental log writing. |
| ├─├─ writerWorkerNum | integer | The number of concurrent incremental log writers. |
| ├─ throttleRps | integer | The maximum allowed transfer rate in terms of the number of records per second during reverse incremental synchronization. Set this parameter to null to indicate no restriction on RPS. |
| ├─ throttleIOPS | integer | The maximum allowed transfer rate in terms of bytes per second during reverse incremental synchronization. Set this parameter to null to indicate no restriction on IOPS. |
| ├─ supportDDLTypes | array | The supported DDL types.
|
Parameters in databases/databasesBlack
Parameter Type Description mappedName string The name of the database in the target. name string The name of the database in the source. id string The ID of the database. tables array The configurations of table objects to be transferred in the current database. When modeisWILDCARD, this parameter specifies the configurations of blacklisted or whitelisted tables and views.├─ mappedName string The mapped name of a table. ├─ name string The name of a table. ├─ id string The ID of a table. ├─ whereClause string A standard SQL WHEREclause for configuring row filtering.Notice
When
modeisWILDCARD, you cannot setwhereClause.├─ filterColumns array A collection of filtered columns. Notice
When
modeisWILDCARD, you cannot setfilterColumns.├─ shardColumns array A collection of sharding columns. Notice
When
modeisWILDCARD, you cannot setshardColumns.views array The configurations of view objects to be transferred in the current database. ├─ mappedName string The mapped name of a view. ├─ name string The name of a view. ├─ id string The ID of a view. ├─ whereClause string A standard SQL WHEREclause for configuring row filtering.Notice
When
modeisWILDCARD, you cannot setwhereClause.├─ filterColumns array A collection of filtered columns. Notice
When
modeisWILDCARD, you cannot setfilterColumns.├─ shardColumns array A collection of sharding columns. Notice
When
modeisWILDCARD, you cannot setshardColumns.specificTables array The configurations of specific table objects. When modeisWILDCARD, this parameter specifies the renaming and row filtering configurations for specific table objects.├─ mappedName string The mapped name of a table. ├─ name string The name of a table. ├─ id string The ID of a table. ├─ whereClause string A standard SQL WHEREclause for configuring row filtering.Notice
When
modeisWILDCARD, you cannot setwhereClause.├─ filterColumns array A collection of filtered columns. Notice
When
modeisWILDCARD, you cannot setfilterColumns.├─ shardColumns array A collection of sharding columns. Notice
When
modeisWILDCARD, you cannot setshardColumns.specificViews array The configurations of specific view objects. When modeisWILDCARD, this parameter specifies the renaming configurations for specific view objects.├─ mappedName string The mapped name of a view. ├─ name string The name of a view. ├─ id string The ID of a view. ├─ whereClause string A standard SQL WHEREclause for configuring row filtering.Notice
When
modeisWILDCARD, you cannot setwhereClause.├─ filterColumns array A collection of filtered columns. Notice
When
modeisWILDCARD, you cannot setfilterColumns.├─ shardColumns array A collection of sharding columns. Notice
When
modeisWILDCARD, you cannot setshardColumns.
Response parameters
| Parameter | Type | Description |
|---|---|---|
| success | boolean | Specifies whether the call is successful. |
| errorDetail | object | Details about the error. |
| code | string | Error code. |
| message | string | Description. |
| advice | string | Suggestion. |
| requestId | string | Request ID. |
| pageNumber | integer[int32] | The page number. This parameter is effective only for paged queries. |
| pageSize | integer[int32] | The number of records per page. This parameter is effective only for paged queries. |
| totalCount | integer[int64] | The total number of records. This parameter is effective only for paged queries. |
| cost | string | The time consumed in the request, in seconds. |
| data | string | The ID of the task. |
Examples
Request example
{
"name":"test",
"type":"MIGRATION",
"labelIds":[
"l_4w****wygjk"
],
"sourceEndpointId":"e_4wg07f****",
"sinkEndpointId":"e_4wfy2****",
"transferMapping":{
"mode":"SPECIFIC",
"databases":[
{
"mappedName":"gez3***",
"name":"ion1****bmysql",
"id":null,
"tables":[
{
"mappedName":"tion12***20obmysql2",
"name":"GRATION129****",
"id":null,
"whereClause":null,
"filterColumns":null,
"shardColumns":null,
}
],
"views":null,
"specificTables":null,
"specificViews":null
}
],
"databasesBlack":null,
"tableAndViewWhiteList":null,
"tableAndViewBlackList":null
},
"commonTransferConfig":{
"tableCategory":null,
"activeActive":false,
"mqSerializerType":null,
"mqPartitionMode":null,
"mqPartition":0,
"datahubTopicType":null,
"rocketMqProducerGroup":null,
"rocketMqMsgTags":null,
"rocketMqEnableMsgTrace":null,
"rocketMqSendMsgTimeout":null,
"dataWorksBusinessName":null
},
"enableStructTransfer":true,
"structTransferConfig":{
"byteCharConvertStrategy":"DO_NOTHING_IF_BYTE_USED",
"deferIndexCreation":true
},
"enableFullTransfer":true,
"enableFullVerify":true,
"fullTransferConfig":{
"nonePkUkTruncateDstTable":null,
"allowDestTableNotEmpty":true,
"throttleRps": 1000,
"throttleIOPS": 1000,
"fullTransferSpeedMode":"NORMAL",
"fullVerifySpeedMode":"NORMAL"
},
"enableIncrTransfer":true,
"enableReverseIncrTransfer":true,
"incrTransferConfig":{
"startTimestamp":1689201369,
"storeLogKeptHour":120,
"recordTypeWhiteList":[
"DELETE",
"INSERT",
"UPDATE"
],
"enableSequencingWithinTxn":true,
"incrSyncConcurrency":64,
"incrLogPullSpeedMode":"NORMAL",
"incrLogPullSpeedConfig":{
"memory":4
},
"incrLogSyncSpeedMode":"CUSTOM",
"incrLogSyncSpeedConfig":{
"memory":4,
"writerWorkerNum":2
},
"throttleRps": 1000,
"throttleIOPS": 1000,
"enableIncrSyncStatistics":true,
"supportDDLTypes":[
"CREATE_TABLE",
"ALTER_TABLE"
],
"incrOnlineDdlConfig":[
"GH_OST",
"PT_OSC",
"ALIYUN_DMS"
]
},
"reverseIncrTransferConfig":{
"startTimestamp":1689201369,
"recordTypeWhiteList":[
"DELETE",
"INSERT",
"UPDATE"
],
"storeLogKeptHour":120,
"incrSyncConcurrency":64,
"incrLogPullSpeedMode":"NORMAL",
"incrLogPullSpeedConfig":{
"memory":4
},
"incrLogSyncSpeedMode":"CUSTOM",
"incrLogSyncSpeedConfig":{
"memory":4,
"writerWorkerNum":2
},
"throttleRps": 1000,
"throttleIOPS": 1000,
"enableIncrSyncStatistics":true,
"supportDDLTypes":[
"CREATE_TABLE",
"ALTER_TABLE"
]
}
}
Response example
{
"Success":true,
"ErrorDetail":{
"Code":"CM-RES***11",
"Level":"WARN",
"Message":"A system error occurred.",
"Proposal":"Contact the administrator."
},
"Code":"INNER_ERROR",
"Message":"A system error occurred.",
"Advice":"Contact the administrator.",
"RequestId":"XCVSADG****DSGDS",
"PageNumber":1,
"PageSize":1,
"TotalCount":1,
"Cost":"27 ms",
"Data":"np_****"
}