Migrate data from a Milvus database to a MySQL-compatible tenant of OceanBase Database

This topic describes how to use OceanBase Migration Service (OMS) to migrate data from a Milvus database to a MySQL-compatible tenant of OceanBase Database.

Background information

Milvus is a high-performance, highly scalable vector database that provides powerful data modeling capabilities. For more information, see Milvus documentation.

The following table describes the key concepts of Milvus.

OMS supports migrating dynamic fields from a Milvus database to a MySQL-compatible tenant of OceanBase Database.

• When you migrate the schema of a Milvus database to a MySQL-compatible tenant of OceanBase Database, OMS migrates the dynamic fields to a column named $meta of the JSON type.

• When you perform a full migration or incremental synchronization from a Milvus database to a MySQL-compatible tenant of OceanBase Database, OMS synchronizes the content of the dynamic fields of the Milvus database to the $meta column as a JSON value. If the column does not exist, an error is returned.

Limitations

• Limitations on operations on the source database

  Do not perform DDL operations to modify the schema of the source database during schema migration or full migration. Otherwise, the data migration task may fail.

• Currently, OMS supports Milvus databases of version V2.3.x and V2.4.x, and OceanBase Database of version V4.3.3 and later.

• OMS does not support triggers on the target database. If a trigger exists on the target database, the data migration task may fail.

• Data source identifiers and user accounts are globally unified in OMS.

• OMS supports only migrating databases, tables, and columns whose names are ASCII characters and do not contain special characters (including line breaks, spaces, and .|"'`()=;/&\).

• When you enable the AutoID attribute for a collection in the source Milvus database, you cannot perform reverse increment synchronization. Even if the precheck ignores the related check error, the reverse increment synchronization will still fail.

• OMS does not support synchronizing schema changes. If a schema change occurs in the source database, data inconsistency may occur. For example, when you delete a partition, the operation cannot be synchronized to the target database.

• For objects that cannot be migrated during schema migration, OMS cannot obtain the schema information of the Milvus database, and you cannot manually modify the SQL statements on the console to migrate them to the target database.

• When you perform full migration from a Milvus database, resuming at breakpoints is not supported. If the full migration task is paused and then resumed, the tables or partitions that have not been migrated will be migrated again.

Considerations

• When you migrate a partitioned table in a Milvus database based on the Partition Key + Num to a key partitioned table in a MySQL-compatible tenant of OceanBase Database, the data may be delivered to different partitions because the hash calculation rules of the two databases are different.

• If a collection in the source Milvus database is configured with the TTL attribute, and the data is purged, the purge operation cannot be synchronized to the target database. This may cause data inconsistency.

• When you migrate data of the BOOLEAN type from a Milvus database to a MySQL-compatible tenant of OceanBase Database, the data is converted to the TINYINT type. true is converted to 1, and false is converted to 0. When you perform reverse increment synchronization from a MySQL-compatible tenant of OceanBase Database to a Milvus database, the data that is not 1 is converted to false.

• When you migrate data of the FLOAT type from a Milvus database to a MySQL-compatible tenant of OceanBase Database, the data is converted to the FLOAT type. However, the precision of the data is limited to six significant digits. If the data type of the source Milvus database is FLOAT, ARRAY, SPARSEFLOATVECTOR, or FLOATVECTOR, and the precision of the data exceeds the limit of the target database, the precision of the data is lost.

• When you migrate a dynamic field from a Milvus database to a MySQL-compatible tenant of OceanBase Database, the NOT NULL attribute of the dynamic field is removed to avoid data write failures in the target database.

• If the data in the source Milvus database exceeds the data type range supported by the MySQL-compatible tenant of OceanBase Database (such as JSON or VARCHAR), the data migration task fails.

  • JSON type: The Milvus database allows you to write string data (for example, "test") to a JSON field. However, the JSON type in the MySQL-compatible tenant of OceanBase Database does not support directly storing JSON data in string format.

  • VARCHAR type: In special scenarios, the Milvus database cannot effectively limit the length of the data written to it. For example, when the primary key field is of the VARCHAR type and the AutoID attribute is enabled, the AutoID generated by the Milvus database is usually 19 characters long. If the length of the VARCHAR field is less than 19 characters, the data will exceed the length limit and write failures will occur.

• The Milvus database allows duplicate primary keys, but the MySQL-compatible tenant of OceanBase Database does not. By default, OMS synchronizes the latest data with the same primary key. However, during full or incremental synchronization, OMS may synchronize duplicate primary key data. In this case, OMS ignores the primary key conflict, retains the data in the target database, and displays a prompt on the console indicating the existence of conflicting data. You need to verify whether the data has been correctly migrated.

• In incremental synchronization of partitioned tables in the Milvus database, OMS does not detect the INSERT, UPSERT, and DELETE operations on the specified partitions. Instead, OMS updates the data in the target database based on the primary key changes in the incremental data. Therefore, if data with the same primary key exists in different partitions of the Milvus database, data inconsistency may occur between the source and target databases. Similarly, during reverse increment synchronization, the operations on specified partitions are not supported. The operations are only applied to the Default partition in the Milvus database.

• When you synchronize new columns from a MySQL-compatible tenant of OceanBase Database to a Milvus database, the new columns cannot be synchronized incrementally. If the new columns have default values, OMS does not update all existing data in the corresponding columns of the Milvus database. This may cause data inconsistency between the source and target databases.

• The primary keys in the Milvus database cannot contain partition keys. During schema migration, OMS changes the primary key and partition key in the source Milvus database to a non-null unique key in the target database.

• When you write the {} value to the SPARSEVECTOR type in a MySQL-compatible tenant of OceanBase Database, an error will be returned when the data is synchronized to the Milvus database.

• If the primary key values in the Milvus database are the same except for the case, the data will be lost during migration to the MySQL-compatible tenant of OceanBase Database due to primary key conflicts.

• If the reverse increment synchronization task is restarted or the checkpoint is pulled again, the data will be written again to the Milvus database. This will result in duplicate primary key records in the Milvus database.

• If you perform incremental synchronization with OceanBase Database as the source or reverse incremental synchronization with OceanBase Database as the target and the database table objects contain UDT columns, the incremental synchronization task may fail due to unsupported UDT columns.

Data type mappings

Procedure

1. Create a data migration task.

   1. Log in to the OMS console.

   2. In the left-side navigation pane, click Data Migration.

   3. On the Data Migration page, click Create Task in the upper-right corner.

2. On the Create Task page, specify a name for the migration task.

   We recommend that you use a combination of Chinese characters, digits, and letters. The name cannot contain spaces and must be no longer than 64 characters.

   Notice

   The task name is a unique identifier in the OMS system. Please specify an appropriate task name.

3. In the Select Source and Target step, configure the parameters.

   

   Parameter	Description
   Source	If you have created a Milvus data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create a data source in the dialog box that appears on the right. For more information, see Create a Milvus data source.
   Target	If you have created a physical data source for OceanBase Database in the MySQL compatible mode, select it from the drop-down list. If not, click New Data Source in the drop-down list and create a data source in the dialog box that appears on the right. For more information, see Create an OceanBase physical data source.
   Tag	Click the text box and select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete a tag. For more information, see Manage data migration tasks by using tags.

4. Click Next. In the Select Migration Type step, select the migration type for the current data migration task.

   Migration Type includes Schema Migration, Full Migration, Incremental Synchronization, and Reverse Increment.

   Migration type	Limitations
   Schema migration	After a schema migration task starts, OMS migrates the data object definitions (tables, indexes, constraints, comments, and views) from the source database to the target database and automatically filters out temporary tables.
   Full migration	After a full migration task starts, OMS migrates the existing data in the source database tables to the corresponding tables in the target database.
   Incremental synchronization	After an incremental synchronization task starts, OMS synchronizes the data that has changed in the source database (new, modified, or deleted data) to the corresponding tables in the target database. Incremental Synchronization has DML synchronization, which includes Insert, Delete, and Update. You can customize the settings as needed. For more information, see Customize DDL/DML settings.
   Reverse increment	After a reverse increment task starts, you can real-time synchronize the changed data generated in the target database after the business switch back to the source database. Reverse increment tasks usually reuse the incremental synchronization settings. You can also customize the settings as needed. You cannot select Reverse Increment in the following cases: Multi-table aggregation is involved. Multiple source schemas map to the same target schema.

5. Click Next. In the Select Migration Objects step, select the objects to be migrated for the current data migration task.

   You can select migration objects by using the Specify Objects and Match by Rule options. This topic describes how to select migration objects by using the Specify Objects option. For more information about how to configure matching rules, see Configure matching rules.

   Notice

   • If the database name or table name contains the "$$" characters, the data migration task cannot be created.

   • OMS automatically filters unsupported objects. For more information about how to query objects, see SQL statements for querying table objects.

   1. In the Select Migration Objects step, select Specify Objects.

   2. In the Source Object(s) list, select the objects to be migrated. You can select one or more tables or views from one or more databases as migration objects.

   3. Click > to add the selected objects to the Target Object(s) list.

   OMS allows you to import objects by using text files. OMS also allows you to rename objects on the target database and remove one or all migration objects.

   Action	Steps
   Import objects	In the Target Object(s) list, click Import Objects in the upper-right corner. In the dialog box that appears, click OK. Notice: Importing objects will overwrite the previous selections. Proceed with caution. In the Import Migration Objects dialog box, import the objects to be migrated. You can rename databases and tables by importing a CSV file. For more information, see Download and import migration object settings. Click Validate. After the validation is successful, click OK.
   Rename	OMS allows you to rename migration objects. For more information, see Rename databases and tables.
   Remove/Remove all	OMS allows you to remove one or more objects temporarily selected to the target database. Remove a single migration object In the Target Object(s) list, hover the pointer over the target object and click Remove. Remove all migration objects In the Target Object(s) list, click Remove All in the upper-right corner. In the dialog box that appears, click OK.

6. Click Next. In the Migration Options step, configure the parameters.

   • Schema migration

     In the Select Migration Type step, select Schema Migration to display the following parameters.

     

     Parameter	Description
     Automatically Enter Next Stage upon Completion	If you select schema migration and any other migration type, you can specify whether to automatically proceed to the next stage after schema migration is completed. The default value is Yes. You can also view and modify this value on the Schema Migration tab of the data migration task details page.
     Normal Index Migration Method	The migration method for non-unique key indexes associated with the migrated table objects. Currently, only Do Not Migrate is supported.

   • Full migration

     After you select Select Migration Type and select Full Migration, the following parameters are displayed.

     

     Parameter	Description
     Full Migration Rate Limit	You can choose whether to limit the full migration rate as needed. If you choose to limit it, you must specify the RPS and BPS. The RPS specifies the maximum rows of data migrated to the target database per second during full migration, and the BPS specifies the maximum amount of data in bytes migrated to the target database per second during full migration. Note The RPS and BPS values specified here are only for throttling. The actual full migration performance is subject to factors such as the settings of the source and target databases and the instance specifications.
     Full Migration Resource Configuration	You can select Small, Medium, or Large to use the corresponding default values of Read Concurrency, Write Concurrency, and Memory. You can also customize the resource configurations for full migration. By setting the resource configuration for the Full-Import component, you can limit the resource consumption of a task in the full migration phase. Notice In the case of custom configurations, the minimum value is 1, and only integers are supported.
     Handle Non-empty Tables in Target Database	Valid values: Ignore and Stop Migration. If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS retains the existing data and records the conflict data. If you select Stop Migration and a target table contains data, an error is returned during full migration, indicating that the migration is not allowed. In this case, you must clear the data in the target table before you can continue with the migration. Notice After an error is returned, if you click Resume in the dialog box, OMS ignores this error and continues to migrate data. Proceed with caution.

   • Incremental synchronization

     After you select Select Migration Type and select Incremental Synchronization, the following parameters are displayed.

     

     Parameter	Description
     Incremental Synchronization Rate Limit	You can choose whether to limit the incremental synchronization rate as needed. If you choose to limit it, you must specify the RPS and BPS. The RPS specifies the maximum rows of data synchronized to the target database per second during incremental synchronization, and the BPS specifies the maximum amount of data in bytes synchronized to the target database per second during incremental synchronization. Note The RPS and BPS values specified here are only for throttling. The actual incremental synchronization performance is subject to factors such as the settings of the source and target databases and the instance specifications.
     Incremental Log Pull Resource Configuration	You can select Small, Medium, or Large to use the corresponding default value of Memory. You can also customize the resource configurations for incremental log pull. By setting the resource configuration for the Store component, you can limit the resource consumption of a task in log pull in the incremental synchronization stage. Notice In the case of custom configurations, the minimum value is 1, and only integers are supported.
     Incremental Data Write Resource Configuration	You can select Small, Medium, or Large to use the corresponding default values of Write Concurrency and Memory. You can also customize the resource configurations for incremental data writes. By setting the resource configuration for the Incr-Sync component, you can limit the resource consumption of a task in data writes in the incremental synchronization stage. Notice In the case of custom configurations, the minimum value is 1, and only integers are supported.
     Incremental Record Retention Duration	The duration that incremental parsed files are cached in OMS. A longer retention duration results in more disk space occupied by the Store component.
     Incremental Synchronization Start Timestamp	If you have selected Full Migration as the migration type, this parameter is not displayed. If you have selected Incremental Synchronization but not Full Migration, specify a point in time after which the data is to be synchronized. The default value is the current system time. For more information, see Set an incremental synchronization timestamp.

   • Reverse increment

     This section is displayed only if you select Reverse Increment in the Select Migration Type step. The configuration parameters of reverse increment are the same as those of incremental synchronization. You can select Reuse Incremental Synchronization Configuration in the upper-right corner.

     

   • Advanced options

     This section is displayed only if the MySQL compatible mode of the target OceanBase Database is V4.3.0 or later, and you select Schema Migration in the Select Migration Type step.

     

     The table object storage types of the target include Default, Row Storage, Column Storage, and Hybrid Row-Column Storage. This configuration determines the storage type of the target table objects during schema migration or incremental synchronization. For more information, see default_table_store_format.

     Note

     The value Default means that other parameters are automatically set based on the parameter configurations of the target database. Table objects in schema migration are written to corresponding schemas based on the specified storage type.

   If the configuration parameters on the page do not meet your requirements, you can click Parameter Configuration at the bottom of the page to configure more parameters. You can also reference a task template or component template that has been configured.

   

7. Click Precheck to perform a precheck on the data migration task.

   In the Precheck step, OMS checks whether the network connections of the databases meet the requirements. You can start a data migration task only after all precheck tasks are passed. If the precheck fails:

   • You can troubleshoot and fix the issue, and then perform the precheck again until the precheck succeeds.

   • You can also click Skip in the Actions column of the failed precheck item. A dialog box appears, prompting you to confirm the impact of skipping the operation. If you confirm that you want to skip the operation, click OK in the dialog box.

8. Click Start Task. If you do not need to start the task, click Save to go to the details page of the data migration task, where you can manually start the task as needed.

   You can click Configure Validation Task in the upper-right corner of the details page to compare the data between the source and target databases. For more information, see Create a data verification task.

   OMS allows you to reduce the number of migration objects during a data migration task. For more information, see View and modify migration objects. After the data migration task is started, it is executed based on the selected migration types. For more information, see the "View migration details" section in View details of a data migration task.