This topic describes how to use OceanBase Migration Service (OMS) Community Edition to migrate data from a Milvus database to OceanBase Database Community Edition.
Background information
Milvus is a high-performance and highly scalable vector database service that provides powerful data modeling features. For more information, see Milvus documentation.
The following table describes the key concepts of Milvus.
| Concept | Description |
|---|---|
| Database | A database is equivalent to a schema. You can create multiple databases in Milvus. A database can contain multiple collections. |
| Collection | A collection is equivalent to a table. It stores detailed data and can contain multiple fields. When you create a collection, you can specify the withEnableDynamicField attribute to allow different records to have different fields. |
| Primary key | The primary key field. A collection contains only one primary key field. |
| Partition | A partition is a division of a collection. A partitioned collection is equivalent to a partitioned table. |
OMS V4.2.9-CE and later allow you to migrate dynamic fields from a Milvus database to OceanBase Database.
If the target table has the
OMS_DYNAMIC_FIELDfield or thesink.omsDynamicFieldparameter is set to a value, the dynamic field data is written to theOMS_DYNAMIC_FIELDfield or the field specified by thesink.omsDynamicFieldparameter as a whole JSON string.If the target table does not have the
OMS_DYNAMIC_FIELDfield and thesink.omsDynamicFieldparameter is not set, the dynamic field is discarded.
Limitations
- Limitations on operations in the source database
Do not perform DDL operations that modify database or table schemas during schema migration or full migration. Otherwise, the data migration task may be interrupted.
At present, data migration tasks from a Milvus database to OceanBase Database Community Edition do not support full verification and reverse increment.
Milvus 2.4.5 and later are supported.
OMS Community Edition supports migrating databases, tables, and columns with ASCII-compliant names that do not contain special characters (spaces, line breaks, or |"'`()=;/&).
To ensure the performance of a data migration task, we recommend that you migrate no more than 1,000 tables at a time.
Data type mapping
| Milvus database | OceanBase Database Community Edition |
|---|---|
| BOOL | BOOL/TINYINT(1) Boolean type. It stores true or false and describes binary states. |
| INT8 | TINYINT 8-bit integer. It stores small-range integer data. |
| INT16 | SMALLINT 16-bit integer. It stores medium-range integer data. |
| INT32 | INT/INTEGER 32-bit integer. It stores general integer data such as the quantity of a product or a user ID. |
| INT64 | BIGINT 64-bit integer. It stores large-range data such as timestamps or identifiers. |
| FLOAT | FLOAT 32-bit floating point number. It stores data that requires general precision, such as grades or temperatures. |
| DOUBLE | DOUBLE 64-bit double-precision floating point number. It stores high-precision data such as financial information or data for scientific computing. |
| VARCHAR | VARCHAR You can specify max_length (the value range is 1 to 65,535) to define the maximum number of bytes that the VARCHAR field can store. |
| FLOAT_VECTOR | VECTOR Vector data type. It stores 32-bit floating point numbers. It is commonly used to represent real numbers in scientific computing and machine learning. It is suitable for scenarios that require high precision, such as distinguishing similar vectors. |
| FLOAT16_VECTOR | VECTOR It stores 16-bit half-precision floating point numbers. It is used in deep learning and GPU computing. It can save storage space in scenarios that do not require high precision, such as the low-precision recall phase of a recommendation system. |
| BFLOAT16_VECTOR | VECTOR It stores 16-bit brain floating point numbers (bfloat16). It is used in scenarios that require fast processing of a large number of vectors, such as large-scale image retrieval. |
| BINARY_VECTOR | VARBINARY It stores binary vectors. |
| SPARSE_FLOAT_VECTOR | JSON It stores sparse vectors. The value has a Map structure similar to KV: {1: 0.6180500771665887, 0: 0.8255673677331509, 4: 0.4227293590979655, 2: 0.09220348558380687}. |
Procedure
Create a data migration task.
Log in to the console of OMS Community Edition.
In the left-side navigation pane, click Data Migration.
On the Data Migration page, click Create Task in the upper-right corner.
On the Select Source and Target page, configure the parameters.
Parameter Description Migration Task Name We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 64 characters in length. Tag (Optional) Click the field and select a tag from the drop-down list. You can also click Manage Tags to create, modify, and delete tags. For more information, see Use tags to manage data migration tasks. Source If you have created a Milvus data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create one in the dialog box that appears on the right. For more information about the parameters, see Create a Milvus data source. Target If you have created an OceanBase-CE data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create one in the dialog box that appears on the right. For more information about the parameters, see Create an OceanBase-CE data source. Click Next. On the Select Migration Type page, configure the parameters.
Options for Migration Type are Schema Migration, Full Migration, and Incremental Synchronization.
Migration type Description Schema Migration After a schema migration task starts, OMS Community Edition migrates the data object definitions (tables, indexes, constraints, comments, and views) from the source database to the destination database and automatically filters out temporary tables.
Manual Table Schema Adjustment Required: OMS Community Edition allows you to customize the table structure. If you select this option, only the table structure SQL statements are obtained during the schema migration phase. After you modify the table structure, the migration task is executed.Full Migration After a full migration task starts, OMS Community Edition migrates the existing data in the source database tables to the corresponding tables in the destination database. Incremental Synchronization After an incremental synchronization task starts, OMS Community Edition synchronizes the data that has changed in the source database (newly added, modified, or deleted data) to the corresponding tables in the destination database.
Incremental Synchronization currently supports onlyInsertandDeletein **DML Synchronization`.Click Next. On the Select Migration Objects page, select the migration objects and migration scope.
You can select Specify Objects or Match Rules to specify the migration objects. The following procedure describes how to specify migration objects by using the Specify Objects option. For information about the procedure for specifying migration objects by using the Match Rules option, see Configure matching rules for migration objects.
Notice
The name of a table to be migrated, as well as the names of columns in the table, must not contain Chinese characters.
If a database or table name contains double dollar signs ("$$"), you cannot create the migration task.
OMS Community Edition allows you to import objects, rename objects, and remove one or all migration objects.
Operation Description Import objects - In the right-side list of the Specify Migration Scope section, click Import Objects in the upper-right corner.
- In the dialog box that appears, click OK.
Notice:
The imported objects will overwrite the previously selected objects. Proceed with caution. - In the Import Migration Objects dialog box, import the objects to be migrated.
You can import a CSV file to rename databases and tables. For more information, see Download and import migration object settings. - Click Validate.
- After the validity check is passed, click OK.
Rename OMS Community Edition allows you to rename migration objects. For more information, see Rename databases and tables. Remove one or all objects OMS Community Edition allows you to remove a single object or all objects to be migrated to the target database during data mapping. - To remove a single migration object:
In the list on the right of the Specify Migration Scope section, move the pointer over the target object and click Remove. - To remove all migration objects:
In the list on the right of the Specify Migration Scope section, click Remove All in the upper-right corner. In the dialog box that appears, click OK.
Click Next. On the Migration Options page, configure the parameters.
To view or modify the parameters of the full migration component, click Configuration Details in the upper-right corner of the Full Migration section. To view or modify the parameters of the incremental synchronization component, click Configuration Details of incr Increment in the upper-right corner of the Incremental Synchronization section. For more information about the parameters, see the Component parameters section.
Full migration
The following parameters are displayed only if you have selected Full Migration on the Select Migration Type page.
Parameter Description Concurrency Speed Valid values: Stable, Normal, Fast, and Custom. The amount of resources to be consumed by a full migration task varies based on the migration performance. If you select Custom, you can set Read Concurrency, Write Concurrency, and JVM Memory as needed. Processing Strategy When Records Exist in Target Object Valid values: Ignore and Stop Migration. - If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Notice
If you select Ignore, data is pulled in IN mode for full verification. In this case, the scenario where the target contains more data than the source cannot be verified, and the verification efficiency will be decreased.
- If you select Stop Migration and a target table contains data, an error is returned during full migration, indicating that the migration is not allowed. In this case, you must clear the data in the target table before you can continue with the migration.
Notice
After an error is returned, if you click Resume in the dialog box, OMS Community Edition ignores this error and continues to migrate data. Proceed with caution.
Writing Method Valid values: SQL (specifies to write data to tables by using INSERTorREPLACE) and Direct Load (specifies to write data through direct load). At present, you cannot write vector data by using direct load.- If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Incremental synchronization
The following parameters are displayed only if you have selected Incremental Synchronization on the Select Migration Type page.
Parameter Description Concurrency Speed Valid values: Stable, Normal, Fast, and Custom. The amount of resources to be consumed by an incremental synchronization task varies based on the synchronization performance. If you select Custom, you can set Read Concurrency, Write Concurrency, and JVM Memory as needed. Incremental Synchronization Start Timestamp - If you have selected Full Migration as the migration type, this parameter is not displayed.
- If you have selected Incremental Synchronization but not Full Migration, specify a point in time after which the data is to be synchronized. The default value is the current system time. For more information, see Set an incremental synchronization timestamp.
Click Precheck to start a precheck on the data migration task.
During the precheck, OMS Community Edition checks the read and write privileges of the database users and the network connectivity of the databases. A data migration task can be started only after it passes all check items. If an error is returned during the precheck, you can perform the following operations:
Identify and troubleshoot the problem and then perform the precheck again.
Click Skip in the Actions column of the failed precheck item. In the dialog box that prompts the consequences of the operation, click OK.
Click Start Task. If you do not need to start the task now, click Save to go to the details page of the task. You can start the task later as needed.
OMS Community Edition allows you to modify the migration objects when the data migration task is running. For more information, see View and modify migration objects. After the data migration task is started, it is executed based on the selected migration types. For more information, see the Migration Details section in the View details of a data migration task topic.