This topic describes how to use OceanBase Migration Service (OMS) Community Edition to migrate data from a Milvus database to OceanBase Database Community Edition.
Background information
Milvus is a high-performance and highly scalable vector database service that provides powerful data modeling features. For more information, see Milvus documentation.
The following table describes the key concepts of Milvus.
| Concept | Description |
|---|---|
| Database | A database is equivalent to a schema. You can create multiple databases in Milvus. A database can contain multiple collections. |
| Collection | A collection is equivalent to a table. It stores detailed data and can contain multiple fields. When you create a collection, you can specify the withEnableDynamicField attribute to allow different records to have different fields. |
| Primary key | The primary key field. A collection contains only one primary key field. |
| Partition | A partition is a division of a collection. A partitioned collection is equivalent to a partitioned table. |
OMS V4.2.9-CE and later allow you to migrate dynamic fields from a Milvus database to OceanBase Database.
If the target table has the
OMS_DYNAMIC_FIELDfield or thesink.omsDynamicFieldparameter is set to a value, the dynamic field data is written to theOMS_DYNAMIC_FIELDfield or the field specified by thesink.omsDynamicFieldparameter as a whole JSON string.If the target table does not have the
OMS_DYNAMIC_FIELDfield and thesink.omsDynamicFieldparameter is not set, the dynamic field is discarded.
Limitations
- Limitations on operations in the source database
Do not perform DDL operations that modify database or table schemas during schema migration or full migration. Otherwise, the data migration task may be interrupted.
At present, data migration tasks from a Milvus database to OceanBase Database Community Edition do not support full verification and reverse increment
Milvus 2.4.5 and later are supported.
To ensure the performance of a data migration task, we recommend that you migrate no more than 1,000 tables at a time.
Data type mapping
| Milvus database | OceanBase Database Community Edition |
|---|---|
| BOOL | BOOL/TINYINT(1) Boolean type. It stores true or false and describes binary states. |
| INT8 | TINYINT 8-bit integer. It stores small-range integer data. |
| INT16 | SMALLINT 16-bit integer. It stores medium-range integer data. |
| INT32 | INT/INTEGER 32-bit integer. It stores general integer data such as the quantity of a product or a user ID. |
| INT64 | BIGINT 64-bit integer. It stores large-range data such as timestamps or identifiers. |
| FLOAT | FLOAT 32-bit floating point number. It stores data that requires general precision, such as grades or temperatures. |
| DOUBLE | DOUBLE 64-bit double-precision floating point number. It stores high-precision data such as financial information or data for scientific computing. |
| VARCHAR | VARCHAR You can specify max_length (the value range is 1 to 65,535) to define the maximum number of bytes that the VARCHAR field can store. |
| FLOAT_VECTOR | VECTOR Vector data type. It stores 32-bit floating point numbers. It is commonly used to represent real numbers in scientific computing and machine learning. It is suitable for scenarios that require high precision, such as distinguishing similar vectors. |
| FLOAT16_VECTOR | VECTOR It stores 16-bit half-precision floating point numbers. It is used in deep learning and GPU computing. It can save storage space in scenarios that do not require high precision, such as the low-precision recall phase of a recommendation system. |
| BFLOAT16_VECTOR | VECTOR It stores 16-bit brain floating point numbers (bfloat16). It is used in scenarios that require fast processing of a large number of vectors, such as large-scale image retrieval. |
| BINARY_VECTOR | VARBINARY It stores binary vectors. |
| SPARSE_FLOAT_VECTOR | JSON It stores sparse vectors. The value has a Map structure similar to KV: {1: 0.6180500771665887, 0: 0.8255673677331509, 4: 0.4227293590979655, 2: 0.09220348558380687}. |
Procedure
Create a data migration task.
Log in to the console of OMS Community Edition.
In the left-side navigation pane, click Data Migration.
On the Data Migration page, click New Task in the upper-right corner.
On the Select Source and Target page, configure the parameters.
Parameter Description Migration Task Name We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 64 characters in length. Tag (Optional) Click the field and select a tag from the drop-down list. You can also click Manage Tags to create, modify, and delete tags. For more information, see Use tags to manage data migration tasks. Source If you have created a Milvus data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create one in the dialog box that appears on the right. For more information about the parameters, see Create a Milvus data source. Target If you have created an OceanBase-CE data source, select it from the drop-down list. If not, click New Data Source in the drop-down list and create one in the dialog box that appears on the right. For more information about the parameters, see Create an OceanBase-CE data source. Click Next. On the Select Migration Type page, select Full Migration.
After a full migration task is started, OMS Community Edition migrates the existing data of tables in the source database to corresponding tables in the target database.
Click Next. On the Select Migration Objects page, select the migration objects and migration scope.
You can select Specify Objects or Match Rules to specify the migration objects. The following procedure describes how to specify migration objects by using the Specify Objects option. For information about the procedure for specifying migration objects by using the Match Rules option, see Configure matching rules for migration objects.
Notice
The name of a table to be migrated, as well as the names of columns in the table, must not contain Chinese characters.
If a database or table name contains double dollar signs ("$$"), you cannot create the migration task.
OMS Community Edition also allows you to rename objects, set row filters, view column information, and remove a single migration object or all migration objects.
Operation Description Rename objects OMS Community Edition allows you to rename migration objects. For more information, see Rename a database table.
Note that the new table name must also be in the format ofTableName$ColumnFamilyName.Configure settings OMS Community Edition allows you to filter rows by using WHEREconditions. For more information, see Use SQL conditions to filter data.
You can also view column information of the migration objects in the View Column section.Remove one or all objects OMS Community Edition allows you to remove a single object or all objects to be migrated to the target database during data mapping. - To remove a single migration object:
In the list on the right of the Specify Migration Scope section, move the pointer over the target object and click Remove. - To remove all migration objects:
In the list on the right of the Specify Migration Scope section, click Remove All in the upper-right corner. In the dialog box that appears, click OK.
Click Next. On the Migration Options page, configure the parameters for full migration.
Parameter Description Concurrency Speed Valid values: Stable, Normal, Fast, and Custom. The amount of resources to be consumed by a full migration task varies based on the migration performance. If you select Custom, you can set Read Concurrency, Write Concurrency, and JVM Memory as needed. Processing Strategy When Records Exist in Target Object Valid values: Ignore and Stop Migration. - If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Notice
If you select Ignore, data is pulled in IN mode for full verification. In this case, the scenario where the target contains more data than the source cannot be verified, and the verification efficiency will be decreased.
- If you select Stop Migration and a target table contains data, an error is returned during full migration, indicating that the migration is not allowed. In this case, you must clear the data in the target table before you can continue with the migration.
Notice
After an error is returned, if you click Resume in the dialog box, OMS Community Edition ignores this error and continues to migrate data. Proceed with caution.
Writing Method Valid values: SQL (specifies to write data to tables by using INSERTorREPLACE) and Direct Load (specifies to write data through direct load). At present, you cannot write vector data by using direct load.To view or modify parameters of the Full-Import component, click Configuration Details in the upper-right corner of the Full Migration section. To view or modify parameters of the Incr-Sync component, click Configuration Details of incr Increment in the upper-right corner of the Incremental Synchronization section. For more information about the parameters, see Component parameters.
- If you select Ignore, when the data to be inserted conflicts with the existing data of a target table, OMS Community Edition retains the existing data and records the conflict data.
Click Precheck to start a precheck on the data migration task.
During the precheck, OMS Community Edition checks the read and write privileges of the database users and the network connectivity of the databases. A data migration task can be started only after it passes all check items. If an error is returned during the precheck, you can perform the following operations:
Identify and troubleshoot the problem and then perform the precheck again.
Click Skip in the Actions column of the failed precheck item. In the dialog box that prompts the consequences of the operation, click OK.
Click Start Task. If you do not need to start the task now, click Save to go to the details page of the task. You can start the task later as needed.
OMS Community Edition allows you to modify the migration objects when the data migration task is running. For more information, see View and modify migration objects. After the data migration task is started, it is executed based on the selected migration types. For more information, see the Migration Details section in the View details of a data migration task topic.