Monitor and maintain vector indexes

This topic describes how to monitor and maintain vector indexes in OceanBase Database.

Monitoring

You can view the basic information and real-time status of vector indexes in system views.

Maintenance

If the amount of incremental data is too large, the search performance will be affected. To reduce the amount of data in the incremental data table, OceanBase Database introduces DBMS_VECTOR to maintain vector indexes.

Incremental refresh

If you write a large amount of data after you create an index, we recommend that you use the REFRESH_INDEX procedure to perform an incremental refresh. For more information, see REFRESH_INDEX.

The system checks for incremental data every 15 minutes. If the number of incremental data exceeds 10,000, an incremental refresh is automatically performed.

Full refresh (rebuild)

Manual full rebuild

If a large amount of data is updated or deleted after the index is created, we recommend that you use the REBUILD_INDEX procedure to perform a full refresh. For more information, see REBUILD_INDEX.

A full refresh is performed every 24 hours. If the new data exceeds 20% of the original data, a full refresh is automatically performed. The full refresh is performed in the background and asynchronously. First, a new index is created, and then the old index is replaced. During the rebuild, the old index remains available, but the overall process is relatively slow.

We also provide the vector_index_memory_saving_mode configuration item to control the memory usage during index rebuild. Enabling this mode can reduce the memory consumption during the rebuild of vector indexes in partitioned tables. Typically, a vector index rebuild requires twice the memory of the index. When the memory-saving mode is enabled, the system temporarily deletes the memory index of a partition after it is built to release memory, thereby effectively reducing the total memory required for the rebuild. For more information, see vector_index_memory_saving_mode.

Consider the following notes:

• When you perform an offline DDL operation (such as ALTER TABLE to modify the table structure or primary key), the index table is rebuilt. Since parallelism cannot be specified during index rebuild, the system uses a single thread by default. Therefore, when the data volume is large, the rebuild process is relatively slow, which affects the overall efficiency of the offline DDL operation.

• When you need to modify the index parameters during index rebuild, you must specify both type and distance in the parameter list, and they must be consistent with the original index type. For example, if the original index type is hnsw and the distance algorithm is l2, you must specify both type=hnsw and distance=l2 during the rebuild.

• During index rebuild, the following operations are supported:

  • Modify the values of m, ef_search, and ef_construction.
  • Online rebuild the ef_search parameter.
  • Rebuild the index type between hnsw and hnsw_sq.
  • Rebuild the index type between ivf_flat and ivf_flat, and between ivf_pq and ivf_pq.
  • Set the parallelism during the rebuild. For more information, see REBUILD_INDEX.

• During index rebuild, the following operations are not supported:

  • Modify the type and distance parameters.
  • Rebuild the index type between hnsw and ivf.
  • Rebuild the index type between hnsw and hnsw_bq.
  • Rebuild the index type between ivf_flat and ivf_pq.

Automatic partition rebuild (recommended)

The current version triggers automatic partition rebuild tasks in the following two scenarios:

• When you execute vector index search statements.

• When a scheduled check is performed. You can manually configure the execution cycle.

  In the oceanbase database, configure the execution cycle by using the vector_index_optimize_duty_time configuration item. Here is an example:

  ALTER SYSTEM SET vector_index_optimize_duty_time='[23:00:00, 24:00:00]';
  After the preceding configuration is completed, the partition rebuild task is executed only between 23:00:00 and 24:00:00. The task is not initiated during other periods. For more information about the parameters, see the corresponding configuration item documentation.
  
  

2. View the task progress and history

   You can query the CDB/DBA_OB_VECTOR_INDEX_TASKS or CDB/DBA_OB_VECTOR_INDEX_TASK_HISTORY view to view the task progress and history.

   Notice

   The preceding views cover partition rebuild tasks, segment merge tasks, and segment merge dump tasks. You can distinguish the task types by using the TASK_TYPE field in the views.

   You can view the current status of a task by using the status field:

   • 0 (PREPARE): The task is waiting to be executed.
   • 1 (RUNNING): The task is being executed.
   • 2 (PENDING): The task is paused.
   • 3 (FINISHED): The task is completed.

   The tasks with the status=FINISHED value are stored in the history table regardless of whether they are successful. For more information, see the corresponding view documentation.

3. Cancel a task

   To cancel a task, you can obtain the trace_id from the DBA_OB_VECTOR_INDEX_TASKS or CDB_OB_VECTOR_INDEX_TASKS view and execute the following command:

   ALTER SYSTEM CANCEL TASK <trace_id>;
   

   Here is an example:

   ALTER SYSTEM CANCEL TASK "Y61480BA2D976-00063084E80435E2-0-1";
   

Manually force a dump or major compaction (Alternative)

The dump and major compaction of incremental segments are usually automatically triggered by the background modules of the database system when specific conditions are met. OceanBase Database also allows you to manually force a dump or major compaction by using the FLUSH_INDEX and COMPACT_INDEX procedures in the DBMS_VECTOR system package. This enables you to perform maintenance tasks as needed.

For syntax and examples, see FLUSH_INDEX and COMPACT_INDEX.

After triggering, you can query the DBA_OB_VECTOR_INDEX_TASKS view. task_type=5 indicates a triggered dump task, and task_type=6 indicates a triggered major compaction task. These tasks may execute very quickly. If no records are found in the above view, you can check the historical task view DBA_OB_VECTOR_INDEX_TASK_HISTORY.

References

• Vector index performance optimization
• The auto-discard and major compaction of an incremental segment can be controlled by setting the threshold. For more information, see Vector index memory management.