View the details of a data migration task

After a data migration task starts, you can view the task information on the details page of the task, such as the basic information, and the progress and status of the task.

Access the details page

1. Log on to the console of OceanBase Migration Service (OMS) Community Edition.

2. In the left-side navigation pane, click Data Migration.

3. On the Data Migration page, click the name of the target task. On the details page that appears, view the basic information and migration details of the task.

   On the Data Migration page, you can search for data migration tasks by tag, status, type, or keywords. A data migration task can be in one of the following states:

   • Not Started: The data migration task has not been started. You can click Start in the Actions column to start the task.

   • Running: The data migration task is in progress. You can view the data migration plan and current progress on the right.

   • Modifying: Migration objects are being modified for the data migration task.

   • Integrating: The data migration task is being integrated with the corresponding migration object modification task.

   • Paused: The data migration task is manually paused. You can click Resume in the Actions column to resume the task.

   • Failed: The data migration task has failed. You can view where the failure occurred on the right. To view the error message, click the task name to go to the task details page.

   • Completed: The data migration task is completed and OMS Community Edition has migrated the specified data to the destination database in the configured migration mode.

   • Releasing: The data migration task is being released. You cannot edit a data migration task in this state.

   • Released: The data migration task is released. After the task is released, OMS Community Edition terminates the current migration and incremental synchronization task.

View basic information

The Basic Information section displays the basic information of the current data migration task.

You can perform the following operations on the current data migration task:

• View migration objects

  Click View Objects in the upper-right corner. The migration objects of the current data migration task are displayed. You can modify the migration objects when the data migration task is running. For more information, see View and modify migration objects.

• View the component monitoring metrics

  Click View Component Monitoring in the upper-right corner to view the information about the Store, Incr-Sync, Full-Import, and Full-Verification components. You can perform the following operations on the components:

  • Start a component: Click Start in the Actions column of the component that you want to start. In the dialog box that appears, click OK.

  • Pause a component: Click Pause in the Actions column of the component that you want to pause. In the dialog box that appears, click OK.

  • Update a component: Click Update in the Actions column of the component that you want to update. On the Update Configuration page, modify the configurations and then click Update.

    Notice

    The system restarts after you update the component. Proceed with caution.

  • View logs: Click View Logs in the Actions column of the component. The View Logs page displays the latest logs. You can search for, download, and copy the logs.

• View or modify parameter configurations

  • For a data migration task in the Running state, click the More icon in the upper-right corner and then select View Parameter Configurations from the drop-down list to view the parameter configurations specified when the data migration task was created.

  • For a data migration task in the Not Started, Paused, or Failed state, click the More icon in the upper-right corner and select Modify Parameter Configurations from the drop-down list. In the Modify Parameter Configurations dialog box, modify the parameters, and click OK.

    The parameters that can be modified vary with the type of the data migration task and the phase of the task.

• Download object configurations

  OMS Community Edition allows you to download the settings of data migration tasks in batches. For more information, see Download and import the settings of migration objects.

• Modify the alert level

  OMS Community Edition allows you to modify the alert level of a data migration task. For more information, see Alert settings.

• Change data sources

  OMS Community Edition allows you to change a data source of a data migration task. However, you can use this feature only in the following scenarios. Otherwise, the data migration task will fail and cannot be recovered.

  • The data source has experienced a primary/standby switchover and you need to replace the IP address of the original primary database with that of the new primary database.

  • The IP address or port number of the data source is changed, but the data source remains unchanged.

  • The username or password for logging on to the data source is changed.

  Perform the following steps to change the data source:

  1. Go to the details page of a data migration task

  2. Click More in the upper-right corner and select Modify Data.

  3. In the Modify Data Source dialog box, select the new source or destination database as needed.

     Notice

     The type of the new data source must be the same as that of the current data source.

  4. Click OK.

View migration details

The Migration Details section displays the status, progress, start time, completion time, and total duration of all subtasks.

Schema migration

The definitions of data objects, such as tables, indexes, constraints, comments, and views, are migrated from the source database to the destination database. Temporary tables are automatically filtered out. If the source database is not of OceanBase Database Community Edition, OMS Community Edition performs SQL format conversion and construction based on the syntax definition and standard of the type of the destination tenant of OceanBase Database, and then replicates the data to the destination database.

When you advance to the forward switchover phase in a data migration task, the hidden columns and unique indexes will be automatically dropped based on the type of the data migration task.

You can view the overall status, start time, completion time, total time consumed, and table and view migration progress for a schema migration task on the Schema Migration page. You can also perform the following operations on an object:

• View Creation Syntax: On the Database or Table tab, click View next to the target object to view the creation syntax of a database, table, or index.

  Compatible DDL syntax executed on the OBServer node is displayed. Incompatible syntax is converted before it is displayed.

• Modify Creation Syntax and Try Again: View the error information, check and modify the definition of the conversion result of a failed DDL statement, and then migrate the data to the destination again.

• Retry/Retry All Failed Objects: You can retry failed schema migration tasks one by one or retry all failed tasks at a time.

• Export All Schemas: You can export the schema information of all objects and save it to your local storage.

• Skip/Batch Skip: You can skip failed schema migration tasks one by one or skip multiple failed tasks at a time. To skip multiple objects at a time, click Batch Skip in the upper-right corner. If you skip an object, its index is also skipped.

• Remove/Batch Remove: You can remove failed schema migration tasks one by one or remove multiple failed tasks at a time. To remove multiple failed tasks at a time, click Batch Remove in the upper-right corner. If you remove an object, its index is also removed.

• View Details: The DDL statements executed on the OBServer node and the execution error information of a failed schema migration task are displayed.

Full data migration

The existing data is migrated from tables in the source database to the corresponding tables in the destination database. On the Full Data Migration page, you can filter objects by source and destination databases, or select View Objects with Errors to filter out objects that hinder the overall migration progress. You can also view related information on the Table Objects, Table Indexes, and Full Data Migration Performance tabs. The status of a full data migration task changes to Completed only after the table objects and table indexes are migrated.

• On the Table Objects tab, you can view the names, source and destination databases, estimated data volume, migrated data volume, and status of tables.

• On the Table Indexes tab, you can view the table objects, source and destination databases, creation time, end time, time consumed, and status. You can also view the index creation syntax and remove unwanted indexes.

• On the Full Data Migration Performance tab, you can view the graphs of performance data such as the RPS and migration traffic of the source database and destination database, average read time and average sharding time of the source database, average write time of the destination database, and performance benchmarks. Such information can help you identify performance issues in a timely manner.

You can combine full data migration with incremental synchronization to ensure data consistency between the source and destination databases. If any objects fail to be migrated during full data migration, the causes of the failure are displayed.

Incremental synchronization

After incremental synchronization starts, OMS Community Edition synchronizes the data that has been changed (added, modified, or deleted) in the source database to the corresponding tables in the destination database. When services are continuously writing data to the source database, OMS Community Edition starts the incremental data pull module to pull incremental data from the source instance, parses and encapsulates the incremental data, and then stores the data in OMS Community Edition, before it starts the full data migration.

After a full data migration task is completed, OMS Community Edition starts the incremental data replay module to pull incremental data from the incremental data pull module. The incremental data is synchronized to the destination database after being filtered, mapped, and converted. If an Incr-Sync exception occurs after you execute a DDL statement on the source database and the data migration task fails, a page appears, displaying the DDL statement that causes the task failure and the Skip button. You can click Skip and confirm your operation.

For a data migration task in the Running state, you can view its latency, current timestamp, and incremental synchronization performance in the incremental synchronization section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a data migration task in the Paused or Failed state, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled. You can also view the specific information about incremental synchronization objects and the incremental synchronization performance.

• The Synchronization Object Statistics tab displays the statistics on table-level DML statements executed for each incremental synchronization object in the current task. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Synchronization Object Statistics tab are the sums of the statistics in the corresponding columns on this tab.

  

• The Incremental Synchronization Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the source database to the destination database, in seconds.

  • Migration traffic: The throughput of data flow for the synchronization from the source database to the target database, measured in KB/s.

  • Average execution time: the average execution time of an SQL statement, in ms.

  • Average commit time: the average commit time of a transaction, in ms.

  • RPS: The number of records processed per second.

  When you create a data migration task, we recommend that you specify related information such as the alert level and alert frequency, to help you understand the task status. OMS Community Edition provides low-level protection by default. You can modify the alert level based on your business requirements. For more information, see Alert settings.

  • When the incremental synchronization latency exceeds the specified alert threshold, the incremental synchronization status stays at Running and the system does not trigger any alerts.

  • When the incremental synchronization latency is less than or equal to the specified alert threshold, the incremental synchronization status changes from Running to Monitoring. After the incremental synchronization status changes to Monitoring, it will not change back to Running when the latency exceeds the specified alert threshold.

Full verification

After the full data migration and incremental data migration are completed, OMS Community Edition automatically initiates a full verification task to verify the data tables in the source and destination data sources.

You can also initiate custom data verification tasks in the incremental data synchronization process. On the Full Verification page, you can view the overall status, start time, end time, total consumed time, estimated total number of rows, number of migrated rows, real-time traffic, and RPS of the full verification task.

The Full Verification page contains the Verified Objects and Verification Performance tabs.

• On the Verified Objects tab, you can view the verification progress and verification object list.

• You can view the names, source and destination databases, full data verification progress and results, and result summary of all migration objects.

• You can filter migration objects by source or destination database.

• You can select View Completed Objects Only to view the basic information of objects that have completed schema migration, such as the object names.

• You can choose Reverify > Restart Full Verification to run a full verification again for all migration objects.

• Take note of the following items for tables with inconsistent verification results:

  If you need to reverify all data in the tables, choose Reverify > Reverify Abnormal Table.

  If you need to reverify only inconsistent data, choose Reverify > Verify Only Inconsistent Records.

  Notice

  Correction operations are not supported if the source database has no corresponding data.

• On the Full Verification Performance tab, you can view the graphs of performance data such as the RPS and verification traffic of the source and destination databases and performance benchmarks. Such information can help you identify performance issues in a timely manner.

  OMS Community Edition allows you to skip full verification for a task that is being verified or has failed verification. On the Full Verification page, click Skip Full Verification in the upper-right corner. In the dialog box that appears, click OK.

  Notice

  If you skip full verification, you cannot resume the verification task for data comparison and correction. You can initiate full verification again only by cloning the current task. Proceed with caution.

  After full verification is completed, you can click Go To Next Stage to start a forward switchover. After you enter the switchover process, you cannot recheck the current verification task to compare or correct data.

Forward switchover

Forward switchover is an abstract and standard process of traditional system cutover and does not involve the switchover of application connections. This process includes a series of tasks that are performed by OMS Community Edition for application switchover in a data migration task. You must make sure that the entire forward switchover process is completed before the application connections are switched over to the destination database.

Forward switchover is required for data migration. OMS Community Edition can ensure the completion of forward data migration in this process, and you can start the reverse incremental synchronization component based on your business needs. The forward switchover process involves the following operations:

1. You must make sure that data migration is completed and wait until forward synchronization is completed.

2. OMS Community Edition automatically supplements CHECK constraints, FOREIGN KEY constraints, and other objects that are ignored in the schema migration phase.

3. OMS Community Edition automatically drops the additional hidden columns and unique indexes that the migration depends on.

   This operation is performed only when you migrate data between instances of OceanBase Database Community Edition. For more information, see Mechanisms for handling hidden columns.

4. You must migrate triggers, functions, and stored procedures in the source database that are not supported by OMS Community Edition to the destination database.

5. You must disable triggers and FOREIGN KEY constraints in the source database. This operation is required only when the data migration task involves reverse incremental migration.

The forward switchover process contains the following steps:

1. Start forward switchover.

   In this step, you can start forward switchover, but no operation is performed in the background. After you confirm that data migration is completed, you can click Start Forward Switchover to start the process.

   Notice

   Before you start forward switchover, make sure that data writing has stopped in the source database.

2. Perform a switchover precheck.

   In this step, OMS Community Edition checks the following items:

   • Synchronization latency between the source and destination databases. If the synchronization latency is within 15 seconds, this check item is passed.

   • Write privilege of the account in the source database. If the data migration task involves reverse incremental migration, OMS Community Edition additionally checks whether the account configured in the source database has the privilege to write data, to ensure that data can be properly written during reverse incremental migration.

   • Privilege of the account in the destination database for incremental data read. If the data migration task involves reverse incremental migration, OMS Community Edition additionally checks whether the account configured in the destination database has the privilege to read data. This ensures that data can be properly written to the destination database during reverse incremental migration.

   • Incremental logs in the destination database. If the data migration task involves reverse incremental migration, OMS Community Edition additionally checks whether the incremental logging configuration in the destination database meets the log extraction requirements of reverse incremental migration.

   If the switchover precheck is passed, OMS Community Edition automatically performs the next step. If the precheck fails, you can click Retry or Skip.

   Notice

   If you click Skip, data loss may occur in the destination database, or the reverse incremental synchronization process may fail. Proceed with caution.

3. Start the destination store.

   Note

   This step is available only when the data migration task involves reverse incremental migration.

   If the precheck for forward switchover is passed, OMS Community Edition automatically starts incremental log pulling for the destination database. This way, OMS Community Edition obtains DML and DDL operations in the destination database and parses and saves related log data to prepare for reverse incremental synchronization. This step takes about 3 to 5 minutes.

4. Confirm that data writing has stopped in the source database.

   In this step, OMS Community Edition checks whether business data is still being written to the source database. After you confirm that no new data is written to the source database, click OK to go to the next step.

5. Confirm the data writing stop timestamp upon synchronization completion.

   In this step, OMS Community Edition checks whether the destination database is synchronized to the data writing stop timestamp in the source database. If not, OMS Community Edition continues to check the destination database until it is synchronized to the timestamp. This way, OMS Community Edition makes sure that all data in the destination database is updated.

6. Stop forward synchronization.

   In this step, you can stop forward synchronization. After forward synchronization is stopped, any data changes in the source database will no longer be synchronized to the destination database. If forward synchronization fails to be stopped, you can click Retry or Skip.

   Notice

   You can click Skip only after you confirm that forward synchronization is completed in the background. Otherwise, data in the source database may be unexpectedly written to the destination database. Proceed with caution.

7. Process database objects.

   In this step, you can process the objects that are ignored in data migration or not supported by OMS Community Edition. This ensures normal operations of your business after the switchover to the destination database.

   • Migrate database objects to the destination database: You must migrate triggers, functions, and stored procedures in the source database that are not supported by OMS Community Edition to the destination database. After you complete the migration, click Mark as Complete.

   • Disable triggers and FOREIGN KEY constraints in the source database: This operation is required only when the data migration task involves reverse incremental synchronization. It prevents data from being affected by triggers or FOREIGN KEY constraints, to avoid failures of reverse incremental synchronization. After you complete this operation, click Mark as Complete.

   • Supplement the objects ignored in schema migration to the destination database: OMS Community Edition automatically supplements the objects that are ignored in schema migration to the destination database, such as check constraints and FOREIGN KEY constraints. The preceding objects are migrated during schema migration by default.

   • Drop hidden columns and unique indexes added by OMS Community Edition: This operation is required only when you migrate data between instances of OceanBase Database Community Edition. OMS Community Edition automatically drops the hidden columns and unique indexes that are added to the destination database to ensure data consistency. This operation runs automatically, and the amount of time required depends on the amount of data in the destination database. You can click Skip to skip this operation, but you must manually perform it later. Proceed with caution. For more information, see Hidden column mechanisms.

8. Start reverse incremental synchronization.

   Note

   This step is available only when the data migration task involves reverse incremental migration.

   In this step, you can start incremental synchronization for the destination database to synchronize incremental DML or DDL operations from the destination database to the source database in real time. The configuration of incremental synchronization is the same as that specified when the task was created. For more information, see Incremental synchronization of DDL operations.

Reverse incremental migration

For a data migration task in the Running state, you can view its latency, current timestamp, and performance of reverse incremental migration in the Reverse Incremental Migration section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a data migration task in the Paused or Failed state, you can enable DDL/DML statistics collection to collect statistics on database operations performed after this feature is enabled. You can also view the specific information about the objects and performance of reverse incremental synchronization.

• The Synchronization Object Statistics tab displays the statistics on table-level DML statements executed for each incremental synchronization object in the current task. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Synchronization Object Statistics tab are the sums of the statistics in the corresponding columns on this tab.

• The Reverse Incremental Migration Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the destination database to the source database, in seconds.

  • Migration traffic: the traffic throughput of incremental data synchronization from the destination database to the source database, in Kbit/s.

  • Average execution time: the average execution time of an SQL statement, in ms.

  • Average commit time: the average commit time of a transaction, in ms.

  • RPS: The number of records processed per second.