Overloaded NIC on an OBServer node

The high load of the network interface controller (NIC) on an OBServer node is related to background tasks such as batch processing, replication, and migration. This topic describes several solutions to this issue. The solutions are similar to those for reducing high I/O load, such as task downgrade and adjustment of migration and replication parameters.

Emergency procedure

To reduce the high load of the NIC on an OBServer node, you can perform the following operations:

1. Pause a running backup task.

   In the OceanBase Cloud Platform (OCP) console, you can check whether a backup task is running on the current node. If yes, pause the backup task to reduce the NIC load. Perform the following steps:

   1. In the left-side navigation pane, click Clusters. The cluster overview page appears.

   2. Find the cluster in the Clusters list, and click its name. In the left-side navigation pane of the page that appears, click Backup and Restore.

   3. On the backup and restore page of the cluster, find the task that is under backup scheduling and pause the task.**** To resume the backup task, perform the following steps:

      1. To pause the backup task, find the task and choose Actions > Pause.

      2. To resume the backup task, choose Actions > Restart to restart the task. The system periodically initiates the task at the specified time.

2. Pause the data import or export task.

   In the OceanBase Migration Service (OMS) console, you can check whether a data transmission task is running on the current node. If yes, pause the task to reduce the NIC load. To pause a data migration project, perform the following steps. You can pause only a running data migration project. After a data migration project is paused, OMS pauses the running subtasks in the orchestration task list and stops scheduling all subsequent subtasks.

   1. Log on to the OMS console.

   2. In the left-side navigation pane, click Data Migration.

   3. On the Data Migration page, click Pause in the Actions column of the project that you want to pause.

      You can also click the project name on the Data Migration page to go to the details page, and click Pause in the upper-right corner.

   You can manually resume a paused or failed migration project. OMS runs the pending subtasks in the orchestration task list in sequence.

   1. Log on to the OMS console.

   2. In the left-side navigation pane, click Data Migration.

   3. On the Data Migration page, click Restore in the Actions column of the project that you want to resume.

      You can also click the project name on the Data Migration page to go to the details page, and click Restore in the upper-right corner.

3. Lower the concurrency of migration tasks.

   If the high load of the NIC is accompanied by unit migration or load balancing tasks in the OceanBase cluster, you can throttle the I/O by limiting the concurrency of migration tasks, which helps reduce the NIC load.

   alter system set migrate_concurrency=5;  -- Default value: 10.
   alter system set data_copy_concurrency=5; -- Default value: 10.
   alter system set server_data_copy_in_concurrency=2; -- Default value: 2. If the value is greater than 2, you can reset it to 2.
   alter system set server_data_copy_out_concurrency=2; -- Default value: 2. If the value is greater than 2, you can reset it to 2.
   

4. Lower the network bandwidth of background tasks.

   Run the following command:

   alter system set sys_bkgd_net_percentage=30;  -- Default value: 60.
   

5. Enable Clog compression.

   You can enable log compression to reduce the high load of the NIC on a node. Set the clog_transport_compress_all parameter to True to enable log compression. The default value False indicates that the log compression feature is disabled. The modification immediately takes effect without a restart. Syntax:

   ALTER SYSTEM SET clog_transport_compress_all = TRUE ;
   

6. Throttle highly concurrent SQL executions.

   If the disk I/O is high because of the execution of an SQL statement, insert the max_concurrent hint to the execution plan bound to the SQL statement to limit the SQL execution concurrency.

   CREATE OUTLINE outline_name ON sql_id USING HINT /*+max_concurrent(1)*/;
   

   For more information, see "SQL statement" in SQL Reference (MySQL Mode) and "SQL statement" in SQL Reference (Oracle Mode).