Node disk I/O high

There are many reasons that can lead to high disk I/O on an OBServer node. Apart from an increase in traffic, it is usually the result of factors such as migration, replication, and major compaction. The usual approach to handling this issue is to downgrade some tasks with high I/O load. This topic provides information on how to handle this issue in emergency situations.

Emergency handling procedure

1. Pause running major compaction tasks.

   If a major compaction is being performed on the node with high disk I/O, pause the major compaction to reduce the I/O load.

   ALTER SYSTEM SUSPEND MERGE [ZONE [=] 'zone'];
   

   Once the I/O pressure is relieved, you can resume the major compaction.

   ALTER SYSTEM RESUME MERGE [ZONE [=] 'zone'];
   

2. Pause running backup tasks.

   You can use OCP to check if the current node is performing backups. If so, pausing the backups can alleviate the I/O pressure.

3. Pause running data transmission, import, and export tasks.

   You can use the TopSQL and Session Management features in OCP to identify SQL statements that are performing batch writing. If you cannot quickly identify which system the batch task is from, you can use OMS to check whether the current node is performing data transmission tasks. If yes, pause them to alleviate the I/O pressure before resuming those tasks.

   You can use ODC to check whether the current node is performing data import tasks. If yes, pause them. For more information, see Import tasks.

   You can also pause the scheduled tasks requested by third-party big data platforms or DataX.

4. Reduce the number of threads for minor compaction

   A high degree of parallelism (DOP) during minor compaction can increase disk I/O. The compaction_mid_thread_score parameter specifies the number of threads for minor compaction. You can decrease the value of this parameter to reduce disk I/O usage. The default value is 0, which indicates that the system adaptively adjusts the number of threads. For 64 CPU cores, it is generally set to 10. You can adjust it according to your specific situation. The modification takes effect immediately without the need for a restart.

   ALTER SYSTEM SET compaction_mid_thread_score= 5;
   

   After the modification, you can execute the SHOW PARAMETERS statement to check whether the modification is successful.

   SHOW PARAMETERS LIKE 'compaction_mid_thread_score';
   

   Alternatively, you can reduce the number of threads for minor compaction by modifying the minor compaction strategy in the OCP console. For more information, see Modify minor compaction settings.

5. Reduce the network bandwidth of background tasks.

   ALTER SYSTEM SET sys_bkgd_net_percentage=30;  -- Default value: 60.
   

6. Apply throttling and add indexes to high-load SQL queries.

   If a particular high-load SQL query is identified, you can limit the concurrency of the SQL query by adding the max_concurrent hint to its execution plan. This achieves throttling the SQL query.

   If a full table scan causes high I/O due to missing indexes, you can add indexes to the relevant tables as needed.

7. Pause running index creation tasks.

   If index creation for a large table is currently in progress in the cluster, you can pause it if necessary and resume the creation after the cluster has been recovered.