Node clog disk full

If the clog disk is fully occupied, the cluster may exhibit symptoms such as election without a leader, data write failure, and lack of replicas.

Scenario

Data writes fail because the clog disk is full.

Emergency handling procedure

To address the full clog disk space issue without compromising cluster data consistency, perform the following steps:

1. Actively stop tenant writes to prevent the temporarily freed clog disk space from being quickly consumed again by business writes, which would prevent recovery.

2. After stopping cluster writes, temporarily increase the clog disk usage threshold from 95% to 98%.

   ALTER system SET log_disk_utilization_limit_threshold  = 98 server ='[IP address]:2882';
   

3. Observe the cluster for a period of time. In most cases, the cluster can recover once the clog catches up. If it does not recover, contact OceanBase Technical Support.