REDO log management

REDO log introduction

REDO logs are a key component of OceanBase Database that are used to recover from downtime and maintain data consistency among multiple replicas. OceanBase Database automatically manages REDO logs, including the creation of log files, synchronization of log files between multiple replicas, reuse of log files, and recovery after downtime.

• Downtime recovery

  REDO logs record all modifications to the MemTable. Before you commit a transaction, you need to write REDO logs to the memory from the buffer first. If the OBServer process exits or the OBServer fails, the OBServer will scan local REDO log files for data recovery after the restart. This ensures the data persistence in OceanBase.

• Maintenance of multi-replica data consistency

  OceanBase Database uses REDO logs to synchronize the data between multiple replicas. OceanBase adopts the Multi-Paxos protocol to synchronize the data between multiple replicas. This protocol ensures that OceanBase returns results to users only after the committed transactions are executed on the majority of nodes. Then, OceanBase Database synchronizes these transactions to all replicas of the table. Followers that provide services for applications reconstruct the MemTable by replaying REDO logs in real time. This allows them to provide services on standby servers such as follower read. OceanBase Database provides stronger disaster recovery capability by persisting REDO logs on the majority of replicas.

REDO log composition

REDO logs of OceanBase Database comprise two parts:

• CLOG

  A commit log (CLOG) records the content of REDO logs. It is located under the storage/clog directory. The file number starts from 1 and increases continuously, while the file ID is not reused and the size of a single file is 64 MB. CLOG files record changes made to the data in OceanBase and guarantee the data persistence.

• ILOG

  An index log (ILOG) records the location information of commit logs with the same file ID in a same partition. These commit logs have become the majority of all logs. They are located under the storage/ilog directory. The file number starts from 1 and increases continuously, while the file ID is not reused and the size of a single file is variable. The log files under the storage/ilog directory are indexes for CLOG files, which are essentially an optimization. The deletion of ILOG files does not affect data persistence, but may affect the system recovery time. ILOG files have no corresponding relationship with CLOG files. An ILOG records much less content in a single log record than a CLOG does. Generally, the number of ILOG files is also much less than that of CLOG files.