Log compression and decompression

Starting from V4.3.1, OceanBase Database supports syslog compression with a compression ratio ranging from 15:1 to 20:1.

Compression can help reduce the sizes of log files, save disk space, and facilitate transmission and storage. Decompression restores compressed log files to the original readable format to facilitate log viewing and analysis.

Log compression

To control syslog compression, perform the following steps:

1. Enable syslog compression.

   After syslog compression is enabled, the observer.log, rootservice.log, election.log, and trace.log files suffixed with a timestamp are compressed, and those suffixed with .wf are not compressed. You can specify a compression algorithm by using the syslog_compress_func parameter to enable syslog compression. Supported compression algorithms are zstd_1.0 and zstd_1.3.8. For example, you can set the syslog compression algorithm to zstd_1.3.8. For more information about the syslog_compress_func parameter, see syslog_compress_func.

   obclient> ALTER SYSTEM SET syslog_compress_func = 'zstd_1.3.8';
   

2. Set the disk space available for syslogs.

   When the total size of existing syslogs is about to reach the specified upper limit, the syslog files are deleted from the oldest ones, including those that are compressed. You can specify the syslog_disk_size parameter to set the maximum disk space available for syslogs. For example, you can set the maximum disk space available for syslogs to 100 GB. For more information about the syslog_disk_size parameter, see syslog_disk_size.

   Note

   The actual disk space available for syslogs is subject to the hardware specifications and actual requirements. If you want to allow syslogs to occupy the entire disk space, you can set this parameter value to 0 GB.

   obclient> ALTER SYSTEM SET syslog_disk_size='100GB';
   

3. Set the number of uncompressed syslog files. You can control the minimum number of uncompressed syslog files for each log type to facilitate problem analysis based on logs. In practice, the number of uncompressed syslog files can exceed the specified value but will not be smaller than this value. You can specify the syslog_file_uncompressed_count parameter to set the number of uncompressed syslog files. For example, you can set the number of uncompressed syslog files to 50. For more information about the syslog_file_uncompressed_count parameter, see syslog_file_uncompressed_count.

   Note

   Set the parameter value based on the actual situation. For example, if you set the parameter value to 50, 200 syslog files are not compressed in total for the four log types, and the total size is about 50 GB (256 MB × 200).

   obclient> ALTER SYSTEM SET syslog_file_uncompressed_count=50;
   

Log decompression

You can view a compressed log file by using either of the following tools:

• Use zstd.

  1. Install zstd.

     Download zstd from the official website.

  2. You can use zstd to view a compressed file in either of the following ways:

     • Use zstd to decompress the file and view its content.

       1. Run the zstd -d xxx.zst command to decompress the file. In the command, xxx.zst is the name of the file to be decompressed.

       2. Run the vim xxx command to view the content of the file. In the command, xxx is the name of the decompressed file.

     • Use zstd to directly view the content of the compressed file.

       Run the zstd -dc xxx.zst command to directly view the content of the compressed file. In the command, xxx.zst is the name of the compressed file.

• Use obdiag.

  1. Install obdiag.

     For more information about how to install obdiag, see Install obdiag.

  2. Use obdiag to decompress the file.

     Run the obdiag gather log [options] command to search for the decompressed log file. For more information about obdiag, see obdiag gather log.