Backup directory structures|V4.2.1| docs|Distributed Database

Backup directory structures

Last Updated：2023-12-25 08:49:41 Updated

Structure of the log archive directory

You can specify the time interval for splitting the log archive directory. By default, the directory is split by day. In other words, the log data directory of one day corresponds to a backup_piece. A backup_piece directory contains a single_piece_info file, which records the backup_piece information, and a tenant_backup_piece_infos file, which records the information about all backup_pieces of the tenant.

rounds in the log archive directory is a placeholder for log archiving. Each placeholder indicates the start and end of a log round. A placeholder is in the format of round_DSETID_ROUNDID_STATE, where:

DSETID indicates the ID of the archive path.
ROUNDID indicates the round of log archiving.
STATE indicates the start or end of the log round.

pieces in the log archive directory is a placeholder for log archiving. Each placeholder indicates the start and end of a log piece. A placeholder is in the format of piece_DSETID_ROUNDID_PIECEID_STATE_DATE, where:

DSETID indicates the ID of the archive path.
ROUNDID indicates the round of log archiving.
PIECEID indicates the ID of the log piece.
STATE indicates the start or end of the log round.
DATE indicates the backup time. The value is in the format of yyyymmddThhmmss. It records the start or end time.

The structure of the log archive directory is as follows:

/log_archive_dest/ # The user-defined root directory for backing up log files.
  ├── format # The format information.
  ├── check_file
  │ ├── 1002_connect_file_20220601T010101 // The connectivity check file, suffixed with date and time in the format of YYYYMMDDTHHMMSS.
  ├── rounds # The rounds placeholder directory.
  |   ├── round_1001_1_start # The round start placeholder.
  |   ├── round_1001_1_end # The round end placeholder.
  |   ├── round_1001_2_end # The round end placeholder.
  ├── pieces # The piece placeholder directory.
  |   ├── piece_d1001r1p1_start_20220601T010101 # The piece start placeholder, in the format of piece_DSETID_ROUNDID_PIECEID_start_DATE.
  |   ├── piece_d1001r1p1_end_20220602T010101   # The piece end placeholder, in the format of piece_DSETID_ROUNDID_PIECEID_end_DATE.
  |   ├── piece_d1001r1p2_start_20220602T010101 # The piece start placeholder, in the format of piece_DSETID_ROUNDID_PIECEID_start_DATE.
  |   ├── piece_d1001r1p2_end_20220602T120000 # The piece start placeholder, in the format of piece_DSETID_ROUNDID_PIECEID_start_DATE.
  |   ├── piece_d1001r2p3_start_20220603T080000 # The piece start placeholder, in the format of piece_DSETID_ROUNDID_PIECEID_start_DATE.
  ├── piece_d1001r1p1 # The piece directory (frozen piece).
  |   ├──piece_d1001r1p2_20220601T010101_20220602T010101
  |   ├──single_piece_info
  |   ├──tenant_archive_piece_infos
  |   ├──checkpoint # The checkpoint directory.
  |   ├── ├──checkpoint_info_0
  |   ├── ├──checkpoint_info_1
  |   ├──piece_file_info # The list of all log stream files.
  |   ├──logstream_0 # The log stream directory.
  |   ├── ├──ls_file_info # The list of log stream files.
  |   |   ├── 1 # The log stream files.
  |   |   ├── 2
  |   |   ├── 3
  |   ├──logstream_1 # The log stream directory.
  |   ├── ├──ls_file_info # The list of log stream files.
  |   |   ├── 1 # The log stream files.
  |   |   ├── 2
  |   |   ├── 3
  ├── piece_d1001r1p2 # The piece directory (frozen piece).
  |   ├──piece_d1001r1p2_20220602T010101_20220602T120000
  |   ├──single_piece_info
  |   ├──tenant_archive_piece_infos
  |   ├──checkpoint # The checkpoint directory.
  |   ├── ├──checkpoint_info_0
  |   ├──piece_file_info # The list of all log stream files.
  |   ├──logstream_0 # The log stream directory.
  |   ├── ├──ls_file_info # The list of log stream files.
  |   |   ├── 4 # The log stream files.
  |   |   ├── 5
  |   ├──logstream_1 # The log stream directory.
  |   ├── ├──ls_file_info # The list of log stream files.
  |   |   ├── 4 # The log stream files.
  ├── piece_d1001r2p3 # The piece directory (active piece).
  |   ├──tenant_archive_piece_infos
  |   ├──checkpoint # The checkpoint directory.
  |   ├── ├──checkpoint_info_0
  |   ├──logstream_0 # The log stream directory.
  |   |   ├── 6 # The log stream files.
  |   ├──logstream_1 # The log stream directory.
  |   |   ├── 5 # The log stream files.

Structure of the data backup directory

Each data backup corresponds to a backup_set directory. Each time the ALTER SYSTEM BACKUP DATABASE statement is executed, a new backup_set directory is generated, containing all data that is backed up this time. A backup_set directory contains a single_backup_set_info file, which records the backup_set information, and a tenant_backup_set_infos file, which records the information about all backup_sets of the tenant.

Backup_sets in the data backup directory are placeholders for data backup. Each placeholder indicates the start and end of a data backup round. The placeholder is in the format of backup_set_N_TYPE_DATE_STATE, where:

N indicates the round of data backup.
The value of TYPE can be FULL (full backup) or INC (incremental backup).
DATE indicates the data backup start or end time. The value is in the format of YYYYMMDDTHHMMSS.
The value of STATE can be start (indicating that the data backup starts), end_success (indicating that the data backup succeeded), or end_failed (indicating that the data backup failed).

The structure of the data backup directory is as follows:

/data_backup_dest/ # The user-defined data backup path.
├── format // The format information.
├── check_file
│   └── 1002_connect_file_20220601T010101          // The connectivity check file, suffixed with date and time in the format of YYYYMMDDTHHMMSS.
├── backup_sets  // The backup_set placeholder for the list.
│   ├── backup_set_1_full_start                   // The start placeholder for a full backup.
│   ├── backup_set_1_full_end_success_20220602T0200510   // The end placeholder for a full backup.
│   ├── backup_set_2_inc_start                  // The start placeholder for an incremental backup.
│   └── backup_set_2_inc_end_success_20220609T021001   // The end placeholder for an incremental backup.
├── backup_set_1_full
│   ├── backup_set_1_full_20220602T010000_20220602T0200510  //The placeholder that represents the start and end time of the backup.
│   ├── single_backup_set_info               //The metadata of the current backup_set.
│   ├── tenant_backup_set_infos              /The full backup_set information of the current tenant.
│   ├── infos
│   │   ├── data_info_turn_1                // The tenant backup files under turn 1.
│   │   │   ├── tablet_log_stream_info      // The file describing the mapping between tablets and log streams.
│   │   │   ├── tenant_major_data_macro_range_index // The major macroblock index.
│   │   │   ├── tenant_major_data_meta_index // The major meta index.
│   │   │   ├── tenant_major_data_sec_meta_index // The mapping between the major logic ID and the physical ID.
│   │   │   ├── tenant_minor_data_macro_range_index // The minor macroblock index.
│   │   │   ├── tenant_minor_data_meta_index // The minor meta index.
│   │   │   └── tenant_minor_data_sec_meta_index // The mapping between the minor logic ID and the physical ID.
│   │   ├── diagnose_info                   // The diagnosis information file of the backup_set.
│   │   ├── locality_info                   // The tenant locality information of the current backup_set.
│   │   └── meta_info                       // The tenant-level log stream metadata file, which contains the metadata of all log streams.
│   │       ├── ls_attr_info_turn_1
│   │       └── ls_meta_infos
│   ├── logstream_1                         // Log stream 1
│   │   ├── major_data_turn_1_retry_0       // The baseline data under turn 1, retry 0.
│   │   │   ├── macro_block_data.0          // A data file sized between 512 MB and 4 GB.
│   │   │   ├── macro_range_index           // The macro index.
│   │   │   ├── meta_index                  // The meta index.
│   │   │   └── sec_meta_index              // The file describing the mapping between the logical ID and the physical ID.
│   │   ├── meta_info_turn_1_retry_0        // The log stream metadata file under turn 1, retry 0.
│   │   │   ├── ls_meta_info                // The log stream metadata.
│   │   │   └── tablet_info                 // The list of log stream tablets.
│   │   ├── minor_data_turn_1_retry_0       // The minor-compacted data under turn 1, retry 0.
│   │   │   ├── macro_block_data.0          
│   │   │   ├── macro_range_index
│   │   │   ├── meta_index
│   │   │   └── sec_meta_index
│   │   └── sys_data_turn_1_retry_0         // The system tablet data under turn 1, retry 0.
│   │       ├── macro_block_data.0
│   │       ├── macro_range_index
│   │       ├── meta_index
│   │       └── sec_meta_index
│   └──  logstream_1001
│       ├── major_data_turn_1_retry_0
│       │   ├── macro_block_data.0
│       │   ├── macro_range_index
│       │   ├── meta_index
│       │   └── sec_meta_index
│       ├── meta_info_turn_1_retry_0
│       │   ├── ls_meta_info
│       │   └── tablet_info
│       ├── minor_data_turn_1_retry_0
│       │   ├── macro_block_data.0
│       │   ├── macro_range_index
│       │   ├── meta_index
│       │   └── sec_meta_index
│       └── sys_data_turn_1_retry_0
│           ├── macro_block_data.0
│           ├── macro_range_index
│           ├── meta_index
│           └── sec_meta_index
├── backup_set_2_inc                         // The incremental backup directory, which is consistent with the full backup directory.