Overview of physical backup and restore

Backup and restore is a core component of the high availability feature of OceanBase Database. It primarily ensures data security by preventing data loss due to storage media damage or user errors. In the case of data loss caused by storage media damage or user errors, data can be restored.

Overview

The backup and restore module of OceanBase Database provides backup, restore, and cleanup features.

OceanBase Database supports physical backup at the tenant level. Physical backup consists of data backup and log archiving. Therefore, physical backup is implemented by combining the data backup and log archiving features. User tenants can be backed up physically. However, the physical backup of the sys tenant and Meta tenant is not supported.

Data backup refers to the feature that backs up data. This feature supports full backup and incremental backup:

• Full backup backs up all macroblocks.

• Incremental backup backs up the macroblocks that are created or modified after the last backup.

Data backup mainly backs up the following information:

• Tenant-related information, such as tenant name, cluster name, zone, locality, and compatibility mode (MySQL or Oracle)

• Data of all user tables

Log archiving refers to the automatic archiving of log data. OBServer nodes regularly archive log data to the specified backup path. This process is fully automated and does not require manual triggering.

The overall architecture of physical restore is as follows:

1. Physical restore supports restore at the tenant level and at the table level.

   • Restore at the tenant level: Tenant restore is the process of rebuilding a new tenant based on the existing backup data. Tenant restore ensures global consistency across tables and partitions.

   • Restore at the table level: Table restore involves restoring specified tables from backup data to an existing tenant. This existing tenant can be the same tenant where the original table was located, a different tenant in the same cluster, or a tenant in a different cluster.

2. Tenant restore supports full restore and quick restore.

   Notice

   A tenant restored through quick restore cannot initiate major compactions manually, cannot be backed up through data backup, and cannot switch to the primary database through Switchover/Failover. It can only exist as a standby database.

   • Full restore: This process involves restoring macroblock data and incremental logs. A tenant is operational only after all its data is restored from backup media to local storage. The full restore process includes the restore and recover processes for the system tables and user tables of the tenant. Restore involves restoring the baseline data to the OBServer nodes of the target tenant, and recover involves restoring the logs corresponding to the baseline to the OBServer nodes.

   • Quick restore: This process allows a tenant to become operational without restoring macroblock data, thereby reducing restore wait time and user costs.

3. Choose a restore time point for your physical restore.

   • Full restore: No specific restore timestamp is specified.

   • Incomplete restore with a specified SCN or timestamp: In this case, the SCN is the precise version number used internally in OceanBase Database. In Oracle mode, the timestamp is precise to nanoseconds without any precision loss. In MySQL mode, the timestamp is precise to microseconds, and the precision beyond microseconds is lost.

The physical restore process is described in Restore process.

Backup media requirements

ApsaraDB for OceanBase supports backup media such as Alibaba Cloud OSS, NFS, AWS S3, and object storage services that are compatible with the S3 protocol (such as Huawei OBS, Google GCS, and Tencent Cloud COS). To use some backup media, you must meet some basic requirements.

SDK version requirements

The following table lists the correspondence between the object storage SDK versions and the observer versions.

Interface requirements

• Alibaba Cloud OSS:

  • Only Alibaba Cloud OSS V1 is supported. The following table lists the APIs on which OSS V1 depends.

    API	Description
    PutObject	Upload a single object
    DeleteObject	Delete a single object
    DeleteObjects	Batch delete objects
    GetObject	Get a specific object
    ListObjects	List all objects in the bucket (strong consistency is required)
    HeadObject	Get the metadata of a specific object
    AppendObject	Upload an object in append mode
    PutObjectTagging (optional)	Set or update the tags of an object
    GetObjectTagging (optional)	Get the tags of an object
    InitiateMultipartUpload	Initialize chunked upload
    UploadPart	Upload a chunk
    CompleteMultipartUpload	Merge uploaded chunks into one object
    AbortMultipartUpload	Cancel chunked upload and delete uploaded chunks
    ListMultipartUploads	List initialized but unfinished or terminated chunks
    ListParts	List completed chunks in the upload task

  • Only the V1 signature algorithm is supported.

• NFS: The version must be NFS 4.1 or later.

• Object storage systems that support the S3 protocol (for example, Huawei OBS, Google GCS, and Tencent Cloud COS):

  1. The following S3 APIs must be supported:

     API	Description
     PutObject	Upload a single object
     DeleteObject	Delete a single object
     DeleteObjects	Batch delete objects
     GetObject	Download a single object
     ListObjects	List all objects in the directory
     HeadObject	Get the metadata of a specific object
     PutObjectTagging (optional)	Set object tags
     GetObjectTagging (optional)	Get object tags
     CreateMultipartUpload	Initialize chunked upload
     UploadPart	Upload a single chunk
     CompleteMultipartUpload	Merge uploaded chunks into one object
     AbortMultipartUpload	Abort chunked upload and delete uploaded chunks
     ListMultipartUploads	List uploaded chunks
     ListParts	List completed chunks in the upload task

  2. Support for Virtual-hosted–style object access URLs are required. For more information about Virtual-hosted–style requests, see AWS S3.

When you select a backup medium, you can use the test_io_device command in the ob_admin tool to verify whether the I/O interface and I/O permissions provided by the backup medium meet the backup and restore requirements. You can also use the io_adapter_benchmark command in the ob_admin tool to view the read and write performance from the OBServer node to the backup medium, which serves as a reference for backup performance. For more information about the test_io_device and io_adapter_benchmark commands, see test_io_device and io_adapter_benchmark.

Directory structure

Data backup directory

The following table describes the directories created for data backup at the destination and the types of files stored in the directories.

data_backup_dest
├── format.obbak                                            // Backup path metadata.
├── check_file
│   └── 1002_connect_file_20230111T193020.obbak             // Connectivity check file.
├── backup_sets                                             // The summary directory of data backup sets, which records all data backup sets.
│   ├── backup_set_1_full_end_success_20230111T193420.obbak // The placeholder for the end of full backup.
│   ├── backup_set_1_full_start.obbak                       // The placeholder for the start of full backup.
│   ├── backup_set_2_inc_start.obbak                        // The placeholder for the start of incremental backup.
│   └── backup_set_2_inc_end_success_20230111T194420.obbak  // The placeholder for the end of incremental backup.
└── backup_set_1_full                                        // The directory of a data backup set. The directory name ending with `full` indicates a full backup, and the name ending with `inc` indicates an incremental backup. A data backup set is generated for each data backup. The backup set will not be modified after the backup is completed.
    ├── backup_set_1_full_20230111T193330_20230111T193420.obbak // The ID and start and end time of the current backup set. This file is used to display information only.
    ├── single_backup_set_info.obbak                        // The metadata of the current backup set, including the backup timestamp and dependent logs.
    ├── tenant_backup_set_infos.obbak                       // The metadata of all backup sets of the current tenant.
    ├── infos
    │   ├── table_list                                      // The file that records the names of full tables.
    │   │   ├── table_list.1702352553000000000.1.obbak      // Table name file 1.
    │   │   ├── table_list.1702352553000000000.2.obbak     // Table name file 2.
    │   │   └── table_list_meta_info.1702352553000000000.obbak  // The metadata file of table name files.
    │   ├── major_data_info_turn_1                           // The tenant-level backup file of major turn 1.
    │   │   ├── tablet_log_stream_info.obbak                 // The mapping file between tablets and log streams.
    │   │   ├── tenant_major_data_macro_range_index.0.obbak  // The major macroblock index.
    │   │   ├── tenant_major_data_meta_index.0.obbak         // The major metaindex.
    │   │   └── tenant_major_data_sec_meta_index.0.obbak     // The mapping file between major logical IDs and physical IDs.
    │   ├── minor_data_info_turn_1                           // The tenant-level backup file of minor turn 1.
    │   │   ├── tablet_log_stream_info.obbak                 // The mapping file between tablets and log streams.
    │   │   ├── tenant_minor_data_macro_range_index.0.obbak  // The minor macroblock index.
    │   │   ├── tenant_minor_data_meta_index.0.obbak         // The minor metaindex.
    │   │   └── tenant_minor_data_sec_meta_index.0.obbak     // The mapping file between minor logical IDs and physical IDs.
    │   ├── diagnose_info.obbak                              // The diagnostics information file of the backup set.
    │   ├── locality_info.obbak                              // The locality information of the current backup set.
    │   └── meta_info                                        // The tenant-level log stream metadata file, which contains the metadata of all log streams.
    │       ├── ls_attr_info.1.obbak                         // The snapshot of the log stream list at the time of backup.
    │       └── ls_meta_infos.obbak                          // The collection of metadata of all log streams.
    ├── logstream_1                                          // The directory that records data of log stream 1, which is the system log stream of the OceanBase tenant.
    │   ├── major_data_turn_1_retry_0                        // The baseline data of major turn 1, retry 0.
    │   │   ├── macro_block_data.0.obbak                     // A data file with a size of 512 MB to 4 GB.
    │   │   ├── macro_range_index.obbak                      // The macroindex.
    │   │   ├── meta_index.obbak                             // The metaindex.
    │   │   └── sec_meta_index.obbak                         // The mapping file between logical IDs and physical IDs.
    │   ├── meta_info_turn_1_retry_0                         // The log stream metadata file of major turn 1, retry 0.
    │   │   ├── ls_meta_info.obbak                           // The log stream metadata.
    │   │   └── tablet_info.1.obbak                          // The tablet metadata list of the log stream.
    │   ├── minor_data_turn_1_retry_0                        // The minor data of major turn 1, retry 0.
    │   │   ├── macro_block_data.0.obbak
    │   │   ├── macro_range_index.obbak
    │   │   ├── meta_index.obbak
    │   │   └── sec_meta_index.obbak
    │   └── sys_data_turn_1_retry_0                          // The system tablet data of major turn 1, retry 0.
    │       ├── macro_block_data.0.obbak
    │       ├── macro_range_index.obbak
    │       ├── meta_index.obbak
    │       └── sec_meta_index.obbak
    └── logstream_1001                                       // The directory that records data of log stream 1001, which is a user log stream of the OceanBase tenant.
        ├── major_data_turn_1_retry_0
        │   ├── macro_block_data.0.obbak
        │   ├── macro_range_index.obbak
        │   ├── meta_index.obbak
        │   └── sec_meta_index.obbak
        ├── meta_info_turn_1_retry_0
        │   ├── ls_meta_info.obbak
        │   └── tablet_info.1.obbak
        ├── minor_data_turn_1_retry_0
        │   ├── macro_block_data.0.obbak
        │   ├── macro_range_index.obbak
        │   ├── meta_index.obbak
        │   └── sec_meta_index.obbak
        └── sys_data_turn_1_retry_0
            ├── macro_block_data.0.obbak
            ├── macro_range_index.obbak
            ├── meta_index.obbak
            └── sec_meta_index.obbak

In the data backup directory, the top-level directories contain the following data:

• format.obbak: records the metadata of the backup path.

• check_file: checks the connectivity of the user data backup directory.

• backup_sets: the summary directory of data backup sets, which records all data backup sets.

• backup_set_1_full: the directory of a data backup set. The directory name ending with full indicates a full backup, and the name ending with inc indicates an incremental backup. A data backup set is generated for each data backup. The backup set will not be modified after the backup is completed.

  In a data backup set, the following data is stored:

  • backup_set_1_full_20230111T193330_20230111T193420.obbak: the ID and start and end time of the current backup set. This file is used to display information only.

  • single_backup_set_info.obbak: the metadata of the current backup set, including the backup timestamp and dependent logs.

  • tenant_backup_set_infos.obbak: the metadata of all backup sets of the current tenant.

  • infos: the directory that records the metadata of data backup sets.

  • logstream_1: the directory that records data of log stream 1, which is the system log stream of the OceanBase tenant.

  • logstream_1001: the directory that records data of log stream 1001, which is a user log stream of the OceanBase tenant.

  In addition, each log stream backup has four subdirectories, where the subdirectories containing retry indicate retries at the log stream level, and the subdirectories containing turn indicate retries at the tenant level:

  • meta_info_xx: the directory that records LS metadata and tablet metadata.
  • sys_data_xx: the directory that records the system tablet data of LS.
  • minor_data_xx: the directory that records the minor data of ordinary tablets.
  • major_data_xx: the directory that records the baseline data of ordinary tablets.

Log archive directory

The following table describes the directories created at the destination for logs and the file types saved in the directories for backup media such as NFS, OSS, and COS.

log_archive_dest
    ├── check_file
    │   └── 1002_connect_file_20230111T193049.obbak // File for connectivity check
    ├── format.obbak                                // Format information of the backup path
    ├── rounds                                      // Placeholder for rounds
    │   └── round_d1002r1_start.obarc               // Placeholder for the start of a round
    ├── pieces                                      // Placeholder for pieces
    │   ├── piece_d1002r1p1_start_20230111T193049.obarc // Placeholder for the start of a piece, with the format of piece_DESTID_ROUNDID_PIECEID_start_DATE
    │   └── piece_d1002r1p1_end_20230111T193249.obarc   // Placeholder for the end of a piece, with the format of piece_DESTID_ROUNDID_PIECEID_end_DATE
    └── piece_d1002r1p1                             // Directory for a piece, named in the format of piece_DESTID_ROUNDID_PIECEID
        ├── piece_d1002r1p1_20230111T193049_20230111T193249.obarc // Records the continuous range of the piece
        ├── checkpoint
        │   └── checkpoint.1673436649723677822.obarc   // Records checkpoint_scn information. The filename follows the format of checkpoint.checkpoint_scn.
        │   └── checkpoint_info.0.obarc             // Records checkpoint metadata
        ├── single_piece_info.obarc                 // Records metadata of the current piece
        ├── tenant_archive_piece_infos.obarc        // Records the metadata of all frozen pieces in the current tenant
        ├── file_info.obarc                         // List of all log stream files
        ├── logstream_1                             // Directory for Log Stream 1
        │   ├── file_info.obarc                    // List of files in Log Stream 1
        │   ├── log
        │   │   └── 1.obarc                         // Archive files in Log Stream 1
        │   └── schema_meta                         // Records metadata of the data dictionary. This directory is generated only in Log Stream 1.
        │       └── 1677588501408765915.obarc
        └── logstream_1001                          // Directory for Log Stream 1001
            ├── file_info.obarc                      // List of files in Log Stream 1001
            └── log
                └── 1.obarc                          // Archive files in Log Stream 1001

In the preceding log archive directory, the top-level directory contains the following data:

• format.obbak: Records the metadata of the backup path, including the tenant that uses the path.

• check_file: Checks the connectivity of the log archive directory.

• rounds: The directory that lists all rounds for log archiving.

• pieces: The directory that lists all pieces for log archiving.

• piece_d1002r1p1: The directory for a log archive piece. The directory is named in the format of piece_DESTID_ROUNDID_PIECEID. In this format, DESTID is the ID of the log_archive_dest parameter, ROUNDID is the ID of the log archiving round, which is a monotonically increasing integer, and PIECEID is the ID of the log archiving piece, which is also a monotonically increasing integer.

  The directory for a log archive piece contains the following data:

  • piece_d1002r1p1_20230111T193049_20230111T193249.obarc: This file displays the ID, start time, and end time of the piece and is used only for information display.

  • checkpoint: The directory records the archive points of active pieces. The ObArchiveScheduler module updates the archive points in this directory periodically. This directory contains the following files:

    • checkpoint.1673436649723677822.obarc: The filename records the checkpoint_scn information of the piece, where 1673436649723677822 is the corresponding checkpoint_scn.

    • checkpoint_info.0.obarc: This file records the metadata of the checkpoint of an active piece. The metadata includes fields such as tenant_id, dest_id, round_id, and piece_id. The metadata remains unchanged within a piece.

  • single_piece_info.obarc: This file records the metadata of the current piece.

  • tenant_archive_piece_infos.obarc: This file records the metadata of all frozen pieces in the current tenant.

  • file_info.obarc: This file records the list of log stream files in the piece.

  • logstream_1: This directory records the log files of Log Stream 1, which is the system log stream of the OceanBase tenant.

  • logstream_1001: This directory records the log files of Log Stream 1001, which is a user log stream of the OceanBase tenant.

  Each log stream backup also contains the following data:

  • file_info.obarc: This file records the list of files in the log stream.

  • log: This directory stores all archive files of the current log stream, with filenames that are the same as those in the source cluster.

  • schema_meta: This directory records the metadata of the data dictionary. This directory is generated only in the system log stream and does not exist in user log streams.

For backup media that support access compatible with the S3 protocol, the directory structure for log archiving is different from that for NFS, OSS, COS, and other backup media. A single log archive file consists of multiple small files and corresponding metadata files. The following table describes the directory structure.

log_archive_dest
    ├── ......
    └── piece_d1002r1p1                             // Directory for a piece, named in the format of piece_DESTID_ROUNDID_PIECEID
        ├── ......                         					// List of all log stream files
        ├── logstream_1                             // Log stream 1
        │   ├── file_info.obarc                    // List of files in Log Stream 1
        │   ├── log
        │   │   └── 1.obarc                         // Archive files in Log Stream 1. The file is prefixed with @APD_PART@.
        |		|       └── @APD_PART@0-32472973.obarc  // The actual data in the archive file, recording data from byte 0 to byte 32472973 in the log file
        |		|		    └── ......
        |		|		    └── @APD_PART@FORMAT_META.obarc // Metadata of the archive file
        |		|		    └── @APD_PART@SEAL_META.obarc   // Metadata of the archive file
        │   └── schema_meta                         // Records metadata of the data dictionary. This file is generated only in Log Stream 1.
        │       └── 1677588501408765915.obarc
        └── logstream_1001                          // Log stream No. 1001
            ├── file_info.obarc                      // List of files in Log Stream 1001
            └── log
                └── 1.obarc                          // Archive files in Log Stream 1001

In the preceding log archive directory, 1.obarc indicates a single log archive file. This file is prefixed with @APD_PART@, and the prefix name is the same as the archive file name. The single archive file contains the following three types of data:

• @APD_PART@FORMAT_META.obarc: When data is written to the archive file for the first time, a format_meta file is written to this directory to record the format of the archive file.

• @APD_PART@0-32472973.obarc: The actual data in the archive file is written to a file whose name contains the starting and ending offsets of each write operation.

• @APD_PART@SEAL_META.obarc: After data is written to the archive file for the last time, a seal_meta file is written to this directory to record the metadata of the archive file.