Overview

Backup and restore is a core component of the high availability feature of OceanBase Database. It ensures data security by preventing data loss due to storage media damage or user errors. If data is lost due to storage media damage or user errors, you can restore the data.

Overview

The backup and restore module of OceanBase Database provides three main features: backup, restore, and cleanup.

OceanBase Database supports tenant-level physical backup. A physical backup consists of data backup and log archiving. Therefore, the physical backup feature is a combination of data backup and log archiving. Here, a tenant refers to a user tenant. Physical backup is not supported for the sys tenant or the Meta tenant.

Data backup refers to the feature of backing up data. This feature is divided into full backup and incremental backup:

• Full backup refers to backing up all macroblocks.

• Incremental backup refers to backing up macroblocks added or modified since the last backup.

The data backed up by data backup includes the following:

• Information about the tenant, including the tenant name, cluster name, time zone (timezone), replica distribution (locality), and compatibility mode (MySQL or Oracle) of the tenant.

• All user table data

  Note

  Data backup backs up system variables and tenant-level configuration items, but does not back up cluster-level configuration items or private system table data.

Log archiving refers to the automatic archiving of log data. OBServer nodes periodically archive log data to the specified backup path. This process is fully automatic and does not require external triggering. After the log archiving service is enabled, the data dictionary is also archived.

The overall architecture of physical restore is as follows:

The backup verification feature of OceanBase Database is mainly used to solve the problems of the integrity and correctness of backup and archive data. Users can actively verify whether the backup set and archive logs are missing files, have physical data damage, or logical inconsistencies without performing actual restore operations.

Backup verification is divided into basic verification and physical verification based on the verification level. Specifically:

• Basic verification: only checks the completeness of the file list.

• Physical verification: checks the correctness and logical consistency of physical data.

Backup verification supports verification at different granularities, including verification of the entire backup and archive path, verification of specified backup sets or log archive pieces, and verification of specified paths across clusters.

Restore

The overall architecture of physical restore is as follows:

1. Physical restore supports tenant-level restore and table-level restore.

   • Tenant-level restore: Tenant-level restore is the process of rebuilding a new tenant based on the existing data backup. Tenant-level restore ensures global consistency across tables and partitions.

   • Table-level restore: Table-level restore is the process of restoring a user-specified table from the backup data to an existing tenant. The existing tenant can be the same as the original tenant, a different tenant within the same cluster, or a tenant in a different cluster.

2. Tenant-level restore supports full restore and quick restore.

   Notice

   After a tenant is restored by using quick restore, the tenant cannot be manually merged, cannot be backed up, and cannot be switched to or failover to the primary database. The tenant can only exist as a standby database.

   • Full restore: Full restore refers to restoring macroblock data and incremental logs. After all data is restored from the backup media to the local environment, the restored tenant can provide services. The full restore process includes the restore and recover processes for tenant system tables and user tables. Restore refers to restoring the baseline data required for the restore to the OBServer node of the target tenant. Recover refers to restoring the logs corresponding to the baseline to the corresponding OBServer node.

   • Quick restore: Quick restore refers to providing services to users without restoring macroblock data. This reduces the restore waiting time and lowers the user's usage cost.

3. Select the restore time point for your physical restore.

   • Complete restore: No restore timestamp is specified.

   • Incomplete restore with a specified SCN or timestamp: SCN is the precise version number of OceanBase Database. In Oracle mode, the timestamp is precise to the nanosecond, with no precision loss. In MySQL mode, the timestamp is precise to the microsecond, with precision loss after the microsecond.

For more information about the physical restore process, see Restore process.

Backup media requirements

OceanBase Database of the current version supports backup media such as Alibaba Cloud OSS, NFS, Azure Blob, AWS S3, and object storage services compatible with the S3 protocol (for example, Huawei OBS and Google GCS). Some backup media require meeting some basic requirements before they can be used.

• Automatic cleanup: Based on the recovery window (recovery_window) mechanism. After users set the expiration time for backup data, the system periodically triggers cleanup tasks to remove expired backup sets and data archive pieces.

• Manual cleanup: Manual cleanup is divided into two types. The first type is manual cleanup of specified backup sets or log archive pieces. The second type is manual cleanup of all backup sets or data archive pieces in the entire path (data backup path or log archive path).

Backup media requirements

OceanBase Database supports the following backup media: Alibaba Cloud OSS, NFS, Azure Blob, AWS S3, and object storage services that are compatible with the S3 protocol (such as Huawei OBS, Google GCS, and Tencent Cloud COS). Some of these backup media require meeting certain basic requirements before they can be used.

SDK version requirements

In the current version, both the standalone node mode (SN mode) and the shared storage mode (SS mode) use obdal to access object storage. obdal is a unified framework for object storage developed based on OpenDAL. For more information about OpenDAL, see OpenDAL.

In addition, if you want to use the SDK of an object storage service, the version of the object storage service's SDK must correspond to the version of the observer component. The following table describes the correspondence between the versions of the object storage service's SDK and the observer component.

Interface requirements

• Alibaba Cloud OSS:

  • The following table describes the interfaces supported by Alibaba Cloud OSS.

    Interface name	Description
    PutObject	Uploads a single object.
    DeleteObject	Deletes a single object.
    DeleteObjects	Deletes objects in batches.
    GetObject	Retrieves an object.
    ListObjects	Lists all objects in the storage space (strong consistency is required).
    HeadObject	Retrieves the metadata of an object.
    AppendObject	Uploads an object in append mode.
    PutObjectTagging (optional)	Sets or updates the tags of an object.
    GetObjectTagging (optional)	Retrieves the tags of an object.
    InitiateMultipartUpload	Initializes a multipart upload.
    UploadPart	Uploads a part.
    CompleteMultipartUpload	Merges the uploaded parts into one object.
    AbortMultipartUpload	Aborts a multipart upload and deletes the uploaded parts.
    ListMultipartUploads	Lists the information of initialized but not completed or not terminated multipart uploads.
    ListParts	Lists the uploaded parts in an upload task.

  • Only the V1 signature algorithm is supported.

• NFS: The version must be NFS 3 or later.

• Object storage services compatible with the S3 protocol (for example, Huawei OBS and Google GCS):

  1. The following table describes the S3 APIs that must be supported.

     Interface name	Description
     PutObject	Uploads a single object.
     DeleteObject	Deletes a single object.
     DeleteObjects	Deletes objects in batches.
     GetObject	Downloads a single object.
     ListObjects	Lists all objects under the specified path.
     HeadObject	Retrieves the metadata of an object.
     PutObjectTagging (optional)	Sets the tags of an object.
     GetObjectTagging (optional)	Retrieves the tags of an object.
     CreateMultipartUpload	Initializes a multipart upload.
     UploadPart	Uploads a single part.
     CompleteMultipartUpload	Merges the uploaded parts into one object.
     AbortMultipartUpload	Aborts a multipart upload and deletes the uploaded parts.
     ListMultipartUploads	Lists the uploaded parts.
     ListParts	Lists the uploaded parts in an upload task.

  2. The object access URL must support the virtual-hosted-style access method. For more information about virtual-hosted-style requests, see AWS S3 documentation.

Before you select a backup media, you can run the test_io_device command in the ob_admin tool to verify whether the I/O interfaces and I/O permissions provided by the backup media meet the requirements for backup and restore. You can also run the io_adapter_benchmark command in the ob_admin tool to view the read and write performance between an OBServer node and the backup media, which can be used as a reference for backup performance. For more information about the test_io_device and io_adapter_benchmark commands, see test_io_device and io_adapter_benchmark.

Directory structure

Backup destination directory

The following table shows the directories created by the backup destination and the types of files stored in each directory.

data_backup_dest
├── format.obbak                                            // The format information of the backup destination.
├── check_file
│   └── 1002_connect_file_20230111T193020.obbak             // The connectivity check file.
├── backup_sets                                             // The directory that contains all the data backup sets.
│   ├── backup_set_1_full_end_success_20230111T193420.obbak // The placeholder indicating the end of a full backup.
│   ├── backup_set_1_full_start.obbak                       // The placeholder indicating the start of a full backup.
│   ├── backup_set_2_inc_start.obbak                        // The placeholder indicating the start of an incremental backup.
│   └── backup_set_2_inc_end_success_20230111T194420.obbak  // The placeholder indicating the end of an incremental backup.
└── backup_set_1_full                                        // The full backup set. The file name ends with full, indicating a full backup. The file name ends with inc, indicating an incremental backup.
    ├── backup_set_1_full_20230111T193330_20230111T193420.obbak // The placeholder indicating the start and end time of a full backup.
    ├── single_backup_set_info.obbak                        // The metadata of the current backup set.
    ├── tenant_backup_set_infos.obbak                       // The full backup set information of the current tenant.
    ├── infos
    │   ├── table_list                                      // The file containing the names of all tables.
    │   │   ├── table_list.1702352553000000000.1.obbak      // The file containing the names of tables 1.
    │   │   ├── table_list.1702352553000000000.2.obbak     // The file containing the names of tables 2.
    │   │   └── table_list_meta_info.1702352553000000000.obbak  // The metadata file of the names of tables.
    │   ├── major_data_info_turn_1                           // The tenant-level backup files under major turn 1.
    │   │   ├── tablet_log_stream_info.obbak                 // The mapping file between tablets and log streams.
    │   │   ├── tenant_major_data_macro_range_index.0.obbak  // The macro block index of major data.
    │   │   ├── tenant_major_data_meta_index.0.obbak         // The meta index of major data.
    │   │   └── tenant_major_data_sec_meta_index.0.obbak     // The mapping file between the logical and physical IDs of major data.
    │   ├── minor_data_info_turn_1                           // The tenant-level backup files under minor turn 1.
    │   │   ├── tablet_log_stream_info.obbak                 // The mapping file between tablets and log streams.
    │   │   ├── tenant_minor_data_macro_range_index.0.obbak  // The macro block index of minor data.
    │   │   ├── tenant_minor_data_meta_index.0.obbak         // The meta index of minor data.
    │   │   └── tenant_minor_data_sec_meta_index.0.obbak     // The mapping file between the logical and physical IDs of minor data.
    │   ├── diagnose_info.obbak                              // The diagnostic information file of the backup set.
    |   ├── tenant_parameter.obbak                           // The tenant-level configuration information of the current tenant, excluding default settings.
    │   ├── locality_info.obbak                              // The locality information of the tenant to which the current backup set belongs, including the resource configuration and replica distribution information of the tenant.
    │   └── meta_info                                        // The log stream metadata file of the tenant level, containing the metadata of all log streams.
    │       ├── ls_attr_info.1.obbak                         // The snapshot of the log stream list at the time of backup.
    │       └── ls_meta_infos.obbak                          // The collection of metadata of all log streams.
    ├── logstream_1                                          // The log stream 1.
    │   ├── major_data_turn_1_retry_0                        // The baseline data under turn 1, retry 0.
    │   │   ├── macro_block_data.0.obbak                     // A data file of 512 MB to 4 GB.
    │   │   ├── macro_range_index.obbak                      // The macro index.
    │   │   ├── meta_index.obbak                             // The meta index.
    │   │   └── sec_meta_index.obbak                         // The mapping file between the logical and physical IDs.
    │   ├── meta_info_turn_1_retry_0                         // The log stream metadata file under turn 1, retry 0.
    │   │   ├── ls_meta_info.obbak                           // The log stream metadata.
    │   │   └── tablet_info.1.obbak                          // The list of log stream tablet metadata.
    │   ├── minor_data_turn_1_retry_0                        // The minor data under turn 1, retry 0.
    │   │   ├── macro_block_data.0.obbak
    │   │   ├── macro_range_index.obbak
    │   │   ├── meta_index.obbak
    │   │   └── sec_meta_index.obbak
    │   └── sys_data_turn_1_retry_0                          // The system tablet data under turn 1, retry 0.
    │       ├── macro_block_data.0.obbak
    │       ├── macro_range_index.obbak
    │       ├── meta_index.obbak
    │       └── sec_meta_index.obbak
    └── logstream_1001                                       // The log stream 1001.
        ├── major_data_turn_1_retry_0
        │   ├── macro_block_data.0.obbak
        │   ├── macro_range_index.obbak
        │   ├── meta_index.obbak
        │   └── sec_meta_index.obbak
        ├── meta_info_turn_1_retry_0
        │   ├── ls_meta_info.obbak
        │   └── tablet_info.1.obbak
        ├── minor_data_turn_1_retry_0
        │   ├── macro_block_data.0.obbak
        │   ├── macro_range_index.obbak
        │   ├── meta_index.obbak
        │   └── sec_meta_index.obbak
        └── sys_data_turn_1_retry_0
            ├── macro_block_data.0.obbak
            ├── macro_range_index.obbak
            ├── meta_index.obbak
            └── sec_meta_index.obbak

The top-level directory in the backup directory contains the following three types of data:

• format.obbak: stores the metadata of the backup path.

• check_file: checks the connectivity of the user data backup directory.

• backup_sets: a summary directory of the list of data backup sets, which records the list of all data backup sets.

• backup_set_1_full: the directory indicates a data backup set. If the directory name ends with full, it indicates a full backup. If it ends with inc, it indicates an incremental backup. A backup set is generated for each data backup. After the data backup is completed, the backup set will no longer be modified.

  The following data is mainly stored in a data backup set:

  • backup_set_1_full_20230111T193330_20230111T193420.obbak: this file shows the ID, start time, and end time of the current backup set. This file is only for information display.

  • single_backup_set_info.obbak: this file records the metadata of the current backup set, including the backup point and the dependent logs.

  • tenant_backup_set_infos.obbak: this file records the metadata of all existing backup sets of the current tenant.

  • infos: this directory records the metadata of the data backup set.

  • logstream_1: this directory records all data of log stream 1. Log stream 1 is the system log stream of the OceanBase Database tenant.

  • logstream_1001: this directory records all data of log stream 1001. Log streams with numbers greater than 1000 are the user log streams of the OceanBase Database tenant.

  Each log stream backup contains four directories. A directory with retry indicates log stream-level retries. A directory with turn indicates tenant-level retries:

  • meta_info_xx: this directory records the metadata of the log stream (LS) and tablet.
  • sys_data_xx: this directory records the data of the internal system tablets in the LS.
  • minor_data_xx: this directory records the minor compaction data of the ordinary tablets.
  • major_data_xx: this directory records the major compaction data of the ordinary tablets.

Cluster-level parameter backup directory

Each time a cluster-level parameter backup is initiated, the system generates a backup file in the specified directory. The directory structure is as follows:

cluster_parameters_backup_dest
├── cluster_parameter.20240710T103610.obbak # Information of non-default cluster-level parameters, file naming format: `cluster_parameter.[timestamp]`
└── cluster_parameter.20241018T140609.obbak

Log archive directory

For backup media such as NFS and OSS, the log archive directory is created at the destination, and the file types saved in each directory are as follows:

log_archive_dest
    ├── check_file
    │   └── 1002_connect_file_20230111T193049.obbak // Connectivity check file
    ├── format.obbak                                // Formatted information of the backup path
    ├── rounds                                      // Placeholder directory for Rounds
    │   └── round_d1002r1_start.obarc               // Placeholder for Round start
    ├── pieces                                      // Placeholder directory for Pieces
    │   ├── piece_d1002r1p1_start_20230111T193049.obarc // Placeholder for Piece start, named as piece_DESTID_ROUNDID_PIECEID_start_DATE
    │   └── piece_d1002r1p1_end_20230111T193249.obarc   // Placeholder for Piece end, named as piece_DESTID_ROUNDID_PIECEID_end_DATE
    └── piece_d1002r1p1                             // Piece directory, named as piece_DESTID_ROUNDID_PIECEID
        ├── piece_d1002r1p1_20230111T193049_20230111T193249.obarc // Records the continuous interval of the Piece
        ├── checkpoint
        │   └── checkpoint.1673436649723677822.obarc   // Records checkpoint_scn information, named as checkpoint.checkpoint_scn
        │   └── checkpoint_info.0.obarc             // Records metadata of the checkpoint
        ├── single_piece_info.obarc                 // Records metadata of the current Piece
        ├── tenant_archive_piece_infos.obarc        // Records metadata of all frozen Pieces in the current tenant
        ├── file_info.obarc                         // List of all log stream files
        ├── logstream_1                             // Log stream 1
        │   ├── file_info.obarc                    // List of files in log stream 1
        │   ├── log
        │   │   └── 1.obarc                         // Archive files under log stream 1
        │   └── schema_meta                         // Records metadata of the data dictionary, generated only in log stream 1
        │       └── 1677588501408765915.obarc
        └── logstream_1001                          // Log stream 1001
            ├── file_info.obarc                      // List of files in log stream 1001
            └── log
                └── 1.obarc                          // Archive files under log stream 1001

In the above log archive directory, the top-level directory contains the following three types of data:

• format.obbak: Records metadata of the archive path, including information such as the tenant using the path.

• check_file: Used for connectivity check of the user log archive directory.

• rounds: A summary list of all Rounds in the log archive, recording the list of all Rounds.

• pieces: A summary list of all Pieces in the log archive, recording the list of all Pieces.

• piece_d1002r1p1: The directory for a Piece in the log archive, named as piece_DESTID_ROUNDID_PIECEID. Here, DESTID refers to the id corresponding to log_archive_dest; ROUNDID refers to the id of the log archive Round, which is a monotonically increasing integer; and PIECEID refers to the id of the log archive Piece, which is also a monotonically increasing integer.

  In a log archive Piece directory, the following data is included:

  • piece_d1002r1p1_20230111T193049_20230111T193249.obarc: This file displays the id, start, and end times of the current Piece and is only for informational purposes.

  • checkpoint: A directory for recording the archive point of the active Piece. The ObArchiveScheduler module periodically updates the archive point information in this directory. Specifically:

    • checkpoint.1673436649723677822.obarc: The filename records the checkpoint_scn information of the Piece, where 1673436649723677822 is the corresponding checkpoint_scn.

    • checkpoint_info.0.obarc: This file records the metadata of the active Piece's checkpoint, including tenant_id, dest_id, round_id, piece_id, etc. The metadata remains unchanged within a Piece.

  • single_piece_info.obarc: This file records the metadata of the current Piece.

  • tenant_archive_piece_infos.obarc: This file records the metadata of all frozen Pieces in the current tenant.

  • file_info.obarc: This file records the list of log stream files within the Piece.

  • logstream_1: This directory records the log files of log stream 1, which is the system log stream of the OceanBase Database tenant.

  • logstream_1001: This directory records the log files of log stream 1001, which is a user log stream of the OceanBase Database tenant.

  In addition, each log stream backup contains three types of data:

  • file_info.obarc: This file records the list of files within the log stream.

  • log: This directory stores all archive files of the current log stream, with filenames consistent with those in the source cluster.

  • schema_meta: This directory records metadata of the data dictionary. It exists only in the system log stream and is absent in user log streams.

For backup media such as AWS S3 and those accessible via the S3 protocol, the directory structure for log archive differs from that of OSS and NFS. Each archive file consists of multiple small files and corresponding metadata files. The specific directory structure is as follows:

log_archive_dest
    ├── ......
    └── piece_d1002r1p1                             // Piece directory. The directory name is in the piece_DESTID_ROUNDID_PIECEID format.
        ├── ......                         					// List of all log stream files.
        ├── logstream_1                             // Log stream 1.
        │   ├── file_info.obarc                    // List of files in log stream 1.
        │   ├── log
        │   │   └── 1.obarc                         // Archive files in log stream 1. The archive files are identified by a prefix.
        |		|       └── @APD_PART@0-32472973.obarc  // Actual data in the archive file. The data records the contents of the log file from the 0th to the 32472973rd byte.
        |		|		    └── ......
        |		|		    └── @APD_PART@FORMAT_META.obarc // Format of the archive file.
        |		|		    └── @APD_PART@SEAL_META.obarc   // Metadata of the archive file.
        │   └── schema_meta                         // Metadata of the data dictionary. This file is generated only in log stream 1.
        │       └── 1677588501408765915.obarc
        └── logstream_1001                          // Log stream 1001.
            ├── file_info.obarc                      // List of files in log stream 1001.
            └── log
                └── 1.obarc                          // Archive files in log stream 1001.

In the preceding log archive directory, 1.obarc indicates an archive file. Each archive file is identified by a prefix, and the prefix name is the same as the archive file name. The archive file contains the following three types of data:

• @APD_PART@FORMAT_META.obarc: When you write data to the archive file for the first time, the system writes a format_meta file to this directory. This file records the format of the archive file.

• @APD_PART@0-32472973.obarc: The actual data is written to a file with the specified prefix. The start and end offsets of the data written in each write operation are recorded in the file name.

• @APD_PART@SEAL_META.obarc: After you write data to the archive file for the last time, the system generates a seal_meta file in this directory. This file records the metadata of the archive file.

Differences from V3.x/V2.x

Log archiving

Data backup

Physical restore

Partner certifications

For a list of partners that have completed certification with OceanBase Database, see Industrial partners.

On the Industrial partners page, select Data Integration & Backup > Backup & Recovery under Select Solution Type. This will display a list of partners that have completed certification with OceanBase Database, along with the certified versions and the validity periods of the certificates.

References

For more information about physical backup and restore, see Backup and restore.