Database transparent encryption overview

Data storage encryption refers to transparent encryption (TDE) of data and clog files stored on disk. During TDE, data is automatically encrypted before being written to a storage device and automatically decrypted when being read. The processes are transparent to users. Hackers and malicious users cannot read sensitive data from files, database backups, or disks.

The user data and backups in OceanBase Database are ultimately stored in binary form on disks or other permanent storage. Unauthorized users may obtain these storage devices and read and decode the stored data, which leads to data leaks. Although OceanBase Database data is stored in compressed form and has a certain degree of confidentiality, there is still a risk of being cracked. To protect the security of data stored in memory and disks, OceanBase Database supports the international mainstream encryption algorithm AES, as well as China's encryption algorithm SM4.

Two-level key system

TDE uses a two-level key system to implement encryption and decryption. The minimum granularity for enabling encryption is a table in the database, and the table that needs to be encrypted needs to be placed in an encrypted tablespace. The encryption unit of OceanBase Database is tablespace, which is designed for Oracle compatibility and can be easily understood as a collection of tables. Each encrypted tablespace is configured with an encryption algorithm and a corresponding data key, which are used to encrypt the data in the tablespace. Each tenant has a master key for encrypting the data key in the tablespace. To prevent unauthorized decryption, the master key is usually stored in a dedicated Keystore.

To ensure data security, OceanBase Database does not allow users to use Keystore to view the master key or encryption key or to specify the master key or encryption key. The master key and encryption key are generated by the system and are not directly stored in plaintext on the disk, which significantly improves system security.

Keystore is the master key management module that provides key management services (KMS). The features of Keystore are as follows:

• Generation of the master key: The master key is generated by Keystore based on the encryption algorithm, and cannot be specified by the user, which is more secure.
• Storage of master key-related information: Keystore obtains the master key based on relevant information. It also keeps multiple copies of information, ensures information consistency, and provides fault handling.
• Master key management and multi-version control: Keystore is responsible for modifying and multi-version control of the master key. The multi-version control mechanism allows newly generated master keys to be gradually replaced without immediately taking effect.
• Obtaining high availability services related to the master key.
• Processing of Keystore commands.

When enabling TDE, you must specify the storage method for the master key, which is controlled by the tde_method parameter.

In OceanBase Database, the "private cloud" deployment currently supports transparent tablespace encryption only in internal mode. In internal mode, the encryption information of the master key is managed in internal tables, and clogs are not encrypted to avoid circular dependency during log replay.

When encryption is enabled, the master key is used to encrypt data keys. The ciphertext of the data keys is stored in internal tables, macroblock headers, and clog headers. The data keys are not stored in plaintext anywhere. To encrypt and decrypt data, the master key is used to decrypt the ciphertext and obtain the data keys, which are then used to encrypt or decrypt user data in macroblocks or clogs.

Limitations

• Encryption cannot be enabled for the sys tenant.
• Once a tenant has TDE configured, it cannot switch to other encryption modes unless the tenant is re-created.
• TDE for the bkmi and obcloud modes supports clog encryption, which must be manually enabled by using the enable_clog_encryption parameter.

Encryption mechanism

User data stored on the disk includes two types: clog and macroblock. Encryption takes effect only when the data is actually written to the disk. The data in memory is not encrypted. If an originally unencrypted table is changed to encrypted, the previously unencrypted clog and macroblock data stored on the disk will not be encrypted, and only new data written to the disk later will be encrypted. Since each clog and each macroblock record encryption metadata, encrypted and unencrypted data can coexist.

Because clogs are written to the disk using an append-only method, old unencrypted clogs will never become encrypted. After encryption has been enabled for a period of time, when old clogs are recycled, all clog data on the disk will become encrypted. Macroblock data is written to the disk only during minor or major compactions. You can manually trigger a full compaction to rewrite unencrypted macroblock data as encrypted.

After you enable TDE, clog encryption is disabled by default. To enable clog encryption for the bkmi or obcloud mode (only supported for these modes), manually enable it by setting the enable_clog_encryption parameter.

Supported encryption algorithms

Currently, OceanBase Database supports the international mainstream encryption algorithm AES, as well as China's encryption algorithm SM4.