OceanBase Database Community Edition V4.2.1|V4.2.1| docs|Distributed Database

OceanBase Database Community Edition V4.2.1

Last Updated：2023-12-29 07:37:25 Updated

Version information

Release date: December 7, 2023
Version: V4.2.1_CE_BP2
RPM version: oceanbase-ce-4.2.1.2-102000042023120514

Enhanced features

The MySQL mode of OceanBase Database now supports paging queries while maintaining the same sorting method for each query.
A new command TRANSFER PARTITION is provided to manually adjust the distribution of partitions. With this command, you can migrate specific partitions to specific log streams to aggregate or scatter partitions.
OBKV now supports data compression for remote procedure calls (RPCs), reducing network bandwidth usage and user costs.
This version now allows you to set a threshold for switching the storage method of LOB data from INROW (stored in the primary table by using row storage) to OUTROW (stored in a separate LOB auxiliary table). This enhancement aims to improve the performance of LOB data queries.

Product behavioral changes

The backup_data_file_size parameter, which is related to backup and restore, is changed from a cluster-level parameter to a tenant-level parameter.
The tableapi_transport_compress_func parameter, which is related to TableAPI, is renamed as kv_transport_compress_func.
A new parameter kv_transport_compress_threshold is introduced to specify the minimum threshold for compressing OBKV query result sets.

The value range for adaptive adjustment of the server_cpu_quota_min and server_cpu_quota_max parameters is increased to meet requirements of larger specifications.

cpu_count	(0C, 8C)	[8C, 16C)	[16C, 32C)	[32C, +∞)
server_cpu_quota_min	1C	2C	3C	4C
server_cpu_quota_max	1C	2C	3C	4C

Bug fixes

Fixed the issue where the disk bandwidth was used up during a major compaction after a large amount of data was deleted. This issue occurred because, for each deleted data row, a macroblock was opened to verify whether the data row exists in the macroblock.
Fixed the issue where a core dump occurred on an OBServer node due to a concurrency error in an earlier version of the OpenSSL library during concurrent calls to encryption functions.
Fixed the issue where, during the execution of a transfer task, errors were encountered and LOB data writes were repeatedly retried due to the failure to refresh the location cache.
Fixed the issue where, during the concurrent creation of indexes on tables with many partitions, the concurrent execution of INNER SQL statements internally could result in a livelock, causing the task to be unable to progress.
Fixed the issue where the period of time consumed in executing UPDATE operations on tables containing LOB data was long.
Fixed the issue where excessive disk space was occupied due to the retention of data snapshots during bypass imports of tables containing LOB data.
Fixed the issue where modifying the auto-increment column attribute in a table schema and performing a Truncate operation would result in Error 4016 due to the incorrect updates in internal tables.
Fixed the issue where data of multiple versions may exist when expired data was deleted based on MaxVersions in the HBase mode of OBKV.
Fixed the issue of memory leaks within the SysJobMgr module that caused Error 4013 due to memory exhaustion in the sys500 tenant during the execution of table-level restore tasks.
Fixed the issue where the units remained in the WAIT GC state for a long period of time and failed to release after deleting a tenant.
Fixed the issue where, even after setting IOPS for resource isolation, the number of I/O operations would still exceed the limits in some scenarios.
Fixed the issue where a long period of time was consumed in a major compaction that involved millions of tablets.
Optimized the log synchronization and physical restore performance.
Optimized the performance of spatial index queries by reducing the number of table access operations and minimizing the performance overhead in spatial relationship calculations.

Considerations

Before V4.2.1_CE_BP2, there was a problem with the limitation of PX threads, potentially resulting in high CPU load on OBServer nodes. However, this issue has been resolved in V4.2.1_CE_BP2, and an upgrade is recommended.
In the current version, there are some instability issues when using the INSERT INTO SELECT statement for bypass import, particularly when triggering a data transfer. Therefore, it is not recommended to use the INSERT INTO SELECT statement for bypass import.

V4.2.1_CE_BP1_HF1

Version information

Release date: November 13, 2023
Version: V4.2.1_CE_BP1_HF1
RPM version: oceanbase-ce-4.2.1.1-101010012023111012

Overview

This version addressed several customer and internal testing issues to enhance product stability.

Bug fixes and improvements

Added a hidden parameter to maintain the same sorting method for each query when paging queries were performed. This feature is compatible with MySQL Database.
Fixed the issue where a TTL task occupied too much memory when the OBKV was scanning expired data.
Fixed the issue where the original leader of the log stream failed to release the memory occupied by the OBKV TTL task after the log stream switched to a new leader.
Fixed the memory leak issue in the CreateContext module that might be caused by frequent creation of new connections during the transmission of large objects and text data.
Fixed the issue where excessive logs were printed when there was a change in tenant roles.

V4.2.1_CE_BP1

Version information

Release date: October 30, 2023
Version: V4.2.1_CE_BP1
RPM version: oceanbase-ce-4.2.1.1-101000042023102715

Overview

This version improved the compatibility of user-defined variables with MySQL Database.
This version added two views GV$OB_KV_CONNECTIONS and V$OB_KV_CONNECTIONS to query all active connections of the current tenant in an OBKV database.
The maximum length of the system variable ob_tcp_invited_nodes was changed to 64 KB.
The default value of the ls_gc_delay_time parameter was changed to 0.
This version addressed several customer and internal testing issues to enhance product stability.

Bug fixes

Fixed the issue where Error 5150 was constantly reported because the tablet location cache of the auto-increment column was not refreshed after the number of units was adjusted for scaling.
Fixed the issue in V4.x_CE where a 50 ms timeout limit was set when obtaining GTS snapshot versions, causing failure to successfully obtain Global Timestamp Service (GTS) snapshot versions in high-latency networks and continuous error OB_GTS_NOT_READY.
Fixed the issue where using the HBaseAPI of the OBKV client for batch PUT operations could potentially cause OBServer node crashes.
Fixed the issue where performing InsertOrUpdate operations on the OBKV client with a column specified as the time-to-live (TTL) column using CURRENT_TIMESTAMP could cause OBServer node crashes.
Fixed the issue where a core dump might be triggered when the SUBPLAN FILTER (SPF) operator is used for batch rescan. A parameter was added to disable this feature by default.
Fixed the issue where SQL statements in user-defined functions (UDFs) failed to hit the plan cache when tx_read_only is set to True.
Fixed the issue where Error 4002 was reported in PS mode when the default value NULL of a complex data type is used as an input parameter of a stored procedure.
Fixed the issue where some parallel DML (PDML) features might not take effect in multi-table joins.
Fixed the issue where memory leak ocurred in the TenantTTLMgr module after TTL tasks were executed in the HBase storage model.
Fixed the potential issue of data and load imbalance after importing data into non-partitioned tables during parallel table creation.

V4.2.1_CE

Version information

Release date: October 13, 2023
Version: V4.2.1_CE
RPM version: oceanbase-ce-4.2.1.0-100000102023092807

Overview

OceanBase Database V4.2.1_CE is the first long-term support (LTS) version of the V4.2.x_CE series. This version comes with a range of MySQL-compatible features, such as the VALUES statement and JSON_TABLE expression. It also introduced the Workload Repository (WR)-based data collection framework and enhanced end-to-end diagnostic capabilities for commercialization. In addition, this version optimized system resource utilization and supports table-level restore and Tencent Cloud Object Storage (COS). This version is recommended in production environments for regular business operations.

Key features

Compatibility with MySQL

VALUES statement

The VALUES statement was introduced in MySQL 8.0.19. It can be used to quickly construct a table of data or as a standalone DML statement with support for ORDER BY and Limit clauses. It can also be used as a special table in regular DML statements. OceanBase Database V4.2.1_CE is compatible with MySQL Database and supports the VALUES statement, making table value construction more convenient and easy to use.
RENAME COLUMN

The RENAME COLUMN syntax was supported since MySQL 8.0. It allows users to rename columns without changing their definitions, which is more lightweight than the CHANGE COLUMN syntax. OceanBase Database is compatible with the RENAME COLUMN syntax of MySQL 8.0 since V4.2.1_CE.

OBKV features

JSON_TABLE expression

JSON is a semi-structured data format where users can extract specific values from the JSON strings based on a path. The JSON_TABLE expression allows users to convert semi-structured data into structured data. The earlier versions of OceanBase Database already supported the JSON data type, but the JSON_TABLE expression was not implemented under the MySQL mode. To meet the needs of users who want to convert JSON data into structured data for computations, V4.2.1_CE also supports this expression under the MySQL mode.
OBKV TTL

Time-to-live (TTL) is a commonly used feature in NoSQL databases, designed to automatically expire data. For example, in businesses that deal with historical data, there is a need to periodically scan the data, determine its expiration time, and delete it accordingly. Many businesses currently rely on the TableAPI of OBKV. To simplify the management of periodic data, OceanBase Database provided the OBKV TTL capability since V4.2.1_CE. Users can define TTL attributes at the table or row level using the CREATE TABLE statement so that expired data is not processed during read and write operations in OBKV.

Performance improvements

Concurrent execution of CREATE TABLE

In the earlier versions of OceanBase Database, DDL requests were executed serially in RootService queues. Starting from OceanBase Database V4.1.0_CE, the concurrent DDL capability was introduced, with support for concurrent execution of TRUNCATE TABLE. To address the performance issues related to schema migration for tables with millions of data rows, V4.2.1_CE now supports concurrent execution of CREATE TABLE, providing a performance improvement of tenfold or even more compared with serial execution.
Improved foreign key check performance

Prior to OceanBase Database V4.2.1_CE, foreign key checks were implemented using nested SQL statements, incurring significant overhead. From V4.2.1_CE, foreign key checks are performed using Database Autonomy Service (DAS) tasks, and batch checks in single-table DML operations is supported. Additionally, with the check result caching strategy, noticeable performance improvements have been achieved in INSERT and UPDATE scenarios. In a single-node scenario (where the parent table and child table are on the same node), the performance degradation caused by foreign key checks during INSERT and UPDATE operations on foreign key tables is generally controlled to around 10% compared with tables without foreign keys. In a distributed scenario (where the parent table and child table are on different nodes), the performance degradation is generally controlled to around 30% compared with tables without foreign keys during INSERT and UPDATE operations.
Dynamic sampling on MemTables

OceanBase Database V4.2.0_CE introduced the dynamic sampling feature, which samples data during the generation of an execution plan. When the number of rows in the SSTable exceeds that in the MemTable, dynamic sampling is performed on SSTable blocks. When the number of rows in the MemTable exceeds that in the SSTable, dynamic sampling is performed on the MemTable. In V4.2.0_CE, this was achieved through full read of the MemTable, which was relatively inefficient. However, in V4.2.1_CE, direct sampling for the MemTable is implemented, utilizing RANGE partitioning to perform interval reads. This enhancement significantly improves the efficiency of dynamic sampling on the MemTable.
Parallel read of archived logs

In OceanBase Database V4.1.0_CE, OceanBase Change Data Capture (CDC) can read archived logs through the CDC service or directly read logs from the archive destination. In V4.2.0_CE, a network-based standby database reuses the log pulling framework of OceanBase CDC to pull logs from the primary database by using the CDC service. After the CDC service receives a log pulling remote procedure call (RPC) request from OceanBase CDC or a network-based standby database, the service first attempts to read logs from the local disk. If the corresponding logs do not exist in the local disk due to reasons such as log stream garbage collection (GC) or log recycling, and if the tenant has enabled log archiving, the CDC service reads logs from the log archive destination, such as the network file system (NFS) or Alibaba Cloud Object Storage Service (OSS). The log reading performance of the CDC service determines the log consumption speed at the downstream. If the performance of the CDC service in reading archived logs from OSS is poor, the synchronization link or the network-based standby database may lag behind the primary database and may be unable to become synchronized with the primary database. To address this issue, V4.2.1_CE allows the CDC service to read OSS-archived log files based on the basic capability of parallel read of OSS files.

Resource optimization

This version optimized the utilization of system resources in public cloud environments, including user tenant memory, memory of the SYS500 tenant, and operating system memory. For example, for Elastic Compute Service (ECS) instances with specifications of 128 GB or higher, the memory utilization of system resources has been reduced from around 30% to approximately 15%.

High availability enhancement

Table-level restore

Prior to OceanBase Database V4.2.1_CE, tenant-level data restore was supported. If data within a table was damaged due to a misoperation, users needed to rely on the physical restore feature to restore the tenant to a previous point in time or use the import/export feature for table-level restore. V4.2.1_CE added support for table-level restore, allowing users to specify which backup data to use, which databases or tables to restore, which tenant to operate on, and the desired recovery point in time on the target cluster.
Support for COS

OceanBase Database V4.1.0_CE supports OSS as the backup media. OceanBase Database V4.2.1_CE supplements this support by allowing users to back up logs and data to COS and read backup data from COS for data restore.

Security enhancement

Data integrity protection: The Security Assessment of Commercial Cryptography Application imposes requirements for confidentiality and integrity protection on stored data in information systems. In the versions earlier than V4.2.1_CE, the transparent encryption feature of OceanBase Database has already supported AES_ECD and SM4_CBC (which are named in the format of encryption algorithm_encryption mode), providing confidentiality protection for stored data. V4.2.1_CE added support for the GCM mode. In addition to data encryption, the checksum of ciphertext is calculated based on the key, and the ciphertext is checked during decryption to determine whether the ciphertext has been changed, thereby protecting data integrity.

Diagnostic capabilities improvement

WR-based data collection framework

The ease of use of the database performance diagnostics feature is closely related to the efficiency of system tuning. Oracle Database 10g introduces proactive automatic diagnostic capabilities such as Active Session History (ASH), Automatic Workload Repository (AWR), and Automatic Database Diagnostic Monitor (ADDM), and provides an inside-out (with information collected by internal components) and top-down (based on the database time and other metrics) database optimization and analysis framework. After decades of development, these capabilities have become industry benchmarks. OceanBase Database V4.2.1_CE implements a data collection framework similar to AWR of Oracle Database. The framework collects existing data such as ASH reports, statistics, wait events, and SQL execution details, and periodically persists performance-related views of OceanBase Database. OceanBase Database V4.2.1_CE introduces WR views and supports configuration adjustment and snapshot management by using DBMS_WORKLOAD_REPOSITORY packages. The new version also unifies, streamlines, enriches, and enhances SYSSTAT, ASH, and other statistical metrics for higher comprehensiveness and accuracy. The performance diagnostic capabilities will be further enhanced in future OceanBase Database versions.
Enhanced ASH feature

The ASH feature was introduced in OceanBase Database V4.0.0_CE. It samples the status of all active sessions in the system at an interval of 1 second and records the information in internal tables. In OceanBase Database V4.2.1_CE, additional sampling points related to transactions, storage, and DAS were added. The ASH data is collected into the WR internal table at a default sampling rate of 10:1 and a collection interval of 1 hour. By analyzing ASH data, you can easily identify issues such as long wait events, slow SQL statements, and resource utilization of active sessions, which helps in performance optimization.
End-to-end diagnostics feature

OceanBase Database V4.2.0_CE supports visualized end-to-end tracing of SQL requests, and in V4.2.1_CE, the configuration management capability for end-to-end diagnosis is further enhanced. A new view GV$OB_FLT_TRACE_CONFIG is added to record the configuration policies for end-to-end diagnostics at different levels. The GV$OB_PROCESSLIST view now includes LEVEL, SAMPLE_PERCENTAGE, and RECORD_POLICY columns to display the monitoring level, sampling frequency, and recording/printing policy that are effective for each session. Additionally, the GV$OB_SQL_AUDIT view includes a new column FLT_TRACE_ID to record the end-to-end trace information, enabling comprehensive SQL tracing. OceanBase Cloud Platform (OCP) will also provide corresponding page functionality to support this feature.

Product behavioral changes

The following table describes the product behavioral changes made in this version.

Feature	Change description
Triggers disabled for the `LOAD DATA` statement	The `LOAD DATA` statement is executed in parallel for fast data import. When parallel execution involves triggers, trigger correctness issues may occur. Therefore, triggers are disabled for the `LOAD DATA` statement in OceanBase Database V4.2.1_CE.
Semantic change for `tenant=all`	In the versions earlier than OceanBase Database V4.2.1_CE, the semantics of `tenant=all` was ambiguous. For example, `tenant=all` indicates all tenants in system parameter settings, but indicates user tenants in backup and restore settings. Since OceanBase Database V4.2.1_CE, the semantics of `tenant=all` is changed and specifies "to include only all user tenants and to exclude SYS and META tenants". `tenant=all_user` and `tenant=all_meta` are added, which respectively indicate all user tenants and all META tenants. After the change, `tenant=all` and `tenant=all_user` share the exact same semantics. Therefore, `tenant=all` will be deprecated in future versions.
Refined load balancing control	OceanBase Database V4.2.0_CE introduced the load balancing and transfer capabilities, which are specified by the tenant-level parameter `enable_rebalance`. However, when the `enable_rebalance` parameter is disabled, you cannot modify `unit_num`, `primary_zone`, and other parameters in O&M commands. OceanBase Database V4.2.1_CE introduced the tenant-level parameter `enable_transfer`, which allows you to disable the transfer feature for a tenant. You can use it with the `enable_rebalance` parameter to control the load balancing strategy for a tenant. `enable_rebalance=true` and `enable_transfer=true`: specify to dynamically adjust the number of log streams for a tenant based on the load balancing algorithm and automatically perform leader balancing and partition balancing for the tenant. `enable_rebalance=true` and `enable_transfer=false`: specify not to dynamically adjust the number of log streams or perform transfer but only to perform log stream balancing based on existing log streams. `enable_rebalance=false` and `enable_transfer`: invalid. In this case, no load balancing related O&M commands are allowed.
Case sensitivity in OceanBase CDC	OceanBase CDC is case insensitive in the versions earlier than V4.2.1_CE. The new version supports case sensitivity. The database and table names are output according to the setting of the `lower_case_table_names` parameter, whose valid values are `0`, `1`, and `2`.
Transport Layer Security (TLS) protocol version for SSL encryption	In the versions earlier than V4.2.1_CE, there were no restrictions on the TLS protocol version for SSL encryption. Therefore, clients can use the TLS protocol of an early version for encrypted communication with databases. However, some scenarios require TLS1.2 and TLS1.3 for higher security. For compatibility with clients of earlier versions, OceanBase Database V4.2.1_CE introduces the `sql_protocol_min_tls_version` parameter, which allows you to specify the minimum version of the TLS protocol. The default value is compatible with previous versions.
Calling a stored procedure with an output parameter without returning a result set	MySQL allows you to call a stored procedure with an output parameter without returning the result set of the output parameter. In the versions earlier than OceanBase Database V4.2.1_CE, a result set of an output parameter is returned, causing the native MySQL driver to report an error. In OceanBase Database V4.2.1_CE, after you execute a `CALL` statement by using OBClient, the result set of the output parameter is no longer displayed.
zlib compression disabled	In OceanBase Database V4.2.1_CE, you cannot use `zlib` compression for new tables or change the compression algorithm of an existing table to `zlib`.

View changes

The following table describes the changes to views made in this version.

View	Change type	Description
CDB/DBA_WR_CONTROL	New	Stores WR-related configurations.
[G]V$OB_TENANT_RUNTIME_INFO	New	Displays the running details of tenants.
[G]V$OB_SQL_AUDIT	Modified	The `FLT_TRACE_ID` column is added to record end-to-end trace information.

Parameter changes

The following table describes the parameter changes made in this version.

Parameter	Change type	Description
memory_limit_percentage	Modified	The upper limit of the percentage of available memory for OBServer nodes is changed from 90% to 95%. OBServer nodes are allowed to use a higher percentage of resources in servers with high specifications.
sql_protocol_min_tls_version	New	The minimum version of the SSL/TLS protocol used by SSL connections for SQL statements. This is a cluster-level parameter. The default value is `none`, which means there are no restrictions on the TLS version for connections. Note This parameter controls only the version number of SSL. To enable SSL, you must set the `ssl_client_authentication` parameter.
enable_transfer	New	Specifies whether to allow transfer within the tenant. This is a tenant-level parameter. This parameter is invalid when `enable_rebalance` is disabled. The default value is `True`, which indicates that transfer within the tenant is allowed.
compaction_dag_cnt_limit	New	The maximum number of directed acyclic graphs (DAGs) in a compaction DAG queue. This parameter is a tenant-level parameter. The default value is `15000`. In scenarios with millions of partitions, you can increase the value of this parameter to speed up compaction, which however increases memory consumption.
compaction_schedule_tablet_batch_cnt	New	The maximum number of partitions that can be scheduled in each batch for batch compaction scheduling. This parameter is a tenant-level parameter. The default value is `50000`. In scenarios with millions of partitions, you can increase the value of this parameter to speed up compaction, which however increases CPU consumption and results in longer thread scheduling time.
server_cpu_quota_min	Modified	The default value is changed from `1` to `0`, and the value range is changed to [0, 16]. The value `0` indicates that the specifications of the `sys` tenant are adaptive.
server_cpu_quota_max	Modified	The default value is changed from `1` to `0`, and the value range is changed to [0, 16]. The value `0` indicates that the specifications of the `sys` tenant are adaptive.

Recommended versions of tools

The recommended platform tool versions for OceanBase Database V4.2.1_CE are described in the following table:

Tool	Version
OceanBase Database Proxy (ODP)	V4.2.1
OCP	V4.2.0
OceanBase Deployer (OBD)	V2.3.1
OceanBase All in One	V4.2.1
OceanBase Developer Center (ODC)	V4.2.2
OceanBase CDC	V4.2.1
OceanBase Migration Service (OMS)	V4.2.0
OBClient	V2.2.3
OceanBase Connector/C	V2.2.3

Upgrade notes

An online upgrade of OceanBase Database Community Edition from V4.2.0_CE to V4.2.1_CE is supported.
An online upgrade of OceanBase Database Community Edition from V4.1.0_CE to V4.2.1_CE is supported.
We recommend that you upgrade OceanBase Database before you upgrade ODP.
During the upgrade, the system automatically disables major compactions and DDL operations. After the upgrade is complete, the system resumes normal operation.
An online upgrade of OceanBase Database Community Edition from V3.x_CE to V4.2.1_CE is not supported.
You can upgrade OceanBase Database Community Edition from V3.x_CE to V4.2.1_CE through logical data migration by using OMS.

OceanBase Database Community Edition V4.2.1

V4.2.1_CE_BP2

Version information

Enhanced features

Product behavioral changes

Bug fixes

Considerations

V4.2.1_CE_BP1_HF1

Version information

Overview

Bug fixes and improvements

V4.2.1_CE_BP1

Version information

Overview

Bug fixes

V4.2.1_CE

Version information

Overview

Key features

Compatibility with MySQL

OBKV features

Performance improvements

Resource optimization

High availability enhancement

Security enhancement

Diagnostic capabilities improvement

Product behavioral changes

View changes

Parameter changes

Note

Recommended versions of tools

Upgrade notes