OceanBase Database Community Edition V4.3.5

V4.3.5_CE_BP1

Version information

• Release date: March 21, 2025
• Version: V4.3.5_CE_BP1
• RPM version: oceanbase-ce-4.3.5.1-101000042025031818

Upgrade notes

• Supports upgrading from any version of V4.3.x to V4.3.5 BP1.
• Does not support upgrading from the V4.2.x series or earlier versions to V4.3.5 BP1.
• Since V4.3.2, the persistence format for multi-source data has been restructured, requiring conversion between old and new formats during the upgrade. Allocate ample time for the upgrade if there are many partitions.

Feature enhancements

• Support for heap-organized tables

  OceanBase Database adopts the clustered-index table model, which performs excellently in OLTP environments. This model stores primary keys and main table data in the same table to optimize primary key query speed and unique value verification performance. However, this model has limitations for OLAP scenarios that require efficient data import and complex data analysis. The import process requires sorting of full data, and primary key row fusion is mandatory during queries, which affects performance. The new version introduces support for heap-organized tables. In this mode, primary keys serve only for uniqueness constraints, and queries depend on the main table. When user data is sorted by time, skip indexes can be effectively used to improve query efficiency. Moreover, by decoupling primary keys from data, the need for sorting main table data during the import process is eliminated, thereby enhancing import performance.

• Support for the STRING data type

  Some AP business scenarios tolerate longer columns and often use variable-length string types to store data, requiring these columns to be set as primary keys or indexes. In OceanBase Database, CHAR and VARCHAR types can store strings and be used as primary keys or index keys, but their length must be specified. On the other hand, LOB types like MEDIUMTEXT and TEXT can store variable-length strings but cannot be used as primary keys or index keys. To better meet AP business requirements, the new STRING data type is introduced. This data type does not require a specified length and has a default maximum length of 16 MB. Columns of the STRING data type can be used as primary keys or index keys if their length is less than 16 KB and does not exceed the value of lob_inrow_threshold.

• Optimization of vector indexes

  OceanBase Database has supported HNSW indexes since V4.3.3, which are graph-based vector indexes that offer excellent index performance but require constant memory residence, leading to high costs. When the scale reaches a certain level, it becomes challenging to store all HNSW indexes in memory. The new version introduces HNSW_SQ indexes, offering construction speed, query performance, and recall rates similar to HNSW indexes while reducing total memory usage to 1/3 to 1/4 of HNSW indexes. This makes HNSW_SQ indexes suitable for scenarios requiring high performance and recall rates. Additionally, the new version supports IVF indexes, which do not need to reside in memory, making them suitable for scenarios with high data volume and cost sensitivity where performance requirements are not extremely high. The new version also optimizes vector queries with filter conditions, improving vector index performance in partitioned tables by 40%. Furthermore, the new version allows the creation of both vector and full-text indexes on the same table.

• Performance and resource optimization for table-level restore

  The table-level restore process consists of three steps: auxiliary tenant physical restore, cross-tenant table import, and auxiliary tenant cleanup. Currently, the auxiliary tenant physical restore step uses a full restore approach, where all data is restored to the auxiliary tenant from backup media. This approach has the following issues:

  1. Users must reserve sufficient CPU, memory, and disk resources in the cluster to support the temporary restoration of the auxiliary tenant.
  2. Restoring all data from backup media to the auxiliary tenant consumes a large amount of time and network bandwidth.

  The new version optimizes the mechanism for auxiliary tenant physical restore. Instead of using full restore, it now employs quick restore, reducing the disk resources required for temporary auxiliary tenants and decreasing the time needed for auxiliary tenant restore by dozens of times. This significantly improves table-level restore performance.

• Support for incremental direct load to heap tables with unique indexes

  Direct load is divided into full direct load and incremental direct load. Incremental direct load outperforms full direct load when data already exists in the table. However, incremental direct load did not previously support importing data to a heap table with a unique index. The new version now supports using incremental direct load to import data to a heap table with a local unique index.

• Support for HASH partitioning and concurrent partition import in direct load

  HASH partition-level direct load was added starting from V4.3.5 BP1. Concurrent partition import is supported in incremental direct load, but when multiple import tasks involve overlapping partitions, concurrent partition import is not supported.

• Performance improvement in direct load

  Vectorization optimization, streamlined temporary file processes, and improved computing performance at each stage have collectively enhanced the performance of importing tables in Clickbench by 17% compared with V4.3.5.

• Tenant-level disaster recovery

  All changes related to replicas, such as permanently removing members, adding or deleting replicas due to locality changes, and replica migration during scaling, were handled by the disaster recovery module of the RootService. If the RootService disaster recovery service fails, it could affect the disaster recovery needs of all tenants. In the new version, the disaster recovery service is separated for each tenant, allowing each tenant to have its own independent disaster recovery. This prevents mutual interference between the RootService and tenants.

• Decoupling of parameter modifications and execution of certain DDL operations from RootService status

  The new version decouples the modification of parameters, execution of certain DDL operations (such as CREATE/ALTER/DROP DATABASE, CREATE/CREATE_LIKE/ALTER/DROP/RENAME/TRUNCATE TABLE, CREATE/DROP/UPDATE STATUS INDEX, TABLEGROUP/TRIGGER/PACKAGE/SYNONYM/OUTLINE operations), and RootService status. This means that even if RootService is unavailable, users can still modify system and tenant parameters, and execute common DDL statements, enhancing system availability.

• Enhanced DDL performance diagnostics

  Offline DDL operations often take a long time. To facilitate performance diagnostics and progress monitoring of DDL operations, the new version enriches the output information of the [G]V$SESSION_LONGOPS view, including the number of rows processed, progress, and estimated completion time.

• Support for adding subpartitions

  When the subpartition is of type RANGE [COLUMNS] or LIST [COLUMNS], new rows cannot be inserted once the data exceeds the upper limit of the initially created maximum partition. If you cannot dynamically add subpartitions, the only solution is to rebuild the table, which causes significant inconvenience for maintenance. The new version extends support for new syntax to add and remove subpartitions, enhancing the ease and flexibility of partition management.

• Optimization of statistics collection

  Statistics collection for columnstore tables has been adapted to improve performance. A progressive automatic statistics collection strategy has been introduced, where statistics are collected in batches at the partition level if there are many partitions. Asynchronous statistics collection now includes a failure flag. If statistics collection for a large table fails three times, asynchronous collection scheduling for that table will be stopped to prevent a single large table from blocking the entire collection window.

• Decoupling of PX computing nodes and data nodes

  In the current system architecture, storage resources and computing resources are highly coupled. For non-leaf DFOs, the existing PX scheduling mechanism allocates computing tasks only on the machines where the data is located. This tight coupling may limit the efficient utilization of resources in some scenarios. To enable large SQL queries to utilize more machines for computing, the new version supports decoupling between PX computing nodes and data nodes. Users can decide the candidate resource pools for non-leaf DFOs through parameters or hints (PX_NODE_POLICY). Additionally, hints (PX_NODE_ADDRS, PX_NODE_COUNT) allow users to explicitly specify the machines or the number of machines for non-leaf DFOs.

• Performance optimization for CAST, IN, and logical expressions

  In columnstore scenarios, the new version improves conversion performance using the CAST operator by an average of 70%, and enhances performance for the IN and NOT IN expressions by 10% to 70%. Logical expressions such as IS NULL and IS NOT NULL see performance improvements ranging from 20% to 80%.

• Invisible columns

  The new version supports invisible columns. When a column is marked as invisible, it is ignored unless explicitly specified in an SQL statement.

• Optimization of foreign key functionality

  The new version allows foreign keys to reference columns with non-unique indexes. It also relaxes the type constraints for parent and child keys and disables circular self-referencing foreign key cascading updates.

• New character sets and collations

  The new version introduces Asian character sets and collations such as ujis, euckr, gb2312, cp932, eucjpms, cp850, hp8, macroman, and swe7.

• Improved MySQL compatibility

  In earlier versions, users could create indexes on ENUM and SET columns, but the query ranges for these indexes could not be correctly extracted. The new version enhances the capability to extract query ranges for ENUM and SET columns, enabling the use of indexes to improve performance. The new version also maintains compatibility with MySQL's type downgrade behavior, aligning constants with column types in comparisons to enhance efficiency. In standard SQL semantics, STRING values are compared as strings when compared with INT or DECIMAL values. However, in some AP scenarios, comparing STRING values can achieve better performance. The new version introduces a feature to enable non-standard comparison. This can be done by setting the hidden parameter _non_standard_comparison_level to equal (suitable for equality comparison) or range ( (suitable for both equality and range comparisons) ) or using the hint opt_param('non_standard_comparison_level', 'xxx').

• Support for reading files from HDFS in external tables

  HDFS is a common storage medium in data lake architectures and serves as a platform for multi-engine data sharing. The new version of OceanBase Database allows direct reading of files from HDFS using external tables and supports connection to HDFS clusters with Kerberos authentication.

• Support for reading and writing data from/to MaxCompute (ODPS) data sources via external tables

  The new version supports reading data from MaxCompute using external tables or URL external tables and writing data from OceanBase to MaxCompute.

• Enhanced full-text index capabilities

  Starting from V4.3.1, OceanBase Database has supported full-text indexing. As the version evolves, additional features and extensions have been gradually added. The new version of V4.3.5 BP1 introduces the built-in Chinese IK full-text indexing engine, supporting two modes: SMART and MAX_WORD. It also allows users to add custom full-text indexing engines in plugin form. The new version also adds a new attribute, parser_properties, for full-text indexing indexes, allowing configuration at the table level. Apart from the existing natural language mode for full-text indexing, the new version also supports boolean mode, which can be activated using the phrase IN BOOLEAN MODE in the MATCH AGAINST clause. Boolean mode supports boolean operators such as +, -, (), and no operation. Additionally, this release supports merging full-text indexes, using the index union merge computation to enhance query performance. The update/ deletion of full-text indexed columns in the primary table is optimized to improve update/delete performance.

• URL external tables

  When OceanBase Database accesses external data sources, users typically need to create an external table first and then query it using SQL statements. The new version of V4.3.5 BP1 introduces support for URL external tables, allowing users to directly query external CSV, Parquet, and ORC files or data from MaxCompute using SELECT statements without the need to create an external table first. Additionally, the new version extends the LOAD DATA syntax to directly reference URL external tables for data import, reducing the operational complexity and cost of data import.

• Optimization of columnstore replicas

  The columnstore replica feature released in V4.3.3 had some limitations, such as supporting only one C replica and not allowing direct full import of columnstore data to a C replica. The new version addresses these limitations by allowing multiple C replicas to be specified in a cluster and supporting direct full direct load of columnstore data to C replicas. The new version also provides the system view CDB/DBA_OB_CS_REPLICA_STATS for displaying the conversion progress at the partition level and log stream level, allowing users to monitor the status of columnstore replicas.

• Enhanced materialized views

  The refresh parallelism of materialized views is closely related to the efficiency of incremental and full refreshes. The new version supports complete control mechanisms for refresh parallelism, allowing users to set the refresh degree of parallelism (DOP) through the system variable mview_refresh_dop, the DBMS_MVIEW.refresh subprogram, or the PARALLEL clause when creating materialized views. The new version also supports the LOB data type for incremental materialized views and provides the ability to directly modify materialized views or their attributes using ALTER MATERIALIZED VIEW and ALTER MATERIALIZED VIEW LOG commands. The new version also modifies the parallelism, background refresh task cycle, log table parallelism, background cleanup task cycle, and LOB inline storage length threshold for materialized views. Additionally, the new version supports adding columns to the base table of a materialized view log (MLOG), including adding columns to the end or middle of the base table, simplifying the process.

• Support for querying down in rowstore data addition scenarios

  During rowstore scans, for data with no intersection, the scan is pushed down to the storage layer for processing. After major compactions, when new columns are added, theoretically, the scan can also be pushed down. However, due to the parsing failure of new columns, the pushdown fails, and it reverts to a less efficient row-by-row read. The new version optimizes this scenario to support query pushing down to the storage layer, enhancing query performance.

• OceanBase Database startup acceleration

  The new version optimizes the time required for bootstrapping a 1-1-1 cluster and creating a tenant to within 40 seconds.

• Improved functions for ARRAY data types

  OceanBase Database supported the ARRAY data type and some common functions and operators in V4.3.3. The new version further refines these functions and introduces new ones, including array_prepend, array_concat, array_compact, array_filter, array_sort, array_sortby, array_length, array_range, array_sum, array_first, array_difference, array_min, array_max, array_avg, array_position, array_slice, and reverse.

• Enhanced PL stability

  The new version improves the dependency collection capabilities for PL objects, enhancing the thread stability of PL debugging.

• Support for cell-level TTL in OBKV

  Time To Live (TTL) is a mechanism used in HBase to control the storage duration of data. It automatically deletes data exceeding the specified duration to help manage storage space and keep data up to date. HBase supports two types of TTL: Column Family TTL and Cell TTL. Earlier versions of OBKV-HBase supported table-level TTL, corresponding to HBase's Column Family TTL. The new version adds support for Cell TTL, providing smaller-grained automatic expiration for cells, applicable to scenarios such as short-term data storage, session data cleanup, and social network data cleanup.

• Support for full-text indexing in OBKV

  OBKV-Table models adapt the full-text indexing capabilities. In scenarios with full-text indexes, OBKV-Table supports DML operations including single and batch insert, update, delete, insertOrUpdate, replace, increment, and append. Full-text indexing capabilities are also adapted for OBKV-Table, supporting synchronization or asynchronous queries.

• Enhanced compatibility and features of OBKV-HBase

  Earlier versions of OBKV-HBase provided compatibility with most features of HBase 1.x. The new version further enhances compatibility with HBase 1.x and 2.x, including support for multiple namespaces, ColumnRangeFilter, ColumnValueFilter, CheckAndMutateBuilder, hybrid batch processing, and CheckAndMutateBuilder. The new version also allows users to batch execute Get, Put, and Delete operations using the batch interface in the OBKV-HBase client. Group commit is supported for Put operations to address the issue of insufficient qualifiers written in a single Put operation, which hinders the performance advantage of batch processing.

• Enhanced features of OBKV-Table

  In previous versions, if the checkAndInsUp operation failed, the system would do nothing. In some customer scenarios, checkAndInsUp is used as an atomic operation for compare-and-swap (CAS). An atomic operation requires that if the checkAndInsUp operation fails, an error should be returned. To meet this requirement, the new version supports RollbackWhenCheckFailed for checkAndInsUp. If the check fails, the system returns the error [-10518][OB_KV_CHECK_FAILED] and rolls back the transaction.

• Interface model for OBKV-Redis

  The new version introduces the OBKV-Redis interface model, designed to be a persistent database compatible with Redis interfaces. It aims to solve the issues of data inconsistency, complex architecture management, poor scalability, lack of transaction support, and high O&M costs associated with traditional solutions combining Redis with relational databases. By persistently storing data, OBKV-Redis reduces business costs and provides transaction support to ensure data consistency, meeting 80% of user RT and throughput requirements. It is compatible with native Redis versions 3.2, 4.0, and 5.0, supporting five basic data types: String, Hash, List, Set, and Zset, as well as public commands like TTL, EXPIRE, and DEL. Additionally, the new version supports histogram statistics for command response times in OBKV-Redis, allowing users to query sampled data in the [G]V$OB_QUERY_RESPONSE_TIME_HISTOGRAM view and visualize the results using Prometheus.

• Compatibility with illegal date types in MySQL mode

  The new version adds compatibility with illegal date types. Users can control whether to allow storage of illegal dates using the sql_mode parameter. For example, specific SQL_MODE values like ALLOW_INVALID_DATES and NO_ZERO_IN_DATE can be used to control the storage of dates like 2024-2-30 and 2022-01-00. By default, this feature is enabled in new clusters and disabled in older versions. To enable it in older versions, users can set the hidden parameter _enable_mysql_compatible_dates to True.

• Enhanced PL functionality

  The new version optimizes the performance of using index tables within anonymous blocks and query performance for the mysql.proc system table. In addition, alter procedure and function compile syntaxes are added to manually compile stored procedures or functions.

• Changes in replicated table attributes

  Replicated tables, supported since V4.2.0, allow users to read the latest data modifications from any "healthy" replica. Converting a replicated table to a regular table or vice versa requires retable creation and data migration, which is time-consuming and complex. The new version introduces the ability to change replicated table attributes using the ALTER TABLE command, reducing user costs.

• Performance optimization for small-scale OLTP

  Small-scale sysbench performance has improved by 10% overall compared to V4.3.5. For single-row query scenarios, the performance has improved by 20% by avoiding range scans and using table gets instead.

• Enhanced SQL query interface

  V4.3.5 BP1 introduced the read-only replica access isolation feature, which defines internal data request routing rules and strictly prohibits forwarding requests sent to read-only zones to non-read-only replicas, achieving resource isolation between TP and AP and ensuring long-term stable HTAP operation. Non-read-only replica routing is not restricted by these rules. Additionally, the new version reconstructs the metadata information of SQL protocol result sets. OceanBase Server obtains server-side metadata based on tenant compatibility mode. OceanBase JDBC then derives client-side metadata based on the server-side metadata to respond to user metadata access requests.

Feature updates

• AP parameter template disabling NLJ

  After the optimizer generates a nested loop join plan, the plan is sensitive to the number of rows in the driving table. If the estimated number of rows in the driving table is small, the actual execution performance of the plan can significantly degrade. In OLAP scenarios, data volumes are generally large, and the benefits of using nested loop join plans are limited. Therefore, by default, the system generates hash join plans instead of nested loop join plans unless a non-equality join is required.

• Client disconnects after file import times out

  When importing large files, if the client's timeout period is set too short, the server attempts to receive the remaining file content after the timeout and then returns a timeout message to the client (according to protocol design). This can occupy a session connection for a long time. The new version updates this behavior by disconnecting the client if no messages are received for an extended period.

Parameter changes

System variable changes

View changes

Function changes

Syntax changes

Bug fixes

• Fixed the issue where system variable settings might fail during connection reset.
• Fixed the issue where error code 4016 might be returned during statistics collection for hybrid row-column storage partitioned tables.
• Fixed the issue where querying data in some full-text index scenarios produced incorrect results.
• Fixed a performance degradation issue in direct load caused by lock conflicts between INSERT OVERWRITE operations for large data volumes and writing statistics information.
• Adjusted the freeze_cnt value to reset after daily major compaction.
• Fixed the issue where transaction data could not be read in some column-store merge scenarios, causing merge timeouts
• Fixed the issue where SQL audit might fail to evict records in some scenarios.
• Fixed the issue where refreshing materialized views containing data types such as arrays without precision would throw errors.
• Fixed a memory bloat issue in HashPartInfra during rescan scenarios.
• Fixed the issue where querying data in some batch get scenarios in OceanBase Database's Key-Value mode produced incorrect results.
• Fixed the issue where error code 4019 was returned when using REPLACE INTO in some full-text indexing scenarios.

V4.3.5_CE

Version information

• Release date: January 6, 2025
• Version: V4.3.5_CE
• RPM version: oceanbase-ce-4.3.5.0-100000202024123117

Version overview

OceanBase Database V4.3.5 is the first LTS version for AP scenarios. On the basis of V4.3.4, the new version refines product capabilities, supports nested materialized views, and enhances full-text index and vector index features. Data import and export capabilities are also improved: direct load is supported for specified partitions, external tables can read ORC files, and the SELECT INTO OUTFILE statement now supports exporting Parquet and ORC files. The optimizer capabilities are enhanced with SPM evolution based on the baseline priority strategy, improved optimizer row estimation, cost validation optimization for recursive inline common table expressions (CTEs), and strengthened query range extraction.

In terms of performance optimization, V4.3.5 optimizes table-level restore performance, direct load performance, DML performance, and DDL performance to varying degrees. The new version supports online conversion between row-based storage and columnar storage, quick intermediate column addition, and extended parallel DDL operations. Statistics and clog submission are integrated with resource isolation, and I/O isolation is implemented to enhance the resource isolation mechanism of OceanBase Database. The new version also supports memory usage limits at the SQL level, optimized memory usage for stored procedures, and reduced tenant memory resource consumption through a shared thread pool, improving stability in various scenarios. In terms of ease of use, diagnostic capabilities for ASH and WR are enhanced, and log stream replica parallel migration optimization and monitoring views for log transmission links to enhance log synchronization link observability are supported, making OceanBase Database easier to use. The new version supports more ARRAY and XML expressions and optimizes memory usage for the execution of JSON, XML, and GIS data. Compatibility with OBKV-HBase and HBase is enhanced, and P95 and P99 metrics are supported for OBKV.

Features

Kernel enhancements

• Support for nested materialized views

  In versions prior to V4.3.5, materialized views could only be created on user tables. In data warehouse scenarios, materialized views are used for data processing. To support lightweight real-time data warehouses, V4.3.5 introduced the ability to create materialized views based on existing ones, known as nested materialized views. Nested materialized views support both full and incremental refresh, just like non-nested materialized views. Consequently, V4.3.5 enables the creation of materialized view logs based on materialized views. The freshness of data in a nested materialized view depends on the freshness of its base table. To ensure the freshness of data in a nested materialized view, you must first refresh the underlying materialized view.

• Specific partition-based direct load

  In versions prior to V4.3.5, OBServer nodes only supported full-table direct load. If you wanted to perform direct load on only specific partitions, you had to use full-table direct load and then exchange the partitions. This process was inconvenient for users. To better support partition-level data import, V4.3.5 allows you to specify partitions for direct load by using the LOAD DATA and INSERT INTO SELECT statements. However, you cannot specify the last-level partitions with HASH or KEY partitioning types.

• Enhancements to full-text indexing functionality

  V4.3.5 continues to enhance the full-text indexing functionality based on V4.3.4, with the following main improvements:

  • Supports creating full-text indexes after using the CREATE FULLTEXT INDEX or ALTER TABLE ADD FULLTEXT INDEX statements.

  • Supports cost-based full-text index plan selection. For SQL queries with multiple filters that include MATCH AGAINST, it allows selecting the index with lower overhead for scanning based on cost.

  • Supports Functional Lookup:

    • Allows multiple MATCH AGAINST expressions in query statements.
    • Supports using full-text indexes alongside other indexes.
    • Allows MATCH AGAINST without filter semantics to be output as projection columns.
    • Supports using symbols like <= or < for filtering with MATCH AGAINST that has filter semantics.
    • Allows logical AND/OR linking of MATCH AGAINST with filter semantics and other filter semantics.

• Enhancements to SELECT INTO OUTFILE functionality

  V4.3.5 enhances the SELECT INTO export functionality introduced in V4.3.0 by supporting exporting data in the Parquet and ORC formats and allowing you to specify a compression algorithm when exporting data in the CSV format. Additionally, V4.3.5 introduces new syntaxes for SELECT INTO.

  • You can use the format syntax to set export options.
  • You can specify a compression algorithm when exporting data in the CSV format.
  • You can export data in the Parquet format.
  • You can export data in the ORC format.

• Enhancements to external table functionality

  In V4.3.5, external tables can read data in the ORC format.

• Support for specifying directories and wildcard-matched files in LOAD DATA statements

  V4.3.0 introduced support for using wildcards to match files during data import via LOAD DATA in direct load mode, whereas regular import only supports single file import. To import multiple files, multiple LOAD DATA statements are necessary. V4.3.5 enhances regular import by allowing multiple file imports, similar to direct load, with support for comma-separated file lists and wildcard file description formats.

• Control HASH/NL CONNECT BY by using hints

  In CONNECT BY hierarchical queries, two join methods are available: nested loop join and hash join. Currently, the system automatically selects the join method based on rules, using hash join only when no index is available. However, in some cases, hash join might be the better choice even when an index exists. In V4.3.5, you can manually control whether to use nested loop join or hash join in hierarchical queries with the use_hash hint, and you can also bind an outline for more precise adjustments.

• Correctness management for ALTER SEQUENCE

  The execution of sequence involves internal tables and local caches. Internal tables ensure that the values generated globally are unique, while local caches accelerate value generation. If local caches are not cleared during ALTER SEQUENCE execution, it can lead to inconsistencies between the cache and SEQUENCE parameters, causing correctness issues when new parameters are used with old cache values. To address this, OceanBase Database V4.3.5 supports clearing local caches during ALTER SEQUENCE and synchronizing any remaining cached values back to the internal table to reduce sequence value gaps.

• Binding global indexes with primary tables in SHARDING = NONE table groups

  Prior to V4.3.5, global indexes were not included in table groups. This means that the primary table bound to a global index and the global index itself may not be in the same node. Starting with V4.3.5, for DDL operations related to global indexes such as CREATE TABLE, CREATE TABLE LIKE, CREATE INDEX, ALTER TABLE ADD INDEX, TRUNCATE TABLE, and TRUNCATE PARTITION, if the primary table is in a table group with SHARDING = NONE, the global index table must be distributed in the same way as the primary table. Additionally, if the primary table is moved into a table group with SHARDING = NONE using the ALTER TABLE statement, the global index will align with the primary table for consistent table group binding.

• Enhanced capability of query range extraction

  V4.3.5 implements a new query range extraction logic, which expands predicate pushdown scenarios to make range extraction in vector-form predicates more precise. It also addresses memory amplification issues found in the previous version. This update not only optimizes range extraction performance but also enhances the range extraction capability. For example, it modifies the way NOT IN expressions generate query ranges, improving extraction performance. It supports range graph trimming to reduce the number of final query range extractions, thereby lowering memory consumption. Additionally, it expands index hint functionality and supports trimming interesting ordering index paths according to rules, optimizing strategies like LIMIT recalculation to resolve performance issues in ORDER BY LIMIT scenarios due to suboptimal plan selection. The update also supports range extraction for spatial type columns, adds implicit conversion support, and enables fast path extraction in nested loop join rescan scenarios. Furthermore, query range trimming is integrated with a cost model to determine the extent of column extraction and the range of query range based on cost. It also supports constant extraction with exec_param and NVL expression extraction.

• Improvements to optimizer row estimation system

  The optimizer's cost estimation relies on accurate row estimation for each operator, which in turn depends on the selection of estimation strategies and the accuracy of statistical information. V4.3.5 further enhances the row estimation system based on existing capabilities, improving accuracy in certain scenarios:

  • By defining the lower bound of selectivity for complex joins, such as those with multiple types of join conditions or cyclic joins involving multiple tables, this version sets a rule that when INNER/OUTER JOIN conditions are complex, each row of the smaller table must connect with at least one row of the larger table. This ensures that the selectivity is no less than one divided by the current number of rows in the larger table, addressing the issue of overly small row estimates in cyclic joins.
  • The number of rows returned by joins is corrected after filtering, improving the accuracy of row count estimation.
  • For ANTI JOIN queries with predicates on t1 and t2, if the predicate on t1 has strong filtering power, the system assumes simple inclusion instead of basic inclusion, improving the accuracy of cardinality estimation in this scenario.

Improvement in CTE extraction/inline cost validation

CTE extraction and inline are two approaches we use for handling public computation modules. During CTE extraction, the result of a CTE is materialized using the `WITH` clause, and the stored result is referenced when needed. In contrast, CTE inline does not materialize the CTE result but recalculates it within each usage context. V4.3.5 abstracts a common cost verification module for CTE inline, enhancing the cost verification process and improving the accuracy of local cost validation.

• Baseline-prioritized SPM evolution

  In V4.3.4 and earlier, SQL Plan Management (SPM) evolution was conducted online. Specifically, when a new plan appeared, it would immediately be compared with the baseline plan. If the new plan was verified to be better than the baseline plan (based on the average CPU time required for execution), it would replace the baseline plan as the new baseline. While effective in most cases, this approach encountered issues in some scenarios. For example, if a new plan was generated when the dataset was very small, SPM might mistakenly identify it as superior to the old plan. Once the dataset grew significantly, the new plan became unsuitable. V4.3.5 introduces a new SPM evolution mode that prioritizes baseline plans. In this mode, the optimizer always uses the baseline plan unless all baseline plans fail. Here, all baseline plans are considered fixed.

• Reduced cost skew and execution time

  Query rewrite rules can be categorized into rule-based and cost-based rewrites. Unlike rule-based rewrites, cost-based rewrites are not optimal for all scenarios. They depend on factors like data distribution and the availability of suitable indexes. Therefore, cost-based rewrites require an additional cost verification step. This step introduces two main issues: high hard parsing time and plan deviation. V4.3.5 reduces hard parsing time by optimizing the cost verification process and adds a mechanism to limit the number of attempts in cost-based rewrites, ensuring a balance between efficiency and accuracy. Additionally, it addresses plan deviation in three scenarios where cost-based rewrites often lead to issues: grouping pullup, OR expansion, and grouping pushdown.

• Optimized DAS path selection

  This feature expands support for batch rescan and optimizes its cost calculation method. It also introduces rule-based plan pruning for batch rescan. Additionally, it adjusts the strategy for generating DAS paths in rescan plans and refines the cost calculation method for DAS rescan. Specifically, it models the RPC overheads in the DAS rescan process and adjusts the cost calculation for DAS dynamic partition pruning, enabling the optimizer to select a more efficient execution plan.

• Optimized UNION parser and parameterized display of EXPLAIN plans

  To address the issue of high hard parsing time for some complex queries, V4.3.5 optimizes interfaces related to same_as and equal sets, reducing hard parsing time. Furthermore, V4.3.5 introduces new error checks and explain information to promptly expose issues such as plan convergence failure, constraint errors, constraint inflation, and plan divergence, thereby improving the quality of parsing and rewriting.

• Optimizations to SQL optimizer features

  V4.3.5 enhances the SQL optimizer features to generate better execution plans:

  • Pushes down GROUP BY in queries with window functions.
  • Pushes down GROUP BY to the inner part of UNION ALL.
  • Optimizes the runtime filter allocation strategy.
  • Rewrites the SUM aggregate function.
  • Checks for duplicate values.
  • Pushes down DISTINCT in HASH INTERSECT DISTINCT plans.
  • Rewrites the DISTINCT aggregate function.

• Automatic partition splitting

  The automatic partition splitting feature, which was an experimental feature in V4.3.4, has been enhanced for stability in veion V4.3.5. In V4.3.5, this feature has reached General Availability (GA) standards.

• Read-only columnstore replicas

  The read-only columnstore replica feature introduced in V4.3.3 was an experimental feature. After iterations in V4.3.4 and V4.3.5, this feature has reached the GA standards in V4.3.5.

MySQL compatibility

• Support for Latin1 collations

  Starting from V4.1, OceanBase Database supports the Latin1 character set and provides support for latin1_swedish_ci and latin1_bin collations. To improve compatibility with MySQL, V4.3.5 extends support for the Latin1 collations to include latin1_german1_ci, latin1_danish_ci, latin1_german2_ci, latin1_general_ci, latin1_general_cs, and latin1_spanish_ci.

• Support for the SELECT ... FOR UPDATE SKIP LOCKED clause

  V4.3.5 introduces compatibility with the MySQL syntax SELECT ... FOR UPDATE SKIP LOCKED. This allows the system to skip locked rows by other transactions when a lock conflict occurs and immediately execute the query to return the result.

• Redesign of ENUM and SET types

  The ENUM and SET types are used to store strings of limited varieties, saving storage space and improving data processing efficiency. These types store values numerically but use character arrays to describe the data when created. This feature redesigns the metadata storage for ENUM and SET types. By storing complex character array information in the subschema context, it reduces developers' concerns about passing ENUM and SET metadata. Moreover, the generalized type conversion logic makes type inference for ENUM and SET simpler and more efficient.

• Intermediate fast column addition

  In V4.3.5, the DDL operation for adding intermediate columns has been changed from offline DDL to online DDL, enhancing the performance of intermediate column addition. It supports two scenarios: intermediate ADD COLUMN and a combination of intermediate ADD COLUMN with MODIFY COLUMN using online DDL.

Performance improvements

• Optimization of pushdown predicate filtering

  In earlier versions, the SQL layer of OceanBase relied on estimated values for filtering predicates, failing to utilize real-time information. Similarly, the storage layer lacked real-time information when handling predicate filtering, potentially leading to suboptimal query performance. V4.3.5 introduces the collection of real-time statistics related to predicates, such as execution overhead and the number of filtered rows, at the storage layer. Filters are then reordered based on these real-time statistics to optimize query performance.

• Support for asynchronous online DDL in row-to-column conversion operations

  V4.3.0 has supported three table storage formats: row-based, columnar, and hybrid row-column storage. Converting a row-based table to a hybrid row-column or columnar table was an offline DDL operation that could impact customers. V4.3.5 addresses this by supporting asynchronous conversion of row-based tables to columnar tables, enabling online DDL. Users can specify the delayed keyword in DDL statements to modify table schemas without blocking data writes. Additionally, baseline data reorganization for columnar storage is now performed asynchronously during baseline data merges. V4.3.5 now supports online DDL for converting row-based tables to columnar and hybrid row-column storage formats.

• Vectorization optimization for direct load path

  Prior to V4.3.5, the implementation of direct load lacked vectorization, with write functions processing rows sequentially, resulting in high function call overhead. Additionally, the process of writing SSTables was executed in the background without targeted optimization. V4.3.5 introduces vectorization for the direct load path and optimizes columnar encoding, significantly enhancing direct load performance. For tables without primary keys, performance improves by approximately 2 times.

• Performance optimization for DML operations

  OceanBase Database already supports performance improvement for large tables and indexes through parallel execution mechanisms of PDML for insertions, updates, and deletions. However, some scenarios remain unaddressed, including REPLACE INTO ..., INSERT ... ON DUPLICATE KEY, INSERT ALL ..., and MERGE INTO ... for updating primary keys, INSERT with auto-increment columns, and multi-table UPDATE. In previous versions, these scenarios utilized single-threaded writing, leading to long response times for large datasets, negatively impacting user experience. To overcome these performance bottlenecks for large-scale scenarios, V4.3.5 introduces concurrent write capabilities at the DAS layer, significantly enhancing DML performance using mechanisms similar to those of PDML. Additionally, for INSERT ... VALUES ... ON DUPLICATE KEY UPDATE ... and REPLACE ... VALUES ... statements, support for MULTI QUERY scenarios with BATCH processing further boosts DML performance. Furthermore, MERGE INTO statements with UNIQUE KEY support PDML, improving execution efficiency in large-scale scenarios.

• Performance optimization for insertup/replace operations

  In V4.x, the insertup (INSERT ... VALUES ... ON DUPLICATE) and replace (REPLACE INTO ... VALUES ...) statements do not perform partition pruning row by row. Therefore, SQL statements involving multiple tablet writes will uniformly use a distributed execution plan, obtaining a Global Timestamp (GTS) once before executing the statement. In scenarios with small data volumes, especially when there are many primary key/unique key conflicts, the GTS retrieval operation can cause significant performance degradation. The new version refines the write scenarios: when the write table does not involve a unique global index or involves a unique global index but the data being written is within the same log stream, the GTS retrieval operation is omitted, optimizing the execution process and significantly improving Insertup/Replace performance.

• Performance optimization for table-level restore operations

  The overall process for table-level restore consists of three steps:

  1. First, auxiliary tenants are restored to a specified point in time from backup data (physical restore).
  2. Next, the specified tables are imported from the auxiliary tenant to the target tenant across tenants.
  3. Finally, the auxiliary tenant is cleared.

  In versions prior to V4.3.5, the table import task scheduler in the cross-tenant import step restricted multiple table import tasks to serial execution. Additionally, the degree of parallelism was low during the primary table data supplementation and index reconstruction phases. To enhance table-level restore performance, V4.3.5 introduces parallel optimizations in the following phases:

  4. The maximum number of concurrent table import tasks can be controlled using the tenant parameter recover_table_concurrency, enhancing parallelism in multi-table imports.
  5. Parallel index creation is supported using parameters provided by DDL for controlling parallelism in table-level restore.

Reliability enhancements

• Support for backup of parameters and tenant resource configuration

  V4.3.5 supports the backup of both cluster-level and tenant-level parameters. Cluster-level parameters can be backed up separately by specifying path information using the ALTER SYSTEM command. Tenant-level parameters are included in the backup data set and will be backed up when data backup is performed. During tenant-level physical restore, we recommend that the target tenant's resource configuration matches the source tenant to ensure a high success rate of physical restore and subsequent production stability. To provide a reference for configuring resources for the target tenant, V4.3.5 also supports backing up tenant resource configuration information and replica distribution information.

• New parameter addressing_model for S3 Object Storage Access

  Starting from V4.3.0, OceanBase Database supports AWS S3 object storage and S3-compatible object storage. Prior to V4.3.5, OceanBase Database only supported the virtual-hosted style address format. However, many self-developed object storage vendors that are compatible with the S3 protocol only support path-style, preventing access to some S3-compatible object storage through OceanBase Database. V4.3.5 introduces an optional parameter addressing_model for S3 path configuration, with valid values: virtual_hosted_style and path_style. This allows users to choose the format mode based on the actual compatibility of the object storage.

Resource optimization

• Hierarchical isolation for I/O manager

  OceanBase Database's I/O scheduling algorithm for resource isolation includes three features: reserved bandwidth, shared bandwidth, and limited bandwidth. While the reserved and limited bandwidth features perform well, there is room for improvement in shared bandwidth. For example, a small tenant with a large resource group might be prioritized over a large tenant with a small resource group during scheduling, which hinders the achievement of a true serverless architecture. This occurs because the existing scheduling method flattens all tenants' resource groups to allocate I/O resources. Therefore, in version V4.3.5, the I/O scheduling logic was adjusted to enable true I/O resource sharing. The scheduling method transitioned from flat scheduling of resource groups to a hierarchical tree structure: system -> tenant -> resource group.

• Reduced tenant memory resource consumption through a shared thread pool

  To meet serverless requirements, it is necessary to optimize the number of OBServer threads to reduce resource consumption. Before V4.3.5, each ObTimer instance had its own thread, and adding a tenant would create dozens of new threads. These threads were not always active, leading to some degree of waste. V4.3.5 addresses the issue of excessive thread numbers by supporting timer pooling, allowing each tenant to share a thread pool. Tasks scheduled by timers are executed by threads within this pool. The timer thread pool has dynamic scalability, adapting the number of threads based on the load. Additionally, non-tenant modules share a global timer thread pool.

• Integration of statistics and clog submission into resource isolation

  In high-performance computing environments, proper resource allocation and isolation are crucial for ensuring system stability and efficiency. Effective resource isolation strategies prevent competition and interference between tasks, improving resource utilization and overall service quality. Currently, OceanBase Database has implemented resource isolation between tenants using tenant units and within tenants using the resource manager, covering key resources like CPU and I/O. To further isolate background tasks within tenants and reduce their impact on frontend user requests, statistics collection and clog submission have been adapted to resource isolation in V4.3.5. The I/O resource isolation parameters include MIN_IOPS, MAX_IOPS, and WEIGHT_IOPS, representing minimum IOPS, maximum IOPS, and IOPS weight, respectively. The I/O scheduling strategy prioritizes meeting MIN_IOPS and then schedules based on WEIGHT_IOPS without exceeding MAX_IOPS. The specific isolation strategies are as follows:

  1. Limit the MAX_IOPS for clog submission to 10, indicating that at most 10% of IOPS can be used for clog submission.
  2. Limit the MAX_IOPS for statistics collection to 20, indicating that at most 20% of IOPS can be used for statistics collection.
  3. Limit the WEIGHT_IOPS for frontend tasks and statistics collection. By setting the default value of WEIGHT_IOPS for frontend tasks to 100 and the value for statistics collection to 20, the ratio of bandwidth for frontend tasks to that for statistics collection is 5:1.

  Note that background resource isolation allows for the isolation of both CPU and I/O resources. While CPU isolation requires enabling the cgroup feature, I/O isolation can still take effect even if the cgroup feature is not enabled.

• Memory usage control at the SQL level

  OceanBase Database V4.3.4 did not support tracking memory usage during individual SQL queries. In scenarios with large or poor SQL queries or suboptimal optimizer plans, high memory usage by a single SQL query could compromise the system's overall stability. Therefore, OceanBase Database V4.3.5 introduced support for tracking memory usage during individual SQL queries. When memory usage exceeds a specific threshold, actions such as terminating execution with an error or printing a WARN log can be taken to prevent out-of-memory (OOM) situations or help diagnose OOM causes. A new tenant-level parameter query_memory_limit_percentage is provided to specify the percentage of tenant memory available for a single SQL query.

• Optimization of stored procedure memory usage

  V4.3.5 replaced the general memory allocator used in stored procedures with a custom stored procedure memory allocator that supports automatic memory release and shortens the stored procedure memory lifecycle. The design of the symbol table, memory usage of basic types, management of complex types like records and arrays, and expression memory management have also been optimized to resolve various memory leak issues, enhance memory usage stability for stored procedures, and improve performance in some expression computations.

• Removing the restriction that the cgroup feature takes effect only after a cluster restart

  On cloud platforms, the cgroup feature can be enabled or disabled only after the cluster is deployed. However, enabling or disabling cgroups requires restarting the cluster, which affects cloud-based businesses. V4.3.5 removes the requirement to restart the cluster for the cgroup feature to take effect, ensuring business continuity.

• Enhanced lock conflict management

  Local lock conflict management has been optimized to eliminate dependency on the fallback wake-up mechanism, reducing unnecessary waiting delays. Remote lock conflicts are now managed using a combination of proactive probing at the source and wake-up notifications at the target, avoiding blind retries and reducing CPU and network overhead caused by remote lock conflicts. In hotspot row scenarios, efforts are made to wake up locked parties in the order of their requests to prevent new requests from cutting in line and blocking older ones, or multiple requests from retrying simultaneously, which can cause CPU jitter. The solution also minimizes wake-up latency in abnormal scenarios.

• System resource optimization

  • Meta tenant and user tenant memory are now shared. This addresses the issue of insufficient meta memory for large tenants and low meta tenant memory utilization for small tenants.
  • OceanBase Database's commonly used thread pools now support dynamic thread creation. The number of threads has been changed from a static setting to adaptively increase or decrease based on the request queue, and some background thread pools have been modified to use this thread pool.
  • Thread-local memory has been streamlined. By removing unnecessary thread-local variables, reducing the memory footprint of local variables, and minimizing the lifecycle of local variables, thread-local memory usage is reduced.

Security hardening

• Fully compatible with the Rocky Linux 9 operating system.

Usability improvements

• Enhanced diagnostic capabilities of ASH and WR

  Starting with OceanBase Database V4.2.1, the Workload Repository (WR) framework was supported, enabling the persistent collection of internal view statistics and enhancing the Active Session History (ASH) functionality. By analyzing ASH data, we can easily identify issues such as long wait events, slow SQL statements, and resource utilization of active sessions, facilitating performance optimization. Building on this, V4.3.5 has further strengthened the diagnostic capabilities of ASH and WR by adding more statistical items, metrics, and diagnostic information. It also supports new diagnostic frameworks, such as SQLSTAT and histograms by SQL type. These features significantly enhance the usability of OceanBase Database.

• Optimization of parallel migration of log stream replicas

  Before V4.3.5, for load balancing tasks like adding or removing replicas that required modifying paxos_replica_num, the RS layer had to wait for a member change to complete on a log stream and obtain the accurate paxos_replica_num before generating the next task. This restriction meant that multiple replica tasks (such as migration tasks) for a single log stream could only be executed serially. This limitation led to lower efficiency in some load balancing tasks and maintenance operations. For example, when a user wanted to migrate two units in two zones simultaneously, the two replicas of the same log stream located on these units had to be migrated sequentially, not in parallel. Starting from V4.3.5, replicas of a log stream in different zones can be migrated in parallel. This change accelerates scaling operations, improves load balancing efficiency, and reduces the likelihood of standby replica rebuilds. The needs for parallel migration differ between primary and standby tenants: primary tenants primarily require it for accelerating scaling in POC scenarios and might not enable parallel migration during regular use due to the potential impact on I/O performance from large data transfers during simultaneous migrations. Standby databases have a strong demand for parallel migration, as it effectively reduces the probability of standby replica rebuilds. The parallel migration optimization introduces a tenant-level parameter, replica_parallel_migration_mode, to control the parallel migration mode for log stream replicas.

• Monitoring downstream log synchronization links (CDC/standby tenant) for the primary tenant

  In the design of direct network connections between primary and standby tenants, the primary tenant is not aware of the standby tenant. Therefore, there is no concept of connection or session between them, making it difficult for the primary database to directly perceive the status of the standby tenant. Since downstream log synchronization links consume primary tenant resources when requesting logs, there is currently no straightforward way to observe how many downstream synchronization link requests the primary tenant is handling. This also makes it challenging to limit log requests from downstream on the primary database. This feature enhances the observability of log synchronization links by implementing a monitoring view for the log transmission links.

OBKV/Multi-model

• Performance improvement of RoaringBitmap expressions - Two-phase optimization

  The business wants to use rb_build_agg and rb_cardinality instead of count distinct for UV calculations. However, currently, rb_build_agg does not support two-phase aggregation, which limits parallelism in complex queries and results in suboptimal performance. To address this, V4.3.5 supports two-phase aggregation for the rb_build_agg expression to enhance its performance.

• Support for complex DML operations on JSON multi-valued indexes

  In V4.3.5, JSON multi-valued indexes are enhanced to support complex DML statements such as UPDATE and DELETE. This includes support for multi-valued predicates in these statements, as well as complex DML operations that involve joining tables with multi-valued indexes.

• Support for more ARRAY expressions

  OceanBase Database supported the ARRAY data type in V4.3.3. To better meet business needs for using the ARRAY data type, V4.3.5 introduces support for additional ARRAY expressions, expanding the range of business scenarios that can be addressed. The new version includes new expressions such as array_append, array_distinct, arrayMap, array_remove, cardinality, element_at, array_contains_all, array_overlaps, array_to_string, array_to_string, array_agg, unnest, and rb_build.

• Enhancements to vector index functionality

  In V4.3.3, OceanBase Database introduced basic support for vector indexes. To better meet user needs, V4.3.5 enhances the stability of existing vector index features while increasing the maximum supported vector dimension from 2,000 to 4,096. Additionally, the cosine distance algorithm is now supported, allowing users to specify it when creating or modifying indexes, and use the cosine_distance expression as a filter condition in SELECT queries.

• Enhanced OBKV-HBase compatibility

  In V4.3.4, OceanBase Database achieved compatibility with most features of HBase 1.x. V4.3.5 further improves compatibility with HBase 1.x/2.x, including support for multiple namespaces, ColumnRangeFilter, ColumnValueFilter, CheckAndMutateBuilder, mixed Batch, and CheckAndMutateBuilder functionalities.

• Memory usage optimization for JSON, XML, and GIS data

  V4.3.5 optimizes memory usage during JSON expression execution. When querying JSON data using operators like JSON_KEYS, JSON_LENGTH, and JSON_SEARCH, memory usage is reduced by half. For output operations on JSON types like JSON_PRETTY and JSON_UNQUOTE, memory usage is reduced to about one-third. Memory usage is also optimized for inserting and updating XML data, with operations like GetClobVal and XmlCast reducing memory usage by about 2-3 times. Insertion and updating of XML data (UpdateXml and InsertChildXml) is optimized without unnecessary parsing.

  Additionally, V4.3.5 includes memory optimizations for GIS data input, output, and certain query scenarios. For GIS input operations (such as st_geomfromtext, st_geomfromwkb, st_geogfromtext, and geometrycollection), memory usage is reduced by 3 times. For GIS output operations (such as st_astext and st_aswkb), memory usage is reduced by half. Query scenarios involving spatial relationship expressions (such as st_crosses, st_overlaps, and _st_touches) and GIS attribute retrieval expressions (such a st_geometrytype and st_iscollection) have also seen significant reductions in memory amplification.

• Response time histogram for OBKV

  Building on the response time histogram feature introduced in V4.3.4, the new version adds OBKV-related statistics to the [G]V$OB_QUERY_RESPONSE_TIME_HISTOGRAM system view. These statistics include categories such as TABLEAPI SELECT, TABLEAPI INSERT, TABLEAPI DELETE, TABLEAPI UPDATE, TABLEAPI REPLACE, TABLEAPI QUERY AND MUTATE, TABLEAPI OTHER, HBASE SCAN, HBASE PUT, HBASE DELETE, HBASE APPEND, HBASE INCREMENT, HBASE CHECK AND PUT, HBASE CHECK AND DELETE, and HBASE HYBRID BATCH.

Compatibility changes

Product behavior changes

The following changes are introduced:

View changes

The following changes are introduced:

Parameter changes

The following changes are introduced:

Syntax changes

Recommended versions of components and tools

The following table lists the recommended versions of components and tools for OceanBase Database V4.3.5.

Upgrade notes

• Supports smooth upgrades to V4.3.5 from V4.3.0 Beta and later minor versions, V4.3.1 Beta and later minor versions, V4.3.2 Beta Hotfix1 and later minor versions, V4.3.3 GA and later minor versions, and V4.3.4 GA and later minor versions.
• Currently does not support upgrades from the V4.2.x series or earlier versions to V4.3.5. Future updates will include upgrade paths from V4.2.x to V4.4.x as the versions evolve.
• Since V4.3.2, the persistent format for multi-source data has been restructured. During upgrades, conversion between old and new multi-source data formats is necessary, requiring relatively ample upgrade time when there are many partitions.