Scale Out Without Downtime: How OceanBase Adds Nodes Online

When businesses grow rapidly, database scaling often becomes one of the biggest challenges for operations teams. For many relational databases, horizontal scaling is never as simple as "adding one more machine." It usually comes with service interruption risk and requires teams to manually redesign sharding strategies, migrate data, and carefully validate the results.

Mainstream cloud-native databases offer natural advantages in scaling reads, but writes are often constrained by the architectural bottleneck of a single primary node. When write traffic surges, teams often have to upgrade to a larger instance type, which still involves a database restart or a primary-standby switchover.

As a native distributed database, OceanBase makes adding nodes much simpler. Users only need to adjust a parameter, and the new nodes can automatically start taking over traffic. The change is largely transparent to applications, and services remain available throughout the process. This turns what was once a high-risk scaling operation into a routine task that can be performed whenever needed.

How OceanBase Horizontal Scaling Works

Adjust One Parameter to Scale Out and Rebalance Data Automatically

In "OceanBase Tenant-Level Scaling: Adjust Resources in Seconds," we compared an OceanBase cluster to a large business park, where each physical machine is like an office building. When a company, or tenant, moves into the park, it leases a package of offices distributed across different buildings. These offices correspond to the resource units allocated to the tenant.

In that article, we explained that the fastest way to handle sudden traffic spikes is to add more desks and chairs inside the existing offices—that is, to increase the resource quota within each machine. But when a single machine is already close to its physical limit, the system needs to scale out by adding nodes.

In OceanBase, horizontal scaling is mainly achieved in two ways:

1. Increase the number of resource units.
   This applies to capacity expansion within the same data center. In the business park analogy, this is like adding new office buildings to the same park and leasing additional offices in those buildings. In technical terms, new nodes are added to the cluster, and the tenant receives more resource units across those nodes. Heavy workloads that were previously concentrated in a small number of offices are redistributed across more offices and processed in parallel, breaking through the compute and storage limits of a single machine.
2. Increase the number of primary zones.
   This applies to service expansion across data centers, allowing nodes in more regions to serve traffic together. It is like a company with offices in cities A, B, and C. Originally, only city A handles customer reception and order processing, while cities B and C mainly handle accounting synchronization, or data backup. When city A reaches its service capacity limit, the branch office in city B can also be promoted into a business center that handles orders. In this way, nodes in multiple regions can directly process read and write requests at the same time. Without adding new physical machines, the system can make better use of previously idle resources.

No matter which parameter is adjusted, adding nodes and redistributing workloads, or data rebalancing, are handled automatically by the database kernel. After new nodes join the cluster, OceanBase automatically and smoothly migrates some data to the new nodes based on data volume, storage usage, and hotspot distribution.

For example, if three nodes originally manage 100 data segments, after two new nodes are added, the system may redistribute roughly 20 segments to each new node. Eventually, all five nodes share the workload together, with no service interruption or manual intervention.

How Does OceanBase Maintain Consistency During Data Migration?

During horizontal scaling, OceanBase does not take the entire database offline or move all data to a new set of nodes at once. Instead, it migrates data gradually at the partition level.

In a multi-replica architecture, read and write traffic for each partition always follows the corresponding leader replica. During scaling, the system first creates new follower replicas for selected partitions on the new nodes and starts synchronizing data. This synchronization is based on OceanBase's internal physical logs, known as clog, and the Multi-Paxos protocol.

Compared with the binlog and relay log processing chain used in MySQL primary-standby replication, this mechanism is more lightweight. It reduces synchronization overhead in high-concurrency scenarios and helps new replicas catch up with the source nodes more quickly.

After the replicas on the new nodes catch up, the system initiates a smooth leader switchover based on the Multi-Paxos protocol. The current leader does not step down until the necessary log commits are complete. Through majority election and lease mechanisms, the system ensures that the replica on the new node can safely take over as the new leader.

This mechanism ensures that the same partition never has two valid leaders at the same time, preventing data inconsistency caused by split-brain scenarios. After the switchover is complete, the frontend proxy component automatically detects the change, refreshes routing information, and forwards subsequent requests to the node where the new leader resides.

The entire handoff is completed at the kernel and routing layers, while business applications continue running steadily.

Note: OceanBase's Multi-Paxos protocol and related mechanisms play a decisive role in balancing data reliability and availability. For details, see the article on how OceanBase uses Multi-Paxos for consensus and disaster recovery.

Why Are Applications Unaware of the Change?

After data migration is complete, how does the application know where to find the data?

The answer is that it does not need to know.

As mentioned above, OceanBase provides a transparent routing proxy layer, OceanBase Database Proxy (ODP, also known as OBProxy). No matter how the underlying nodes change or where data is migrated, the application continues to connect through a single stable database endpoint.

Developers do not need to modify database connection strings or change SQL syntax. They can use a continuously expanding cluster just as they would use a conventional single-node database.

Performance Scales Near-Linearly as Nodes Are Added

The real value of scaling lies in whether performance actually improves as more nodes are added.

In the publicly available TPC-C benchmark, when an OceanBase cluster expanded from 3 nodes to 1,500 nodes, its transaction throughput per minute, or tpmC, achieved near-linear growth. The cluster maintained this near-linear scaling even when 10% of the workload consisted of distributed transactions.

Many distributed architectures maintain consensus and synchronization independently for replicas of each data partition. This means that as the number of partitions increases, the total overhead required to maintain data consistency also rises. OceanBase's design ties this overhead to the number of nodes rather than the number of partitions. As a result, when the cluster scales horizontally, overhead grows smoothly with the number of nodes instead of spiraling out of control as data partitions multiply.

By contrast, mainstream cloud-native architectures are still essentially based on a primary-standby model. The cluster has only one write node, and simply adding read-only nodes cannot break through the physical ceiling of write capacity.

Scaling Duration and Business Impact

A practical question for engineers is: how long does the operation take, and how much will it affect the business?

The time required for scaling is mainly determined by the amount of business data and the available cluster resources at the time, such as network bandwidth and disk I/O. The upper limit of migration speed can be roughly estimated as follows:

Note: Available bandwidth equals the hardware's physical limit minus the resources currently consumed by the business. The default migration quota ratio is 60%.

Put simply, migration can only run as fast as the more constrained resource allows, whether that constraint is disk throughput or network bandwidth. If scaling is performed during peak business hours, business traffic may already be consuming a large amount of these resources. With fewer resources left for migration tasks, scaling naturally takes longer.

During scaling, background data migration tasks share node resources with frontend business traffic. To avoid resource contention, OceanBase provides resource isolation mechanisms. Operations teams can centrally configure the network bandwidth quota, disk IOPS limit, and CPU weight available to migration tasks. During off-peak hours, they can loosen these limits to accelerate migration. During peak hours, they can tighten them to protect normal business traffic.

The actual impact on the business depends on cluster load at the time of scaling.

• When the cluster is lightly loaded, such as when CPU and I/O utilization are below 60%, the business is almost unaffected.
• When the cluster is heavily loaded, resource isolation can proportionally limit the bandwidth and CPU consumed by migration tasks, keeping QPS fluctuations within 15% and restoring stability within a few minutes.

When Should You Use Horizontal Scaling?

OceanBase provides multiple elastic scaling options, and enterprises can choose based on their needs. As the final topic in this series, this article summarizes the recommended approaches for different scenarios:

Note: In the pay-as-you-go mode of OceanBase Cloud, billing based on the new specification starts as soon as the scaling operation is initiated, and compute costs increase as nodes are added. After scale-in, node charges also stop immediately, so you do not need to continue paying for idle capacity.

Summary

Database scaling should not be a disruptive, high-risk engineering project. It should be a routine resource adjustment.

OceanBase achieves this through a combination of kernel-level mechanisms: consensus protocols protect data during migration, the routing layer keeps applications unaware of topology changes, and the database kernel handles load rebalancing automatically. This design allows the database to scale flexibly with business demand, while helping enterprises avoid long-term over-provisioning for uncertain growth.

Further Reading

• Add a node
• Delete a node
• sys_bkgd_net_percentage
• ha_high_thread_score
• OceanBase Database Documentation