How to Eliminate Manual Database Sharding and Scale Writes Without Application Complexity

As data volumes and workloads grow, engineers almost inevitably face the same difficult question: what happens when a single-node database can no longer keep up?

For many years, the industry's standard answer has been horizontal scaling through database sharding. A large table is split into multiple pieces and distributed across independent database instances. A middleware layer is then introduced to process SQL requests from the application, route each request to the target shard based on predefined sharding rules, aggregate results from multiple shards when necessary, and return the final result to the application.

This approach works. But as systems continue to grow, the development and operational effort involved in sharding can gradually become a heavy burden.

The Pros and Cons of Database Sharding

The idea behind sharding is straightforward: it allows a system to grow beyond the storage and throughput limits of a single-node database. However, because this approach moves scalability logic outside the database rather than building it into the database itself, its costs grow over time. It pushes distributed-system complexity onto the application layer, middleware, and DBAs.

On the development side, SQL design becomes tightly coupled with how data is distributed. If a query does not include the sharding key, the middleware has no choice but to scan all database shards, which leads to poor performance. When updates span multiple shards, data consistency can also become harder to guarantee in complex scenarios.

The operational side is no easier. Whenever data volume grows rapidly and the system needs to scale out, the process becomes a complex engineering project: redesigning the sharding strategy, building custom dual-write migration tools, and carefully validating migrated data. Even more challenging, as the business keeps growing, the number of shards continues to expand. Once the number of shards reaches the hundreds, even small architectural changes can bring extremely high management costs.

Can Cloud-Native Databases Put an End to Sharding?

When looking for alternatives, many companies turn to cloud-native databases such as Amazon Aurora.

Aurora adopts an architecture that separates compute from storage, relying on distributed shared storage underneath. This removes the storage capacity ceiling of a single machine. Read requests can also be scaled horizontally by adding read replicas.

However, many shared-storage cloud-native relational databases still face a fundamental architectural constraint: writes are typically handled by a single primary node, or Writer. As a result, the Writer node can easily come under heavy load. For example, when executing a mixed SQL statement such as INSERT INTO ... SELECT ..., which includes both complex reads and writes, the entire operation must be executed on the Writer node to ensure data consistency. This can make the primary node a performance bottleneck.

When workloads grow rapidly and the write capacity of a single node reaches its limit, users usually have to upgrade to a larger instance type, often with a restart or a brief service interruption. This problem is even more pronounced in write-intensive scenarios such as social interactions, gaming, and live-streaming e-commerce. If the largest single Writer instance still cannot handle write spikes, users have little choice but to split data across independent clusters and reintroduce database sharding.

In short, cloud-native architectures based on shared storage solve the problem of storage capacity, but they do not fundamentally solve the problem of write scalability. In this sense, they are closer to an enhanced primary-standby architecture than to a fully distributed write-scalable database.

OceanBase: Hiding Distributed Complexity in the Database Kernel

OceanBase takes a different path: a shared-nothing, native distributed architecture. Each node in the cluster has its own CPU, memory, and local storage resources, rather than relying on a central shared-storage layer. This architecture eliminates the single-writer bottleneck found in shared-storage systems, so manual sharding is no longer required.

For developers and DBAs, OceanBase presents itself as a single logical database. So how does it distribute data internally while preserving the experience of a single-node database?

Storing One Large Table Across Multiple Machines

In OceanBase, even if a large table contains tens of billions of rows, the system can split the data into multiple partitions based on partitioning rules and place those partitions across multiple commodity servers. At the logical layer, however, it remains a single complete table. Application developers generally no longer need to manually split tables or maintain complex sharding logic.

After Data Is Distributed, How Does a Query Find the Right Node?

Once data is distributed, where should a request be sent? No application-layer code needs to be modified. OceanBase Database Proxy (ODP, also known as OBProxy) handles this automatically. It parses incoming SQL predicates, locates the target partition and node based on the partitioning key, and forwards the request accordingly.

From the application's perspective, connecting to OceanBase is no different from connecting to a single-node MySQL database. There is no need to rewrite routing logic.

What If the SQL Query Does Not Include the Partitioning Key?

One of the biggest concerns in sharded architectures is queries that do not include the sharding key. These queries usually cause the middleware to scan all shards, significantly degrading performance.

This is a common challenge for any distributed partitioning architecture. OceanBase addresses it through built-in global indexes. A global index does not have to follow the primary table's partitioning scheme, so indexes can be created on columns other than the partitioning key. Even if a SQL query does not include the partitioning key, the system can use a global index to narrow the query down to a small number of target nodes, avoiding inefficient full-shard scans.

Can ACID Still Be Guaranteed for Cross-Node Reads and Writes?

In traditional sharded architectures, if a business operation such as a transfer or an order update involves data distributed across different nodes, the application layer often needs to introduce distributed transaction middleware as a safeguard. This not only increases architectural complexity but also makes data inconsistency more likely in exceptional scenarios.

OceanBase builds consistency guarantees for cross-node transactions directly into the database kernel. Regardless of how many nodes the data spans, each transaction either commits completely or rolls back completely. Full atomicity, consistency, isolation, and durability (ACID) semantics are handled by the database itself, without requiring additional processing at the application layer. OceanBase also uses the Multi-Paxos protocol to ensure strong consistency across replicas.

OceanBase's cross-node transaction guarantees can be understood in two layers.

• The first layer is transaction-level coordination, which guarantees atomicity. When a transfer or order operation involves multiple partitions, the system first asks the relevant nodes to enter a "prepare to commit" state. The transaction is committed only after all participants confirm success. If any participant fails, the entire transaction is rolled back. By using an optimized two-phase commit protocol, OceanBase prevents partial commits, where one part of the operation succeeds while another fails.
• The second layer is replica-level consistency, based on a majority mechanism. Each piece of data usually has multiple replicas stored across different nodes. OceanBase uses the Multi-Paxos majority consensus mechanism: a data change does not need confirmation from every replica; it can be considered committed once a majority of replicas have persisted it. Mathematically, any two majorities must overlap in at least one node—just as two groups that each include more than half of all members must share at least one person. This avoids split-brain behavior, where two isolated groups of nodes both believe they can commit changes independently. As a result, even when a minority of nodes fail or become disconnected, the system can continue serving requests as long as a majority remains available, while preserving strong consistency and durability.

One real-world example comes from transaction reconciliation at a large live-streaming platform that uses OceanBase. In the past, the platform used a traditional database sharding solution. The middleware had limited support for cross-database consistency and transaction atomicity. As a result, the system often encountered data inconsistencies in complex or abnormal situations—for example, refunds that were not issued or inaccurate debit amounts—which in turn caused financial losses. These are precisely the kinds of consistency issues that OceanBase's native distributed transactions are designed to address at the kernel level.

Will Data Distribution Become Unbalanced as Data Grows or Nodes Are Added and Removed?

Automatic rebalancing is one of the features that significantly reduces the operational burden on database administrators (DBAs). When the cluster is nearing capacity and new nodes are added, OceanBase performs online rebalancing: it smoothly migrates data partitions to the new nodes and automatically rebalances read and write traffic across the cluster.

The entire process requires no manual intervention, no routing-rule changes, and no downtime for data migration. As a result, horizontal scaling becomes transparent to applications.

Together, these five mechanisms—transparent partitioning, automatic routing, global indexes, native distributed transactions, and automatic rebalancing—form the foundation that allows OceanBase to eliminate the need for manual sharding. These capabilities are built into the database kernel and remain transparent to applications.

Trade-offs Worth Knowing

Eliminating manual sharding doesn't mean eliminating distributed-systems physics. Three trade-offs are worth understanding:

• Global indexes aren't free. Because a global index can live on different nodes than the primary table, writes that touch it become distributed transactions, adding write overhead. For write-heavy tables, local indexes on the partitioning key remain the cheaper option.
• Cross-node transactions cost more than single-node ones. OceanBase's optimized two-phase commit narrows the gap, but a transaction spanning three nodes will never be as cheap as one hitting a single machine. Good partitioning keeps most transactions single-node.
• Partition strategy still matters. You no longer write sharding logic, but choosing a sensible partitioning key still determines how often queries stay local.

Architectural Comparison of Three Approaches

OceanBase in Production: Scaling a Large-Scale Social and Live-Streaming Platform

What do these architectural differences mean in real-world business scenarios? A good example is Inkeverse, an OceanBase customer that runs a large-scale social and live-streaming platform.

Like many internet companies, Inkeverse built most of its systems on cloud infrastructure and relied on a managed relational database service with a primary-standby architecture. As the company grew and expanded into overseas markets, the team began to run into two major limitations.

The first was the rising cost of scaling reads. Social applications generate massive amounts of data and place heavy pressure on the database with highly concurrent queries. To keep the primary database from being overwhelmed, the team had to provision additional database instances as read replicas. These read replicas became a major line item on the company's monthly cloud bill.

The second was delayed service response during scaling operations. Live-streaming traffic can change rapidly, requiring frequent scale-out and scale-in operations. In overseas scenarios where Inkeverse used traditional cloud databases, each scaling operation often introduced more than 10 seconds of response delay—an unacceptable hit to user experience for real-time services such as live streaming.

To overcome these limitations, the team migrated its core systems to OceanBase. This architectural shift directly addressed both problems.

• Read/Write-Capable Nodes and High Compression Cut Database Costs by 40% to 50%

   Thanks to OceanBase's distributed architecture, data replicas are spread across nodes for high availability, and nodes can process both reads and writes for the data they serve. When business volume increases, the team can add nodes to improve overall read and write throughput, without provisioning dedicated read replicas.

   Together with OceanBase's data compression technology, this combination of architectural redundancy reduction and storage optimization reduced overall database costs by 40% to 50%.

• True Horizontal Scaling with Smooth, Application-Transparent Expansion

   When facing sudden traffic spikes, the team can now simply add nodes horizontally. OceanBase uses its distributed mechanisms to smoothly migrate data partitions and rebalance request traffic across the expanded cluster.

   Because scaling no longer requires restarting or upgrading a single primary instance the way it used to, the 10-second-plus scaling delays are largely a thing of the past. The scaling process is smooth enough that the front-end live-streaming application layer doesn't perceive the underlying changes.

Summary

Database sharding "outsources" distributed complexity. In the short term, it gives the system room to grow, but in the long run, it increases operational burden and architectural complexity. Cloud-native architectures greatly ease storage constraints, but many remain limited by the single-writer bottleneck and the high cost of scaling.

OceanBase takes a different path: it handles distributed complexity inside the database kernel while presenting a simple logical database to applications.

A native distributed architecture without manual sharding is the foundation of OceanBase's elastic scaling capabilities. Application developers can focus on business logic, while DBAs no longer need to worry about data migration or read/write bottlenecks. Built on this foundation, OceanBase's tenant-level dynamic resource isolation and smooth node scale-out and scale-in remain transparent to applications and help achieve zero downtime for the business.

Elastic scaling is only useful if it's transparent — and transparency has to be built into the kernel, not bolted on above it. If you're weighing a move away from sharded MySQL, the OceanBase Quickstart is the fastest way to see the single-logical-database experience firsthand.

Further Reading

• Distributed transactions
• OceanBase Database architecture
• OceanBase Database Documentation