Background
As a platform serving tens of millions of monthly active users, E-surfing Pay offers a variety of daily services but requires the storage and processing of relevant data. Considering its original database issues, such as insufficient storage space, high analysis latency, difficulties in cost control, and increased O&M complexity, the company needs to replace the original architecture with a new one to ensure the long-term business continuity. After an in-depth investigation of the distributed database solutions available in the market, E-surfing Pay decided to adopt OceanBase.
Challenges
1. Insufficient system capabilities
The previous architecture of the payment system could not ensure a high availability of 99.999% if the number of transactions per second (TPS) and active users increased by 20 times.
2. Tenant interference
E-surfing Pay deployed numerous databases in the previous database cluster, with resources shared among them. When any database experienced traffic surges during peak hours, it affected the services of other databases, significantly impacting the user experience.
3. High hardware costs
To ensure business stability during the execution of O&M tasks, business modules were physically isolated and ran in different database environments. Consequently, many databases were deployed. Although isolating business modules achieved business stability, the costs inccured were high.
Solution
1. Two editions of OceanBase Database
OceanBase Database is available in Enterprise Edition and Community Edition and is compatible with MySQL and Oracle databases. It helps E-surfing Pay build an integrated database solution with an all-in-one technology stack. The solution has replaced foreign-branded centralized databases, such as MySQL and TiDB.
2. Disaster recovery of OCP in multiple IDCs
A MetaDB cluster of OceanBase Cloud Platform (OCP) contains three replicas. In the event of a disaster causing one replica to fail, the remaining two replicas, as the majority, can continue providing services. Each of the three replicas resides in an Internet data center (IDC), also known as a zone of OceanBase Database. Based on primary zone settings of the MetaDB cluster, if IDC 1 (Zone 1) is the primary zone, and fails in the event of a disaster, the service is automatically switched to IDC 2 (Zone 2) without manual intervention. In other words, IDC 2 and IDC 3 work normally when IDC 1 fails. The disaster recovery switchover is automatically performed in the business OceanBase cluster and OCP.
3. Guaranteed high stability
The data of a business table is stored in the form of shards. In the distributed architecture of OceanBase Database, the data of a single table can be evenly distributed across different nodes. E-surfing Pay no longer needs a complex database and table sharding solution. Additionally, OceanBase Database stores three replicas for a set of data by default and uses the Paxos protocol for leader election. This ensures that upper-layer business applications are not affected, and the business data is not lost when a minority of nodes or replicas fails or becomes abnormal.
4. Multitenancy and resource isolation
E-surfing Pay needs to deploy multiple databases for some business modules, such as the online mall. To avoid resource contention among these databases, which may affect other business modules, each database is contained in an OceanBase Database tenant. Resources for each tenant are isolated, allowing E-surfing Pay to control resource allocation for each database. Furthermore, the amount of tenant resources can be adjusted as needed, providing E-surfing Pay with the ability to adjust resources for each database. This ensures the business stability and flexibility.
Phase 1: Deploy the initial architecture, which contains a cluster of three replicas in a single IDC in Nanjing.
Phase 2: Deploy two additional IDCs in Ma'anshan and Wuhu and upgrade the database architecture of three replicas in a single IDC to the five replicas across three IDCs. The upgraded architecture supports IDC-level disaster recovery. In this phase, two read/write replicas were added to the Wuhu IDC and one read-only replica was added to the Ma'anshan IDC. The number of replicas in the Nanjing IDC was also scaled in to two. The upgraded cluster contains five replicas across three IDCs in 2-2-1 mode. The Nanjing and Wuhu IDCs handle the majority of the business traffic, which can be switched between the two IDCs in the event of a disaster.
Benefits
1. Significant reduction in storage cost
The multitenancy feature of OceanBase Database enables resource isolation, allowing E-surfing Pay to migrate business modules from the billing center and message center from 14 servers to only 6 servers in OceanBase Database, which reduces hardware costs by 57%. Additionally, OceanBase Database uses advanced data compression algorithms, which reduce the required disk space by 85% and increase resource utilization by more than 8 times, achieving higher efficiency at lower costs.
2. Higher business processing capacity
The computing power and storage capacity can be increased by adding nodes, and the integrated architecture consumes less resource to process remote procedure call (RPC) requests between components and within the same component of the distributed cluster. Business processing capacity is 30 times higher, and performance is improved by 5 times. The efficiency of single table analysis is improved by 10%-20%. OceanBase Database also supports online linear scale-out to accomodate business growth.
3. High availability and high-level disaster recovery
The Paxos-based architecture consists of three IDCs in two regions (Nanjing and Shanghai). The time required for disaster recovery is reduced from hours to seconds, meeting the Level 6 disaster recovery requirements of the financial industry, with a recovery point objective (RPO) of 0 and a recovery time objective (RTO) of less than 30 seconds.
4. Business upgrade with zero modifications
E-Sufing Pay uses DSG tools for data migration and OceanBase Migration Service (OMS) for incremental data verification. This combination ensures smooth data migration and flexible, faster business cutover with zero application modifications. Although the data volume was 4.75 TB in the smallest database and 43 TB in the largest database of the previous system, the incremental verification was completed within 10 minutes during the business cutover.
