Background

As a leading player in the trendy toy market, POP MART has captured the hearts of tens of millions of young people with its viral one of a kind series, including Molly, Dimoo, and Skullpanda. Its online and offline stores have accumulated more than 26 million members. In 2022, POP MART's revenue reached CNY 4.62 billion, with 41.8% generated from online stores. An essential contributor to this success was the blind box picking mini-program.

The mini-program, launched on September 2, 2018, allows fans to try their luck at online blind box picking anytime, anywhere. This virtual blind box vending machine offers consumers the opportunity to experience similar excitement and fun online, contributing to the loyalty of a substantial fan base through a variety of innovative elements.

Challenges

1. Difficulties in dealing with traffic peaks

When POP MART launch a new product, it will lead to a sudden surge increase in business traffic due to popularity of the launch. On one hand, it is very difficult to estimate the required system capacity. To guarantee service availability during the weekly product release event, the company often scales up its system capacity based on the maximum possible traffic, resulting in significant resource waste. On the other hand, the O&M team faces high levels of stress on cost management and managing the additional resources. In order to minimize the impact on the business systems, the team must scale up or down their database system during the early hours of each day.

2. Slow cluster scaling

In response to fluctuating business traffic, POP MART needs to flexibly adjust the capacity of its business database. The system is expected to run stably with fewer resources during off-peak hours to minimize resource waste and be instantly scaled out during peak hours to ensure system stability during events like a new product release. However, conventional standalone databases, such as MySQL databases, show unsatisfactory scalability. Increasing the database capacity requires users to replace servers with higher-specification ones, leading to significant business disturbances. 

3. Complex management of a large number of database instances

POP MART has deployed many database instances to support its various business modules. Resources are scattered and inefficiently utilized, posing challenges to database cost control and O&M efficiency management. 

Solutions

1. Multi-level auto scaling

OceanBase Cloud supports triple-level scalability. Specifically, users can scale their databases by adjusting tenant and server specifications, as well as the number of servers. Leveraging this feature, POP MART's O&M team has developed flexible scaling strategies, allowing them to handle traffic fluctuations easily and cost-effectively.

Level 1 scaling: adjustment of tenant specifications

Adjustments to the specifications of an OceanBase Cloud tenant take effect within seconds and have no impact on applications. In OceanBase Cloud, a tenant is a resource group. The adjustment of tenant specifications is completed within the OceanBase Database kernel and does not involve changes to physical resources.

Level 2 scaling: adjustment of server specifications

If the Level 1 scaling does not meet the business needs, users can increase server specifications, commonly known as vertical scaling. To scale up a MySQL database, users must perform a primary/standby switchover, which causes transient application disconnections and poses high risks. OceanBase Cloud synchronizes data among nodes using the Paxos protocol, which features automatic leader election from three replicas of the same set of data and ensures that the logs are committed. Therefore, the entire process of scale-up is transparent to applications.

Level 3 scaling: adjustment of the number of servers

Built on a native distributed architecture, OceanBase Cloud supports distributed transactions and can be horizontally scaled out without impacting applications—a capability not achievable with a MySQL database in the primary/standby architecture. Once users add servers to an OceanBase cluster, business traffic is automatically directed to the new servers. This process is transparent to applications. Users can use this multiserver cluster the same way as a single-server MySQL database.

2. A large multitenant cluster

OceanBase Cloud supports resource isolation and virtualization of multiple tenants within the same cluster. This means that multiple legacy database instances can be contained in a single OceanBase cluster, with each tenant corresponding to one instance. Data and resources are isolated for each tenant. The new solution accommodates dozens of database instances deployed by POP MART for different business lines in three OceanBase clusters, with each instance corresponding to one tenant in a cluster. This simplifies system O&M and achieves a balance between overall costs and resource allocation.

Benefits

1. Tenant-level scalability with less time required for scaling

POP MART can handle most minor business traffic peaks by adjusting the specifications of core business tenants anytime and anywhere without paying additional costs or interrupting applications. The adjustments take effect in seconds, cutting the time required for scaling by 90%. 

2. Handling steep traffic increase during flash sales with cluster-level scalability

In the event of drastic traffic surges, POP MART can upgrade the specifications of each node (vertical scaling) and add more nodes to a cluster (horizontal scaling) as needed. The new system easily handles the traffic surges, which may increase 100 times during a flash sales event, by distributing the traffic load across more nodes with higher specifications.   This ensures a system availability of 99.999% during the release of new products. The scaling is transparent to upper-layer applications. 

3. Significant improvement in O&M efficiency

As dozens of database instances are aggregated into three OceanBase clusters, the efficiency of POP MART's O&M team is significantly improved. The team can adjust the number of CPU cores and memory size of a tenant to smoothly improve the maximum transactions that can be processed per second (TPS) by the tenant at any given time. For example, they can adjust the resources during normal business hours.