OceanBase provides stable support for massive clusters of Kwai

300+MySQL shards smoothly migrated
3/4Reduction in data synchronization latency
75%Reduction in storage cost

Kwai is a short video mobile application. Formerly known as GIF Kuaishou, the application was first released in 2011, transformed into a short video community in 2012, and rebranded as Kwai in 2014. Kwai Technology has been listed on the main board of the Hong Kong Stock Exchange since 2021. As of the end of 2021, Kwai boasted an average of 308 million daily active users and 544 million monthly active users, making it a tier-1 short video platform.

Background

Kwai is one of the top short video and live streaming apps player, aiming to capture content from all facets of everyday life. With the power of technology, Kwai hopes to enhance the unique sense of happiness for everyone. Users can record moments in their lives by taking photos and creating short videos. Additionally, they can interact with their followers in real-time through live streaming. Founded in 2011, Kwai has been listed since 2021 and has attracted hundreds of millions of daily active users. This, on one hand, drives the rapid growth of the company's e-commerce businesses, and on the other hand, places an unprecedented load on the underlying system.

Challenges

1. Storage bottleneck
- When Kwai's order management business total data volume of orders hit 150 TB, the storage bottleneck and performance issues of its standalone centralized MySQL database worsened and impact customer experence.
2. Increasing number of MySQL database shards
- To mitigate the business impact, Kwai attempted database and table sharding to deal with storage and performance issues. However, the number of database shards increased along with business growth. Before migrating to OceanBase Database, Kwai had created more than 300 MySQL database shards, increasing database management complexities.
3. Exhausting application adaptation
- The increasing number of database shards not only made O&M more costly and complex but also led to endless application adaptations. Kwai realized that its workarounds only alleviated the business issues and would never provide a permanent solution. What it needed was a solution that could reduce the O&M complexity and meet performance requirements.

Solution

1. High compatibility with MySQL
- The O&M team can use OceanBase Database in the same way as they use a MySQL database. Furthermore, OceanBase Database is compatible with MySQL protocols and syntax, allowing legacy data to be smoothly migrated and significantly reducing the costs of business migration and transformation.
2. Three IDCs within the same city
- OceanBase Database can be deployed across three IDCs within the same city. This deployment mode reduces the recovery point objective (RPO) to zero and the recovery time objective (RTO) to less than 30 seconds in case of disasters. A read-only zone can be added in another city to provide local read services to improve query efficiency. Features such as zone-disaster recovery and local read further enhance the business stability and performance.
3. Flexible resource scaling
- The capacity of OceanBase Database can be linearly scaled in response to the actual business development to support the storage and computing of massive amounts of data. This allows Kwai to get ready for future business growth.
4. Ultra-high data compression ratio
- Compared with a conventional centralized MySQL database, OceanBase Database compresses data in the storage at an ultra-high data compression ratio, which significantly reduces the hardware costs of enterprises.

Scenarios

Kwai has deployed eight OceanBase clusters to support its short video platform, managing over 800 TB of data across more than 200 servers. The data volume of the largest cluster is over 400 TB and growing. Kwai upgraded all clusters from OceanBase Database V3.1 to V4.2 to provide online services, including clusters that host the core transaction verification system and payment system, as well as clusters that handle high-concurrency data writes in place of the previous master MySQL database. After the upgrade to OceanBase Database V4.1, the clusters demonstrated significant improvement in business benefits and stability.
Transaction verification
As a short video platform, Kwai provides e-commerce services to meet the needs of prospective shoppers when they swipe through videos. Typically, e-commerce services attract stable traffic, generating 80,000 to 90,000 queries per second (QPS) daily. However, during a live streaming event with a massive audience, the traffic increases dramatically, and the QPS value often soars by 10 or even 100 times, reaching over a million. The compressed data volume can exceed 100 TB.
The distributed architecture of OceanBase Database is horizontally scalable. Faced with an increasing amount of data, Kwai has the option to scale out the storage and computing resources of the clusters to deal with large table queries and storage issues. OceanBase Database also provides native distributed capabilities and demostrated better performance in processing distributed transactions. After OceanBase Database is put into operation, each data record written to shards of the MySQL cluster from upstream business modules is synchronized to OceanBase Database in real time by using binlogs. A data query in the upstream MySQL cluster also triggers the same query in OceanBase Database. The query results are compared to ensure the consistency of the order status in the entire accounting system.
Payment business
Payment data must be processed in real time in e-commerce business. On one hand, merchants and customer service staff check the sales in a live streaming event based on the payment data. On the other hand, the payment gateway uses the payment data for aggregate queries. With the online scalability of OceanBase Database, the storage of large amounts of data is no longer a problem. Additionally, the hybrid transaction/analytical processing (HTAP) capability of OceanBase Database allows the system to handle real-time query and analysis while ensuring data writes. In this business scenario, OceanBase Database has demonstrated performance equivalent to analyzing complex SQL statements as Elasticsearch. Furthermore, OceanBase Database supports online indexing. This feature allows Kwai to perform DDL operations at any time. By replacing the previous MySQL + Elasticsearch solution with OceanBase Database, Kwai not only saves the cost of Elasticsearch services and hardware but also significantly reduces the costs of MySQL hardware. Compared with the previous solution, OceanBase Database meets query requirements with 50% fewer server resources.

Benefits

1. More efficient and convenient O&M
- More than 300 MySQL database shards/instances are aggregated into one OceanBase cluster, making management easier with greatly reduced O&M costs.
2. Significant storage cost reduction
- OceanBase Database stores hundreds of terabytes of data with guaranteed performance and stability with Paxos replication protocol. Compared with the previous solution, the storage cost of the OceanBase solution is slashed by 75%.
3. Improved data synchronization performance
- The latency from data writing in the upstream business modules to the response of the downstream OceanBase cluster is reduced, making data synchronization faster. The synchronization latency time is reduced by 75%.