Trends and Challenges of Migrating Databases to the Cloud

Photo by aziz ayad on Unsplash

This article is a summary based on a conversation between Bai Yushan, Oracle ACE/OCM, and Han Fusheng, OceanBase's founding engineer and senior R&D director.

Since the release of Amazon Web Services (AWS) Managed Services about ten years ago, cloud databases have experienced the fastest growth in the database market, setting the trend of migrating local databases to the cloud. According to The Register, a Gartner report expects that 75% of databases around the globe will be deployed or migrated to the cloud by 2022. This trend of moving to the cloud will improve database analytical performance and push Database-as-a-Service (DaaS) products. Factors that affect the costs of cloud migration, such as compatibility and ease of use and database security must be considered. Solving these issues is also a significant challenge.

What are the challenges of migrating databases to the cloud?

This is indeed an extensive topic. A large enterprise that runs hundreds or even thousands of systems must overcome different challenges from those of a small or medium-sized enterprise that runs much fewer systems. Generally, we can look at this from two perspectives.

The first is about the technical architecture of the enterprise. To migrate its on-premises business systems, middleware, or databases to a cloud platform, an enterprise has to upgrade the entire technical architecture, which is an innovative transformation that involves the selection of a new overall IT infrastructure of the enterprise.

The second is about the database. The upgrade to the cloud poses a challenge to the expertise of technical teams. One of the biggest benefits of a cloud-based database solution is that it solves database stability and reliability issues. These issues usually cause big troubles to users of on-premises databases or require a lot of energy to deal with. Cloud databases make it easier for enterprises to meet their needs on database service. Technical talents, such as developers and database administrators (DBAs) dedicated to maintaining system stability and reliability, can be freed up to jobs that are more beneficial and help application systems to use the database with improved overall efficiency better.

While enterprises save resources in that regard, their technical teams are facing new challenges. After all, cloud databases have their unique characteristics. For example, components are associated differently than those of on-premises databases. Therefore, it is necessary to learn about how to use services on the cloud better and give full play to the benefits of cloud databases. Database users must be familiar with both the database and the cloud platform, which requires different knowledge.

When it comes to migrating a database to the cloud, a major technical challenge is the overall planning of data migration. OceanBase provides a lot of useful tools for data migration, such as OceanBase Migration Service (OMS) which helps enterprises easily migrate all existing and incremental data from their active database systems to OceanBase databases on the cloud. OMS is a great hassle-saver. All you need to do is to make sure that the data migration meets your requirements and verify the data consistency during the migration.

Generally, migrating to the cloud involves massive amounts of changes. It requires a comprehensive plan that covers many aspects, including the technical architecture, expertise of technical teams, and how to ensure continuous migration. Many enterprises start from peripheral systems and gradually start migrating the critical ones. Nowadays, more enterprises have joined the trend, including some tech giants, who are trying the hybrid cloud, a deployment mode that integrates conventional on-premises IT infrastructure with more convenient cloud hardware to cope with their new business needs.

What are the technical requirements for cloud databases?

First, high availability. Mainstream cloud platforms and database products today already have features to support high availability and multi-replica deployment modes for disaster recoveries, such as three data centers across two regions and five data centers across three regions. Commercial IT products have demonstrated higher performance than most self-managed systems. For example, many database vendors claim a recovery point objective (RPO) of about zero, and the recovery time objective (RTO) of OceanBase can be within 8 seconds.

Second, automatic scaling. Hard to achieve in a conventional architecture, automatic scaling is a native feature of a cloud system. The question is, can we make it better? For example, scaling only the CPU or only the memory, and using CPUs of a higher frequency, say 3 GHz, to meet various resource flexibility requirements.

Third, storage resources. I know that vendors provide different resource plans, such as high-capacity plans and high-performance plans. It’s better to offer more diverse plans, like cell phone plans, for customers to deal with more business scenarios.

Fourth, performance improvement. This is a lasting topic and database vendors have their solutions in this regard. For example, slow SQL statements are often automatically captured or held in a queue for a while. Nowadays, many up-to-date database products provide optimization suggestions, which is good. It would be best if an optimization feature can support the automatic implementation of rules preset by users based on the optimization suggestions, such as the rules of automatic indexing, which is a simple business scenario. If an enterprise purchases a cloud database capable of providing optimization suggestions, it’s like the database comes with a senior DBA for performance tuning.

Fifth, intelligent O&M. Conventional IT O&M relies on human operators, who work like first aiders in the case of emergency. Cloud vendors provide products that come with built-in monitoring systems. Most of those monitoring systems work based on threshold values. For example, you can set the disk usage threshold to 90% and the CPU utilization threshold to 8%, and your monitoring system will send you an alert when the measured value goes above the threshold. Now, we often mention AI-powered intelligent O&M. This is an excellent early-warning mechanism based on experience or historical data, which helps you notice the signs of problems.

Last but not least, an exception can be fixed by software operations. While this capability reduces the O&M costs, it is difficult to achieve, because users may run into a variety of issues and it is hard to specifically preset rules or provide suggestions for every single issue.

Will multi-cloud deployment be the new trend of cloud computing?

As fundamental information technology, cloud computing will support the development of more Platform-as-a-service (PaaS) and Software-as-a-service (SaaS) products. Cloud databases have gained great traction these days. On one hand, a database hosts the core digital assets of an enterprise. On the other hand, many of our customers are interested in cross-cloud architectures, worrying that they might be locked on one single cloud platform.

That makes sense because enterprises have different needs. Some want to extend their business globally and must choose global vendors to deploy underlying services all over the world. Others would like to focus on the local market but may need to provide differentiated services in different regions, use services by a specific vendor, or cooperate with eco-partners.

Cross-cloud or multi-cloud operations can maximize the efficiency of the entire software industry or computer industry. Only in that way can the market introduce sufficient competition, which brings higher efficiency.

What capabilities should DBAs have to stay competitive? Have their roles changed?

Ten years ago or even earlier, DBAs were O&M engineers doing jobs such as database installation and debugging, data backup, performance tuning, and disaster recovery.

Now we have O&M DBAs, development DBAs, Oracle DBAs, MySQL DBAs, PostgreSQL DBAs, and DBAs who know enough about cloud computing. As the development of technology adds more duties to the job description of a DBA, one must be versatile to take the role. DBA, in terms of career classification, is a critical position in an enterprise, because it handles data, a vital asset of an enterprise.

A good DBA is probably the most versatile person in an enterprise, as he or she must be an all-rounder getting the tricks about storage, network, system, and business. Therefore, a DBA can take the seat of an architect or solution engineer, and give suggestions about the IT system of an enterprise against the bigger picture.

The DBA of a conventional database may need to do jobs in person, such as writing commands and scripts, preparing backups, and monitoring the system for disaster recovery. For example, in-depth database tuning and exception recovery were must-have skills for an Oracle DBA. The development of technology, cloud computing, in particular, brings changes to the industry. For example, extensive experience is not necessary for basic O&M of databases on the cloud. Proper training is enough for a green hand to get on track. In addition, modern cloud software is usually capable of providing optimization suggestions, showing intelligence comparable to a senior DBA. New DBAs may have to work hard for a long time to catch up, which puts huge pressure on them. Many tasks, such as database monitoring, disaster recovery, and performance tuning, can now be executed on the cloud with a few clicks. Some DBA skills are indeed replaced by cloudware.

What can DBAs do in response?

First things first, hone your essential skills. Whether it is a cloud platform, conventional IT architecture, network, system, database, or storage, the underlying technology remains unchanged.

Second, as a versatile person, you can do more than being a DBA. Be a T-shaped talent and find your way up to show your strengths at higher positions.

Welcome to continue the discussion by leaving a comment below!