Upgrade an OceanBase cluster

This topic describes how to use the upgrade script to upgrade an OceanBase cluster.

Prerequisites

You have installed Python 2 on all OBServer nodes, and you have installed the mysql.connector module compatible with Python 2.

Procedure

1. Analyze the oceanbase_upgrade_dep.yml file.
2. Confirm the upgrade process.
3. Back up the cluster parameters.
4. Upload and install the target RPM package.
5. Run the upgrade_checker.py script.
6. Run the upgrade_pre.py script.
7. Upgrade the cluster.
8. Run the upgrade_post.py script.
9. Restore the cluster parameters.

Step 1: Analyze the oceanbase_upgrade_dep.yml file

After decompressing the RPM package of the target version, you can obtain the oceanbase_upgrade_dep.yml file, which records the topology of the OceanBase cluster version upgrade. The steps are as follows:

1. Obtain the RPM package of the target version of OceanBase.

   Please click Resources in the upper-left corner of the OceanBase console and then click Download Center to download the RPM package of the corresponding version. If the RPM package of the corresponding version is not found, contact Technical Support to obtain the RPM package of the target version.

2. Decompress the OceanBase RPM package.

   You can use the following command to decompress the OceanBase RPM package to the current directory:

   [xxx@xxx $rpm_dir]# sudo rpm2cpio $rpm_name | cpio -div
   

   In this command, $rpm_name indicates the name of the RPM package.

   Note

   After decompressing the RPM package, the home and use directories are generated in the directory where the RPM package is stored. The upgrade-related scripts and the oceanbase_upgrade_dep.yml file are stored in the home/admin/oceanbase/etc directory.

3. Analyze the oceanbase_upgrade_dep.yml file.

   The meanings of the attributes in the oceanbase_upgrade_dep.yml file:

   • version: the version number of the current OceanBase database, referred to as the current version.
   • can_be_upgraded_to: the target version number to which the current OceanBase database can be upgraded, referred to as the target version.
   • deprecated: indicates whether the current version can be used as the target version for upgrade. If the value is true, the current version cannot be used as the target version. For example, the 4.1.0.0-100000982023031415 version cannot be used as the target version.
   • require_from_binary: indicates whether the current version is a barrier version in the upgrade sequence. If the current version is a barrier version, you must upgrade to the current version before upgrading to the target version.
     • value: when the value is true, it is used in conjunction with the when_come_from attribute.
     • when_come_from: a list. If when_come_from is not defined, it indicates that any version must be upgraded to the current barrier version before being upgraded to the target version. If when_come_from is defined, it indicates that the versions in the list must be upgraded to the current barrier version before being upgraded to the target version. For example, when upgrading the OceanBase clusters of the 4.0.0.0 and 4.1.0.0-100000982023031415 versions to the V4.2.0.0 version, you must first upgrade them to the V4.1.0.1(barrier) version.

   Example:

   Assume that the upgrade dependencies of the OceanBase cluster versions in the oceanbase_upgrade_dep.yml file are as follows:

   - version: 4.0.0.0
     can_be_upgraded_to:
         - 4.1.0.0
   
   - version: 4.1.0.0-100000982023031415
     can_be_upgraded_to:
       - 4.1.0.0
     deprecated: True
   
   - version: 4.1.0.0
     can_be_upgraded_to:
         - 4.1.0.1
   
   - version: 4.1.0.1
     can_be_upgraded_to:
         - 4.2.0.0
     require_from_binary:
       value: True
       when_come_from: [4.0.0.0, 4.1.0.0-100000982023031415]
   

   By analyzing the example file, you can obtain the following upgrade sequences:

   • V4.0.0.0 > V4.1.0.0 > V4.1.0.1(barrier) > V4.2.0.0
   • V4.1.0.0-100000982023031415 > V4.1.0.0 > V4.1.0.1(barrier) > V4.2.0.0
   • V4.1.0.0 > V4.1.0.1 > V4.2.0.0
   • V4.1.0.1 > V4.2.0.0

   Notice

   If the upgrade sequence contains a barrier version, you must also obtain the OceanBase RPM packages of all barrier versions in the upgrade path.

Step 2: Confirm the upgrade process

Based on the upgrade sequence in Step 1, you can obtain the following information:

• If you are using an OceanBase cluster of the V4.0.0.0 or V4.1.0.0-100000982023031415 version, you must first upgrade the OceanBase cluster to the V4.1.0.1(barrier) version and then upgrade it from the V4.1.0.1(barrier) version to the V4.2.0.0 version.

• If you are using an OceanBase cluster of the V4.1.0.0 or V4.1.0.1 version, you can directly upgrade the OceanBase cluster to the V4.2.0.0 version.

Example:

This example describes the upgrade process of a three-replica OceanBase cluster.

• If the current version of the OceanBase cluster is V4.0.0.0, the upgrade process is as follows:

  1. You must first upgrade Zone1 from the V4.0.0.0 version to the V4.1.0.1(barrier) version, then upgrade Zone2 from the V4.0.0.0 version to the V4.1.0.1(barrier) version, and then upgrade Zone3 from the V4.0.0.0 version to the V4.1.0.1(barrier) version. At this point, the entire cluster has been upgraded from the V4.0.0.0 version to the V4.1.0.1(barrier) version.
  2. You must then upgrade Zone1, Zone2, and Zone3 from the V4.1.0.1(barrier) version to the V4.2.0.0 version in sequence. After all zones are upgraded to the V4.2.0.0 version, the cluster upgrade is complete.

• If the current version of the OceanBase cluster is V4.1.0.0, the upgrade process is as follows:

  You must first upgrade Zone1 from the V4.1.0.0 version to the V4.2.0.0 version, then upgrade Zone2 from the V4.1.0.0 version to the V4.2.0.0 version, and then upgrade Zone3 from the V4.1.0.0 version to the V4.2.0.0 version. After all zones are upgraded to the V4.2.0.0 version, the cluster upgrade is complete.

Step 3: Back up cluster configuration items

Before you start the upgrade process, you need to back up the values of the following configuration items to restore them after the upgrade:

• server_permanent_offline_time
• enable_rebalance
• enable_rereplication

Run the following SQL statement in the sys tenant (system tenant) to obtain the values of the configuration items to be backed up:

SELECT SVR_IP,ZONE,NAME,VALUE FROM OCEANBASE.GV$OB_PARAMETERS WHERE NAME IN ("server_permanent_offline_time", "enable_rebalance", "enable_rereplication");

Step 4: Upload and install the target RPM package

1. Use the scp command to upload the OceanBase RPM package required for the upgrade to the OBServer node.

   scp $rpm_name admin@$observer_ip:/$rpm_dir
   

   Parameter description:

   • $rpm_name: the name of the RPM package.
   • $observer_ip: the IP address of the OBServer node.
   • $rpm_dir: the directory where the RPM package is stored.

   Here is an example:

   scp oceanbase-4.2.0.0-100010022023081911.el7.x86_64.rpm admin@10.10.10.1:/home/admin/rpm
   

2. Run the following command to install the OceanBase RPM package.

   Notice

   • If there is a barrier version in your upgrade path, install the corresponding barrier version RPM package first. After the cluster is successfully upgraded to the barrier version, install the target version RPM package.
   • If there are multiple barrier versions in the upgrade path, install and upgrade them in sequence until the cluster is successfully upgraded to the last barrier version, then install the target version RPM package.

   rpm -Uvh $rpm_name
   

   Here, $rpm_name indicates the name of the RPM package.

   Note

   The upgrade-related scripts and oceanbase_upgrade_dep.yml file are stored in the /home/admin/oceanbase/etc directory. The /home/admin/oceanbase/etc directory is the default installation directory of OceanBase Database.

3. Repeat steps 1 and 2 until the target version RPM package is installed on all OBServer nodes.

Step 5: Run the upgrade_checker.py script

Specify the login information of the sys tenant to directly connect to the OBServer node on any OBServer node and run the upgrade_checker.py script for pre-upgrade checks. If the script runs successfully, you can proceed with the upgrade. The command is as follows:

python upgrade_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user who has the read, write, and modification permissions on the global system parameters of the sys tenant.
• $password: the user password.

Here is an example:

1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Run the following command to execute the upgrade_checker.py script for pre-upgrade checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 6: Run the upgrade_pre.py script

Specify the login information of the sys tenant to directly connect to the OBServer node on any OBServer node and run the upgrade_pre.py script. If the script runs successfully, you can proceed with the upgrade. The command is as follows:

python upgrade_pre.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user who has the read, write, and modification permissions on the global system parameters of the sys tenant.
• $password: the user password.

The upgrade_pre.py script will execute the following commands and actions:

1. alter system begin upgrade.
2. alter system begin rolling upgrade.
3. special pre action: disable and adjust the configuration items.
4. health check: cluster-level health check.

The complete upgrade_pre.py script is as follows:

./upgrade_pre.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. begin_upgrade
                    2. begin_rolling_upgrade
                    3. special_action
                    4. health_check
                    5. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=begin_upgrade,begin_rolling_upgrade,special_action
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_pre.py -h 127.0.0.1 -P 2881 -u ******  -p ******

Here is an example:

1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Run the following command to execute the upgrade_pre.py script for pre-upgrade checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_pre.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 7: Upgrade the cluster

Perform the following steps for each zone. Here is an example for zone1.

1. Run the following command to check the health of the cluster.

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password
   

   Parameters:

   • $user_name: The user who has the permissions to read, write, and modify global system parameters in the sys tenant.
   • $password: The password of the user.

   Example:

   1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Run the following command to execute the upgrade_health_checker.py script to check the health of the cluster.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p******
      

2. Stop the zone (STOP ZONE).

   Log in to the sys tenant of the cluster as the root user and run the following command to stop the zone, where $zone_name is the name of the zone. The command is as follows:

   ALTER SYSTEM STOP ZONE '$zone_name';
   

   The STOP ZONE command will return successfully only after all leader replicas of the zone have been switched.

   Example:

   obclient [(none)]> ALTER SYSTEM STOP ZONE 'zone1';
   

   You can run the following SQL command to view the status of the zone.

   obclient [(none)]> SELECT ZONE,STATUS FROM oceanbase.DBA_OB_ZONES;
   

   The return result is as follows:

   +-------+----------+
   | ZONE  | STATUS   |
   +-------+----------+
   | zone1 | INACTIVE |
   | zone2 | ACTIVE   |
   | zone3 | ACTIVE   |
   +-------+----------+
   3 rows in set
   

3. Stop the processes on the nodes in the zone and restart the nodes with the new binary.

   Stop the observer process and then restart the observer process.

   1. Switch to the admin user.

      [root@xxx /home/admin/oceanbase/etc]# su - admin
      

   2. Stop the observer process.

      -bash-4.2$ kill -9 `pidof observer`
      

   3. Restart the observer process.

      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer
      

      Note

      When you restart the observer process, you do not need to specify the startup parameters because the previous startup parameters have been written to the parameter file.

4. Run the following command to check the health of the zone.

   The health check of the zone basically reuses the health check logic of the cluster. It mainly verifies the status of all nodes in the specified zone. The command is as follows:

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password -z '$zone_name'
   

   Parameters:

   • $user_name: The user who has the permissions to read, write, and modify global system parameters in the sys tenant.
   • $password: The password of the user.
   • $zone_name: The name of the zone.

   Example:

   1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Run the following command to execute the upgrade_health_checker.py script to check the health of the zone.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p****** -z 'zone1'
      

5. Start the zone (start zone).

   Log in to the sys tenant of the cluster as the root user and run the following command to start the zone, where $zone_name is the name of the zone. The command is as follows:

   ALTER SYSTEM START ZONE '$zone_name';
   

   Example:

   obclient [(none)]> ALTER SYSTEM START ZONE 'zone1';
   

6. Repeat steps 1 to 5 until all zones are upgraded.

Step 8: Execute the upgrade_post.py script

On any OBServer node, specify the login information of the sys tenant to directly connect to the OBServer node and execute the upgrade_post.py script to complete the main upgrade operations and related checks. The command is as follows:

python upgrade_post.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user with read/write and modify permissions for global system parameters in the sys tenant.
• $password: the user password.

The upgrade_post.py script performs the following upgrade actions:

1. Cluster-level health check.
2. alter system end rolling upgrade.
3. Execute begin upgrade for each tenant.
4. Execute system variable correction for each tenant.
5. Execute system table correction for each tenant.
6. Execute virtual table/view correction for each tenant.
7. Execute version-related upgrade actions for each tenant.
8. Execute internal table self-check for each tenant.
9. Execute end upgrade for each tenant.
10. alter system end upgrade: ends the cluster upgrade status.
11. Upgrade post check: re-enables some configurations that were disabled during the upgrade and executes checks.

The steps from 3 to 5, 7, and 9 are not executed in the following upgrade scenarios:

1. Same version upgrade.
2. Cross-version upgrade where the data versions of the original and target versions are consistent.

The complete capabilities of the upgrade_post.py script are as follows:

./upgrade_post.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. health_check
                    2. end_rolling_upgrade
                    3. tenant_upgrade
                    4. end_upgrade
                    5. post_check
                    6. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=health_check,begin_rolling_upgrade
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_post.py -h 127.0.0.1 -P 2881 -u *** -p ******

Here is an example:

1. Use the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Use the following command to execute the upgrade_post.py script to complete the main upgrade operations and related checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_post.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 9: Restore cluster configurations

Based on the old values of the configurations backed up in Step 3, log in to the sys tenant of the cluster as the root user and execute the following command to restore the configurations.

ALTER SYSTEM SET $parameter value= $parameter_value 

Parameter description:

• $parameter: the name of the configuration item.
• $parameter_value: the value of the configuration item.

Here is an example:

• Restore the value of server_permanent_offline_time.

  obclient [(none)]> ALTER SYSTEM SET server_permanent_offline_time = '3600s';
  

• Restore the value of enable_rebalance.

  obclient [(none)]> ALTER SYSTEM SET enable_rebalance = 'True';
  

• Restore the value of enable_rereplication.

  obclient [(none)]> ALTER SYSTEM SET enable_rereplication = 'True';
  

What to do next

Verify the upgrade

Verify whether the upgrade is completed. For more information, see Verify the upgrade.

Reconfigure cgroups

If cgroups are configured for the OceanBase cluster before the upgrade, reconfigure cgroups for the OceanBase cluster after the upgrade. For more information, see Configure cgroups.

References

• For more information about how to connect to OceanBase Database, see Overview of connection methods.
• For more information about how to start a zone, see Start a zone.
• For more information about how to modify a configuration parameter, see ALTER SYSTEM PARAMETER.