Upgrade an OceanBase cluster

This topic describes how to upgrade an OceanBase cluster by using an upgrade script.

Prerequisites

You have installed Python 2 on all OBServer nodes and installed the mysql.connector module that is compatible with Python 2.

Procedure

1. Analyze the oceanbase_upgrade_dep.yml file.
2. Confirm the upgrade process.
3. Back up the cluster parameters.
4. Upload and install the target RPM package.
5. Run the upgrade_checker.py script.
6. Run the upgrade_pre.py script.
7. Upgrade the cluster.
8. Run the upgrade_post.py script.
9. Restore the cluster parameters.

Step 1: Analyze the oceanbase_upgrade_dep.yml file

After you decompress the OceanBase RPM package of the target version, you can obtain the oceanbase_upgrade_dep.yml file, which records the topology of the OceanBase cluster version upgrade. The steps are as follows:

1. Obtain the OceanBase RPM installation package of the target version.

   You can click Resources in the upper-left corner of the OceanBase console and choose Download Center to obtain the RPM package of the target version. If you cannot find the RPM package of the target version, contact Technical Support for assistance.

2. Decompress the OceanBase RPM package.

   You can run the following command to decompress the OceanBase RPM package to the current directory:

   [xxx@xxx $rpm_dir]# sudo rpm2cpio $rpm_name | cpio -div
   

   In the preceding command, $rpm_name indicates the name of the RPM package.

   Note

   After you decompress the RPM package, the home and use directories are generated in the directory where the RPM package is stored. The upgrade-related scripts and oceanbase_upgrade_dep.yml file are stored in the home/admin/oceanbase/etc directory.

3. Analyze the oceanbase_upgrade_dep.yml file.

   The following table describes the attributes in the oceanbase_upgrade_dep.yml file:

   • version: the version number of the current OceanBase Database, which is referred to as the current version.
   • can_be_upgraded_to: the target version number to which the current OceanBase Database can be upgraded, which is referred to as the target version.
   • deprecated: indicates whether the current version can be used as the target version. If the value is true, the current version cannot be used as the target version. For example, the 4.1.0.0-100000982023031415 version cannot be used as the target version.
   • require_from_binary: indicates whether you must upgrade to the current version before you upgrade to the target version. In other words, whether the current version is the barrier version in the upgrade sequence.
     • value: when the value is true, it is used in combination with the when_come_from attribute.
     • when_come_from: a list. If when_come_from is not defined, it indicates that you must upgrade to the current barrier version before you upgrade to the target version from any version. If when_come_from is defined, it indicates that you must upgrade to the current barrier version before you upgrade to the target version from the versions in the list. For example, when you upgrade the OceanBase clusters of the 4.0.0.0 and 4.1.0.0-100000982023031415 versions to the V4.2.0.0 version, you must first upgrade them to the V4.1.0.1(barrier) version.

   Here is an example:

   Assume that the upgrade dependencies of the OceanBase clusters of different versions in the oceanbase_upgrade_dep.yml file are as follows:

   - version: 4.0.0.0
     can_be_upgraded_to:
         - 4.1.0.0
   
   - version: 4.1.0.0-100000982023031415
     can_be_upgraded_to:
       - 4.1.0.0
     deprecated: True
   
   - version: 4.1.0.0
     can_be_upgraded_to:
         - 4.1.0.1
   
   - version: 4.1.0.1
     can_be_upgraded_to:
         - 4.2.0.0
     require_from_binary:
       value: True
       when_come_from: [4.0.0.0, 4.1.0.0-100000982023031415]
   
   # 4.3.0.x
   - version: 4.3.0.0
     can_be_upgraded_to:
         - 4.3.0.1
   
   - version: 4.3.0.1
     can_be_upgraded_to:
         - 4.3.1.0
   

   By analyzing the preceding example, you can obtain the following upgrade sequences:

   1. V4.0.0.0 > V4.1.0.0 > V4.1.0.1(barrier) > V4.2.0.0
   2. V4.1.0.0-100000982023031415 > V4.1.0.0 > V4.1.0.1(barrier) > V4.2.0.0
   3. V4.1.0.0 > V4.1.0.1 > V4.2.0.0
   4. V4.3.0.0 > V4.3.0.1 > V4.3.1.0

   Notice

   If an upgrade sequence contains a barrier version, you must also obtain the OceanBase RPM packages of all barrier versions in the upgrade sequence.

Step 2: Confirm the upgrade process

By analyzing the upgrade sequences in Step 1, you can obtain the following information:

1. If you are using the OceanBase cluster of the V4.0.0.0 or V4.1.0.0-100000982023031415 version, you must first upgrade the OceanBase cluster to the V4.1.0.1(barrier) version and then upgrade it to the V4.2.0.0 version.

2. If you are using the OceanBase cluster of the V4.1.0.0 or V4.1.0.1 version, you can directly upgrade the OceanBase cluster to the V4.2.0.0 version.

3. If you are using the OceanBase cluster of the V4.3.0.0 or V4.3.0.1 version, you can directly upgrade the OceanBase cluster to the V4.3.1.0 version.

Here is an example:

The following example describes the upgrade process of a three-zone OceanBase cluster.

• The upgrade sequence contains a barrier version. If the current version of the OceanBase cluster is V4.0.0.0, the upgrade process is as follows:

  1. You must first upgrade Zone1 from the V4.0.0.0 version to the V4.1.0.1(barrier) version, upgrade Zone2 from the V4.0.0.0 version to the V4.1.0.1(barrier) version, and upgrade Zone3 from the V4.0.0.0 version to the V4.1.0.1(barrier) version. At this time, the entire cluster has been upgraded from the V4.0.0.0 version to the V4.1.0.1(barrier) version.
  2. You must then upgrade Zone1, Zone2, and Zone3 from the V4.1.0.1(barrier) version to the V4.2.0.0 version in sequence. After all zones are upgraded to the V4.2.0.0 version, the cluster upgrade is complete.

• The upgrade sequence does not contain a barrier version. If the current version of the OceanBase cluster is V4.3.0.0, the upgrade process is as follows:

  You must first upgrade Zone1 from the V4.3.0.0 version to the V4.3.1.0 version, upgrade Zone2 from the V4.3.0.0 version to the V4.3.1.0 version, and upgrade Zone3 from the V4.3.0.0 version to the V4.3.1.0 version. After all zones are upgraded to the V4.3.1.0 version, the cluster upgrade is complete.

Step 3: Back up cluster configuration items

Before you start the upgrade process, you need to back up the values of the following configuration items to restore them after the upgrade:

• server_permanent_offline_time
• enable_rebalance
• enable_rereplication

Execute the following SQL statement in the sys tenant (system tenant) to obtain the values of the configuration items to be backed up:

SELECT SVR_IP,ZONE,NAME,VALUE FROM OCEANBASE.GV$OB_PARAMETERS WHERE NAME IN ("server_permanent_offline_time", "enable_rebalance", "enable_rereplication");

Step 4: Upload and install the target RPM package

1. Use the scp command to upload the OceanBase RPM package required for the upgrade to the OBServer node.

   scp $rpm_name admin@$observer_ip:/$rpm_dir
   

   Parameter description:

   • $rpm_name: the name of the RPM package.
   • $observer_ip: the IP address of the OBServer node.
   • $rpm_dir: the directory where the RPM package is stored.

   Here is an example:

   scp oceanbase-4.2.0.0-100010022023081911.el7.x86_64.rpm admin@10.10.10.1:/home/admin/rpm
   

2. Execute the following command to install the OceanBase RPM package.

   Notice

   • If there is a barrier version in your upgrade path, install the corresponding barrier version RPM package first. Wait until the cluster is successfully upgraded to the barrier version before installing the target version RPM package.
   • If there are multiple barrier versions in the upgrade path, install and upgrade them in sequence until the cluster is successfully upgraded to the last barrier version, then install the target version RPM package.

   rpm -Uvh $rpm_name
   

   In the preceding command, $rpm_name indicates the name of the RPM package.

   Note

   The upgrade-related scripts and oceanbase_upgrade_dep.yml file are stored in the /home/admin/oceanbase/etc directory. The /home/admin/oceanbase/etc directory is the default installation directory of OceanBase Database.

3. Repeat steps 1 and 2 until the target version RPM package is installed on all OBServer nodes.

Step 5: Execute the upgrade_checker.py script

Specify the login information of the sys tenant directly connected to the OBServer node on any OBServer node and execute the upgrade_checker.py script for pre-upgrade checks. If the script execution is successful, you can proceed with the upgrade. The command is as follows:

python upgrade_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user with read, write, and modification permissions for global system parameters in the sys tenant.
• $password: the user password.

Here is an example:

1. Execute the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Execute the following command to run the upgrade_checker.py script for pre-upgrade checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 6: Execute the upgrade_pre.py script

Specify the login information of the sys tenant directly connected to the OBServer node on any OBServer node and execute the upgrade_pre.py script. If the script execution is successful, you can proceed with the upgrade. The command is as follows:

python upgrade_pre.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user with read, write, and modification permissions for global system parameters in the sys tenant.
• $password: the user password.

The upgrade_pre.py script performs the following commands and actions:

1. alter system begin upgrade.
2. alter system begin rolling upgrade.
3. special pre action: disable and adjust configuration items.
4. health check: cluster-level health check.

The complete upgrade_pre.py script is as follows:

./upgrade_pre.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. begin_upgrade
                    2. begin_rolling_upgrade
                    3. special_action
                    4. health_check
                    5. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=begin_upgrade,begin_rolling_upgrade,special_action
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_pre.py -h 127.0.0.1 -P 2881 -u ******  -p ******

Here is an example:

1. Execute the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Execute the following command to run the upgrade_pre.py script for pre-upgrade checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_pre.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 7: Upgrade the cluster

Upgrade the cluster by zone. Perform the following steps for each zone, using zone1 as an example.

1. Run the following command to perform a cluster-level health check.

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password
   

   Parameter description:

   • $user_name: The user who has the privileges to read, write, and modify global system parameters of the sys tenant.
   • $password: The password of the user.

   Here is an example:

   1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Run the following command to execute the upgrade_health_checker.py script for a cluster-level health check.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p******
      

2. Stop the zone (STOP ZONE).

   Notice

   For a standalone or single-replica OceanBase Database, you do not need to stop the zone (STOP ZONE).

   Log in to the sys tenant of the cluster as the root user, and run the following command to stop the zone. In this example, $zone_name specifies the name of the zone. The command is as follows:

   ALTER SYSTEM STOP ZONE '$zone_name';
   

   The STOP ZONE command returns successfully only after all leader replicas of the zone have been switched.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM STOP ZONE 'zone1';
   

   You can run the following SQL statement to view the status of the zone.

   obclient [(none)]> SELECT ZONE,STATUS FROM oceanbase.DBA_OB_ZONES;
   

   The return result is as follows:

   +-------+----------+
   | ZONE  | STATUS   |
   +-------+----------+
   | zone1 | INACTIVE |
   | zone2 | ACTIVE   |
   | zone3 | ACTIVE   |
   +-------+----------+
   3 rows in set
   

3. Stop the processes on the nodes in the zone, replace the new binary files, and restart the processes.

   Stop the observer process and then restart the observer process.

   1. Switch to the admin user.

      [root@xxx /home/admin/oceanbase/etc]# su - admin
      

   2. Stop the observer process.

      -bash-4.2$ kill -9 `pidof observer`
      

   3. Restart the observer process.

      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer
      

      Note

      When you restart the observer process, you do not need to specify the startup parameters because the previous startup parameters have been written to the parameter file.

4. Run a zone-level health check.

   The zone-level health check basically reuses the cluster-level health check logic. It mainly verifies the status of all nodes in the specified zone. The command is as follows:

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password -z '$zone_name'
   

   Parameter description:

   • $user_name: The user who has the privileges to read, write, and modify global system parameters of the sys tenant.
   • $password: The password of the user.
   • $zone_name: The name of the zone.

   Here is an example:

   1. Run the following command to switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Run the following command to execute the upgrade_health_checker.py script for a zone-level health check.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p****** -z 'zone1'
      

5. Start the zone (start zone).

   Log in to the sys tenant of the cluster as the root user, and run the following command to start the zone. In this example, $zone_name specifies the name of the zone. The command is as follows:

   ALTER SYSTEM START ZONE '$zone_name';
   

   Here is an example:

   obclient [(none)]> ALTER SYSTEM START ZONE 'zone1';
   

6. Repeat steps 1 to 5 until all zones are upgraded.

Step 8: Execute the upgrade_post.py script

On any OBServer node, specify the login information for the sys tenant to connect directly to the OBServer node and execute the upgrade_post.py script to complete the main upgrade operations and related checks. The command is as follows:

python upgrade_post.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

Parameter description:

• $user_name: the user with read/write and modify permissions for global system parameters in the sys tenant.
• $password: the user password.

The upgrade_post.py script performs the following upgrade actions:

1. Cluster-level health check.
2. Execute alter system end rolling upgrade.
3. Execute begin upgrade for each tenant.
4. Execute system variable correction for each tenant.
5. Execute system table correction for each tenant.
6. Execute virtual table/view correction for each tenant.
7. Execute version-related upgrade actions for each tenant.
8. Execute internal table self-check for each tenant.
9. Execute end upgrade for each tenant.
10. Execute alter system end upgrade to end the cluster upgrade status.
11. Execute upgrade post check: re-enable the configuration items disabled during the upgrade process and perform checks.

The steps from 3 to 5, 7, and 9 are not executed in the following upgrade scenarios:

1. Same-version upgrade.
2. Cross-version upgrade where the data version of the original version is the same as the target version.

The complete upgrade_post.py script is as follows:

./upgrade_post.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. health_check
                    2. end_rolling_upgrade
                    3. tenant_upgrade
                    4. end_upgrade
                    5. post_check
                    6. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=health_check,begin_rolling_upgrade
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_post.py -h 127.0.0.1 -P 2881 -u *** -p ******

Here is an example:

1. Use the following command to switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Use the following command to execute the upgrade_post.py script to complete the main upgrade operations and related checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_post.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 9: Restore cluster configuration items

Based on the old values of the configuration items backed up in Step 3, log in to the sys tenant of the cluster as the root user and use the following command to restore the related configuration items.

ALTER SYSTEM SET $parameter value= $parameter_value 

Parameter description:

• $parameter: the name of the configuration item.
• $parameter_value: the value of the configuration item.

Here is an example:

• Restore the value of server_permanent_offline_time.

  obclient [(none)]> ALTER SYSTEM SET server_permanent_offline_time = '3600s';
  

• Restore the value of enable_rebalance.

  obclient [(none)]> ALTER SYSTEM SET enable_rebalance = 'True';
  

• Restore the value of enable_rereplication.

  obclient [(none)]> ALTER SYSTEM SET enable_rereplication = 'True';
  

What to do next

Post-upgrade checks

To verify whether the upgrade is completed, see Post-upgrade checks.

Reconfigure cgroup

If you configured cgroup for the OceanBase cluster before the upgrade, you need to reconfigure cgroup after the upgrade. For more information, see Configure cgroup.

References

• For more information about how to connect to OceanBase Database, see Overview of connection methods.
• For more information about how to start a zone, see Start a zone.
• For more information about how to modify a configuration parameter, see ALTER SYSTEM PARAMETER.