Upgrade an OceanBase cluster

This topic describes how to use an upgrade script to upgrade an OceanBase cluster.

Prerequisites

You have installed Python 2 and the mysql.connector module that is compatible with Python 2 on all OBServer nodes.

Procedure

1. Analyze the oceanbase_upgrade_dep.yml file.
2. Confirm the upgrade process.
3. Back up the cluster parameters.
4. Upload and install the RPM package of the target version.
5. Execute the upgrade_checker.py script.
6. Execute the upgrade_pre.py script.
7. Upgrade the cluster.
8. Execute the upgrade_post.py script.
9. Restore the cluster parameters.

Step 1: Analyze the oceanbase_upgrade_dep.yml file

After you decompress the OceanBase RPM package of the target version, obtain the oceanbase_upgrade_dep.yml file, which records the version upgrade topology of OceanBase Database. Perform the following steps:

1. Obtain the RPM package of the target OceanBase Database version.

   In the upper-left corner of the OceanBase website, choose Resources > Download and choose the RPM package of the target version. If no installation package of the target version is available on the website, contact OceanBase Technical Support for help.

2. Decompress the OceanBase RPM package.

   Run the following command to decompress the OceanBase RPM package to the current directory:

   [xxx@xxx $rpm_dir]# sudo rpm2cpio $rpm_name | cpio -div
   

   Here, $rpm_name indicates the name of the RPM package.

   Note

   After the RPM package is decompressed, the home and use directories are generated in the directory where the RPM package is located. The upgrade script and the oceanbase_upgrade_dep.yml file are stored in the home/admin/oceanbase/etc directory.

3. Analyze the oceanbase_upgrade_dep.yml file.

   Fields in the oceanbase_upgrade_dep.yml file are described as follows:

   • version: the version of the current OceanBase database, hereinafter referred to as the current version.
   • can_be_upgraded_to: the target version to which the current OceanBase Database version can be upgraded.
   • deprecated: indicates whether the current version can be used as the target version for upgrade. If the value is true, the current version cannot be used as the target version for upgrade. For example, the version 4.1.0.0-100000982023031415 in this example cannot be used as the target version.
   • require_from_binary: indicates whether an upgrade to the current version is required when the current version is a barrier version in the upgrade sequence.
     • value: This parameter is used together with when_come_from when the value is true.
     • when_come_from: a list of versions that require an upgrade to the current barrier version before they are upgraded to the target version. If when_come_from is not defined, all versions require an upgrade to the current barrier version before they are upgraded to the target version. For example, when you upgrade an OceanBase cluster of 4.0.0.0 or 4.1.0.0-100000982023031415 to V4.2.0.0, you must upgrade the cluster to V4.1.0.1 first.

   Here is an example:

   Assume that the oceanbase_upgrade_dep.yml file defines the upgrade dependencies as follows for an OceanBase cluster:

   - version: 4.0.0.0
     can_be_upgraded_to:
         - 4.1.0.0
   
   - version: 4.1.0.0-100000982023031415
     can_be_upgraded_to:
       - 4.1.0.0
     deprecated: True
   
   - version: 4.1.0.0
     can_be_upgraded_to:
         - 4.1.0.1
   
   - version: 4.1.0.1
     can_be_upgraded_to:
         - 4.2.0.0
     require_from_binary:
       value: True
       when_come_from: [4.0.0.0, 4.1.0.0-100000982023031415]
   
   # 4.3.0.x
   - version: 4.3.0.0
     can_be_upgraded_to:
         - 4.3.0.1
   
   - version: 4.3.0.1
     can_be_upgraded_to:
         - 4.3.1.0
   

   Based on the file, the upgrade sequence is as follows:

   1. V4.0.0.0 > V4.1.0.0 > V4.1.0.1 (barrier version) > V4.2.0.0
   2. V4.1.0.0-100000982023031415 > V4.1.0.0 > V4.1.0.1 (barrier version) > V4.2.0.0
   3. V4.1.0.0 > V4.1.0.1 > V4.2.0.0
   4. V4.3.0.0 > V4.3.0.1 > V4.3.1.0

   Notice

   If the upgrade sequence involves barrier versions, you must also obtain the OceanBase RPM packages of all barrier versions in the upgrade path.

Step 2: Confirm the upgrade process

You can obtain the following information from the upgrade sequence obtained in Step 1:

1. If you are upgrading an OceanBase cluster of V4.0.0.0 or V4.1.0.0-100000982023031415, you must first upgrade the cluster to V4.1.0.1 and then to V4.2.0.0.

2. If you are using an OceanBase cluster of V4.1.0.0 or V4.1.0.1, you can directly upgrade the cluster to V4.2.0.0.

3. If you are using an OceanBase cluster of V4.3.0.0 or V4.3.0.1, you can directly upgrade the cluster to V4.3.1.0.

The following example upgrades a three-replica OceanBase cluster.

• Scenario 1: The upgrade sequence involves barrier versions. If the current OceanBase cluster version is V4.0.0.0, the upgrade process is as follows:

  1. Upgrade Zone1, Zone2, and Zone3 in sequence from V4.0.0.0 to V4.1.0.1, which is the barrier version. Then, the whole cluster is upgraded to the barrier version V4.1.0.1.
  2. Upgrade Zone1, Zone2, and Zone3 in sequence from V4.1.0.1 to V4.2.0.0. The whole cluster is then upgraded to V4.2.0.0.

• Scenario 2: The upgrade sequence does not involve barrier versions. If the current OceanBase cluster version is V4.3.0.0, the upgrade process is as follows:

  Upgrade Zone 1, Zone 2, and Zone 3 in sequence from V4.3.0.0 to V4.3.1.0. Then, the whole cluster is upgraded to V4.3.1.0.

Step 3: Back up the cluster parameters

Before the upgrade, back up the values of the following parameters, which are used for restoration after the upgrade.

• server_permanent_offline_time
• enable_rebalance
• enable_rereplication

Execute the following SQL statement in the sys tenant to obtain the parameters to be backed up:

SELECT SVR_IP,ZONE,NAME,VALUE FROM OCEANBASE.GV$OB_PARAMETERS WHERE NAME IN ("server_permanent_offline_time", "enable_rebalance", "enable_rereplication");

Step 4: Upload and install the target RPM package

1. Run the scp command to upload the OceanBase RPM package to the OBServer node.

   scp $rpm_name admin@$observer_ip:/$rpm_dir
   

   where

   • $rpm_name indicates the name of the RPM package.
   • $observer_ip indicates the IP address of the OBServer node.
   • $rpm_dir indicates the directory where the RPM package is stored.

   Here is an example:

   scp oceanbase-4.2.0.0-100010022023081911.el7.x86_64.rpm admin@10.10.10.1:/home/admin/rpm
   

2. Install the OceanBase RPM package.

   Notice

   • If the upgrade sequence involves barrier versions, install the OceanBase RPM packages of the barrier versions in the upgrade path first. Install the RPM package of the target version after the cluster is successfully upgraded to the final barrier version.
   • If the upgrade path involves multiple versions, install and upgrade the versions in sequence until the cluster is upgraded to the final barrier version, and then install the RPM package of the target version.

   rpm -Uvh $rpm_name
   

   Here, $rpm_name indicates the name of the RPM package.

   Note

   Upgrade-related scripts and the oceanbase_upgrade_dep.yml file are stored in the /home/admin/oceanbase/etc directory.

3. Repeat steps 1 to 2 until the RPM package of the target version is installed on all OBServer nodes.

Step 5: Execute the upgrade_checker.py script

On any OBServer node, specify the logon information of the sys tenant to directly connect to the OBServer node. Run the upgrade_checker.py script to perform the pre-upgrade check. If the script is successfully executed, the upgrade can proceed. The syntax of the command is as follows:

python upgrade_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

where

• $user_name indicates a user that has the permissions to read, write, and modify global system parameters in the sys tenant.
• $password indicates the user password.

Here is an example:

1. Switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Execute the upgrade_checker.py script to perform the pre-upgrade check.

   [root@xxx /rpm_dir/home/admin/oceanbase/etc]# python upgrade_checker.py -h 127.0.0.1 -P 2881 -u xxxx@sys -p******
   

Step 6: Execute the upgrade_pre.py script

On any OBServer node, specify the logon information of the sys tenant to directly connect to the OBServer node and execute the upgrade_pre.py script. If the script is successfully executed, the upgrade can proceed. The syntax of the command is as follows:

python upgrade_pre.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

where

• $user_name indicates a user that has the permissions to read, write, and modify global system parameters in the sys tenant.
• $password indicates the user password.

The upgrade_pre.py script will execute the following commands and perform the following actions:

1. alter system begin upgrade
2. alter system begin rolling upgrade
3. special pre action: to disable and adjust parameters.
4. health check: to perform a cluster-level health check.

Complete capabilities of the upgrade_pre.py script:

./upgrade_pre.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. begin_upgrade
                    2. begin_rolling_upgrade
                    3. special_action
                    4. health_check
                    5. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=begin_upgrade,begin_rolling_upgrade,special_action
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_pre.py -h 127.0.0.1 -P 2881 -u ******  -p ******

Here is an example:

1. Switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Execute the upgrade_pre.py script for pre-upgrade checks.

   [root@xxx /rpm_dir/home/admin/oceanbase/etc]# python upgrade_pre.py -h 127.0.0.1 -P 2881 -u xxxx@sys -p******
   

Step 7: Upgrade the cluster

The cluster is upgraded zone by zone. Therefore, you must perform the following steps for each zone. The following describes the upgrade procedure by taking zone1 as an example.

1. Perform a health check at the cluster level.

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password
   

   where

   • $user_name indicates a user that has the permissions to read, write, and modify global system parameters in the sys tenant.
   • $password indicates the user password.

   Here is an example:

   1. Switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Execute the upgrade_health_checker.py script for a cluster-level health check.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p******
      

2. Execute the STOP ZONE statement to stop the zone.

   Notice

   When you upgrade a standalone or single-replica OceanBase database, you do not need to execute the STOP ZONE statement to stop the zone.

   Log on to the sys tenant of the cluster as the root user and stop the zone with the following command, where $zone_name specifies the name of the zone. The syntax of the command is as follows:

   ALTER SYSTEM STOP ZONE '$zone_name';
   

   The STOP ZONE command will not return a success until the leader role is switched to another replica.

   Here is an example:

   obclient [(none)]> ALTER SYSTEM STOP ZONE 'zone1';
   

   You can run the following SQL statement to check the zone status:

   obclient [(none)]> SELECT ZONE,STATUS FROM oceanbase.DBA_OB_ZONES;
   

   The result is as follows:

   +-------+----------+
   | ZONE  | STATUS   |
   +-------+----------+
   | zone1 | INACTIVE |
   | zone2 | ACTIVE   |
   | zone3 | ACTIVE   |
   +-------+----------+
   3 rows in set
   

3. Stop the observer processes on the OBServer nodes in the zone. Replace the binary files with those of the new database version and then restart the processes.

   Stop and then restart the observer process.

   1. Switch to the admin user.

      [root@xxx /home/admin/oceanbase/etc]# su - admin
      

   2. Stop the observer process.

      -bash-4.2$ kill -9 `pidof observer`
      

   3. Restart the observer process.

      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer
      

      Note

      You do not need to specify the startup parameters when you restart the arbitration server process because the startup parameters of the earlier version are automatically written to the parameter file.

4. Perform a zone-level health check.

   The logic of the zone-level health check is the same as that of the cluster-level health check and aims to verify the status of all OBServer nodes in the zone. The syntax of the command is as follows:

   python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password -z '$zone_name'
   

   where

   • $user_name indicates a user that has the permissions to read, write, and modify global system parameters in the sys tenant.
   • $password indicates the user password.
   • $zone_name indicates the name of the zone.

   Here is an example:

   1. Switch to the /home/admin/oceanbase/etc directory.

      cd /home/admin/oceanbase/etc
      

   2. Execute the upgrade_health_checker.py script for a zone-level health check.

      [root@xxx /home/admin/oceanbase/etc]# python upgrade_health_checker.py -h 127.0.0.1 -P 2881 -u root@sys -p****** -z 'zone1'
      

5. Start the zone.

   Log on to the sys tenant of the cluster as the root user and start the zone with the following command, where $zone_name specifies the name of the zone. The syntax of the command is as follows:

   ALTER SYSTEM START ZONE '$zone_name';
   

   Here is an example:

   obclient [(none)]> ALTER SYSTEM START ZONE 'zone1';
   

6. Repeat steps 1 to 5 until the cluster is upgraded in all zones.

Step 8: Execute the upgrade_post.py script

On any OBServer node, specify the logon information of sys tenant to directly connect to the OBServer node, and execute the upgrade_post.py script to complete the main upgrade operations and related checks. The syntax of the command is as follows:

python upgrade_post.py -h 127.0.0.1 -P 2881 -u $user_name@sys -p$password

where

• $user_name indicates a user that has the permissions to read, write, and modify global system parameters in the sys tenant.
• $password indicates the user password.

The upgrade_post.py script performs the following upgrade actions:

1. Perform a cluster-level health check.
2. alter system end rolling upgrade
3. Perform the BEGIN UPGRADE operation.
4. Adjust system variables by tenant.
5. Correct system tables by tenant.
6. Correct virtual tables and views by tenant.
7. Perform related upgrade operations by tenant.
8. Perform a self-check on internal tables by tenant.
9. Perform the END UPGRADE operation.
10. End the upgrade state of the cluster by using the ALTER SYSTEM END UPGRADE statement.
11. Enable the parameters disabled during the upgrade and perform a check.

Skip steps 3 to 5, step 7, and step 9 in the following scenarios:

1. Intra-version upgrade
2. Cross-version upgrade in which the source and target versions have the same version of data

Complete capabilities of the upgrade_post.py script:

./upgrade_post.py [OPTIONS]

-I, --help          Display this help and exit.
-V, --version       Output version information and exit.
-h, --host=name     Connect to host.
-P, --port=name     Port number to use for connection.
-u, --user=name     User for login.
-p, --password=name Password to use when connecting to server. If password is
                    not given it's empty string "".
-t, --timeout=name  Cmd/Query execute timeout(s).
-m, --module=name   Modules to run. Modules should be a string combined by some of
                    the following strings:
                    1. health_check
                    2. end_rolling_upgrade
                    3. tenant_upgrade
                    4. end_upgrade
                    5. post_check
                    6. all: "all" represents that all modules should be run.
                    They are splitted by ",".
                    For example: -m all, or --module=health_check,begin_rolling_upgrade
-l, --log-file=name Log file path. If log file path is not given it's ./upgrade_pre.log


Maybe you want to run cmd like that:
./upgrade_post.py -h 127.0.0.1 -P 2881 -u *** -p ******

Here is an example:

1. Switch to the /home/admin/oceanbase/etc directory.

   cd /home/admin/oceanbase/etc
   

2. Execute the upgrade_post.py script to complete the main upgrade operations and related checks.

   [root@xxx /home/admin/oceanbase/etc]# python upgrade_post.py -h 127.0.0.1 -P 2881 -u root@sys -p******
   

Step 9: Restore the cluster parameters

Log on to the sys tenant of the cluster as the root user and restore the parameters backed up in Step 3.

ALTER SYSTEM SET $parameter value= $parameter_value

where

• $parameter indicates the name of the parameter.
• $parameter_value indicates the value of the parameter.

Here is an example:

• Restore the value of server_permanent_offline_time.

  obclient [(none)]> ALTER SYSTEM SET server_permanent_offline_time = '3600s';
  

• Restore the value of enable_rebalance.

  obclient [(none)]> ALTER SYSTEM SET enable_rebalance = 'True';
  

• Restore the value of enable_rereplication.

  obclient [(none)]> ALTER SYSTEM SET enable_rereplication = 'True';
  

What to do next

Verify whether the upgrade is completed. For more information, see Check upgrade results.

References

• For more information about how to connect to OceanBase Database, see Overview.
• For more information about how to start a zone, see Start a zone.
• For more information about modifying parameters, see ALTER SYSTEM PARAMETER.