Upgrade in multi-node deployment mode

This topic describes how to upgrade OceanBase Migration Service (OMS) Community Edition in multi-node deployment mode to V4.2.10.

Upgrade procedure

1. If the high availability (HA) feature is enabled, disable it.

   1. Log in to the OMS Community Edition console.

   2. In the left-side navigation pane, click System Management > System Parameters.

   3. On the System Parameters page, find ha.config.

   4. Click the edit icon next to Value.

   5. In the Modify Value dialog box, set enable to false to disable the HA feature.

2. Back up the database.

   1. Log in to IP1 and IP2, respectively, and stop the OMS Community Edition V4.2.9 containers on the two servers.

      sudo docker stop ${CONTAINER_NAME}
      

      Note

      CONTAINER_NAME is the name of the created container.

   2. Log in to the CM heartbeat database specified in the configuration file, delete some unnecessary records in the database to save backup time.

      # Log in to the CM heartbeat database specified in the configuration file.
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -Dcm_hb_429
      
      # Delete unnecessary records.
      # heatbeat_sequence is used to report heartbeats and obtain auto-increment IDs.
      delete from heatbeat_sequence where id < (select max(id) from heatbeat_sequence);
      

   3. Run the following command to manually back up the rm, cm, and cm_hb databases to SQL files and confirm that the sizes of the files are not 0.

      In multi-region scenarios, you need to back up the cm_hb databases in all regions. For example, if two regions are involved, you need to back up four databases: rm, cm, cm_hb1, and cm_hb2.

      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false rm_429 > /home/admin/rm_429.sql
      
      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false cm_429 > /home/admin/cm_429.sql
      
      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false cm_hb_429 > /home/admin/cm_hb_429.sql
      

      The following table describes the backup parameters.

      Parameter	Description
      -h	The host information to be exported.
      -P	The port number for connecting to the database.
      -u	The username for the connection.
      -p	The password for connecting to the database.
      --triggers	Specifies whether to export triggers. The default value is false (not to export).
      rm_429, cm_429, cm_hb_429	This parameter specifies to back up the rm, cm, and cm_hb databases to SQL files. The format is database name > path of SQL file.sql. You must replace the values with those in your actual environment.

3. Load the OMS Community Edition installation package to the local image repository of the Docker container.

   docker load -i <OMS Community Edition installation package>
   

4. Start the OMS Community Edition V4.2.10 container.

   OMS Community Edition supports accessing the OMS Community Edition console through HTTP and HTTPS protocols. If you want to securely access OMS Community Edition, you can configure an HTTPS self-signed certificate and mount it to the specified directory in the container. If you want to access OMS Community Edition through the HTTP protocol, you do not need to configure it.

   Notice

   • When you start the OMS Community Edition V4.2.10 container, the paths of the three mounted disks must be consistent with those before the upgrade.
     You can run the sudo docker inspect ${CONTAINER_NAME} | grep -A5 'Binds' command to view the paths of the disks mounted in the previous OMS Community Edition container.

   • OMS Community Edition V3.3.1 and later versions support the new parameter -e IS_UPGRADE=true. This parameter is only applicable in upgrade scenarios and must be added in such scenarios.

   • Replace work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} with the name of the image actually imported by executing the docker load -i command.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you want to mount the HTTPS certificate in the OMS Community Edition container, you need to set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   -e IS_UPGRADE=true \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

   The following table describes the startup parameters.

   Parameter	Description
   OMS_HOST_IP	The IP address of the host. Notice: The OMS_HOST_IP values of each node are different.
   CONTAINER_NAME	The name of the created container.
   IMAGE_TAG	After you load the OMS Community Edition installation package to Docker, you can run the docker images command to obtain the [IMAGE ID] or [REPOSITORY:TAG] of the loaded image, which is the unique identifier of the loaded image, namely, <OMS_IMAGE>.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	/data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run can be replaced with the mounted directories created on the OMS Community Edition deployment server. These directories store log files generated during OMS Community Edition operation, log pull component files, and synchronization component files. The files are persisted on the server. Notice: The mounted directories must be retained in the same location during subsequent re-deployments and upgrades.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are fixed directories in the container. The paths cannot be modified.
   /data/oms/https_crt (optional) /data/oms/https_key (optional)	The mounted path of the HTTPS certificate in the OMS Community Edition container. If the HTTPS certificate is mounted, the OMS Community Edition container runs the Nginx service in HTTPS mode. You must access OMS Community Edition in HTTPS mode to use the OMS Community Edition console.
   IS_UPGRADE	In upgrade scenarios, you must set IS_UPGRADE=true. Please note that IS_UPGRADE must be in uppercase.
   privileged	Grants extended permissions to the container.
   pids-limit	Specifies the number of processes in the container. -1 indicates no limit.
   ulimit nproc	Specifies the maximum number of user processes.

5. Enter the new container.

   docker exec -it ${CONTAINER_NAME} bash  
   

   Note

   CONTAINER_NAME is the name of the created container.

6. In the root directory, execute the metadata initialization operation.

   sleep 300
   bash /root/docker_init.sh
   

   Note

   • After you run the preceding command, the script automatically applies schema changes to the three databases of OMS Community Edition.

   • If the configuration files of multiple nodes in the same region are the same (mainly the cm_nodes file), you need to run the docker_init.sh script only on one node in the region.
     If the configuration files of multiple nodes in the same region are different (mainly the cm_nodes file), you need to run the docker_init.sh script on each node.

   If the preceding command returns the error Version before 4.1.1 is not allowed to upgrade., perform the following steps.

   1. Confirm that the OMS Community Edition version to be upgraded is not earlier than V4.1.1-CE.

   2. Execute the following SQL statements in the rm database:

      CREATE TABLE IF NOT EXISTS `oms_version` (
         `version` varchar(64) NOT NULL COMMENT 'Version',
         `gmt_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
         PRIMARY KEY (`version`)
         ) DEFAULT CHARSET = utf8mb4 COMMENT = 'Version table';
      
      REPLACE INTO `oms_version`(`version`) VALUES('x.x.x-CE');  -- Enter the actual OMS Community Edition version before the upgrade.
      

   3. Run the initialization command again.

7. After the docker_init.sh script is executed, confirm that the server list is normal and all servers are in the Online state.

   1. Log in to the OMS Community Edition console.

   2. In the left-side navigation pane, click OPS & Monitoring > Servers.

   3. On the Servers page, view the server list and confirm that the status of all servers is Online.

8. After the two nodes are upgraded, enable the HA feature and set related parameters on the System Parameters page.

   1. Log in to the OMS Community Edition console.

   2. In the left-side navigation pane, click System Management > System Parameters.

   3. On the System Parameters page, find ha.config.

   4. Click the edit icon next to Value.

   5. In the Modify Value dialog box, set enable to true to enable the HA feature and record the time T2.

      You can also set perceiveStoreClientCheckpoint to true at the same time. If you set it to true, you do not need to record T1 and T2.

   6. If you set perceiveStoreClientCheckpoint to false, you need to modify the value of refetchStoreIntervalMin (the time interval for pulling data, in minutes) based on your business needs. The value of refetchStoreIntervalMin must be greater than T2-T1.

      If you set perceiveStoreClientCheckpoint to true, you do not need to modify the value of refetchStoreIntervalMin. HA starts the Store based on the earliest timestamp of the downstream component minus the value of refetchStoreIntervalMin. For example, if the earliest timestamp of the downstream connector/jdbc-connector is 12:00:00 and the value of refetchStoreIntervalMin is 30 minutes, HA starts the Store at 11:30:00.

      After the HA feature is enabled, HA performs the following operations on components:

      Status of the task/component before the upgrade	Status after the upgrade (HA is not enabled)	Status after the upgrade (HA is enabled)
      The task is running normally. The Incr-Sync component is running normally. The Store component is running normally.	The status of the task remains unchanged. The Incr-Sync component fails. The Store component fails.	The status of the task remains unchanged. HA automatically starts the Incr-Sync component. HA creates a new Store component based on the configuration. For more information, see Modify HA configurations.
      The task fails. The Incr-Sync component fails. The Store component fails.
      The task is paused. The Incr-Sync component is paused. The Store component is running normally.	The status of the task remains unchanged. The Incr-Sync component is paused. The Store component fails.	The status of the task remains unchanged. The Incr-Sync component is paused. HA creates a new Store component based on the configuration. For more information, see Modify HA configurations.

9. (Optional) If you need to roll back, perform the following steps.

   1. Disable the HA feature following the operation in Step 1.

   2. Stop the new container after the upgrade and record the time T3.

      sudo docker stop ${CONTAINER_NAME}
      

   3. Connect to the metaDB and run the following command.

      drop database rm_429;
      drop database cm_429;
      drop database cm_hb_429;
      
      create database rm_429;
      create database cm_429;
      create database cm_hb_429;
      

   4. Restore the backed-up SQL files to the original database.

      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/rm_429.sql" -Drm_429
      
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/cm_429.sql" -Dcm_429
      
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/cm_hb_429.sql" -Dcm_hb_429
      

   5. Start the OMS Community Edition V4.2.9 container.

      sudo docker restart ${CONTAINER_NAME}
      

   6. On the System Parameters page, enable the HA feature and set the refetchStoreIntervalMin parameter to the required value.

      If you perform a rollback, HA automatically takes over the Store component, but you may need to manually restore the Incr-Sync or Full-Import component.

10. After the upgrade is completed, clear the browser cache and log in.