Upgrade OMS in multi-node deployment mode

This topic describes how to upgrade OceanBase Migration Service (OMS) to V4.0.0-CE in multiple-node deployment mode.

Procedure

1. If high availability (HA) is enabled, disable it first.

   1. Log on to the OMS console.

   2. In the left-side navigation pane, choose System Management > System Parameters.

   3. On the System Parameters page, find ha.config.

   4. Click the edit icon in the Value column of the parameter.

   5. In the Modify Value dialog box, set enable to false to disable HA.

2. Back up the databases.

   1. Log on to the two hosts where the container of OMS V3.3.1-CE is deployed by using their respective IP addresses, and suspend the container.

      sudo docker stop ${CONTAINER_NAME}
      

      Notice:

      CONTAINER_NAME specifies the name of the container.

   2. Log on to the cluster management (CM) heartbeat database specified in the configuration file and delete some useless records to reduce the backup time.

      # Log on to the CM heartbeat database specified in the configuration file.
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -Dcm_hb_331
      
      # Delete useless records.
      # Obtain the auto-increment ID from heatbeat_sequence, which reports the heartbeat.
      delete from heatbeat_sequence where id < (select max(id) from heatbeat_sequence);
      

   3. Run the following commands to back up the rm, cm, and cm_hb databases as SQL files and make sure that the sizes of the files are not 0.

      If you have deployed databases in multiple regions, you must back up the cm_hb database in all regions. For example, if you have deployed databases in two regions, you must back up the following four databases: rm, cm, cm_hb1, and cm_hb2.

      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false rm_331 > /home/admin/rm_331.sql
      
      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false cm_331 > /home/admin/cm_331.sql
      
      mysqldump -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> --triggers=false cm_hb_331 > /home/admin/cm_hb_331.sql
      

      Parameter	Description
      -h	The IP address of the host from which the data is exported.
      -P	The port number used to connect to the database.
      -u	The username used to connect to the database.
      -p	The password used to connect to the database.
      --triggers	The data export trigger. The default value is false, which disables data export.
      rm_331, cm_331, and cm_hb_331	Specify to back up the rm, cm, and cm_hb databases to SQL files named in the format of database name > SQL file storage path.sql. You need to replace the values based on the actual environment.

3. Load the downloaded OMS image file to the local image repository of the Docker container.

   docker load -i <Storage path of the OMS image>
   

4. Start the new container of OMS V4.0.0-CE.

   You can access the OMS console by using an HTTP or HTTPS URL. To securely access the OMS console, install a self-signed SSL certificate and mount it to the specified directory in the container. The certificate is not required for HTTP access.

   Notice:

   • Before you start the container of OMS V4.0.0-CE, make sure that the three disk mounting paths of OMS are the same as those before the upgrade.
     You can run the sudo docker inspect ${CONTAINER_NAME} | grep -A5 'Binds' command to view the paths of disks mounted to the old OMS container.

   • The -e IS_UPGRADE=true parameter is provided in OMS V3.3.1-CE or later. This parameter is provided only to support OMS upgrades and must be specified when you upgrade OMS.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate in the OMS container, you must set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   -e IS_UPGRADE=true \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   reg.docker.alibaba-inc.com/oboms/oms-all-in-one:${IMAGE_TAG}
   

   Parameter	Description
   OMS_HOST_IP	The IP address of the host. Notice: The value of OMS_HOST_IP is different for each node.
   CONTAINER_NAME	The name of the container in the oms_xxx format.
   IMAGE_TAG	The tag of the image in the feature_x.x.x-ce format. Specify x.x.x based on the actual OMS version. For example, if you use OMS V4.0.0-CE, the value is feature_4.0.0-ce.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	You can replace /data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run with the mount directories created on the server where OMS is deployed. The mount directories store the logs generated during the operation of OMS and files generated by the store and synchronization components, respectively, to persistently retain the files on the server. Notice: The mount directories must remain unchanged during subsequent redeployment or upgrades.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are default directories in the container and cannot be modified.
   /data/oms/https_crt (optional) /data/oms/https_key (optional)	The mount directory of the SSL certificate in the OMS container. If you mount an SSL certificate, the NGINX service in the OMS container runs in HTTPS mode. In this case, you can access the OMS console by using only the HTTPS URL.
   IS_UPGRADE	To upgrade OMS, you must set the IS_UPGRADE parameter to true.
   privileged	Specifies whether to grant extended privileges on the container.
   pids-limit	Specifies whether to limit the number of container processes. The value -1 indicates that the number is unlimited.
   ulimit nproc	The maximum number of user processes.

5. Access the new container.

   docker exec -it ${CONTAINER_NAME} bash  
   

   Note:

   CONTAINER_NAME specifies the name of the container.

6. Perform metadata initialization in the root directory.

   bash /root/docker_init.sh
   

   Note:

   • After you run the preceding command, the script automatically implements schema changes of the three OMS databases.

   • If the same configuration file is used for multiple cm_nodes in the same region, you need to execute the docker_init.sh script only once in a region.
     If different configuration files are used for multiple cm_nodes in the same region, you need to execute the docker_init.sh script once on each node.

7. After the docker_init.sh script is executed, confirm whether the server list is normal and whether all servers are in the Online state.

   1. Log on to the OMS console.

   2. In the left-side navigation pane, choose OPS & Monitoring > Server.

   3. On the Servers page, check whether the server list is normal. Check whether all servers are in the Online state.

8. After you upgrade OMS on two nodes, enable HA on the System Parameters page, and configure the parameters.

   1. Log on to the OMS console.

   2. In the left-side navigation pane, choose System Management > System Parameters.

   3. On the System Parameters page, find ha.config.

   4. Click the edit icon in the Value column of the parameter.

   5. In the Modify Value dialog box, set enable to true to enable HA, and record the time T2.

      You can also set the perceiveStoreClientCheckpoint parameter to true. After that, you do not need to record T1 and T2.

   6. If you set the perceiveStoreClientCheckpoint parameter to false, you need to modify the value of the refetchStoreIntervalMin parameter based on your business needs, which specifies the time interval, in minutes, for pulling data from the store component. The value must be greater than T2 minus T1.

      If you set the perceiveStoreClientCheckpoint parameter to true, you can use the default value of the refetchStoreIntervalMin parameter. HA is enabled, so the system starts the store component based on the earliest request time of downstream components minus the value of the refetchStoreIntervalMin parameter. For example, if the earliest request time of the downstream connector or JDBC-connector component is 12:00:00 and the refetchStoreIntervalMin parameter is set to 30 minutes, the system starts the store component at 11:30:00.

   The following table shows the changes in components after HA is enabled.

   Link type	Link or component status before upgrade	After upgrade (HA disabled)	After upgrade (HA enabled)
   Migration	The link and components run normally.	The link runs normally. Exceptions occur in the Store and JDBCWriter components.	JDBCWriter is automatically started.
   Migration	The link fails.	The link fails. Exceptions occur in the Store and JDBCWriter components.	JDBCWriter is automatically started. You must manually recover the link.
   Synchronization	The link and components run normally.	The link runs normally. Exceptions occur in the Store and Connector components.	Data is pulled from the Store component at the interval specified by the refetchStoreIntervalMin parameter. The default interval is 30 minutes. The Connector component is restarted.
   Synchronization	The link is suspended. The Connector component is suspended. The Store component runs normally.	The link is suspended. The Connector component is suspended. The Store component becomes abnormal.	Data is pulled from the Store component at the interval specified by the refetchStoreIntervalMin parameter.`` The default interval is 30 minutes. The Connector component is restarted. You must manually recover the link.

9. Optional. To roll back, perform the following steps:

   1. Follow the instructions in Step 1 to disable HA.

   2. Suspend the new container and record the time T3.

      sudo docker stop ${CONTAINER_NAME}
      

   3. Connect to the MetaDB and run the following commands:

      drop database rm_331;
      drop database cm_331;
      drop database cm_hb_331;
      
      create database rm_331;
      create database cm_331;
      create database cm_hb_331;
      

   4. Recover the original databases based on the SQL files created in Step 2.

      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/rm_331.sql" -Drm_331
      
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/cm_331.sql" -Dcm_331
      
      mysql -hxxx.xxx.xxx.1 -P<port> -u<username> -p<password> -e "source /home/admin/cm_hb_331.sql" -Dcm_hb_331
      

   5. Restart the container of OMS V3.3.1-CE.

      sudo docker restart ${CONTAINER_NAME}
      

   6. On the System Parameters page, enable HA and set the refetchStoreIntervalMin parameter.

   When you perform the rollback operation, the HA feature supports automatic disaster recovery. However, you may need to manually recover the JDBCWriter or connector component.

10. After the upgrade is complete, clear the browser cache before you log on to OMS.