Deploy OMS Community Edition on multiple nodes in a single region

This topic describes how to deploy OceanBase Migration Service (OMS) Community Edition on multiple nodes in a single region by using the deployment tool.

Background information

• You can deploy OMS Community Edition on a single node first and then scale out to multiple nodes.

• If you choose to deploy OMS Community Edition with the config.yaml configuration file, note that the settings are slightly different from those for the single-node deployment mode. For more information, see the "Template and example of a configuration file" section of this topic.

• To deploy OMS Community Edition on multiple nodes, you must apply for a virtual IP address (VIP) and use it as the mount point for the console of OMS Community Edition. In addition, you must configure the mapping rules for Ports 8088 and 8089 in the VIP-based network strategy.

  You can use the VIP to access the console of OMS Community Edition even if a node of OMS Community Edition fails.

Prerequisites

• The installation environment meets the system and network requirements. For more information, see System and network requirements.

• You have created a MetaDB cluster for OMS Community Edition.

• The server to deploy OMS Community Edition can connect to all other servers.

• All servers involved in the multi-node deployment can connect to each other and you can obtain root permissions on a node by using its username and password.

• You have obtained the installation package of OMS Community Edition, which is generally a tar.gz file whose name starts with oms. You can download the installation package from OceanBase Download Center.

• You have downloaded the installation package of OMS Community Edition and loaded it to the local image repository of the Docker container on each server node.

  docker load -i <Installation package of OMS Community Edition>
  

• You have prepared a directory for mounting the container of OMS Community Edition. In the mount directory, OMS Community Edition will create the /home/admin/logs, /home/ds/store, and /home/ds/run directories for storing the component information and logs generated during the running of OMS Community Edition.

• (Optional) You have prepared a time-series database for storing performance monitoring data and DDL/DML statistics of OMS Community Edition.

Terms

You need to replace variable names in some commands and prompts. A variable name is enclosed with angle brackets (<>).

• Mount directory of the container of OMS Community Edition: the directory to which the container of OMS Community Edition is mounted.

• IP address of the server: the IP address of the host that executes the script.

• OMS_IMAGE: the unique identifier of the loaded image. After you load the installation package of OMS Community Edition by using Docker, run the docker images command to obtain the [IMAGE ID] or [REPOSITORY:TAG] of the loaded image. The obtained value is the unique identifier of the loaded image. Here is an example:

  $sudo docker images
  REPOSITORY                               TAG                 IMAGE ID          
  work.oceanbase-dev.com/obartifact-store/oms feature_4.0.0-ce    2786e8a6eccd        
  

  In this example, <OMS_IMAGE> can be work.oceanbase-dev.com/obartifact-store/oms:feature_4.0.0-ce or 2786e8a6eccd. Replace the value of <OMS_IMAGE> in related commands with the preceding value.

• Directory of the config.yaml file: If you want to deploy OMS Community Edition based on an existing config.yaml configuration file, this directory is the one where the configuration file is located.

Multi-node deployment architecture

The following figure shows the multi-node deployment architecture, in which Store is the log pulling component and Incr-Sync is the incremental synchronization component. When a single point of failure (SPOF) occurs on Node A of OMS Community Edition, the Store and Incr-Sync components running on this node are guarded by the high availability (HA) service and dynamically switch to Node B or Node C of OMS Community Edition.

Deployment procedure without a configuration file

1. Log in to the server where OMS Community Edition is to be deployed.

2. (Optional) Deploy a time-series database.

   If you need to collect and display the monitoring data of OMS Community Edition, deploy a time-series database. Otherwise, you can skip this step. For more information, see Deploy a time-series database.

3. Run the following command to obtain the deployment script docker_remote_deploy.sh from the loaded image:

   sudo docker run -d --name oms-config-tool <OMS_IMAGE> bash && sudo docker cp oms-config-tool:/root/docker_remote_deploy.sh . && sudo docker rm -f oms-config-tool
   

   Here is an example:

   sudo docker run -d --net host --name oms-config-tool work.oceanbase-dev.com/obartifact-store/oms:feature_4.0-ce bash && sudo docker cp oms-config-tool:/root/docker_remote_deploy.sh . && sudo docker rm -f oms-config-tool
   

4. Use the deployment script to start the deployment tool.

   bash docker_remote_deploy.sh -o <Mount directory of the OMS Community Edition container> -i <IP address of the server> -d <OMS_IMAGE>
   

   The deployment tool of OMS Community Edition automatically verifies the CPU, memory, and disk resources. If any resource item does not meet the requirement, the deployment tool will display a message prompting that insufficient resources will affect the data migration speed.

5. Complete the deployment as prompted. After you set each parameter, press Enter to move on to the next parameter.

   1. Select a deployment mode.

      Select Multiple Nodes in Single Region.

   2. Select a task.

      Select No Configuration File. Deploy OMS Starting from Configuration File Generation.

   3. Enter the configuration information of the MetaDB cluster as follows:

      1. Enter the IP address, port, username, and password of the MetaDB cluster.

      2. Set a prefix for names of databases in the MetaDB cluster.

         For example, when the prefix is set to oms, the databases in the MetaDB cluster are named oms_rm, oms_cm, and oms_cm_hb.

      3. Confirm your settings.

         If the settings are correct, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

         If the system displays The specified database names already exist in the MetaDB. Are you sure that you want to continue?, it indicates that the database names you specified already exist in the MetaDB cluster. This may be caused by repeated deployment or upgrade of OMS Community Edition. You can enter y and press Enter to proceed, or enter n and press Enter to re-specify the settings.

         If you enter y and press Enter, the message The specified database names already exist in the MetaDB. Do you want to reinitialize the MetaDB? is displayed. You can enter y and press Enter to reinitialize the MetaDB cluster.

   4. Enter the cluster configuration information for OMS Community Edition as follows:

      1. Set whether to define component ports and configure related parameters.

         You can enter y and press Enter to configure custom ports for components. Enter the ports of the Nginx, GHANA, CM, Supervisor, and sshd services as prompted.

      2. Specify the URL of the CM service, which is the VIP or domain name to which all CM servers in the region are mounted. The corresponding parameter in the configuration file is cm-url.

         Enter the VIP or domain name as the URL of the CM service. You can enter the IP address and port number in the URL in sequence, or use a colon (:) to join the IP address and port number in the <IP address>:<port number> format.

         Note

         The http:// prefix in the URL is optional.

      3. Enter the IP addresses of all servers in the region. Separate them with commas (,).

      4. Confirm whether the displayed settings of OMS Community Edition cluster are correct.

         If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

   5. Determine whether to monitor historical data of OMS Community Edition.

      • If you have deployed a time-series database in Step 2, enter y and press Enter to go to the step of configuring the time-series database and enable monitoring of historical data for OMS Community Edition.

      • If you chose not to deploy a time-series database in Step 2, enter n and press Enter to go to the step of determining whether to enable the audit log feature and configure Simple Log Service (SLS) parameters. In this case, OMS Community Edition does not monitor the historical data after deployment.

   6. Configure the time-series database.

      Perform the following operations:

      1. Confirm whether you have deployed a time-series database.

         Enter the value based on the actual situation. If yes, enter y and press Enter. If no, enter n and press Enter to go to the step of "determining whether to enable the audit log feature and setting SLS parameters".

      2. Set the type of the time-series database to INFLUXDB.

         Notice

         At present, only INFLUXDB is supported.

      3. Enter the URL, username, and password of the time-series database as prompted.

      4. Confirm whether the displayed settings are correct.

         If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

   7. Determine whether to enable the audit log feature and write the audit logs to SLS.

      To enable the audit log feature, enter y and press Enter to go to the next step to specify the SLS parameters.

      Otherwise, enter n and press Enter to start the deployment. In this case, OMS Community Edition does not audit the logs after deployment.

   8. Specify the SLS parameters.

      1. Set the SLS parameters as prompted.

      2. Confirm whether the displayed settings are correct.

      If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

   9. If the configuration file passes the check, all the settings are displayed. If the settings are correct, enter n and press Enter to proceed. Otherwise, enter y and press Enter to modify the settings.

      If the configuration file fails the check, modify the settings as prompted.

   10. Complete deployment settings on each node as follows:

       1. Specify the directory to which OMS Community Edition is mounted on the node.

          Specify a directory with a large capacity.

       2. Confirm whether the image file of OMS Community Edition can be named <OMS_IMAGE>.

          If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

       3. Determine whether to mount an SSL certificate to the container of OMS Community Edition.

          If yes, enter y, press Enter, and specify the https_key and https_crt directories as prompted. Otherwise, enter n and press Enter.

       4. If the current node is a remote node, the system finds that the IP address of the deployment mode is inconsistent with the IP address of the server where the deployment tool resides, and asks you whether to remotely execute the script. This requires that the remote node support SSH connection and you have the sudo privilege on the remote node.

          If you enter y, check whether the port of the node can be connected through an SSH connection. If SSH connection is supported, enter the username and password for logging in to the remote node as prompted.

          If you leave the password empty, the system will prompt that the password is an empty string and ask you whether to use trusted login. If you enter y, enter the password of the sudo user again. If no password is required, directly press Enter.

       5. Confirm whether the path to which the config.yaml configuration file will be written is correct.

       If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

   If the deployment fails, you can log in to the container of OMS Community Edition and view logs in the .log files prefixed with docker_init in the /home/admin/logs directory. If the container of OMS Community Edition fails to be started, you cannot obtain logs.

Deployment procedure with a configuration file

1. Log in to the server where OMS Community Edition is to be deployed.

2. (Optional) Deploy a time-series database.

   If you need to collect and display the monitoring data of OMS Community Edition, deploy a time-series database. Otherwise, you can skip this step. For more information, see Deploy a time-series database.

3. Run the following command to obtain the deployment script docker_remote_deploy.sh from the loaded image:

   sudo docker run -d --name oms-config-tool <OMS_IMAGE> bash && sudo docker cp oms-config-tool:/root/docker_remote_deploy.sh . && sudo docker rm -f oms-config-tool
   

4. Use the deployment script to start the deployment tool.

   bash docker_remote_deploy.sh -o <Mount directory of the OMS Community Edition container> -c <Directory of the existing config.yaml file> -i <IP address of the server> -d <OMS_IMAGE>
   

   The deployment tool of OMS Community Edition automatically verifies the CPU, memory, and disk resources. If any resource item does not meet the requirement, the deployment tool will display a message prompting that insufficient resources will affect the data migration speed.

   Note

   • The value of the -c option must be the absolute path of the config.yaml configuration file.
   • For more information about settings of the config.yaml file, see the "Template and example of a configuration file" section.

5. Complete the deployment as prompted. After you set each parameter, press Enter to move on to the next parameter.

   1. Select a deployment mode.

      Select Multiple Nodes in Single Region.

   2. Select a task.

      Select Use Configuration File Uploaded with Script Option [-c].

   3. If the system displays The specified database names already exist in the MetaDB. Are you sure that you want to continue?, it indicates that the database names you specified already exist in the MetaDB cluster in the original configuration file. This may be caused by repeated deployment or upgrade of OMS Community Edition. You can enter y and press Enter to proceed, or enter n and press Enter to re-specify the settings.

   4. If the configuration file passes the check, all the settings are displayed. If the settings are correct, enter n and press Enter to proceed. Otherwise, enter y and press Enter to modify the settings.

      If the configuration file fails the check, modify the settings as prompted.

   5. Complete deployment settings on each node as follows:

      1. Specify the directory to which the container of OMS Community Edition is mounted on the node.

         Specify a directory with a large capacity.

         For a remote node, the username and password for logging in to the remote node are required. The corresponding user account must have the sudo privilege on the remote node.

      2. Confirm whether the image file of OMS Community Edition can be named <OMS_IMAGE>.

         If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

      3. Determine whether to mount an SSL certificate to the container of OMS Community Edition.

         If yes, enter y, press Enter, and specify the https_key and https_crt directories as prompted. Otherwise, enter n and press Enter.

      4. If the current node is a remote node, the system finds that the IP address of the deployment mode is inconsistent with the IP address of the server where the deployment tool resides, and asks you whether to remotely execute the script. This requires that the remote node support SSH connection and you have the sudo privilege on the remote node.

         If you enter y, check whether the port of the node can be connected through an SSH connection. If SSH connection is supported, enter the username and password for logging in to the remote node as prompted.

         If you leave the password empty, the system will prompt that the password is an empty string and ask you whether to use trusted login. If you enter y, enter the password of the sudo user again. If no password is required, directly press Enter.

      5. Confirm whether the path to which the config.yaml configuration file will be written is correct.

         If yes, enter y and press Enter to proceed. Otherwise, enter n and press Enter to modify the settings.

      If the deployment fails, you can log in to the container of OMS Community Edition and view logs in the .log files prefixed with docker_init in the /home/admin/logs directory. If the container of OMS Community Edition fails to be started, you cannot obtain logs.

To modify the configuration after deployment, perform the following steps:

1. Log in to the container of OMS Community Edition.

2. Modify the config.yaml file in the /home/admin/conf/ directory based on business needs.

3. Run the python -m omsflow.scripts.units.oms_init_manager --init-config-file command.

4. Run the supervisorctl restart oms_console oms_drc_supervisor command.

Template and example of a configuration file

Configuration file template

# Information about the MetaDB cluster for OMS Community Edition
oms_meta_host: ${oms_meta_host}
oms_meta_port: ${oms_meta_port}
oms_meta_user: ${oms_meta_user}
oms_meta_password: ${oms_meta_password}
     
# You can define the names of the following three databases, which are created in the MetaDB cluster during the deployment of OMS Community Edition.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}
     
# Configurations of OMS Community Edition
# To deploy OMS Community Edition on multiple nodes in a single region, you must set the cm_url parameter to a VIP or domain name to which all CM servers in the region are mounted.
cm_url: ${cm_url}
cm_location: ${cm_location}
# The cm_region parameter is not required for single-region deployment.
# cm_region: ${cm_region}
# The cm_region_cn parameter is not required for single-region deployment.
# cm_region_cn: ${cm_region_cn}
cm_is_default: true
cm_nodes:
 - ${host_ip1}
 - ${host_ip2}
     
# Configurations of the time-series database
# The default value of `tsdb_enabled`, which specifies whether to configure a time-series database, is `false`. To enable metric reporting, set the parameter to `true` and delete the comments for the parameter.
# tsdb_enabled: false 
# If the `tsdb_enabled` parameter is set to `true`, delete comments for the following parameters and specify the values based on your actual configurations.
# tsdb_service: 'INFLUXDB'
# tsdb_url: '${tsdb_url}'
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

# Custom component ports
# Specify the port number for the GHANA service.
ghana_server_port: xxxxx
# Specify the port number for the Nginx service.
nginx_server_port: xxxxx
# Specify the port number for the CM service.
cm_server_port: xxxxx
# Specify the port number for the Supervisor service.
supervisor_server_port: xxxxx
# Specify the port number for the sshd service.
sshd_server_port: xxxxx

Sample configuration file

Replace related parameters with the actual values in the target deployment environment.

oms_meta_host: xxx.xxx.xxx.xxx
oms_meta_port: 2883
oms_meta_user: oms_meta_user
oms_meta_password: *********
drc_rm_db: oms_rm
drc_cm_db: oms_cm
drc_cm_heartbeat_db: oms_cm_heartbeat
cm_url: http://VIP:8088
cm_location: 100
cm_region: cn-anhui
cm_region_cn: cn-anhui
cm_is_default: true
cm_nodes:
 - xxx.xxx.xxx.xx1
 - xxx.xxx.xxx.xx2
tsdb_service: 'INFLUXDB'
tsdb_enabled: true
tsdb_url: 'xxx.xxx.xxx.xxx:8086'
tsdb_username: username
tsdb_password: ********
ghana_server_port: xxxxx
nginx_server_port: xxxxx
cm_server_port: xxxxx
supervisor_server_port: xxxxx
sshd_server_port: xxxxx