Scale-out OMS

This topic describes how to scale out OceanBase Migration Service (OMS) from a single region or a single node.

Scale out OMS from a single node

The following two scenarios are possible:

• A virtual IP (VIP) address is specified for the cm_url parameter of the initial node.

• The IP address of a physical server is specified for the cm_url parameter of the initial node.

A VIP is specified for the cm_url parameter in the initial environment

Perform the same steps as in single-node deployment. For more information, see Deploy OMS on a single node.

The following example shows how to change the value of the cm_nodes parameter in the config.yaml configuration file form xxx.xxx.xxx.1 for the initial node to xxx.xxx.xxx.2 for the node after scale-out.

# Information about the OMS MetaDB
oms_meta_host: ${oms_meta_host}
oms_meta_port: ${oms_meta_port}
oms_meta_user: ${oms_meta_user}
oms_meta_password: ${oms_meta_password}

# When you scale out OMS from one node to multiple nodes, make sure that the names of the following three databases are consistent with those of the initial node.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}

# Configurations of the OMS cluster
cm_url: http://VIP:8088
cm_location: ${cm_location}
cm_region: ${cm_region}
cm_region_cn: ${cm_region_cn}
cm_is_default: true
cm_nodes:
 - xxx.xxx.xxx.1
 - xxx.xxx.xxx.2

# Configurations of the time-series database
# Default value: false. To enable metric reporting, set the parameter to `true` and delete the comments for the parameter.
# tsdb_enabled: false 
# If the `tsdb_enabled` parameter is set to `true`, delete comments for the following parameters and specify the values based on your actual configurations.
# tsdb_service: 'INFLUXDB'
# tsdb_url: '${tsdb_url}'
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

The IP address of a physical server is specified for the cm_url parameter of the initial environment

In this scenario, you need to perform the following operations to modify the database configurations:

1. Configure the VIP, SLB, or domain name and bind the IP addresses of all physical servers in the region.

   For example, the VIP is bound to physical servers whose IP addresses are xxx.xxx.xxx.1 and xxx.xxx.xxx.2.

2. Stop the processes in the original container. Sample statement for the oms_330 container:

   sudo docker exec -it oms_330 bash  
   supervisorctl stop all
   

3. Use the drc_rm_db database and the cluster_info table as examples. Log on to the database and perform the following operations:

   1. Query and back up the data in the cluster_info table.

      select * from cluster_info;
      

   2. Delete the data in the cluster_info table.

      delete from cluster_info;
      

4. Modify the configuration file in the oms_330 container.

   # The /home/admin/conf/config.yaml directory is fixed in Docker.
   sudo vi /home/admin/conf/config.yaml
   
   # Modify cm_url and add cm_nodes.
   cm_url: http://VIP:8088
   cm_nodes:
    - xxx.xxx.xxx.1
    - xxx.xxx.xxx.2
   

5. Reinitialize the data.

   sudo sh /root/docker_init.sh
   

   After the initialization is completed, check whether the value of the cm_url parameter in the cluster_info table of the database is consistent with the VIP.

6. Replicate the configuration file and start the OMS container on another host.

   Notice

   When you start the container, the value of -e OMS_HOST_IP must be the IP address of the current host.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate in the OMS container, you must set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

Scale out OMS from a single region

The procedure is generally the same as that for deploying OMS on a single node. A difference is that the drc_cm_heartbeat_db parameter in the config.yaml file must be different from the database used by the initial OMS host. In addition, you need to modify the following parameters for the new region: cm_url, cm_location, cm_region, cm_region_cn, cm_is_default, and cm_nodes.

1. Log on to the server where OMS is to be deployed.

2. Optional. Deploy a time-series database.

   If you need to collect and display OMS monitoring data, deploy a time-series database. Otherwise, you can skip this step. For more information, see Deploy a time-series database.

3. Prepare the configuration file.

   Edit the OMS configuration file in the appropriate directory. For example, you can create the config.yaml configuration file in the /root/ directory.

   You must replace the sample values of required parameters based on your actual deployment environment. The following example describes how to set the config.yaml file and scale out OMS deployed on the host xxx.xxx.xxx.1 in the region Jiangsu:

   Notice

   In the config.yaml file, you must specify the parameters in the key: value format, with a space after the colon (:).

   # Information about the OMS MetaDB
   oms_meta_host: ${oms_meta_host}
   oms_meta_port: ${oms_meta_port}
   oms_meta_user: ${oms_meta_user}
   oms_meta_password: ${oms_meta_password}
   
   # Keep the values of the drc_rm_db and drc_cm_db parameters consistent with those of the initial node.
   drc_rm_db: ${drc_rm_db}
   drc_cm_db: ${drc_cm_db}
   # The name of the drc_cm_heartbeat_db database must be different from that of the heartbeat_db database for the initial node.
   drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}
   
   # Configurations of the OMS cluster
   # Specify the following parameters for the region to be scaled.
   cm_url: http://xxx.xxx.xxx.1:8088
   cm_location: ${cm_location}
   cm_region: cn-jiangsu
   cm_region_cn: cn-jiangsu
   cm_is_default: true
   cm_nodes:
    - xxx.xxx.xxx.1
   
   # Configurations of the time-series database
   # Default value: false. To enable metric reporting, set the parameter to `true` and delete the comments for the parameter.
   # tsdb_enabled: false 
   # If the `tsdb_enabled` parameter is set to `true`, delete comments for the following parameters and specify the values based on your actual configurations.
   # tsdb_service: 'INFLUXDB'
   # tsdb_url: '${tsdb_url}'
   # tsdb_username: ${tsdb_user}
   # tsdb_password: ${tsdb_password}
   

   Parameter	Description	Required
   oms_meta_host	The IP address of the MetaDB, which can be the IP address of a MySQL database or a MySQL tenant of OceanBase Database. Notice: This parameter is valid only in OceanBase Database V2.0 and later.	Yes
   oms_meta_port	The port number of the MetaDB.	Yes
   oms_meta_user	The username of the MetaDB.	Yes
   oms_meta_password	The user password of the MetaDB.	Yes
   drc_rm_db	The name of the database for the OMS console. The name must be the same as that for the initial node.	Yes
   drc_cm_db	The name of the MetaDB for the CM service. The name must be the same as that for the initial node.	Yes
   drc_cm_heartbeat_db	The name of the heartbeat database for the CM service. Specify a name that is different from that of the heartbeat_db database for the initial node.	Yes
   cm_url	The URL of the OMS CM service. Example: http://xxx.xxx.xxx.1:8088. **Note:**In single-node deployment mode, the IP address of the server where OMS is deployed is used. We recommend that you do not set it to http://127.0.0.1:8088, which cannot be used for scaling out to multiple nodes in multiple regions.	Yes
   cm_location	The code of the region. Value range: [0,127]. You can select one number for each region.	Yes
   cm_region	The name of the region. Example: cn-jiangsu. **Notice:**If you use OMS with the Alibaba Cloud Multi-Site High Availability (MSHA) service in an active-active disaster recovery scenario, use the region configured for the Alibaba Cloud service.	No
   cm_region_cn	The value here is the same as the value of cm_region.	No
   cm_nodes	The IP addresses of servers used for OMS CM service. In examples in this topic, the value is xxx.xxx.xxx.1.	Yes
   cm_is_default	Indicates whether the OMS CM service is enabled by default. Notice When multiple regions exist, the cm_is_default parameter of only one region can be set to true, which indicates that the OMS CM service is enabled.	No. Default value: true.
   tsdb_enabled	Indicates whether metric reporting is enabled for monitoring. Valid values: true and false.	No. Default value: false.
   tsdb_service	The type of the time-series database. Valid values: INFLUXDB and CERESDB.	No. Default value: CERESDB.
   tsdb_url	The IP address of the server where InfluxDB is deployed. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	No
   tsdb_username	The username used to connect to the time-series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true. After you deploy the time-series database, manually create a user and specify the username and password.	No
   tsdb_password	The password used to connect to the time-series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	No

4. Load the downloaded OMS installation package to the local image repository of the Docker container.

   docker load -i <OMS installation package>
   

5. Run the following command to start the container.

   You can access the OMS console by using an HTTP or HTTPS URL. To securely access the OMS console, install a Secure Socket Layer (SSL) certificate and mount it to the specified directory in the container. The certificate is not required for HTTP access.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate in the OMS container, you must set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

   Parameter	Description
   OMS_HOST_IP	The IP address of the host.
   CONTAINER_NAME	The name of the container in the oms_xxx format. Specify xxx based on the actual OMS version. For example, if you use OMS V3.1.0, the value is oms_310.
   IMAGE_TAG	The unique identifier of the loaded image. After you load the OMS installation package by using Docker, run the docker images command to obtain the [IMAGE ID] or [REPOSITORY:TAG] of the loaded image. The obtained value is the unique identifier of the loaded image.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	You can replace /data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run with the mount directories created on the server where OMS is deployed. The mount directories store the logs generated during the operation of OMS and files generated by the store and synchronization components, respectively, to persistently retain the files on the server. Notice: The mount directories must remain unchanged during subsequent redeployment or upgrades.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are default directories in the container and cannot be modified.
   /data/oms/https_crt (optional) /data/oms/https_key (optional)	The mount directory of the SSL certificate in the OMS container. Specify the directory based on your actual configurations. If you mount an SSL certificate, the NGINX service in the OMS container runs in HTTPS mode. In this case, you can access the OMS console by using only the HTTPS URL.
   privileged	Specifies whether to grant extended privileges on the container.
   pids-limit	Specifies whether to limit the number of container processes. The value -1 indicates that the number is unlimited.
   ulimit nproc	The maximum number of user processes.

6. Go to the new container.

   docker exec -it  ${CONTAINER_NAME} bash  
   

   Notice

   CONTAINER_NAME specifies the name of the container. in the oms_xxx format. Specify xxx based on the actual OMS version. For example, if you use OMS V3.1.0, the value is oms_310.

7. Perform metadata initialization in the root directory.

   bash /root/docker_init.sh
   

   After you run the command, the following initialization process is performed:

   1. Initialize the data in the MetaDB.

   2. Generate configuration files for respective components.

   3. Restart all components.

   4. Initialize OMS resource tags and resource groups.

   In the execution of the docker_init.sh script, pay attention to the output of command lines. When the execution is completed, the following message is displayed: [Completed] All initialization steps are executed.

   Notice

   The execution takes 2 to 4 minutes to complete. Be patient and do not interrupt the process.