Scale out OMS Community Edition

This topic describes how to scale out OceanBase Migration Service (OMS) Community Edition from a single region or a single node.

Scale out OMS Community Edition from a single node

The following two scenarios are possible:

• A virtual IP (VIP) address is specified for the cm_url parameter of the initial node.

• The IP address of a physical server is specified for the cm_url parameter of the initial node.

A VIP is specified for the cm_url parameter in the initial environment

In this scenario, you can replicate the config.yaml configuration file and append one or more rows of values to the cm_nodes parameter. Other parameters and operations are the same as those of the initial node.

Notice:

When you start the container, the value of -e OMS_HOST_IP must be the IP address of the current host.

The following example shows how to change the value of the cm_nodes parameter in the config.yaml configuration file form xxx.xxx.xxx.1 for the initial node to xxx.xxx.xxx.2 for the node after scale-out.

# The information about OMS MetaDB.
oms_meta_host: ${oms_meta_host}
oms_meta_port: ${oms_meta_port}
oms_meta_user: ${oms_meta_user}
oms_meta_password: ${oms_meta_password}

# When you scale out OMS from one node to multiple nodes, make sure that the names of the following three databases are consistent with those of the initial node.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}

# The user that consumes the incremental data of OceanBase Database.
# To read the incremental logs of OceanBase Database, create the user in the SYS tenant.
drc_user: ${drc_user}
drc_password: '${drc_password}'

# Configure the OMS cluster.
cm_url: http://VIP:8088
cm_location: ${cm_location}
cm_region: ${cm_region}
cm_region_cn: ${cm_region_cn}
cm_is_default: true
cm_nodes:
 - xxx.xxx.xxx.1
 - xxx.xxx.xxx.2

# Configurations of the time series database.
# Default value: false. To enable metric reporting, set the parameter to true and delete comments for the parameter.
# tsdb_enabled: false 
# If the tsdb_enabled parameter is set to true, delete comments for the following parameters and specify the values based on your actual configurations.
# tsdb_service: 'INFLUXDB'
# tsdb_url: '${tsdb_url}'
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

The IP address of a physical server is specified for the cm_url parameter of the initial environment

In this scenario, you need to perform the following operations to modify the database configurations:

1. Configure the VIP, SLB, or domain name and bind the IP addresses of all physical servers in the region.

   For example, the VIP is bound to physical servers whose IP addresses are xxx.xxx.xxx.1 and xxx.xxx.xxx.2.

2. Stop the processes in the original container. Sample statement for the oms_330 container:

   sudo docker exec -it oms_330 bash  
   supervisorctl stop all
   

3. Use the drc_rm_db database and the cluster_info table as examples. Log on to the database and perform the following operations:

   1. Query and back up the data in the cluster_info table.

      select * from cluster_info;
      

   2. Delete the data in the cluster_info table.

      delete from cluster_info;
      

4. Modify the configuration file in the oms_330 container.

   # The /home/admin/conf/config.yaml directory is fixed in Docker.
   sudo vi /home/admin/conf/config.yaml
   
   # Modify cm_url and add cm_nodes.
   cm_url: http://VIP:8088
   cm_nodes:
    - xxx.xxx.xxx.1
    - xxx.xxx.xxx.2
   

5. Reinitialize the data.

   sudo sh /root/docker_init.sh
   

   After the initialization is completed, check whether the value of the cm_url parameter in the cluster_info table of the database is consistent with the VIP.

6. Replicate the configuration file and start the OMS container on another host.

   Notice:

   When you start the container, the value of -e OMS_HOST_IP must be the IP address of the current host.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate in the OMS container, you need to set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} 
   

Scale out OMS Community Edition from a single region

The procedure is generally the same as that for deploying OMS Community Edition on a single node. A difference is that the drc_cm_heartbeat_db parameter in the config.yaml file must be different from the database used by the initial OMS Community Edition host. In addition, you need to modify the following parameters for the new region: cm_url, cm_location, cm_region, cm_region_cn, cm_is_default, and cm_nodes.

1. Log on to the server where OMS Community Edition is deployed.

2. Optional. Deploy a time series database.

   If you need to collect and display OMS Community Edition monitoring data, deploy a time series database. Otherwise, you can skip this step. For more information, see Deploy a time series database.

3. Prepare the configuration file.

   Edit the OMS Community Edition configuration file in the appropriate directory. For example, you can create the config.yaml configuration file in the /root/ directory.

   You must replace the sample values of required parameters based on your actual deployment environment. The following example describes how to set the config.yaml file and scale out OMS Community Edition deployed on the host xxx.xxx.xxx.1 in the region Jiangsu.

   Notice:

   In the config.yaml file, you must specify the value in the parameter: value format. A space is required after the colon (:).

   # The information about OMS MetaDB.
   oms_meta_host: ${oms_meta_host}
   oms_meta_port: ${oms_meta_port}
   oms_meta_user: ${oms_meta_user}
   oms_meta_password: ${oms_meta_password}
   
   # Keep the values of the drc_rm_db and drc_cm_db parameters consistent with those of the initial node.
   drc_rm_db: ${drc_rm_db}
   drc_cm_db: ${drc_cm_db}
   # The name of the drc_cm_heartbeat_db database must be different from that of the heartbeat_db database for the initial node.
   drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}
   
   # The user that consumes the incremental data of OceanBase Database.
   # To read the incremental logs of OceanBase Database, create the user in the SYS tenant.
   drc_user: ${drc_user}
   drc_password: '${drc_password}'
   
   # Configure the OMS cluster.
   # Specify the following parameters for the region to be scaled.
   cm_url: http://xxx.xxx.xxx.1:8088
   cm_location: ${cm_location}
   cm_region: cn-jiangsu
   cm_is_default: true
   cm_nodes:
    - xxx.xxx.xxx.1
   
   # Configurations of the time series database.
   # Default value: false. To enable metric reporting, set the parameter to true and delete comments for the parameter.
   # tsdb_enabled: false 
   # If the tsdb_enabled parameter is set to true, delete comments for the following parameters and specify the values based on your actual configurations.
   # tsdb_service: 'INFLUXDB'
   # tsdb_url: '${tsdb_url}'
   # tsdb_username: ${tsdb_user}
   # tsdb_password: ${tsdb_password}
   

   Parameter	Description	Required or optional
   oms_meta_host	The IP address of MetaDB, which can be the IP address of a MySQL database or a MySQL tenant of OceanBase Database. Notice: OceanBase Database V2.0 or later is required.	Required
   oms_meta_port	The port number of MetaDB.	Required
   oms_meta_user	The username of MetaDB.	Required
   oms_meta_password	The password of the MetaDB username.	Required
   drc_rm_db	The name of the database for the OMS Community Edition console. The name must be the same as that for the initial node.	Required
   drc_cm_db	The name of the MetaDB for the CM service. The name must be the same as that for the initial node.	Required
   drc_cm_heartbeat_db	The name of the heartbeat database for the CM service. Specify a name that is different from that of the heartbeat_db database for the initial node.	Required
   drc_user	The user that reads the incremental logs of OceanBase Database. You need to create the user in the SYS tenant. For more information, see the "User privileges" topic in OMS Community Edition User Guide.	Optional
   drc_password	The password of the drc user.	Optional
   cm_url	The URL of the OMS Community Edition CM service. Example: http://xxx.xxx.xxx.1:8088. Note: In single-node deployment mode, the IP address of the server where OMS Community Edition is deployed is used. We recommend that you do not set it to http://127.0.0.1:8088, which cannot be used for scaling out to multiple nodes in multiple regions.	Required
   cm_location	The code of the region. Value range: [0,127]. You can select one number for each region.	Required
   cm_region	The name of the region. Example: cn-jiangsu. Notice: If you use OMS Community Edition with the Alibaba Cloud Multi-Site High Availability (MSHA) service in an active-active disaster recovery scenario, use the region configured for the Alibaba Cloud service.	Optional
   cm_nodes	The IP addresses of servers used for OMS Community Edition CM service. In examples in this topic, the value is xxx.xxx.xxx.1.	Required
   cm_is_default	Indicates whether the OMS Community Edition CM service is enabled by default. Notice: When multiple regions exist, the cm_is_default parameter of only one region can be set to true, which indicates that the OMS Community Edition CM service is enabled.	Optional. Default value: true
   tsdb_enabled	Indicates whether metric reporting is enabled for monitoring. Valid values: true and false.	Optional. Default value: false
   tsdb_service	The type of the time series database. Valid values: INFLUXDB and CERESDB.	Optional. Default value: CERESDB
   tsdb_url	The IP address of the server where InfluxDB is deployed. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	Optional
   tsdb_username	The username used to connect to the time series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true. After you deploy the time series database, you need to manually create a user and specify the username and password.	Optional
   tsdb_password	The password used to connect to the time series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	Optional

4. Load the downloaded OMS Community Edition image file to the local image repository of the Docker container.

   docker load -i <The storage path of the OMS Community Edition image>
   

5. Run the following command to start the container.

   You can access the OMS Community Edition console by using a HTTP or HTTPS URL. To securely access the OMS Community Edition console, install an SSL certificate and mount it to the specified directory in the container. The certificate is not required for HTTP access.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate in the OMS container, you need to set the following two parameters.
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} 
   

   Parameter	Description
   OMS_HOST_IP	The IP address of the host.
   CONTAINER_NAME	The name of the container in the oms_xxx format. Specify xxx based on the actual OMS Community Edition version.
   IMAGE_TAG	The tag of the image. To view the image tag, execute the docker images statement after you load the image on the host. The image tag is in the feature_x.x.x-ce format.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	You can replace /data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run with the mount directories created on the server where OMS Community Edition is deployed. The mount directories store the logs generated during the operating of OMS Community Edition and generated by stores and synchronization components, respectively, to persistently retain the logs on the server. Notice: The mount directories must remain unchanged during subsequent redeployment or upgrades.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are default directories in the container and cannot be modified.
   /data/oms/https_crt (optional) /data/oms/https_key (optional)	The mount directory of the SSL certificate in the OMS container. Specify the directory based on your actual configurations. If you mount an SSL certificate, the NGINX service in the OMS Community Edition container runs in HTTPS mode. In this case, you can access the OMS Community Edition console by using only the HTTPS URL.
   privileged	Specifies whether to grant the container scaling privilege.
   pids-limit	Specifies whether to limit the number of container processes. The value -1 indicates that the number is unlimited.
   ulimit nproc	The maximum number of user processes.

6. Access the new container.

   docker exec -it  ${CONTAINER_NAME} bash  
   

   Notice:

   CONTAINER_NAME represents the name of the container in the oms_xxx format. Specify xxx based on the actual OMS Community Edition version.

7. Perform metadata initialization in the root directory.

   bash /root/docker_init.sh
   

   After you run the command, the following initialization process is executed:

   1. Initialize the data in the MetaDB.

   2. Generate configuration files for respective components.

   3. Restart all components.

   4. Initialize OMS Community Edition resource tags and resource groups.

   In the execution of the docker_init.sh script, pay attention to the output of command lines. When the execution is completed, the following message is displayed: [Completed] All initialization steps are executed.

   Notice:

   The execution takes 2 to 4 minutes to complete. Be patient and do not interrupt the process.