Scale out OMS Community Edition

This article introduces how to scale out OMS Community Edition from a single region, a single node, and multiple nodes in a single region.

Scale out OMS Community Edition from a single node

The following two scenarios are possible:

• Scenario 1: A virtual IP address (VIP) is specified for the cm_url parameter of the initial node.

• Scenario 2: The IP address of a physical server is specified for the cm_url parameter of the initial node.

Scenario 1

Perform the same steps as in single-node deployment. For more information, see Deploy OMS Community Edition on a single node.

Here is a sample config.yaml configuration file where the IP address of the original node is xxx.xxx.xxx.1 and that of the added node is xxx.xxx.xxx.2

# Information about the MetaDB cluster for OMS Community Edition
oms_meta_host: ${oms_meta_host}
oms_meta_port: ${oms_meta_port}
oms_meta_user: ${oms_meta_user}
oms_meta_password: ${oms_meta_password}

# When you scale out OMS Community Edition from one node to multiple nodes, make sure that the names of the following three databases are consistent with those of the initial node.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}

# Configurations of OMS Community Edition
cm_url: http://VIP:8088
cm_location: ${cm_location}
cm_region: ${cm_region}
cm_region_cn: ${cm_region_cn}
cm_is_default: true
cm_nodes:
 - xxx.xxx.xxx.1
 - xxx.xxx.xxx.2

# Configurations of the time-series database
# The default value of `tsdb_enabled`, which specifies whether to configure a time-series database, is `false`. To enable metric reporting, set the parameter to `true` and delete the comments for the parameter.
# tsdb_enabled: false 
# If the `tsdb_enabled` parameter is set to `true`, delete comments for the following parameters and specify the values based on your actual configurations.
# tsdb_service: 'INFLUXDB'
# tsdb_url: '${tsdb_url}'
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

Scenario 2

In this scenario, you need to correct data as follows:

1. Configure the VIP, Server Load Balancer (SLB), or domain name and bind the IP addresses of all physical servers in the region.

   For example, the VIP is bound to physical servers whose IP addresses are xxx.xxx.xxx.1 and xxx.xxx.xxx.2.

2. Stop the processes in the original container. Here is an example of stopping the processes in the oms_330 container:

   sudo docker exec -it oms_330 bash  
   supervisorctl stop all
   

3. Take the drc_rm_db database and the cluster_info table for example. Log in to the database and perform the following operations:

   1. Query and back up the data in the cluster_info table.

      select * from cluster_info;
      

   2. Delete the data from the cluster_info table.

      delete from cluster_info;
      

4. Modify the configuration file in the oms_330 container.

   # The /home/admin/conf/config.yaml directory is fixed in Docker.
   sudo vi /home/admin/conf/config.yaml
   
   # Modify cm_url and cm_nodes.
   cm_url: http://VIP:8088
   cm_nodes:
    - xxx.xxx.xxx.1
    - xxx.xxx.xxx.2
   

5. Reinitialize the data.

   sudo sh /root/docker_init.sh
   

   After the initialization is completed, check whether the value of the cm_url parameter in the cluster_info table of the database is consistent with the VIP.

6. Replicate the configuration file and start the container of OMS Community Edition on another server.

   Notice

   • When you start the container, the value of -e OMS_HOST_IP must be the IP address of the current server.

   • You need to replace work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} with the image name specified in the docker load -i command.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate to the container of OMS Community Edition, you must set the following two parameters:
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

Scale out OMS Community Edition from a single region

The procedure is generally the same as that for deploying OMS on a single node. A difference is that the value of the drc_cm_heartbeat_db parameter in the config.yaml file must be different from the name of the heartbeat_db database used by the initial node of OMS Community Edition. In addition, you need to modify the following parameters for the new region: cm_url, cm_location, cm_region, cm_region_cn, cm_is_default, and cm_nodes.

1. Log in to the server where OMS Community Edition is to be deployed.

2. (Optional) Deploy a time-series database.

   If you need to collect and display the monitoring data of OMS Community Edition, deploy a time-series database. Otherwise, you can skip this step. For more information, see Deploy a time-series database.

3. Prepare the configuration file.

   Edit the configuration file of OMS Community Edition in the appropriate directory. For example, you can create the config.yaml configuration file in the /root/ directory.

   You must replace the sample values of required parameters based on your actual deployment environment. The following example describes how to set the config.yaml file and scale out OMS Community Edition deployed on the host xxx.xxx.xxx.xx1 in the Jiangsu region:

   Notice

   In the config.yaml file, you must specify the parameters in the key: value format, with a space after the colon (:).

   # Information about the MetaDB cluster for OMS Community Edition
   oms_meta_host: ${oms_meta_host}
   oms_meta_port: ${oms_meta_port}
   oms_meta_user: ${oms_meta_user}
   oms_meta_password: ${oms_meta_password}
   
   # Keep the values of the drc_rm_db and drc_cm_db parameters consistent with those of the initial node.
   drc_rm_db: ${drc_rm_db}
   drc_cm_db: ${drc_cm_db}
   # The name of the drc_cm_heartbeat_db database must be different from that of the heartbeat_db database for the initial node.
   drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}
   
   # Configurations of OMS Community Edition
   # Specify the following parameters for the region to be scaled.
   cm_url: http://xxx.xxx.xxx.1:8088
   cm_location: ${cm_location}
   cm_region: cn-jiangsu
   cm_region_cn: cn-jiangsu
   cm_is_default: true
   cm_nodes:
    - xxx.xxx.xxx.1
   
   # Configurations of the time-series database
   # The default value of `tsdb_enabled`, which specifies whether to configure a time-series database, is `false`. To enable metric reporting, set the parameter to `true` and delete the comments for the parameter.
   # tsdb_enabled: false 
   # If the `tsdb_enabled` parameter is set to `true`, delete comments for the following parameters and specify the values based on your actual configurations.
   # tsdb_service: 'INFLUXDB'
   # tsdb_url: '${tsdb_url}'
   # tsdb_username: ${tsdb_user}
   # tsdb_password: ${tsdb_password}
   

   Parameter	Description	Required?
   oms_meta_host	The IP address of the MetaDB cluster, which can be that of a MySQL database or a MySQL-compatible tenant of OceanBase Database Community Edition.	Yes
   oms_meta_port	The port number of the MetaDB cluster.	Yes
   oms_meta_user	The username used for accessing the MetaDB cluster.	Yes
   oms_meta_password	The user password used for accessing the MetaDB cluster.	Yes
   drc_rm_db	The name of the database for the console of OMS Community Edition. The name must be the same as the setting for the initial node.	Yes
   drc_cm_db	The name of the database for the CM service. The name must be the same as the setting for the initial node.	Yes
   drc_cm_heartbeat_db	The name of the heartbeat database for the CM service. Specify a name that is different from that of the heartbeat_db database for the initial node.	Yes
   cm_url	The URL of the CM service of OMS Community Edition, for example, http://xxx.xxx.xxx.1:8088. Note In single-node deployment mode, the URL of the CM service is generally set to the IP address of the server where OMS Community Edition is deployed. We recommend that you do not set it to http://127.0.0.1:8088, which cannot be used for scaling out to multiple nodes in multiple regions.	Yes
   cm_location	The code of the region. Value range: [0, 127]. You can select one number for each region.	Yes
   cm_region	The name of the region, for example, cn-jiangsu. Notice If you use OMS Community Edition in combination with the Alibaba Cloud Multi-Site High Availability (MSHA) service in an active-active disaster recovery scenario, use the region configured for the Alibaba Cloud service.	No
   cm_region_cn	The value here is the same as the value of cm_region.	No
   cm_nodes	The IP addresses of servers on which the CM service of OMS Community Edition is deployed. In this example, the value is xxx.xxx.xxx.1.	Yes
   cm_is_default	Specifies whether the CM service is enabled for OMS Community Edition by default. Notice When multiple regions exist, the cm_is_default parameter can be set to true only for one region, which indicates that the CM service is enabled for OMS Community Edition by default.	No. Default value: true.
   tsdb_enabled	Specifies whether metric reporting is enabled for monitoring. Valid values: true and false.	No. Default value: false.
   tsdb_service	The type of the time-series database. Valid values: INFLUXDB and CERESDB.	No. Default value: INFLUXDB.
   tsdb_url	The IP address of the server where InfluxDB is deployed. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	No
   tsdb_username	The username used for accessing the time-series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true. After you deploy a time-series database, manually create a user and specify the username and password.	No
   tsdb_password	The password used for accessing the time-series database. You need to modify this parameter based on the actual environment if you set the tsdb_enabled parameter to true.	No

4. Load the downloaded installation package of OMS Community Edition to the local image repository of the Docker container.

   docker load -i <Installation package of OMS Community Edition>
   

5. Run the following command to start the container.

   You can access the console of OMS Community Edition by using an HTTP or HTTPS URL. To securely access the console of OMS Community Edition, install an SSL certificate and mount it to the specified directory in the container. The certificate is not required for HTTP access.

   OMS_HOST_IP=xxx
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # If you mount the SSL certificate to the container of OMS Community Edition, you must set the following two parameters:
   -v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   -v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

   Parameter	Description
   OMS_HOST_IP	The IP address of the host.
   CONTAINER_NAME	The name of the container, in the oms_xxx format. Specify xxx based on the actual OMS version.
   IMAGE_TAG	The unique identifier of the loaded image. After you load the installation package of OMS Community Edition by using Docker, run the docker images command to obtain the [IMAGE ID] or [REPOSITORY:TAG] value of the loaded image. The obtained value is the unique identifier (<OMS_IMAGE>) of the loaded image. Notice You need to replace work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} with the image name specified in the docker load -i command.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	/data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run can be replaced with the mount directories created on the server where OMS Community Edition is deployed to respectively store the runtime log files of OMS Community Edition, the files generated by the Store component, and the files generated by the Incr-Sync component, for local data persistence. Notice The mount directories must remain unchanged during subsequent redeployment or upgrades.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are default directories in the container and cannot be modified.
   /data/oms/https_crt (optional) /data/oms/https_key (optional)	The mount directory of the SSL certificate in the container of OMS Community Edition. Specify the directory based on the actual situation. If you mount an SSL certificate, the Nginx service in the container of OMS Community Edition runs in HTTPS mode. In this case, you can access the console of OMS Community Edition by using only the HTTPS URL.
   privileged	Specifies whether to grant extended privileges on the container.
   pids-limit	Specifies whether to limit the number of container processes. The value -1 indicates that the number is unlimited.
   ulimit nproc	The maximum number of user processes.

6. Go to the new container.

   docker exec -it  ${CONTAINER_NAME} bash  
   

   Notice

   CONTAINER_NAME specifies the name of the container.

7. Perform metadata initialization in the root directory.

   bash /root/docker_init.sh
   

   The initialization process is as follows:

   1. Initialize the data in the MetaDB.

   2. Generate configuration files for the components.

   3. Restart all components.

   4. Initialize the resource tags and resource groups of OMS Community Edition.

   During the execution of the docker_init.sh script, pay attention to the command output. When the execution is completed, the following message is displayed: [Completed] All initialization steps are executed.

   Notice

   The execution takes 2 to 4 minutes. Be patient and do not interrupt the execution process.

   Scale out OMS Community Edition from multiple nodes in a single region

Scale out OMS Community Edition from two nodes in a single region, a virtual IP address (VIP) is specified for cm_url

This example describes how to scale out OMS Community Edition V4.2.7-CE from multiple nodes in a single region docker_remote_deploy.sh.

1. Pull the OMS Community Edition V4.2.7-CE image and obtain the deployment script from the loaded image.

   sudo docker pull <OMS_IMAGE>
   
   sudo docker run -d --net host --name oms-config-tool <OMS_IMAGE> bash && sudo docker cp oms-config-tool:/root/docker_remote_deploy.sh . && sudo docker rm -f oms-config-tool
   

2. Run the deployment script to start the deployment tool.

sudo sh docker_remote_deploy.sh -o <OMS mount directory> -i <public IP address> -d <OMS_IMAGE>

3. Deploy OMS in a single region with two nodes based on the prompts from the tool. For more information, see Deploy OMS in a single region.

4. After the initialization is completed, log in to the OMS Community Edition console to verify.

Procedure

In this example, the cm_url parameter is set to the value of the VIP http://xxx.xxx.xxx.1:1188 based on the initial configuration, and the cm_nodes parameter is set to xxx.xxx.xxx.2 for the new node to be added. The procedure for scaling out OMS from multiple nodes in a single region is as follows.

1. Configure the VIP, SLB, or domain name and add the new node.

2. Copy the configuration file and add the new node to be expanded in the config.yaml file. For example, add xxx.xxx.xxx.2 to the cm_nodes parameter.

3. Run the following command to start a container.

   OMS_HOST_IP=xxx.xxx.xxx.2
   CONTAINER_NAME=oms_xxx
   IMAGE_TAG=feature_x.x.x-ce
   docker run -dit --net host \
   -v /data/config.yaml:/home/admin/conf/config.yaml \
   -v /data/oms/oms_logs:/home/admin/logs \
   -v /data/oms/oms_store:/home/ds/store \
   -v /data/oms/oms_run:/home/ds/run \
   # The following two parameters are required only if you mount the HTTPS certificate in the OMS Community Edition container.
   #-v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
   #-v /data/oms/https_key:/etc/pki/nginx/oms_server.key
   -e OMS_HOST_IP=${OMS_HOST_IP} \
   --privileged=true \
   --pids-limit -1 \
   
   --ulimit nproc=65535:65535 \
   --name ${CONTAINER_NAME} \
   work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}
   

   Parameter	Description
   OMS_HOST_IP	The IP address of the host. This parameter must be set to the IP address of the node to be expanded.
   CONTAINER_NAME	The name of the container. The value of this parameter must be in the format of oms_xxx, where xxx indicates the version number.
   IMAGE_TAG	After you use Docker to load the OMS Community Edition installation package, you can run the docker images command to obtain the [IMAGE ID] or [REPOSITORY:TAG] of the loaded image, which uniquely identifies the image. <OMS_IMAGE>. Notice Replace work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG} with the image name actually imported by running the docker load -i command.
   /data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	You can replace /data/oms/oms_logs, /data/oms/oms_store, and /data/oms/oms_run with mount directories created on the OMS Community Edition deployment host to store log files generated during OMS Community Edition operation, files generated by the log pulling component, and files generated by the synchronization component, respectively. These mount directories are used for persistent storage on the local host. Notice: The mount directories must not be changed when you redeploy or upgrade OMS in the later versions.
   /home/admin/logs /home/ds/store /home/ds/run	/home/admin/logs, /home/ds/store, and /home/ds/run are fixed directories in the container. The paths of these directories cannot be modified.
   /data/oms/https_crt (Optional) /data/oms/https_key (Optional)	The mount directory for the HTTPS certificate in the OMS Community Edition container. If you mount the HTTPS certificate, the Nginx service in the OMS Community Edition container runs in HTTPS mode. You can only use the console service of OMS Community Edition in HTTPS mode.
   privileged	Grants extended privileges to the container.
   pids-limit	Configures the process limit of the container. The value -1 indicates no limit.
   ulimit nproc	Configures the upper limit of user processes.

4. Go to the container.

      docker exec -it  ${CONTAINER_NAME} bash  
   

5. In the root directory, execute the metadata initialization operation.

   bash /root/docker_init.sh
   

6. After the initialization is completed, log in to the OMS Community Edition console to verify.