Deploy OMS Community Edition on multiple nodes in multiple regions|V3.3.1|OceanBase Migration Service| docs|Distributed Database

This topic describes how to deploy OceanBase Migration Service (OMS) Community Edition on multiple nodes in multiple regions.

Background

As more users apply OMS Community Edition in data migration, OMS Community Edition must adapt to increasingly diverse scenarios. In addition to single-region data migration and data synchronization, OMS Community Edition supports data synchronization across regions, data migration between IDCs in different regions, and active-active data synchronization.

You can deploy OMS Community Edition on one or more nodes in each region. OMS Community Edition can be deployed on multiple nodes in a region to build an environment with high availability. In this way, OMS Community Edition can start components on appropriate nodes based on the tasks.

For example, if you want to synchronize data from the region Hangzhou to Heilongjiang, OMS Community Edition starts stores on a node in the Hangzhou region to collect the incremental logs and starts writers on a node in the Heilongjiang region to synchronize the data.

Notes:

You can deploy OMS Community Edition on a single node first and scale out to multiple nodes. For more information, see Scale out OMS Community Edition.
To deploy OMS Community Edition on multiple nodes, you must apply for a virtual IP (VIP) address and use it as the mount point for the OMS Community Edition console. In addition, you must configure the mapping rules of ports 8088 and 8089 in the VIP network strategy.

You can use the VIP to access the OMS Community Edition console even if an OMS Community Edition node fails.

Prerequisites

The installation environment meets the system requirements. For more information, see System requirements.
The MetaOB cluster is prepared as the OMS Community Edition MetaDB.
The OMS Community Edition installation package is obtained. Generally, the package is a tar.gz file whose name starts with oms.

Procedure

To display the monitoring curves of incremental migration, synchronization latency, and traffic in the OMS Community Edition console, deploy a time-series database before OMS Community Edition deployment. For more information, see Deploy a time-series database.

Otherwise, you can skip this step.

Notice:

The example in this topic describes how to deploy two nodes both in the Hangzhou and Heilongjiang regions.

Deploy OMS Community Edition in the Hangzhou region

Prepare the configuration file.

Edit the OMS Community Edition configuration file in the appropriate directory. For example, you can create the config.yaml configuration file in the /root/ directory.

You must replace the sample values of related parameters based on your actual deployment environment.

Notice

When multiple regions exist, you must set the cm_is_default parameter to true for only one region, and set it to false for all other regions. In addition, you must sequentially run commands in each region.
To deploy multiple nodes in the Hangzhou region, specify the IP addresses of all nodes for the cm_nodes parameter.
In the config.yaml file, you must specify the value in the parameter: value format. A space is required after the colon (:).

The following example describes a template of the config.yaml file:

# The information about OMS MetaDB.
oms_meta_host: ${oms_meta_host}
oms_meta_port: ${oms_meta_port}
oms_meta_user: ${oms_meta_user}
oms_meta_password: ${oms_meta_password}

# You can customize the names of the following three databases, which are created in the MetaDB when you deploy OMS.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}

# The user that consumes the incremental data of OceanBase Database.
# To read the incremental logs of OceanBase Database, create the user in the SYS tenant.
# You must create the drc user in the SYS tenant of the OceanBase cluster to be migrated and specify the drc user in the config.yaml file.
drc_user: ${drc_user}
drc_password: '${drc_password}'

# Configure the OMS cluster in the Hangzhou region.
# To deploy OMS on multiple nodes in multiple regions, you must set the cm_url parameter to a VIP or domain name that is accessible to all cluster management (CM) service nodes in the region.
cm_url: ${cm_url}
cm_location: ${cm_location}
cm_region: ${cm_region}
cm_region_cn: ${cm_region_cn}
cm_is_default: true
cm_nodes:
 - ${host_ip1}
 - ${host_ip2}

# Configurations of the time-series database.
# Default value: false. To enable metric reporting, set the parameter to true.
# tsdb_enabled: false 
# If the tsdb_enabled parameter is set to true, delete comments for the following parameters and specify the values based on your actual configurations.
# tsdb_service: 'INFLUXDB'
# tsdb_url: '${tsdb_url}'
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

Parameter	Description	Required
oms_meta_host	The IP address of MetaDB, which can be the IP address of a MySQL database or a MySQL tenant of OceanBase Database. Notice: OceanBase Database V2.0 or later is required.	Yes
oms_meta_port	The port number of MetaDB.	Yes
oms_meta_user	The username of MetaDB.	Yes
oms_meta_password	The password of the MetaDB username.	Yes
drc_rm_db	The name of the database for the OMS Community Edition console.	Yes
drc_cm_db	The name of the database for the CM service.	Yes
drc_cm_heartbeat_db	The name of the heartbeat database for the CM service.	Yes
drc_user	The user that reads the incremental logs of OceanBase Database. You need to create the user in the SYS tenant.	No
drc_password	The password of the `drc user`.	No
cm_url	The URL of the OMS Community Edition CM service. Example: http://VIP:8088. Note: To deploy OMS Community Edition on multiple nodes in multiple regions, you must set the `cm_url` parameter to a VIP or domain name that is accessible to all CM service nodes in the region. We recommend that you do not set it to `http://127.0.0.1:8088`.	Yes
cm_location	The code of the region. Value range: [0,127]. You can select one number for each region.	Yes
cm_region	The name of the region. Example: cn-hangzhou. Notice: If you use OMS Community Edition with the Alibaba Cloud Multi-Site High Availability (MSHA) service in an active-active disaster recovery scenario, use the region configured for the Alibaba Cloud service.	Yes
cm_nodes	The IP addresses of servers on which the OMS Community Edition CM service is deployed. In multi-node deployment mode, you must specify multiple IP addresses for the parameter.	Yes
cm_is_default	Indicates whether the OMS Community Edition CM service is enabled by default.	No. Default value: `true`
tsdb_service	The type of the time-series database. Valid values: `INFLUXDB` and `CERESDB`.	No. Default value: `CERESDB`
tsdb_enabled	Indicates whether metric reporting is enabled for monitoring. Valid values: `true` and `false`.	No. Default value: `false`
tsdb_url	The IP address of the server where InfluxDB is deployed, which needs to be modified based on the actual environment. You need to modify this parameter based on the actual environment if you set the `tsdb_enabled` parameter to `true`. After the time-series database is deployed, it maps to OMS Community Edition deployed for the whole cluster. This means that although OMS Community Edition is deployed in multiple regions, all regions map to the same time-series database.	No
tsdb_username	The username used to connect to the time-series database. You need to modify this parameter based on the actual environment if you set the `tsdb_enabled` parameter to `true`. After you deploy the time-series database, you must manually create a user and specify the username and password.	No
tsdb_password	The password used to connect to the time-series database. You need to modify this parameter based on the actual environment if you set the `tsdb_enabled` parameter to `true`.	No

The following example describes a template of the config.yaml file:

oms_meta_host: xxx.xxx.xxx.1
oms_meta_port: 2883
oms_meta_user: root@oms#obcluster
oms_meta_password: oms
drc_rm_db: oms_rm
drc_cm_db: oms_cm
drc_cm_heartbeat_db: oms_cm_heartbeat
drc_user: drc_user_name
drc_password: 'OceanBase#oms'
cm_url: http://VIP:8088
cm_location: 1
cm_region: cn-hangzhou
cm_is_default: true
cm_nodes:
 - xxx.xxx.xxx.2
 - xxx.xxx.xxx.3
tsdb_service: 'INFLUXDB'
tsdb_enabled: true
tsdb_url: 'xxx.xxx.xxx.5:8086'
tsdb_username: username
tsdb_password: 123456

Load the downloaded OMS image file to the local image repository of the Docker container.
```
docker load -i <The storage path of the OMS image>
```

Run the following command to start the container.

You can access the OMS console by using an HTTP or HTTPS URL. To securely access the OMS console, install an SSL certificate and mount it to the specified directory in the container. The certificate is not required for HTTP access.

OMS_HOST_IP=xxx
CONTAINER_NAME=oms_xxx
IMAGE_TAG=feature_x.x.x-ce

docker run -dit --net host \
-v /data/config.yaml:/home/admin/conf/config.yaml \
-v /data/oms/oms_logs:/home/admin/logs \
-v /data/oms/oms_store:/home/ds/store \
-v /data/oms/oms_run:/home/ds/run \
# If you mount the SSL certificate in the OMS container, you must set the following two parameters.
-v /data/oms/https_crt:/etc/pki/nginx/oms_server.crt 
-v /data/oms/https_key:/etc/pki/nginx/oms_server.key
-e OMS_HOST_IP=${OMS_HOST_IP} \
--privileged=true \
--pids-limit -1 \
--ulimit nproc=65535:65535 \
--name ${CONTAINER_NAME} \
work.oceanbase-dev.com/obartifact-store/oms:${IMAGE_TAG}

Parameter	Description
OMS_HOST_IP	The IP address of the host. Notice: The value of `OMS_HOST_IP` is different for each node.
CONTAINER_NAME	The name of the container in the oms_xxx format.
IMAGE_TAG	The tag of the image. To view the image tag, execute the `docker images` statement after you load the image on the host. The image tag is in the feature_x.x.x-ce format.
/data/oms/oms_logs /data/oms/oms_store /data/oms/oms_run	You can replace `/data/oms/oms_logs`, `/data/oms/oms_store`, and `/data/oms/oms_run` with the mount directories created on the server where OMS is deployed. The mount directories store the logs generated during the operating of OMS and generated by stores and synchronization components, respectively, to persistently retain the logs on the server. Notice The mount directories must remain unchanged during subsequent redeployment or upgrades.
/home/admin/logs /home/ds/store /home/ds/run	`/home/admin/logs`, `/home/ds/store`, and `/home/ds/run` are default directories in the container and cannot be modified.
/data/oms/https_crt (optional) /data/oms/https_key (optional)	The mount directory of the SSL certificate in the OMS container. Specify the directory based on your actual configurations. If you mount an SSL certificate, the NGINX service in the OMS Community Edition container runs in HTTPS mode. In this case, you can access the OMS Community Edition console by using only the HTTPS URL.
privileged	Specifies whether to grant the container scaling privilege.
pids-limit	Specifies whether to limit the number of container processes. The value -1 indicates that the number is unlimited.
ulimit nproc	The maximum number of user processes.

Access the new container.
```
docker exec -it  ${CONTAINER_NAME} bash  
```
Notice

CONTAINER_NAME represents the name of the container.

Initialize the metadata.

bash /root/docker_init.sh

After you run the command, the following initialization process is performed:

Initialize the data in the MetaDB.
Generate configuration files for respective components.
Restart all components.
Initialize OMS Community Edition resource tags and resource groups.

In the execution of the docker_init.sh script, pay attention to the output of command lines. When the execution is completed, the following message is displayed: [Completed] All initialization steps are executed.

Notice:

In multi-node deployment mode, you must run the docker_init.sh script on each node.

The execution takes 2 to 4 minutes to complete. Be patient and do not interrupt the process.

When you set a MySQL tenant of OceanBase Database as the MetaDB, the following error may occur: ERROR 1235 (0A000): Modify large text/lob column not supported, which can be ignored.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Step 1] Verify the config.yaml configuration file.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python -m omsflow.scripts.units.oms_cluster_config --check /home/admin/conf/config.yaml

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Step 2] Create the OMS MetaDB.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python -m omsflow.scripts.units.oms_init_manager --init-db

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Step 3] Create the configuration files for OMS components.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python -m omsflow.scripts.units.oms_init_manager --init-config-file

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Step 4] Restart all OMS components. This step takes about two minutes to complete.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
supervisorctl restart crond nginx oms_console oms_drc_cm oms_drc_supervisor sshd
supervisorctl status crond nginx oms_console oms_drc_cm oms_drc_supervisor sshd
supervisorctl status crond nginx oms_console oms_drc_cm oms_drc_supervisor sshd

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Step 5] Initialize OMS resource tags and resource groups.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python -m omsflow.scripts.units.oms_cluster_manager add_resource

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# [Completed] All initialization steps are executed.
----------------------------------------------------------

Deploy OMS Community Edition in the Heilongjiang region

The operations are the same as those for deploying OMS Community Edition in the Hangzhou region, except that you must modify the following parameters in the config.yaml file: drc_cm_heartbeat_db, cm_url, cm_location, cm_region, cm_region_cn, cm_is_default, and cm_nodes.

Notice:

When multiple regions exist, you must set the cm_is_default parameter to true for only one region, and set it to false for all other regions.
To deploy multiple nodes in the Heilongjiang region, specify the IP addresses of all nodes for the cm_nodes parameter.
You must execute the docker_init.sh script on at least one node in each region.

# The information about OMS MetaDB.
oms_meta_host: ${meta_ip}
oms_meta_port: ${meta_port}
oms_meta_user: ${meta_user}
oms_meta_password: ${meta_password}

# You can customize the names of the following three databases, which are created in the MetaDB when you deploy OMS.
drc_rm_db: ${drc_rm_db}
drc_cm_db: ${drc_cm_db}
drc_cm_heartbeat_db: ${drc_cm_heartbeat_db}

# The user that consumes the incremental data of OceanBase Database.
# To read the incremental logs of OceanBase Database, create the user in the SYS tenant.
# You must create the drc user in the SYS tenant of the OceanBase cluster to be migrated and specify the drc user in the config.yaml file.
drc_user: ${drc_user}
drc_password: ${drc_passwrord}

# Configure the OMS cluster in the Heilongjiang region.
# To deploy OMS on multiple nodes in multiple regions, you must set the cm_url parameter to a VIP or domain name that is accessible to all cluster management (CM) service nodes in the region.
cm_url: ${cm_url}
cm_location: ${cm_location}
cm_region: ${cm_region}
cm_region_cn: ${cm_region_cn}
cm_is_default: false
cm_nodes:
 - ${host_ip1}
 - ${host_ip2}

# Configurations of the time-series database.
# tsdb_service: 'INFLUXDB'
# Default value: false. Set the value based on your actual configuration.
# tsdb_enabled: false

# The IP address of the server where InfluxDB is deployed.
# You need to modify the following parameters based on the actual environment if you set the tsdb_enabled parameter to true.
# tsdb_url: ${tsdb_url}
# tsdb_username: ${tsdb_user}
# tsdb_password: ${tsdb_password}

The following example describes a template of the config.yaml file:

oms_meta_host: xxx.xxx.xxx.1
oms_meta_port: 2883
oms_meta_user: root@oms#obcluster
oms_meta_password: oms
drc_rm_db: oms_rm
drc_cm_db: oms_cm
drc_cm_heartbeat_db: oms_cm_heartbeat_1
drc_user: drc_user_name
drc_password: 'OceanBase#oms'
cm_url: http://xxx.xxx.xxx.6:8088
cm_location: 2
cm_region: cn-heilongjiang
cm_is_default: false
cm_nodes:
 - xxx.xxx.xxx.6
 - xxx.xxx.xxx.7
tsdb_service: 'INFLUXDB'
tsdb_enabled: true
tsdb_url: 'xxx.xxx.xxx.5:8086'
tsdb_username: username
tsdb_password: 123456

Enterprise Edition

Community Edition

Deploy OMS Community Edition on multiple nodes in multiple regions

Background

Prerequisites

Procedure

Deploy OMS Community Edition in the Hangzhou region

Deploy OMS Community Edition in the Heilongjiang region