Configure an AWS endpoint

If you select the endpoint connection method when creating a data source or a data migration task with AWS as the cloud provider, you need to configure it first. This topic describes how to do so.

Prerequisites

You have created a database instance of the corresponding type and obtained its connection address in your VPC. For example, a MySQL instance typically has a single address pair address:port, while a Kafka instance may have multiple address pairs (broker_address1:broker_port1, broker_address2:broker_port2, etc.).

• You have created an Amazon MSK cluster and associated a key. For more information, see Configure SASL/SCRAM authentication for an Amazon MSK cluster.

• If resources such as the load balancing service or virtual server associated with the endpoint service have security group configurations enabled, you need to add the private network address belonging to the OceanBase Cloud data source to the corresponding security group. For more information, see Private network address range description.

Terms

• AWS PrivateLink: A highly available and scalable technology that provides private connectivity between your VPC and supported AWS services. You can create a VPC endpoint service powered by AWS PrivateLink and allow other services to access it.

• Target Group: A target group in a network load balancer used to route requests to one or more registered targets.

• Network Load Balancer: It uses the protocol and port number specified when creating the target group to route requests received from service users to your service.

• Endpoint service: Your own application or service in a VPC. Other AWS customers can create endpoints in their VPCs to connect to your endpoint service. To use AWS PrivateLink, create a VPC endpoint for your service in your VPC.

Obtain the instance IP address

• MySQL database instances include Aurora MySQL, RDS MySQL, and self-managed MySQL.

• Oracle database instances include RDS Oracle and self-managed Oracle.

• OceanBase Database instances include Cluster Instance (Transactional), Cluster Instance (Flagship), Tenant Instance, Cluster Instance (Analytical), and self-managed database. This topic only describes how to obtain the IP address of a self-managed database. Obtain the appropriate type of IP address based on your actual situation.

• Kafka instances include Cloud Kafka and self-managed Kafka.

• PostgreSQL database instances include Aurora PostgreSQL, RDS PostgreSQL, and self-managed PostgreSQL.

• TiDB database instances currently only support self-managed TiDB.

Obtain the IP address of an Aurora MySQL/PostgreSQL instance

1. Log in to the Databases page of the Amazon RDS console.

2. Click the name of the target Aurora MySQL/PostgreSQL database instance. On the Connections and Security tab, copy the Endpoint Name.

3. If the Aurora MySQL/PostgreSQL database instance allows public access, you can execute the following command on any device with public network access to obtain the instance's IP address.

   // Replace the domain name example.example-region.rds.amazonaws.com with the endpoint name you copied in the previous step.
   nslookup <example.example-region.rds.amazonaws.com>
   

   A sample return result is as follows:

   // ...
   Name: example.example-region.rds.amazonaws.com
   Address: xxx.xxx.xxx.1
   //...
   

4. Log in to the Network Interfaces console and search for the public IP address obtained in the previous step in the search box to find the corresponding Primary Private IPv4 Address.

5. Copy the obtained Primary Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain the IP address of an RDS MySQL/Oracle/PostgreSQL instance

1. Log in to the Databases page of the Amazon RDS console.

2. Click the name of the target RDS MySQL/Oracle/PostgreSQL database instance. On the Connections and Security tab, copy the Endpoint.

3. If the RDS MySQL/Oracle/PostgreSQL database instance allows public access, you can run the following command on any device with public network access to obtain the instance's IP address.

   // Replace the domain name example.example-region.rds.amazonaws.com with the endpoint name you copied in the previous step.
   nslookup <example.example-region.rds.amazonaws.com>
   

   A sample return result is as follows:

   // ...
   Name: example.example-region.rds.amazonaws.com
   Address: xxx.xxx.xxx.1
   //...
   

4. Log in to the Network Interfaces console and search for the public IP address obtained in the previous step in the search box to find the corresponding Primary Private IPv4 Address.

5. Copy the obtained Primary Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain Kafka connection information and configure Nginx

1. Log in to the Amazon MSK console.

2. On the Cluster page, click the name of the target cluster to go to its details page.

3. In the upper-right corner of the Cluster Summary section, click View Client Information.

4. On the View Client Information page, click the Copy icon next to the SASL/SCRAM private endpoint to obtain the Kafka connection information.

5. Use EC2 Instance Connect to connect to the instance.

6. Run the following command to install Nginx.

   amazon-linux-extras install nginx1
   yum install nginx-mod-stream
   

7. Modify the /etc/nginx/nginx.conf configuration.

   load_module '/usr/lib64/nginx/modules/ngx_stream_module.so';
   user  nginx;
   worker_processes  auto;
   
   error_log  /var/log/nginx/error.log warn;
   pid        /var/run/nginx.pid;
   
   events {
       worker_connections  1024;
   }
   
   stream {
       # Each server corresponds to a Kafka Bootstrap Server connection address.
       server {
           listen 19092;
           proxy_pass b-2.omskafkaxxxx.vqn998.c11.kafka.us-east-1.amazonaws.com:9096;
           proxy_timeout 60s;
       }
   
       server {
           listen 19093;
           proxy_pass b-1.omskafkaxxxx.vqn998.c11.kafka.us-east-1.amazonaws.com:9096;
           proxy_timeout 60s;
       }
   }
   

   Note

   • Nginx must support the ngx_stream_module.

   • We recommend that you set the proxy_timeout parameter to 60 seconds or longer in Nginx.

8. After modifying the configuration, start Nginx.

   systemctl enable nginx
   systemctl start nginx
   

Obtain the IP address of a self-managed MySQL/Oracle/OceanBase/PostgreSQL/TiDB instance

1. Log in to the Instances page of the EC2 console.

2. On the Instance page, click the Instance ID of your self-managed database instance.

3. On the instance details page, copy the obtained Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain the connection information of a self-managed RocketMQ instance

For information about how to configure the endpoint service, see Use an endpoint service at the end of this topic.

1. Log in to the Instances page of the EC2 console.

2. Obtain the RocketMQ access point configured for the NameServer.

   1. Log in to the EC2 instance where the NameServer is deployed via SSH and run the following command:

      # Obtain the Private IPv4 Address of the Local Host
      hostname -I | awk '{print $1}'
      # Check the real listening port of the NameServer process (usually 9876 by default).
      sudo netstat -tulnp | grep namesrv
      

   2. Save the private IP address of the local host and the listening port of the NameServer from the returned result.

      # Example of IP address response:
      xx.xx.xx.xx
      
      # Port listening response example:
      tcp6       0      0 :::9876                 :::*
      

3. Obtain the RocketMQ access point of the Broker.

   1. Log in to the EC2 instance where the Broker is deployed via SSH and run the following command:

      # Replace the path with the actual location of your broker.conf file.
      cat /path/to/rocketmq/conf/broker.conf | grep -E "^brokerIP1|^listenPort"
      

   2. Save the real IP address and listening port configured for the Broker from the returned result. If the listenPort field is absent in the configuration file, RocketMQ uses port 10911 by default.

      brokerIP1=xx.xx.xx.xx
      listenPort=10911
      

Create a target group

1. Log in to the Target Groups page of the AWS console.

2. Click Create Target Group in the upper-right corner of the page.

3. On the Specify Group Details page, configure the parameters.

   Parameter	Description
   Select Target Type	Select IP Address.
   Target Group Name	Customize the name of the target group according to the naming convention.
   Protocol	Select TCP.
   Port	Enter the corresponding port number. For Kafka instances, select the corresponding Nginx port.
   IP Address Type	Select IPv4.

4. Click Next.

5. In the IP Address section of the Register Target page, configure the parameters.

   Parameter	Description
   Select Network	From the Network drop-down list, select the VPC where the MySQL database instance is located.
   Specify IP and Define Port	In the Enter IPv4 Address of VPC Subnet text box, enter the IP address of the database instance.
   Port	Enter the corresponding port number. For Kafka instances, select the corresponding Nginx port.

6. Click Include the following pending items.

7. In the View Targets section of the Register Target page, after confirming that the IP targets to be included in the group are configured correctly, click Create Target Group.

Create a network load balancer

1. Create a load balancer of the Network Load Balancer type.

   For more information, see Create a network load balancer.

2. On the Create Network Load Balancer page, configure the parameters.

   Note that the following parameters need to be configured according to this topic. Other parameters are not mandatory.

   • In the Mode section of the Basic Settings area, select Private for Mode and IPv4 for IP Address Type.

   • In the Network Mapping section, select the VPC where the MySQL data source is located from the VPC drop-down list. Usually, select all for Mapping, but you can also select at least one availability zone and one subnet based on your actual situation.

   • In the Security Groups section, you can select a security group to associate with the network load balancer. It is recommended that you do not select any security group here. For more information about security groups, see Security groups for network load balancers.

     Note

     If you associate a security group with the network load balancer, you need to allow access to the backend mount ports.

     When creating a network load balancer for a Kafka instance, do not select a security group.

   • The parameters in the Listener and Routing section are configured as follows.

     Parameter	Description
     Protocol	Select TCP for the listener protocol.
     Port	Select the corresponding port number.
     Default Action	Select the target group created in the Create Target Group step from the drop-down list.

3. After successful creation, return to the Load Balancer page and select the created network load balancer.

4. On the Load Balancer page, click Actions > Edit Load Balancer Attributes in the upper-right corner.

5. In the Edit Load Balancer Attributes page, under the Load Balancer Target Selection Strategy section, select Enable Cross-Zone Load Balancing and click Save Changes.

Create an endpoint service

1. Log in to the Endpoint Services page of the AWS console.

2. Click Create Endpoint Service in the upper-right corner of the page.

3. On the Create Endpoint Service page, configure the parameters.

   Parameter	Description
   Name	Customize the name of the endpoint service.
   Load Balancer Type	Select Network.
   Available Load Balancers	Select the Load Balancer created in the Create Network Load Balancer step. Note If it is not displayed on the page, wait until the Load Balancer status is Active, then refresh the page to select it.

4. In the Other Settings section, configure the parameters.

   Parameter	Description
   Accept Endpoints for Use (Optional)	Whether requests initiated by service users require manual secondary processing. If you select Accept, when Data Migration Service first accesses the private link, it will prompt you to enter the AWS console to perform the Accept Endpoint Connection Request operation for the received endpoint connection request. If you do not select Accept, no prompt will be displayed when Data Migration Service first accesses the private link.
   Enable Private DNS Name	Do not enable a personal domain here.
   Supported IP Address Types	Select IPv4.

5. Click Create to complete the creation of the endpoint service.

6. Authorize the accounts that need access to the created endpoint service.

   1. On the Endpoint Services page, select the created endpoint service.

   2. Click the Allow Principals tab at the bottom of the page.

   3. Click Allow Principals in the upper-right corner of the Allow Principals tab.

   4. Obtain the account that needs authorization for the data source or data migration task.

      On the Create Data Source or Create Data Migration Task page of the OceanBase Cloud console, set Connection Type to Endpoint, and then copy the principal that requires authorization (in the format arn:aws:iam::aws-account-id:<type>/<id>) as prompted on the page. For details on creating a data source, see Create a data source. For details on creating a task, see the corresponding type of document in the Data migration module.

   5. On the Allow principals tab, choose Allow principals. For Principals to add, enter the ARN of the principal, and then choose Allow principals.

   After authorization is completed, the created endpoint service becomes visible to the authorized account.

Use an endpoint service

After creating an endpoint service, copy its Service Name and enter it in the AWS > Endpoint > Connection Information section on the page for creating a data source or data migration task. For details on creating a data source, see Create a data source. For details on creating a data migration task, see the corresponding documentation in the Data Migration module.

To copy the service name, perform the following steps:

1. Log in to the Endpoint Services page of the AWS console.

2. Select the target endpoint service.

3. In the Details section at the bottom of the page, click the copy icon next to Service Name.