Configure an AWS endpoint

If you choose the endpoint connection method when creating a data source or a data migration task with AWS as the cloud provider, you need to configure it first. This topic describes how to do so.

Prerequisites

You have created a database instance of the corresponding type and obtained its connection address within your VPC. For example, a MySQL instance typically has a single address pair in the address:port format, while a Kafka instance may have multiple address pairs (broker_address1:broker_port1, broker_address2:broker_port2, etc.).

• You have created an Amazon MSK cluster and associated a key. For details, see Configure SASL/SCRAM authentication for an Amazon MSK cluster.

• If resources associated with the endpoint service, such as the load balancing service or virtual servers, have security group configurations enabled, you need to add the private IP address belonging to the OceanBase Cloud data source to the corresponding security group. For details, see Private IP address range description.

Terms

• Private link: A high-availability, scalable technology that supports private connections from your VPC to supported AWS services. You can create your own VPC endpoint service (supported by Private Link) and enable other services to access your services.

• Target group: A target group in a network load balancer used to route requests to one or more registered targets.

• Network load balancer: It uses the protocol and port number specified when creating the target group to route requests received from service users to your service.

• Endpoint service: Your own applications or services within your VPC. Other AWS customers can create endpoints from their VPCs to your endpoint service. To use AWS Private Link, create a VPC endpoint for your service in your VPC.

Obtain the instance IP address

• MySQL database instances include Aurora MySQL, RDS MySQL, and self-managed MySQL.

• Oracle database instances include RDS Oracle and self-managed Oracle.

• OceanBase Database instances include Flagship Edition instances, Cluster instances (Analytical), and self-managed databases. This topic only describes how to obtain the IP address for a self-managed database. Obtain the appropriate type of IP address based on your actual situation.

• Kafka instances include cloud Kafka and self-managed Kafka.

• PostgreSQL database instances include Aurora PostgreSQL, RDS PostgreSQL, and self-managed PostgreSQL.

• TiDB database instances currently support only self-managed TiDB.

Obtain the IP address of an Aurora MySQL/PostgreSQL instance

1. Log in to the Amazon RDS console, and go to the Databases page.

2. Click the name of the target Aurora MySQL/PostgreSQL database instance. On the Connections and Security tab, copy the Endpoint Name.

3. If the Aurora MySQL/PostgreSQL database instance allows public access, you can run the following command on any device with public network access to obtain the instance's IP address.

   // Replace the domain name example.example-region.rds.amazonaws.com with the endpoint name you copied in the previous step.
   nslookup <example.example-region.rds.amazonaws.com>
   

   A sample return result is as follows:

   // ...
   Name: example.example-region.rds.amazonaws.com
   Address: xxx.xxx.xxx.1
   //...
   

4. Log in to the Network Interfaces console and search for the public IP address obtained in the previous step in the search box to find the corresponding Primary Private IPv4 Address.

5. Copy the obtained Primary Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain the IP address of an RDS MySQL/Oracle/PostgreSQL instance

1. Log in to the Amazon RDS console, and go to the Databases page.

2. Click the name of the target RDS MySQL/Oracle/PostgreSQL database instance. On the Connections and Security tab, copy the Endpoint.

3. If the RDS MySQL/Oracle/PostgreSQL database instance allows public access, you can run the following command on any device with public network access to obtain the instance's IP address.

   // Replace the domain name example.example-region.rds.amazonaws.com with the endpoint name you copied in the previous step.
   nslookup <example.example-region.rds.amazonaws.com>
   

   A sample return result is as follows:

   // ...
   Name: example.example-region.rds.amazonaws.com
   Address: xxx.xxx.xxx.1
   //...
   

4. Log in to the Network Interfaces console and search for the public IP address obtained in the previous step in the search box to find the corresponding Primary Private IPv4 Address.

5. Copy the obtained Primary Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain Kafka connection information and configure Nginx

1. Log in to the Amazon MSK console.

2. On the Cluster page, click the name of the target cluster to go to its details page.

3. In the upper-right corner of the Cluster Summary section, click View Client Information.

4. On the View Client Information page, click the Copy icon next to the SASL/SCRAM private endpoint to obtain the Kafka connection information.

5. Use EC2 Instance Connect to connect to the instance.

6. Run the following command to install Nginx.

   amazon-linux-extras install nginx1
   yum install nginx-mod-stream
   

7. Modify the /etc/nginx/nginx.conf configuration.

   load_module '/usr/lib64/nginx/modules/ngx_stream_module.so';
   user  nginx;
   worker_processes  auto;
   
   error_log  /var/log/nginx/error.log warn;
   pid        /var/run/nginx.pid;
   
   events {
       worker_connections  1024;
   }
   
   stream {
       # Each server corresponds to a Kafka Bootstrap Server connection address.
       server {
           listen 19092;
           proxy_pass b-2.omskafkaxxxx.vqn998.c11.kafka.us-east-1.amazonaws.com:9096;
           proxy_timeout 60s;
       }
   
       server {
           listen 19093;
           proxy_pass b-1.omskafkaxxxx.vqn998.c11.kafka.us-east-1.amazonaws.com:9096;
           proxy_timeout 60s;
       }
   }
   

   Note

   • Nginx must support the ngx_stream_module.

   • We recommend that you set the proxy_timeout parameter to 60 seconds or longer in Nginx.

8. After modifying the configuration, start Nginx.

   systemctl enable nginx
   systemctl start nginx
   

Obtain the IP address of a self-managed MySQL/Oracle/OceanBase/PostgreSQL/TiDB instance

1. Log in to the Instances page of the EC2 console.

2. On the Instance page, click the Instance ID of your self-managed database instance.

3. On the instance details page, copy the obtained Private IPv4 Address. This address will be used in the Create Target Group step.

Obtain the connection information of a self-managed RocketMQ instance

For information about how to configure the endpoint service, see Use the endpoint service at the end of this topic.

1. Log in to the Instances page of the EC2 console.

2. Obtain the RocketMQ access point configured for the NameServer.

   1. Log in to the EC2 instance where the NameServer is deployed via SSH and run the following command:

      # Obtain the Private IPv4 Address of the Local Host
      hostname -I | awk '{print $1}'
      # Check the actual listening port of the NameServer process (usually 9876 by default).
      sudo netstat -tulnp | grep namesrv
      

   2. Save the private IP address of the local host and the listening port of the NameServer from the returned result.

      # Example of IP address response:
      xx.xx.xx.xx
      
      # Port listening response example:
      tcp6       0      0 :::9876                 :::*
      

3. Obtain the RocketMQ access point of the Broker.

   1. Log in to the EC2 instance where the Broker is deployed via SSH and run the following command:

      # Please replace the path with the actual location of your broker.conf file.
      cat /path/to/rocketmq/conf/broker.conf | grep -E "^brokerIP1|^listenPort"
      

   2. Save the real IP address and listening port configured for the Broker from the returned result. If the listenPort is not specified in the configuration file, RocketMQ uses port 10911 by default.

      brokerIP1=xx.xx.xx.xx
      listenPort=10911
      

Create a target group

1. Log in to the Target Groups page of the AWS console.

2. Click Create Target Group in the upper-right corner of the page.

3. On the Specify Group Details page, configure the parameters.

   Parameter	Description
   Select Target Type	Select IP Address.
   Target Group Name	Customize the name of the target group according to the naming convention.
   Protocol	Select TCP.
   Port	Enter the corresponding port number. For Kafka instances, select the corresponding Nginx port.
   IP Address Type	Select IPv4.

4. Click Next.

5. In the IP Address section on the Register Targets page, configure the parameters.

   Parameter	Description
   Select Network	From the Network drop-down list, select the VPC where the MySQL database instance is located.
   Specify IP and Define Port	In the Enter IPv4 Address of VPC Subnet text box, enter the IP address of the database instance.
   Port	Enter the corresponding port number. For Kafka instances, select the corresponding Nginx port.

6. Click Include the following pending items.

7. In the View Targets section on the Register Targets page, confirm that the IP targets to be included in the group are configured correctly, then click Create Target Group.

Create a network load balancer

1. Create a load balancer of the Network Load Balancer type.

   For details, see Create a network load balancer.

2. On the Create Network Load Balancer page, configure the parameters.

   Note that the following parameters need to be configured according to this topic. Other parameters are not mandatory.

   • In the Mode field of the Basic Settings section, select Private. In the IP Address Type field, select IPv4.

   • In the Network Mapping section, select the VPC where the MySQL data source is located from the VPC drop-down list. Usually, select all for Mapping, but you can also select at least one availability zone and one subnet based on your actual situation.

   • In the Security Groups section, you can select a security group to associate with the network load balancer. It is recommended not to select any security group here. For details about security groups, see Security groups for network load balancers.

     Note

     If you associate a security group with the network load balancer, you need to allow access to the backend mount ports.

     When creating a network load balancer for a Kafka instance, do not select a security group.

   • The parameters in the Listener and Routing section are configured as follows.

     Parameter	Description
     Protocol	Listener protocol. Select TCP.
     Port	Select the corresponding port number.
     Default Action	Select the target group created in the Create Target Group step from the drop-down list.

3. After successful creation, return to the Load Balancer page and select the created network load balancer.

4. Click Actions > Edit Load Balancer Attributes in the upper-right corner of the Load Balancer page.

5. In the Load Balancer Target Selection Strategy section of the Edit Load Balancer Attributes page, select Enable Cross-Region Load Balancing and click Save Changes.

Create an endpoint service

1. Log in to the Endpoint Services page of the AWS console.

2. Click Create Endpoint Service in the upper-right corner of the page.

3. On the Create Endpoint Service page, configure the parameters.

   Parameter	Description
   Name	Customize the name of the endpoint service.
   Load Balancer Type	Select Network.
   Available Load Balancers	Select the Load Balancer created in the Create Network Load Balancer step. Note If it is not displayed on the page, wait for the Load Balancer status to become Active, then refresh the page to select it.

4. In the Other Settings section, configure the parameters.

   Parameter	Description
   Accept Endpoints for Use (Optional)	Whether requests initiated by service users require manual secondary processing. If you select Accept, when Data Migration Service first accesses the private link, it will prompt you to enter the AWS console to perform the Accept Endpoint Connection Request operation for the received endpoint connection request. If you do not select Accept, no prompt is given when Data Migration Service first accesses the private link.
   Enable Private DNS Name	Do not enable a personal domain here.
   Supported IP Address Types	Select IPv4.

5. Click Create to complete the creation of the endpoint service.

6. Authorize the accounts that need access to the created endpoint service.

   1. On the Endpoint Service page, select the created endpoint service.

   2. Click the Allow Principals tab at the bottom of the page.

   3. Click Allow Principals in the upper-right corner of the Allow Principals tab.

   4. Obtain the account that needs authorization for the data source or data migration task.

      On the Create Data Source or Data Migration Task page of the OceanBase Cloud console, select Connection Type as Endpoint, and then copy the account that needs authorization as prompted on the page (in the format arn:aws:iam::aws-account-id:<type>/<id>). For details on creating a data source, see Create a data source. For details on creating a task, see the corresponding documentation for creating a data migration task in the Data Migration module.

   5. On the Grantors page, in the Add Grantor section, enter the account that needs authorization and click Grant.

   After authorization is completed, the created endpoint service becomes visible to the authorized account.

Use the endpoint service

After creating an endpoint service, you need to copy its Service Name and enter it in the Connection Information section on the page for creating a new data source or a data migration task. For details on creating a data source, see Create a data source. For details on creating a data migration task, see the corresponding documentation for creating a data migration task in the Data Migration module.

To copy the service name, perform the following steps:

1. Log in to the Endpoint Services page of the AWS console.

2. Select the target endpoint service.

3. In the Details section at the bottom of the page, click the copy icon next to Service Name.