Configure a Google Cloud endpoint

If you choose the endpoint connection method when creating a data source or a data migration task with Google Cloud as the cloud provider, you need to configure it first. This topic describes how to do so.

Prerequisites

• You have created a database instance of the corresponding type and obtained its connection address within your VPC. For example, a MySQL instance typically has a single address pair in the format address:port, while a Kafka instance may have multiple address pairs (broker_address1:broker_port1, broker_address2:broker_port2, etc.).

• If resources associated with the endpoint service, such as the load balancing service or virtual server, have security group configurations enabled, you need to add the private IP address of the OB Cloud data source to the corresponding security group. For details, see Private IP address range description.

Configure private network connections for Cloud SQL instances

GCP Cloud SQL for MySQL/PostgreSQL supports two private network connection methods: Private Service Connect (PSC) and Private Service Access (PSA). You can go to the Cloud SQL instance page in the Google Cloud console and check the Internal connection method of the target Cloud SQL instance to determine whether it uses PSC mode or PSA mode.

## Create an instance
gcloud compute instances create <instance_name> \
    --project=<project_id> \
    --zone=<zone_id> \
    --machine-type=c2d-standard-2 \
    --network-interface=network-tier=STANDARD,stack-type=IPV4_ONLY,subnet=<subnet_name> \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --tags=lb-health-check \
    --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/sqlservice.admin,https://www.googleapis.com/auth/trace.append \
    --create-disk=auto-delete=yes,boot=yes,device-name=<instance_name>,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20240717,mode=rw,size=20,type=pd-balanced \
    --labels=use=rdtest,product=oms,env=devsit

# Create an instance group and add the previously created instance to it.
gcloud compute instance-groups unmanaged create <instance_group_name> \
    --project=<project_id> \
    --zone=<zone_id> \
&& \
gcloud compute instance-groups unmanaged add-instances <instance_group_name> \
    --project=<project_id> \
    --zone=<zone_id> \
    --instances=<instance_name>

# Create a health check.
gcloud compute health-checks create tcp <health_check_name> \
  --project=<project_id> \
  --port=3306 \
  --proxy-header=NONE \
  --region=us-west1 \
  --no-enable-logging \
  --check-interval=5 \
  --timeout=5 \
  --unhealthy-threshold=2 \
  --healthy-threshold=2

Configure private network connection for Kafka instance

Obtain the Kafka node list

1. Run the following command in the Google Cloud command-line interface to obtain the Kafka cluster information.

   gcloud managed-kafka clusters describe <cluster name> --location us-west1
   

   In the execution result, bootstrapAddress indicates the Bootstrap connection address, and subnet indicates the subnet where the cluster is located.

2. Create an instance that can connect to Kafka. The subnet must be the same as the GCP subnet.

   ## Create an instance
   gcloud compute instances create <instance_name> \
       --project=<project_id> \
       --zone=<zone_id> \
       --machine-type=c2d-standard-2 \
       --network-interface=network-tier=STANDARD,stack-type=IPV4_ONLY,subnet=<subnet_name> \
       --maintenance-policy=MIGRATE \
       --provisioning-model=STANDARD \
       --tags=lb-health-check \
       --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/sqlservice.admin,https://www.googleapis.com/auth/trace.append \
       --create-disk=auto-delete=yes,boot=yes,device-name=<instance_name>,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20240717,mode=rw,size=20,type=pd-balanced \
       --labels=use=rdtest,product=oms,env=devsit
   
   # Create an instance group and add the instance created above to the instance group.
   gcloud compute instance-groups unmanaged create <instance_group_name> \
       --project=<project_id> \
       --zone=<zone_id> \
   && \
   gcloud compute instance-groups unmanaged add-instances <instance_group_name> \
       --project=<project_id> \
       --zone=<zone_id> \
       --instances=<instance_name>
   

3. Log in to the instance and run the following command.

   sudo -i
   yum install -y wget java-1.8.0-openjdk
   
   wget -O kafka_2.13-3.7.2.tgz  https://downloads.apache.org/kafka/3.7.2/kafka_2.13-3.7.2.tgz
   tar xfz kafka_2.13-3.7.2.tgz
   export KAFKA_HOME=$(pwd)/kafka_2.13-3.7.2
   export PATH=$PATH:$KAFKA_HOME/bin
   

4. Authorize the retrieval of SASL authentication information and save the authentication file as client.properties on the instance.

   1. Log in to the IAM page in the Google Cloud console and find a target service account (service_account, used when generating the SASL password).

   2. Obtain the key JSON configuration of the target service account (download and save it when generating the key).

   3. Generate an SASL password.

      Taking the key JSON configuration file named my_service_account.json as an example, run the following command in the instance to obtain a base64-encoded password, which will be generated into password.txt.

      base64 -w 0 < my_service_account.json > password.txt
      

   4. Run the following command in the instance to grant the target service account permission to access the Kafka service on GCP.

      gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member serviceAccount: $SERVICE_ACCOUNT_EMAIL \
      --role roles/managedkafka.client
      

   5. Enter the data source in the data source service via SASL_PLAIN to test the connection. Save this file as client.properties.

      security.protocol=SASL_SSL
      sasl.mechanism=PLAIN
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="oms-xxxx.iam.gserviceaccount.com" \
      password="ewogICxxxxFwaXMuY29tIgp9Cg==";
      

5. Run the following command to obtain the addresses of the broker nodes.

   # --bootstrap-server The Bootstrap connection address obtained from the previous query.
   kafka-configs.sh --bootstrap-server bootstrap.xxxx-test-oms.us-west1.managedkafka.oms-dev-41xxxx02.cloud.goog:9092 --describe --all --entity-type brokers --command-config client.properties | grep advertised.listeners | awk -F'EXTERNAL://|,' '{print $2}'
   

   For example, broker-0_bootstrap.xxxx-test-oms.us-west1.managedkafka.oms-dev-41xxxx02.cloud.goog:9092.

Configure Nginx

Configure Nginx based on the obtained broker node list.

# If Nginx fails to start, you need to set this.
sudo setenforce 0

load_module '/usr/lib64/nginx/modules/ngx_stream_module.so';
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    # Each Server corresponds to a Kafka Bootstrap Server connection address.
    # The number after listen is the Nginx port, which can be different from the port in proxy_pass.
    server {
        listen 19092;
        proxy_pass bootstrap.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }
    
    server {
        listen 19093;
        proxy_pass broker-0.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }

    server {
        listen 19094;
        proxy_pass broker-1.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }

    server {
        listen 19095;
        proxy_pass broker-2.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }
}

Create a health check

gcloud compute health-checks create tcp <health_check_name> \
  --project=<project_id> \
  --port=<any Nginx forwarding port> \
  --proxy-header=NONE \
  --region=<region_id> \
  --no-enable-logging \
  --check-interval=5 \
  --timeout=5 \
  --unhealthy-threshold=2 \
  --healthy-threshold=2

Create a PSC service attachment

Create a load balancer

1. On the Load Balancing page, click + CREATE LOAD BALANCER.

   

2. On the Create a load balancer page, select the following configurations in sequence.

   1. Select Network Load Balancer for Type of load balancer and click Next.

   2. Select Passthrough load balancer for Proxy or passthrough and click Next.

   3. Select Internal for Public facing or internal and click Next.

   4. After the selection is complete, click CONFIGURE.

3. On the Create Internal passthrough Network Load Balancer page, configure the parameters.

   

4. Click CREATE.

Create a publish service

1. On the PUBLISH SERVICES page, click + PUBLISH SERVICE.

   

2. On the Publish Service page, configure the parameters.

   1. Select Internal passthrough Network Load Balancer for Load Balancer Type.

   2. From the Internal load balancer drop-down list, select the created load balancer.

   3. Enter a custom name in the Service name text box.

   4. From the Subnets drop-down list, select the created subnet and click OK. If you need to create a new one, click RESERVE NEW SUBNET.

      Note

      Ensure the Publish Service Subnet can access your backend services.

   5. In the Accepted projects section, click + ADD ACCEPTED PROJECT and enter the name of the authorized project.

      The project name here should be copied from the name displayed below the Connection section on the Data Services > Create Data Source or Create Data Migration Task page.

      

3. After the configuration is complete, click ADD SERVICE.

Obtain Service Attachment

After creating a Publish Service, you need to copy the Service attachment and fill it in the Google Cloud > Endpoint > Connection fields on the page for creating a data source or a data migration task. For details on creating the task, see Migrate data from a MySQL database to the MySQL-compatible mode of OceanBase Database.

To copy the Service attachment, perform the following steps:

1. On the Published services page, click the name of the target Service.

   

2. On the Private Service Connect service details page, click the copy icon next to Service attachment to obtain the service connection.