Configure a Google Cloud endpoint

If you select endpoint as the connection type when you create a data source or data migration task with Google Cloud as the cloud vendor, you must configure a Google Cloud endpoint. This topic describes how to configure it.

Prerequisites

• You have created a database instance of the corresponding type and obtained its connection address in your virtual private cloud (VPC). For example, for a MySQL database instance, the connection address is usually a single address-port pair in the format of address:port. For a Kafka instance, multiple address-port pairs in the format of broker_address1:broker_port1, broker_address2:broker_port2, and so on, are possible.

• If the load balancing service or virtual server associated with the endpoint service has enabled security group, add the private IP address of the OceanBase Cloud data source to the corresponding security group. For more information, see Private IP address segments.

Configure a private network connection for the Cloud SQL instance

Google Cloud Platform (GCP) Cloud SQL for MySQL/PostgreSQL supports two private network connection modes: Private Service Connect (PSC) and Private Services Access (PSA). You can log in to the GCP console and go to the Cloud SQL instance list page. Then, check the value of Internal connection method of the target Cloud SQL instance to determine whether it uses PSC or PSA.

## Create an instance.
gcloud compute instances create <instance_name> \
    --project=<project_id> \
    --zone=<zone_id> \
    --machine-type=c2d-standard-2 \
    --network-interface=network-tier=STANDARD,stack-type=IPV4_ONLY,subnet=<subnet_name> \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --tags=lb-health-check \
    --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/sqlservice.admin,https://www.googleapis.com/auth/trace.append \
    --create-disk=auto-delete=yes,boot=yes,device-name=<instance_name>,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20240717,mode=rw,size=20,type=pd-balanced \
    --labels=use=rdtest,product=oms,env=devsit

# Create an instance group and add the instance created earlier to the instance group.
gcloud compute instance-groups unmanaged create <instance_group_name> \
    --project=<project_id> \
    --zone=<zone_id> \
&& \
gcloud compute instance-groups unmanaged add-instances <instance_group_name> \
    --project=<project_id> \
    --zone=<zone_id> \
    --instances=<instance_name>

# Create a health check.
gcloud compute health-checks create tcp <health_check_name> \
  --project=<project_id> \
  --port=3306 \
  --proxy-header=NONE \
  --region=us-west1 \
  --no-enable-logging \
  --check-interval=5 \
  --timeout=5 \
  --unhealthy-threshold=2 \
  --healthy-threshold=2

Configure a private connection for a Kafka instance

Obtain the list of Kafka nodes

1. Run the following command in the Google Cloud CLI to obtain the Kafka cluster information.

   gcloud managed-kafka clusters describe <cluster name> --location us-west1
   

   In the execution result, the bootstrapAddress indicates the Bootstrap connection address, and the subnet indicates the subnet where the cluster is located.

2. Create an instance that can connect to Kafka, and the subnet must be consistent with the GCP subnet.

   ## Create an instance
   gcloud compute instances create <instance name> \
       --project=<project ID> \
       --zone=<zone ID> \
       --machine-type=c2d-standard-2 \
       --network-interface=network-tier=STANDARD,stack-type=IPV4_ONLY,subnet=<subnet name> \
       --maintenance-policy=MIGRATE \
       --provisioning-model=STANDARD \
       --tags=lb-health-check \
       --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/sqlservice.admin,https://www.googleapis.com/auth/trace.append \
       --create-disk=auto-delete=yes,boot=yes,device-name=<instance name>,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20240717,mode=rw,size=20,type=pd-balanced \
       --labels=use=rdtest,product=oms,env=devsit
   
   # Create an instance group and add the above-created instance to the instance group.
   gcloud compute instance-groups unmanaged create <instance group name> \
       --project=<project ID> \
       --zone=<zone ID> \
   && \
   gcloud compute instance-groups unmanaged add-instances <instance group name> \
       --project=<project ID> \
       --zone=<zone ID> \
       --instances=<instance name>
   

3. Log in to the instance and run the following command.

   sudo -i
   yum install -y wget java-1.8.0-openjdk
   
   wget -O kafka_2.13-3.7.2.tgz  https://downloads.apache.org/kafka/3.7.2/kafka_2.13-3.7.2.tgz
   tar xfz kafka_2.13-3.7.2.tgz
   export KAFKA_HOME=$(pwd)/kafka_2.13-3.7.2
   export PATH=$PATH:$KAFKA_HOME/bin
   

4. Authorize access to obtain the SASL authentication information and save the authentication file as client.properties on the instance.

   1. Log in to the IAM page of the Google Cloud console and find a target service account (service_account) for generating the SASL password.

   2. Obtain the key JSON configuration for the target service account (download and save it when generating the key).

   3. Generate the SASL password.

      For example, if the name of the key JSON configuration file is my_service_account.json, run the following command in the instance to obtain a base64-encoded password and generate it to password.txt.

      base64 -w 0 < my_service_account.json > password.txt
      

   4. Run the following command in the instance to grant the target service account the permission to access the Kafka service on GCP.

      gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member serviceAccount: $SERVICE_ACCOUNT_EMAIL \
      --role roles/managedkafka.client
      

   5. Use the SASL_PLAIN method to test the connection by entering the data source in the data source service and save the file as client.properties.

      security.protocol=SASL_SSL
      sasl.mechanism=PLAIN
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="oms-xxxx.iam.gserviceaccount.com" \
      password="ewogICxxxxFwaXMuY29tIgp9Cg==";
      

5. Run the following command to obtain the address of the broker node.

   # Enter the Bootstrap connection address obtained from the previous query after --bootstrap-server.
   kafka-configs.sh --bootstrap-server bootstrap.xxxx-test-oms.us-west1.managedkafka.oms-dev-41xxxx02.cloud.goog:9092 --describe --all --entity-type brokers --command-config client.properties | grep advertised.listeners | awk -F'EXTERNAL://|,' '{print $2}'
   

   For example, broker-0.bootstrap.xxxx-test-oms.us-west1.managedkafka.oms-dev-41xxxx02.cloud.goog:9092.

Configure Nginx

Configure Nginx based on the list of Broker nodes obtained.

# If Nginx fails to start, you need to set as follows:
sudo setenforce 0

load_module '/usr/lib64/nginx/modules/ngx_stream_module.so';
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    # Each Server corresponds to a Kafka Bootstrap Server connection endpoint.
    # The port specified after listen is the Nginx port, which can be different from the port specified in proxy_pass.
    server {
        listen 19092;
        proxy_pass bootstrap.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }
    
    server {
        listen 19093;
        proxy_pass broker-0.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }

    server {
        listen 19094;
        proxy_pass broker-1.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }

    server {
        listen 19095;
        proxy_pass broker-2.<region_id>.managedkafka.<project_id>.cloud.goog:9092;
        proxy_timeout 60s;
    }
}

Create a health check

gcloud compute health-checks create tcp <health_check_name> \
  --project=<project_id> \
  --port=<any_Nginx_forwarding_port> \
  --proxy-header=NONE \
  --region=<region_id> \
  --no-enable-logging \
  --check-interval=5 \
  --timeout=5 \
  --unhealthy-threshold=2 \
  --healthy-threshold=2

Create a PSC service attachment

Create a load balancer

1. Go to Load Balancing and click + CREATE LOAD BALANCER.

   

2. On the Create a load balancer page, specify the following parameters.

   1. Set Type of load balancer to Network Load Balancer and click Next.

   2. Set Proxy or passthrough to Passthrough load balancer and click Next.

   3. Set Public facing or internal to Internal and click Next.

   4. Click CONFIGURE.

3. On the Create Internal passthrough Network Load Balancer page, configure the parameters.

   

4. Click CREATE.

Create a publish service

1. Go to PUBLISH SERVICES and click + PUBLISH SERVICE.

   

2. On the Publish Service page, configure the parameters.

   1. Set Load Balancer Type to Internal passthrough Network Load Balancer.

   2. Select the created load balancer from the Internal load balancer drop-down list.

   3. Enter a custom name in the Service name field.

   4. Select a created subnet from the Subnets drop-down list and click OK. If you need to create a new subnet, click RESERVE NEW SUBNET.

      Notice

      Make sure that the publish service subnet can access your backend service.

   5. In the Accepted projects section, click + ADD ACCEPTED PROJECT and enter the authorized project name.

      Copy the project name displayed below the Connection section on the page for creating a data source or data migration task and paster it here.

      

3. Click ADD SERVICE.

Obtain a service attachment

After you create a publish service, you need to copy Service attachment and enter it in the Google Cloud > Endpoint > Connection field on the page where you create a data source or a data migration task. For more information, see Migrate data from a MySQL database to the MySQL compatible mode of OceanBase Database.

To copy a service attachment, perform the following steps:

1. On the Published services page, click the name of the target service.

   

2. On the Private Service Connect service details page, click the copy icon next to Service attachment to obtain a service connection.