Migrating OceanBase Data Using AWS Glue

AWS Glue is a serverless data integration service. It helps users easily discover, prepare, move, and integrate data from multiple sources to support analytics, machine learning, and application development. AWS Glue provides comprehensive data integration capabilities, including data discovery, modern ETL, data cleansing and transformation, and centralized cataloging, all integrated into a single service. AWS Glue does not require users to manage infrastructure, supports various workloads such as ETL, ELT, and streaming, and can scale on demand to accommodate any data size and type. This topic uses OceanBase Cloud. AWS Glue migrates data from one OceanBase instance to another. As for data streaming, you need to write ETL scripts and schedule tasks.

Connect OceanBase Cloud by using Glue

Follow these steps to connect to the OceanBase Cloud by using Glue:

1. Create an EC2 instance.

2. Set up the AWS S3 VPC Gateway Endpoint. AWS Glue cannot connect to data sources via public IP, so this topic uses a private IP to connect to OceanBase Cloud. Make sure that your Glue subnet has an Amazon S3 VPC gateway endpoint or a NAT gateway route in the subnet routing table. Follow these steps to set up the AWS S3 VPC Gateway Endpoint:

   1. Open the Amazon VPC console. In the navigation pane, select Endpoints.
   2. Select Create Endpoint.
   3. For Service Name, select com.amazonaws.us-east-1.s3. Make sure that the Type column indicates Gateway.

      Note

      You must replace us-east-1 with the AWS region you choose.

   4. For VPC, select the VPC in which you want to create the endpoint.
   5. For Configure routing table, the system will automatically add routes to the S3 VPC endpoint.
   6. For Policy, retain the default option Full Access.
   7. Select Create Endpoint. After the endpoint is created, you can view the endpoint details and subnet information.

3. Test the Glue Connection.

   1. Open the AWS Glue Console, select JDBC.
   2. Use a private IP for JDBC.
   3. Set Network Options. Use the subnet information configured in the endpoint step for the subnet to S3 VPC link.
   4. Click Next to complete creation.
   5. Test the connection.

Use Glue ETL Tasks

Follow these steps to use Glue ETL tasks, connect to OceanBase instances, and transform data to OceanBase instances.

1. Add the OceanBase Source Database.

   1. Click Connections, select the JDBC type.
   2. Fill in the OceanBase data source information.
   3. Complete creation.
   4. Test the connection.

2. Add the OceanBase Target Database. This topic uses Crawlers for target database mapping.

   1. Click Crawlers -> Create Crawler.
   2. Fill in the Crawler information.
   3. Add data sources.
   4. Select the JDBC connection type and database table.
   5. Click Next and assign users.
   6. Add the location for table mapping.
   7. Select the database and complete, click Next to finish creating the Crawler.
   8. Pre-create the table in the target database.
   9. Run the Crawler and view the table mapping information.

3. Create an ETL Task.

   1. Click Visual ETL to create a job.
   2. Select MySQL for the Source type.
   3. Click the MySQL node, select JDBC Source and Table Name to preview data.
   4. Click Targets, select MySQL for the Target type.
   5. Save and run the task.
   6. View the execution results.