Create a Kafka data source

You must configure data sources before you create a data migration task. Before you start data migration, you can add a Kafka instance as the destination data source. This topic describes how to create a Kafka data source in OceanBase Migration Service (OMS).

Limitations

OMS supports adding a Kafka instance only as the destination for data synchronization.

Prerequisites

You have obtained the endpoint of the self-managed Kafka instance.

Background

Data security is a major concern throughout service connectivity verification, task creation, and data transmission. Based on the security system provided by the Kafka service, OMS can meet most security requirements in data encryption and user authentication.

The following Kafka authentication methods are supported by OMS:

• GSSAPI

  Generic Security Services Application Program Interface (GSSAPI) is a framework that provides generic security services. It supports the Kerberos protocol.

• PLAIN

  PLAIN authentication is simple and does not support dynamic changes of users. In this authentication mode, usernames and passwords are configured in plaintext, resulting in low security.

• SCRAM-SHA-256

  Salted Challenge Response Authentication Mechanism (SCRAM) authenticates users by username and password. SCRAM-SHA-256 can be used together with Transport Layer Security (TLS) for security authentication.

  In this authentication method, users can be dynamically changed, and the user data is stored in Zookeeper. Before a broker is started, a user for communication between the broker and Zookeeper must be created. However, usernames and passwords are configured in plaintext in this authentication method.

• SCRAM-SHA-512

  SCRAM-SHA-512 can be used together with TLS for security authentication.

Procedure

1. Log in to the OMS console.

2. In the left-side navigation pane, click Data Source Management.

3. On the Data Source Management page, click New Data Source in the upper-right corner.

   

4. In the New Data Source dialog box, select Kafka for Data Source Type and configure the following parameters.

   Parameter	Description
   Data Source Identifier	We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 32 characters in length. Notice The data source identifier must be globally unique in OMS.
   Region	Select the region where the data source resides from the drop-down list. The region is the value that you set for the cm_region parameter when you deploy OMS. Notice This parameter is displayed only when multiple regions are available. Make sure that the mappings between the data source and the region are consistent. Otherwise, the migration and synchronization performance can be poor.
   Kafka Endpoint	The endpoint of the self-managed Kafka cluster, which is in the IP address:port number format. You can configure multiple nodes for the Kafka data source. Separate the broker addresses with commas (,), for example, xxx.xxx.xxx.xxx:2883,xxx.xxx.xxx.xxx:2883.
   Enable SSL	Choose whether to enable Secure Sockets Layer (SSL) based on the business needs. To enable SSL, click Upload File and upload an SSL certificate suffixed with .jks.
   CA key	After the certificate is uploaded, enter the CA key for secure identity authentication and data encryption.
   Enable Authentication	Choose whether to enable authentication based on business needs. Kafka provides data encryption and multiple identity authentication mechanisms to ensure the security of user data and services.
   Authentication Method	If authentication is enabled, you must specify the authentication method. OMS supports the following authentication methods: GSSAPI, PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
   KDC Server Address	The IP address or domain name of the Kerberos Key Distribution Center (KDC) server. This parameter is displayed only if you set Authentication Method to GSSAPI.
   User Entity	Enter the username. This parameter is displayed only if you set Authentication Method to GSSAPI.
   SSL Certificate	Click Upload File and upload a key file suffixed with .keytab. This parameter is displayed only if you set Authentication Method to GSSAPI.
   Username	The username of the account used for data migration or synchronization. This parameter is not displayed if you set Authentication Method to GSSAPI.
   Password	The password of the account used for data migration or synchronization. This parameter is not displayed if you set Authentication Method to GSSAPI.
   Remarks (Optional)	Additional information about the data source.

5. Click Test Connection to test the network connection between OMS and the self-managed Kafka instance.

6. After the test is passed, click OK.