OceanBase Migration Service

V4.2.0Community Edition

OMS Documentation
What's New
OMS Community Edition Introduction
- What is OMS Community Edition?
- Terms
- OMS Community Edition HA
- Architecture
  - Overview
  - Hierarchical functional system
  - Basic components
- Limits
Quick Start
- Data migration process
- Data synchronization process
Deploy OMS Community Edition
- Deployment modes
- System and network requirements
- Memory and disk requirements
- Prepare the environment
- Deploy OMS Community Edition on a single node
- Deploy OMS Community Edition on multiple nodes in a single region
- Deploy OMS on multiple nodes in multiple regions
- Scale out OMS Community Edition
- Check the deployment
- Deploy a time-series database (Optional)
OMS Community Edition console
- Log on to the OMS Community Edition console
- Overview
- User center
  - Configure user information
  - Change your logon password
  - Log off
Data migration
- Overview
- Migrate data from a MySQL database to OceanBase Database Community Edition
- Migrate data from OceanBase Database Community Edition to a MySQL database
- Migrate data from HBase to OBKV
- Migrate data within OceanBase Database Community Edition
- Active-active disaster recovery within OceanBase Database Community Edition
- Migrate data from a TiDB database to OceanBase Database Community Edition
- Migrate data from a PostgreSQL database to OceanBase Database Community Edition
- Manage data migration projects
  - View details of a data migration project
  - Change the name of a data migration project
  - View and modify migration objects
  - Manage computing platforms
  - Use tags to manage data migration projects
  - Perform batch operations on data migration projects
  - Download and import settings of migration objects
  - Start and pause a data migration project
  - Release and delete a data migration project
- Features
  - DML filtering
  - DDL synchronization
  - Configure matching rules for migration objects
  - Wildcard rules
  - Rename a database table
  - Use SQL conditions to filter data
  - Create and update a heartbeat table
  - Schema migration mechanisms
  - Schema migration operations
  - Set an incremental synchronization timestamp
- Supported DDL operations in incremental migration and limits
  - DDL synchronization from MySQL database to OceanBase Community Edition
    - Overview of DDL synchronization from a MySQL database to a MySQL tenant of OceanBase Database
    - CREATE TABLE
      - Create a table
      - Create a column
      - Create an index or a constraint
      - Create partitions
    - Data type conversion
    - ALTER TABLE
      - Modify a table
      - Operations on columns
      - Operations on constraints and indexes
      - Operations on partitions
    - TRUNCATE TABLE
    - RENAME TABLE
    - DROP TABLE
    - CREATE INDEX
    - DROP INDEX
    - DDL incompatibilities between MySQL database and OceanBase Community Edition
      - Overview
      - Incompatibilities of the CREATE TABLE statement
        Incompatibilities of CREATE TABLE
        Column types that are supported to create indexes or constraints
      - Incompatibilities of the ALTER TABLE statement
        Incompatibilities of ALTER TABLE
        Change the type of a constrained column
        Change the type of an unconstrained column
        Change the length of a constrained column
        Change the length of an unconstrained column
        Delete a constrained column
      - Incompatibilities of DROP INDEX operations
  - Supported DDL operations in incremental migration from OceanBase Community Edition to a MySQL database and limits
  - Supported DDL operations in incremental migration between MySQL tenants of OceanBase Database
Data synchronization
- Data synchronization overview
- Create a project to synchronize data from OceanBase Database Community Edition to a Kafka instance
- Create a project to synchronize data from OceanBase Database Community Edition to a RocketMQ instance
- Manage data synchronization projects
  - View details of a data synchronization project
  - Change the name of a data synchronization project
  - View and modify synchronization objects
  - Use tags to manage data synchronization projects
  - Perform batch operations on data synchronization projects
  - Download and import the settings of synchronization objects
  - Start and pause a data synchronization project
  - Release and delete a data synchronization project
- Features
  - DML filtering
  - DDL synchronization
  - Rename a topic
  - Use SQL conditions to filter data
  - Column filtering
  - Data formats
Create and manage data sources
- Create data sources
  - Create an OceanBase-CE data source
  - Create a MySQL data source
  - Create a TiDB data source
  - Create a Kafka data source
  - Create a RocketMQ data source
  - Create a PostgreSQL data source
  - Create an HBase data source
- Manage data sources
  - View data source information
  - Copy a data source
  - Edit a data source
  - Delete a data source
- Create a database user
- User privileges
- Enable binlogs for the MySQL database
OPS & Monitoring
- O&M overview
- Go to the overview page
- Server
  - View server information
  - Update the quota
  - View server logs
- Components
  - Store
    - Create a store
    - View details of a store
    - Update the configurations of a store
    - Start and pause a store
    - Delete a store
  - Incr-Sync
    - View details of an Incr-Sync component
    - Start and pause an Incr-Sync component
    - Migrate an Incr-Sync component
    - Update the configurations of an Incr-Sync component
    - Batch O&M
    - Delete an Incr-Sync component
  - Full-Import
    - View details of a Full-Import component
    - Pause a Full-Import component
    - Rerun and resume a Full-Import component
    - Update the configurations of a Full-Import component
    - Delete a Full-Import component
  - Full-Verification
    - View details of a Full-Verification component
    - Pause a Full-Verification component
    - Rerun and resume a Full-Verification component
    - Update the configurations of a Full-Verification component
    - Delete a Full-Verification component
- O&M Task
  - View O&M tasks
  - Skip a task or subtask
  - Retry a task or subtask
System management
- Permission Management
  - Overview
  - Manage users
  - Manage departments
- Alert center
  - View project alerts
  - View system alerts
  - Manage alert settings
- Associate with OCP
- System parameters
  - Modify system parameters
  - Modify HA configurations
OMS Community Edition O&M
- Manage OMS services
- OMS logs
- Component O&M
  - O&M operations for the Supervisor component
  - CLI-based O&M for the Connector component
  - O&M operations for the Store component
- Component tuning
  - Incr-Sync or Full-Import tuning
- Component parameters
  - Coordinator
  - Condition
  - Source Plugin
    - Overview
    - StoreSource
    - DataFlowSource
    - LogProxySource
    - KafkaSource (TiDB)
    - HBaseSource
  - Sink Plugin
    - Overview
    - JDBC-Sink
    - KafkaSink
    - DatahubSink
    - RocketMQSink
    - HBaseSink
  - Store parameters
    - Parameters of a MySQL store
    - Parameters of an OceanBase store
  - Parameters of the CM component
  - Parameters of the Supervisor component
- Set throttling
Reference Guide
- API Reference
  - Overview
  - CreateProject
  - StartProject
  - StopProject
  - ResumeProject
  - ReleaseProject
  - DeleteProject
  - ListProjects
  - DescribeProject
  - DescribeProjectSteps
  - DescribeProjectStepMetric
  - DescribeProjectProgress
  - DescribeProjectComponents
  - ListProjectFullVerifyResult
  - StartProjectsByLabel
  - StopProjectsByLabel
  - CreateMysqlDataSource
  - CreateOceanBaseDataSource
  - ListDataSource
  - CreateLabel
  - ListAllLabels
  - ListFullVerifyInconsistenciesResult
  - ListFullVerifyCorrectionsResult
  - UpdateStore
  - UpdateFullImport
  - UpdateIncrSync
  - UpdateFullVerification
- OMS error codes
- Alert Reference
  - oms_host_down
  - oms_host_down_migrate_resource
  - oms_host_threshold
  - oms_migration_failed
  - oms_migration_delay
  - oms_sync_failed
  - oms_sync_status_inconsistent
  - oms_sync_delay
Upgrade Guide
- Overview
- Upgrade OMS in single-node deployment mode
- Upgrade OMS in multi-node deployment mode
- FAQ
FAQ
- General O&M
  - How do I modify the resource quotas of an OMS container?
  - How do I troubleshoot the OMS server down issue?
  - Deploy InfluxDB for OMS
  - Increase the disk space of the OMS host
- Project diagnostics
  - What do I do when a store does not have data of the timestamp requested by the downstream?
  - What do I do when OceanBase Store failed to access an OceanBase cluster through RPC?
- OPS & monitoring
  - What are the alert rules?
- Data synchronization
  - FAQ about synchronization to a message queue
    - What are the strategies for ensuring the message order in incremental data synchronization to Kafka
- Data migration
  - Full migration
    - How do I query the ID of a checker?
    - How do I query log files of the Checker component of OMS?
    - How do I query the verification result files of the Checker component of OMS?
    - What do I do if the destination table does not exist?
    - What can I do when the full migration failed due to LOB fields?
    - What do I do if garbled characters cannot be written into OceanBase Database V3.1.2?
  - Incremental synchronization
    - How do I skip DDL statements?
    - How do I update whitelists and blacklists?
    - What are the application scope and limits of ETL?
- Installation and deployment
  - How do I upgrade Store?
  - How do I upgrade CDC?
  - What do I do when the "Failed to fetch" error is reported?

Download PDF

OceanBase

OceanBase

OceanBase, A Highly Scalable Database for Transactional, Analytical, and AI Workloads.

Company

About OceanBase Contact Us Partner

Product

OceanBase Cloud OceanBase Enterprise Quick Start Download Pricing

Resources

Docs Blog White Paper Trust Center

Follow us

© OceanBase 2024. All rights reserved | Cloud Service Agreement | Privacy Policy | Legal | Security

Document Feedback

DatahubSink

Last Updated：2023-11-30 06:45:08 Updated

share

share

Starter class

datahub_sink-sink.jar

com.oceanbase.oms.connector.datahub.sink.DatahubSinkBooster

[Metadata] endpoint

Description: specifies the DataHub endpoint.
Default value: None. This parameter is required.
Value type: String.
Supported version: OMS V3.3.1 and later.

[Metadata] accessId

Description: specifies the AccessKey ID of DataHub.
Default value: None. This parameter is required.
Value type: String.
Supported version: OMS V3.3.1 and later.

[Metadata] accessKey

Description: specifies the AccessKey Secret of DataHub.
Default value: None. This parameter is required.
Value type: String.
Supported version: OMS V3.3.1 and later.

[Metadata] protocol

Description: specifies the Protocol for writing data to downstream topics.
Default value: TUPLE.
Valid values: BLOB and TUPLE.
Supported version: OMS V3.3.1 and later.

[Distribution] partitionMode

Description: specifies the distribution logic that determines the partition into which a data record is to be written.
Default value: HASH.
Valid values: HASH (hash value of the primary key) and TABLE (hash value of the table name).
Supported version: OMS V3.3.1 and later.

[Threads] workerNum

Description: specifies the number of data writing threads.
Default value: 4.
Value type: int.
Supported version: OMS V3.3.1 and later.

[DataHub parameters] connectionTimeoutMs

Description: specifies the timeout period of the connection to the DataHub client.
Default value: 10000 ms.
Value type: int.
Supported version: OMS V3.3.1 and later.

[DataHub parameters] enableBinary

Description: specifies whether to enable binary writing.
Default value: true.
Value type: Boolean.
Supported version: OMS V3.3.1 and later.

[Error handling] requestRetrySec

Description: specifies the allowed retry period after an error occurs. After this period expires, the process exits.
Default value: 3600s
Value type: int.
Supported version: OMS V3.3.1 and later.

[Serialization] serializerType

Description: specifies the serialization format.
Default value: DEFAULT.
Valid values: DEFAULT, DEFAULT_WITH_SCHEMA, SHAREPLEX, CANAL, DATAWORKS_V2, and DATAWORKS_V1.
Supported version: OMS V3.3.1 and later.

[Error handling] requestLimitSize

Description: specifies the maximum size of a writing request, which will be dynamically adjusted based on errors.
Default value: 4M.
Value type: int.
Supported version: OMS V3.4.0 and later.

[Error handling] tooLargeRequestErrorCodes

Description: specifies the error code for an excessively large request.
Default value: RequestEntityTooLarge,TooLargePayload.
Value type: String.
Supported version: OMS V3.4.0 and later.

Previous topic

Next topic