Create a task to synchronize data from OceanBase Database Community Edition to a Kafka instance|V4.2.10|OceanBase Migration Service| docs|Distributed Database

OceanBase Migration Service

V4.2.10Community Edition

OMS Documentation
What's New
OMS Community Edition Introduction
- What is OMS Community Edition?
- Terms
- OMS Community Edition HA
- Architecture
  - Overview
  - Hierarchical functional system
  - Basic components
- Limitations
Quick Start
- Data migration process
- Data synchronization process
Deploy OMS Community Edition
- Deployment modes
- System and network requirements
- Memory and disk requirements
- Prepare the environment
- Deploy OMS Community Edition on a single node
- Deploy OMS Community Edition on multiple nodes in a single region
- Deploy OMS Community Edition on multiple nodes in multiple regions
- Integrate the OIDC protocol into OMS Community Edition to implement SSO
- Scale out deployment
- Check the deployment
- Deploy a time-series database (Optional)
OMS Community Edition console
- Log in to the console of OMS Community Edition
- Overview
- User center
  - Configure user information
  - Change your logon password
  - Log off
Data migration
- Overview
- Migrate data from a MySQL database to OceanBase Database Community Edition
- Migrate data from HBase to OBKV
- Migrate incremental data from OBKV to HBase
- Migrate data from a Milvus database to OceanBase Database Community Edition
- Migrate data from a Redis database to an OBKV-Redis database
- Migrate data between tenants of OceanBase Database Community Edition
- Migrate data in active-active disaster recovery scenarios
- Migrate data from a TiDB database to OceanBase Database Community Edition
- Migrate data from a PostgreSQL database to OceanBase Database Community Edition
- Migrate data from Hive to OceanBase Database Community Edition
- Migrate data from an ElasticSearch database to OceanBase Database Community Edition
- Migrate data from OceanBase Database Community Edition to an ElasticSearch database
- Migrate data from a MongoDB database to OceanBase Database Community Edition
- Migrate data from a Qdrant database to OceanBase Database Community Edition
- Manage data migration tasks
  - View details of a data migration task
  - Change the name of a data migration task
  - View and modify migration objects
  - Manage computing platforms
  - Use tags to manage data migration tasks
  - Perform batch operations on data migration tasks
  - Download and import settings of migration objects
  - Start and pause a data migration task
  - Release and delete a data migration task
- Features
  - DML filtering
  - DDL synchronization
  - Configure matching rules for migration objects
  - Wildcard rules
  - Rename a database table
  - Use SQL conditions to filter data
  - Create and update a heartbeat table
  - Schema migration mechanisms
  - Schema migration operations
  - Set an incremental synchronization timestamp
  - Direct load
- Supported DDL operations in incremental migration and limits
  - DDL synchronization from MySQL database to OceanBase Community Edition
    - Overview of DDL synchronization from a MySQL database to a MySQL-compatible tenant of OceanBase Database
    - CREATE TABLE
      - Create a table
      - Create a column
      - Create an index or a constraint
      - Create partitions
    - Data type conversion
    - ALTER TABLE
      - Modify a table
      - Operations on columns
      - Operations on constraints and indexes
      - Operations on partitions
    - TRUNCATE TABLE
    - RENAME TABLE
    - DROP TABLE
    - CREATE INDEX
    - DROP INDEX
    - DDL incompatibilities between MySQL database and OceanBase Community Edition
      - Overview
      - Incompatibilities of the CREATE TABLE statement
        Incompatibilities of CREATE TABLE
        Column types that are supported to create indexes or constraints
      - Incompatibilities of the ALTER TABLE statement
        Incompatibilities of ALTER TABLE
        Change the type of a constrained column
        Change the type of an unconstrained column
        Change the length of a constrained column
        Change the length of an unconstrained column
        Delete a constrained column
      - Incompatibilities of DROP INDEX operations
  - Supported DDL operations in incremental migration between MySQL-compatible tenants of OceanBase Database
Data synchronization
- Data synchronization overview
- Create a task to synchronize data from OceanBase Database Community Edition to a Kafka instance
- Create a task to synchronize data from OceanBase Database Community Edition to a RocketMQ instance
- Manage data synchronization tasks
  - View details of a data synchronization task
  - Change the name of a data synchronization task
  - View and modify synchronization objects
  - Use tags to manage data synchronization tasks
  - Perform batch operations on data synchronization tasks
  - Download and import the settings of synchronization objects
  - Start and pause a data synchronization task
  - Release and delete a data synchronization task
- Features
  - DML filtering
  - DDL synchronization
  - Configure matching rules for synchronization objects
  - Rename a topic
  - Use SQL conditions to filter data
  - Column filtering
  - Data format description
Create and manage data sources
- Create data sources
  - Create an OceanBase-CE data source
  - Create a MySQL data source
  - Create a TiDB data source
  - Create a Kafka data source
  - Create a RocketMQ data source
  - Create a PostgreSQL data source
  - Create an HBase data source
  - Create a Qdrant data source
  - Create a Milvus data source
  - Create a Redis data source
  - Create a Hive data source
  - Create an ElasticSearch data source
  - Create a MongoDB data source
- Manage data sources
  - View data source information
  - Copy a data source
  - Edit a data source
  - Delete a data source
  - Cache system views
- Create a database user
- User privileges
- Enable binlogs for the MySQL database
OPS & Monitoring
- O&M overview
- Go to the overview page
- Server
  - View server information
  - Update the quota
  - View server logs
- Components
  - Store
    - Create a Store component
    - View details of a Store component
    - Update the configurations of a Store component
    - Start and pause a Store component
    - Delete a Store component
  - Incr-Sync
    - View details of an Incr-Sync component
    - Start and pause an Incr-Sync component
    - Migrate an Incr-Sync component
    - Update the configurations of an Incr-Sync component
    - Batch O&M
    - Delete an Incr-Sync component
  - Full-Import
    - View details of a Full-Import component
    - Pause a Full-Import component
    - Rerun and resume a Full-Import component
    - Update the configurations of a Full-Import component
    - Delete a Full-Import component
  - Full-Verification
    - View details of a Full-Verification component
    - Pause a Full-Verification component
    - Rerun and resume a Full-Verification component
    - Update the configurations of a Full-Verification component
    - Delete a Full-Verification component
  - Component parameters
    - Coordinator
    - Condition
    - Source Plugin
      - Overview
      - StoreSource
      - DataFlowSource
      - LogProxySource
      - KafkaSource (TiDB)
      - HBaseSource
      - HiveSource
      - ElasticSearchSource
      - MongoDBSource
    - Sink Plugin
      - Overview
      - JDBC-Sink
      - KafkaSink
      - DatahubSink
      - RocketMQSink
      - HBaseSink
      - HiveSink
    - Store parameters
      - Parameters of a MySQL store
      - Parameters of an OceanBase store
    - Parameters of the CM component
    - Parameters of the Supervisor component
    - Parameters of the Full-Verification component
- O&M Task
  - View O&M tasks
  - Skip a task or subtask
  - Retry a task or subtask
System management
- Permission Management
  - Overview
  - Manage users
  - Manage departments
- Alert center
  - View task alerts
  - View system alerts
  - Manage alert settings
- Associate with OCP
- System parameters
  - Modify system parameters
  - Modify HA configurations
OMS Community Edition O&M
- Manage OMS services
- OMS logs
- Component O&M
  - O&M operations for the Supervisor component
  - CLI-based O&M for the Connector component
  - O&M operations for the Store component
- Component tuning
  - Incr-Sync or Full-Import tuning
- Set throttling
Reference Guide
- API Reference
  - Overview
  - CreateProject
  - StartProject
  - StopProject
  - ResumeProject
  - ReleaseProject
  - DeleteProject
  - ListProjects
  - DescribeProject
  - DescribeProjectSteps
  - DescribeProjectStepMetric
  - DescribeProjectProgress
  - DescribeProjectComponents
  - ListProjectFullVerifyResult
  - StartProjectsByLabel
  - StopProjectsByLabel
  - CreateMysqlDataSource
  - CreateMySQLMasterSlaveDataSource
  - CreateOceanBaseDataSource
  - ListDataSource
  - CreateLabel
  - ListAllLabels
  - ListFullVerifyInconsistenciesResult
  - ListFullVerifyCorrectionsResult
  - UpdateStore
  - UpdateFullImport
  - UpdateIncrSync
  - UpdateFullVerification
- OMS error codes
- Alert Reference
  - oms_host_down
  - oms_host_down_migrate_resource
  - oms_host_threshold
  - oms_migration_failed
  - oms_migration_delay
  - oms_sync_failed
  - oms_sync_status_inconsistent
  - oms_sync_delay
- Telemetry parameters
Upgrade Guide
- Overview
- Upgrade in single-node deployment mode
- Upgrade in multi-node deployment mode
- Upgrade the CDC library
- FAQ
FAQ
- General O&M
  - How do I modify the resource quotas of an OMS container?
  - Clear files in the Store component
  - How do I troubleshoot the OMS server down issue?
  - Deploy InfluxDB for OMS
  - Increase the disk space of the OMS host
- Task diagnostics
  - What do I do when a store does not have data of the timestamp requested by the downstream?
  - What do I do when OceanBase Store failed to access an OceanBase cluster through RPC?
- OPS & monitoring
  - What are the alert rules?
- Data synchronization
  - FAQ about synchronization to a message queue
    - What are the strategies for ensuring the message order in incremental data synchronization to Kafka
  - Performance troubleshooting and tuning for data synchronization from OceanBase Community Edition to Kafka
- Data migration
  - Full migration
    - How do I query the ID of a checker?
    - How do I query log files of the Checker component of OMS?
    - How do I query the verification result files of the Checker component of OMS?
    - Garbled characters in the Latin1 character set
    - What do I do if the target table does not exist?
    - What can I do when the full migration failed due to LOB fields?
    - What do I do if garbled characters cannot be written into OceanBase Database V3.1.2?
  - Incremental synchronization
    - How do I skip DDL statements?
    - How do I update whitelists and blacklists?
    - What are the application scope and limits of ETL?
- Logon and password
  - What do I do if my logon password is locked?
- Installation and deployment
  - How do I upgrade Store?
  - What do I do when the "Failed to fetch" error is reported?
  - Change port numbers for components
  - Switching between the primary and standby OMS MetaDBs