Overview of object storage

Object storage is a core dependency component of an OceanBase AI cluster. When creating an AI cluster, you must pre-configure the connection information for object storage through an external integration module, including the bucket path, endpoint address, AccessKey credentials, and so on.

Background information

AI clusters adopt a storage-and-compute-separation architecture, where data is stored on object storage services rather than local disks. To securely and conveniently manage object storage access credentials, OCP provides an external integration module to centrally manage object storage configurations.

Managing object storage through external integration enables you to:

• Centralized management: Manage the object storage configurations used by multiple clusters in one place, avoiding configuration chaos caused by decentralized management.
• Lifecycle management: Manage the entire lifecycle of object storage, including creation, update, and deletion operations. Deletion only removes the configuration record from OCP; it does not delete the AccessKey credentials on the cloud vendor side.
• Credential security: Securely store AccessKey credentials, which are automatically injected during cluster creation, eliminating the need for manual entry.

Key concepts

• Object storage: A distributed storage service that supports data access via RESTful APIs. Common object storage services include Alibaba Cloud OSS, AWS S3, Tencent Cloud COS, and Huawei Cloud OBS.

• Bucket: A container in object storage used to store objects (files). Each bucket has a unique domain name and access address.

• Endpoint: The access endpoint address for the object storage service. Different regions have different endpoints.

• AccessKey: The credential for accessing object storage, consisting of an AccessKey ID and AccessKey Secret, used for authentication.

Application scenarios

Object storage serves as the primary data storage space for OceanBase AI clusters and their dependent service, Logservice. AI clusters adopt a storage-and-compute-separation architecture, with all data files, log files, and more stored on object storage services rather than local disks.

Considerations

• Independent Buckets: When creating AI clusters and Logservice clusters, use independent Buckets separately; mixed use is not supported. It is recommended to configure independent Buckets for different types of clusters to ensure data isolation and security.

• Least privilege principle: Configure the minimum necessary permissions for object storage AccessKeys, adhering to the least privilege principle.

• Regular rotation: Regularly rotate AccessKeys to reduce the risk of credential leakage.

Supported storage types

OCP supports the following object storage types:

Related operations

• Create object storage configuration
• Delete object storage configuration
• Best practices for object storage