Load an Iceberg table through Catalog

Apache Iceberg is an open-source table format (Lake Table Format). The data files and metadata files of a table are stored in external storage (such as OSS, S3, or HDFS), and the Catalog manages the table's namespace and metadata pointers.

OceanBase Database connects to an external Catalog service (such as HMS Catalog or REST Catalog) via External Catalog, loads the Iceberg table metadata from the corresponding Catalog, and then reads the underlying data files. Iceberg tables cannot be accessed directly through simple file path external tables—they require metadata information such as metadata.json, snapshots, and manifests provided by the Catalog.

In the Iceberg ecosystem, the roles of each component are as follows:

OceanBase Database does not directly parse the object storage directory structure to "discover" Iceberg tables. Instead, it obtains the table definition from an external Catalog service via External Catalog, locates and reads the data files based on the metadata.

Supported capabilities

• Supported types

  Iceberg is a new generation of open table format specification. HMS Catalog extends its support to manage Iceberg tables with Hive metadata, achieving a unified metadata entry point.

  The CREATE EXTERNAL CATALOG syntax and parameters for each Catalog type shall comply with CREATE EXTERNAL CATALOG and the current version of the product documentation.

• Advanced features

  • Iceberg Schema Evolution (column addition and deletion)
  • Iceberg Partition Transform (such as bucket() and year() partition expressions)

Prerequisites

1. Version and mode: A tenant in MySQL-compatible mode. When accessing an Iceberg table via HMS Catalog, OceanBase Database must be V4.4.1 or later (for HMS Catalog configuration, see Load a Hive table through Catalog).
2. Iceberg table readiness: The target Iceberg table is registered in the corresponding Catalog, and its metadata is complete and available.
3. Storage access: All OBServer nodes have read permission for the Iceberg table's warehouse path; Java SDK must be deployed when using HDFS. For details, see Deploy the OceanBase Database JAVA SDK environment.
4. Catalog service accessibility: The network of the OceanBase cluster is accessible to the HMS Thrift endpoint, the REST Catalog service address, or the object storage/file system path.
5. Permissions: The current user must have permissions such as CREATE CATALOG and USE CATALOG. If Kerberos or AccessKey is required at the storage layer, additional Location or Catalog credentials need to be configured.

Create an External Catalog for accessing an Iceberg table

Load via HMS Catalog

When the Iceberg table metadata is in Hive Metastore, create an External Catalog with TYPE = 'HMS':

CREATE EXTERNAL CATALOG iceberg_hms
PROPERTIES (
    TYPE = 'HMS',
    URI = 'thrift://hms.example.com:9083'
);

For the creation procedure, Kerberos authentication, and location configuration, see Load a Hive table through Catalog.

Load by using REST Catalog

REST Catalog provides Iceberg metadata through HTTP APIs. When creating an External Catalog, you must specify the Catalog type as REST and configure parameters such as the REST service address, warehouse, and authentication information.

CREATE EXTERNAL CATALOG iceberg_rest
PROPERTIES (
    TYPE = 'REST',
    URI = '<rest_catalog_api_endpoint>',
    WAREHOUSE = '<warehouse_name_or_path>'
    -- Other parameters are subject to the CREATE EXTERNAL CATALOG syntax documentation.
);

The complete list of parameters for REST Catalog (such as OAuth2 authentication and WAREHOUSE format) varies depending on the implementation of the REST service. For specific information, see the CREATE EXTERNAL CATALOG document.

Query an Iceberg table

Switch Catalogs

SET CATALOG iceberg_hms;
USE analytics_db;

Alternatively, use a three-part identifier to avoid context switching:

SELECT * FROM iceberg_hms.analytics_db.events
WHERE event_date = '2025-04-01'
LIMIT 100;

Sample query

-- Aggregated query
SELECT product_id, SUM(sales_amount)
FROM iceberg_hms.analytics_db.sales
WHERE event_time >= '2025-04-01'
GROUP BY product_id;

-- Federated Query with Internal Tables
SELECT o.order_id, e.event_type
FROM internal.trade_db.orders o
JOIN iceberg_hms.analytics_db.events e ON o.user_id = e.user_id;

View metadata

SHOW CATALOGS;

DESC iceberg_hms.analytics_db.sales;

SHOW CREATE TABLE iceberg_hms.analytics_db.sales;

Iceberg table capabilities

The following capabilities apply to Iceberg tables accessed via an External Catalog. When loaded via HMS Catalog or REST Catalog, the features and column types supported by the Iceberg table are the same; the main difference lies in which Catalog service retrieves the metadata. If write capabilities differ due to version or Catalog type, the behavior of the current version shall prevail.

Read capability

Write capability

Data file format

The underlying data file format for Iceberg tables is primarily Parquet; metadata files include metadata.json, manifest, manifest-list, etc., which are parsed by OceanBase during queries.

Considerations

1. Distinguish between Catalog and storage: The Catalog provides metadata mapping, while the data files remain in external storage such as OSS, S3, or HDFS. Storage credentials must be configured separately.
2. Distinguish between table format and file format: Iceberg is a table format; Parquet is its commonly used underlying file format. These two should not be used interchangeably.
3. Iceberg tables in HMS: Hive 1.2.x / 2.3.x / 3.1.x does not natively manage Iceberg. However, if engines like Spark/Flink have registered Iceberg metadata with HMS, OceanBase can access it through the HMS Catalog. Refer to the Iceberg compatibility notes in Load a Hive table through Catalog.
4. Files not registered in the Catalog: Raw Parquet/ORC files on object storage are not Iceberg tables. You should use an external file table to access them.
5. Limitations: Parameters and behaviors of REST and FILESYSTEM Catalogs are subject to the documentation of the current product version. Iceberg features not listed in this topic are, by default, considered unsupported in the current version or require further confirmation.

References

• Overview of Data Lakes
• Catalog and External Tables
• Load a Hive Table through Catalog
• Hive Tables (HMS)
• CREATE EXTERNAL CATALOG