OceanBase DataStudio: From Stitched Pipelines to Unified AI Data Production

A Request Every ML Data Engineer Knows

"Filter me a batch of data: token count under 1000, non-QA, social science category, not synthetic."

One sentence. Two days to deliver.

The delay isn't in writing SQL — it's in the data stack underneath. Each feature (token count, QA flag, synthetic flag) lives in a different table, computed by a different UDF, with unclear definitions and unverified logic. You explore value ranges one by one, stitch together a long error-prone query, run it, and then verify the output.

The root cause: storage, computation, feature management, data governance, and data serving are split across multiple systems. Data moves between them. Consistency and traceability ultimately depend on people.

• Storage uses file formats in a data lake — adding a feature column means copying data, repeated joins, full table rewrites.
• Features scatter across UDFs in various pipeline components — filtering and computation are tangled, lineage is unclear, new data arrives with empty feature columns requiring manual backfill.
• Filtering is funnel-shaped — one UDF scoring stage feeds the next, no cross-feature analysis, no visibility into feature distributions for free combination.
• Filtered subsets don't record their logic — not reproducible, not auditable, not shareable.

Each problem is patchable individually. The structural issue is that the entire pipeline is stitched together from separate systems.

What OceanBase DataStudio Is

OceanBase DataStudio is a data production, governance, and serving workbench running on top of OceanBase Lakebase. It doesn't build another storage or compute system — it organizes the full chain of "data comes in, gets processed, gets governed, gets served, gets monitored" into one operational workflow.

OceanBase Lakebase provides the foundation: structured, semi-structured, unstructured, and vector data managed under a unified architecture. Database transactions, consistency, and real-time serving combine with the open storage and open compute of a data lake. Scalar, full-text, and vector search run on the same data foundation.

OceanBase DataStudio productizes these capabilities into a workbench that data teams can use directly: data ingestion, data processing, task orchestration, semantic modeling, quality governance, permission management, lineage analysis, dataset publishing, and data services — all in one pipeline.

Within the OceanBase AI Database product family, Lakebase is the core engine, DataStudio is the data development and governance layer, and OceanBase DataPilot is the business-facing data intelligence agent. DataStudio sits in the middle: Lakebase provides the data foundation; DataStudio makes data manageable and serviceable; DataPilot helps users consume data intelligence directly.

Wide-Table Storage: Add a Column, Not a Pipeline

A single data record — from raw file to trainable data to derived features — hangs on one wide table, linked by UUID. Large objects like images and video go into Volumes; the table stores only paths.

Configured as columnar storage, this brings two direct benefits: adding a new feature is just adding a column — no data copying, no repeated joins. Adding or removing feature columns doesn't trigger full table rewrites. Frequent feature column changes become lightweight operations.

Because the foundation is OceanBase, it inherits database capabilities natively: row locks, row-level UPSERT/MERGE INTO by UUID, strong consistency. OceanBase also supports hybrid retrieval — scalar, full-text, and vector search in one query. Training data scenarios frequently need vector recall combined with structured filtering, which is exactly this.

Two additional capabilities come from the base layer:

AI columns function as real-time computed columns on the table. When data is written, model computations like embedding and tagging fire automatically, with results written back to the table. The critical property is transactional consistency: a batch of audio files either all completes embedding and tagging, or all fails and rolls back. No dirty half-computed state — the exact thing that makes feature backfill painful.

Fork Database creates a complete database copy in seconds, like pulling a Git branch. Under the hood, Copy-on-Write means only incremental space is consumed. You can experiment freely with feature processing logic and filtering conditions on the branch. If it works, MERGE back to main. If it doesn't, discard — near-zero cost. Training data work is inherently iterative; this capability lets you experiment aggressively.

Daft on Ray for Multimodal Processing

Feature production frequently handles images and audio — work that calls models. Traditional offline engines like Spark face two problems here: each batch reloads models (expensive), and CPU and GPU can't be fed simultaneously (wasteful).

OceanBase DataStudio uses Daft on Ray as its multimodal computation engine. Ray's Actor model lets models load once and reuse across batches. Each operator is dynamically scheduled by micro-partition, stringing CPU and GPU processing into a pipeline. Large-scale multimodal feature backfill and processing accelerate on these two points.

Building a feature platform on top becomes simpler — not because DataStudio is a feature platform (it isn't), but because it provides the foundational capabilities that a feature platform needs.

Building a Feature Platform on Top

OceanBase DataStudio doesn't directly provide feature registration or backfill. What it does is prepare the foundation — making it far simpler to build training data production capabilities on top than assembling from scratch.

Consider what a feature platform typically needs:

These capabilities are straightforward to build because storage, computation, and hybrid search are already provided by DataStudio. Going back to the opening request: with features registered, distributions visible, cross-filtering available, and results traceable — "token count under 1000, non-QA, social science, not synthetic" becomes a single SQL query.

The complete chain looks like this: raw data enters (images, audio/video, logs batch-imported to the wide table, large objects to Volumes) → processing runs (Daft on Ray orchestrates splitting, frame extraction, embedding, tagging — AI columns auto-supplement on write) → registration (each feature's logic, type, source columns recorded, lineage auto-extracted) → filtering (cross-conditions run on the wide table, generating subsets with recorded logic) → publishing (subsets delivered as reproducible, auditable, shareable datasets for downstream training).

The "two-day" request becomes a single traceable SQL query.

How It Differs from Stitched Approaches

AI training data production already has established tools: Feature Store for features, open table formats for lake data, Spark/Ray/Daft for compute, vector databases and search engines for recall. Each has value individually. But when an enterprise stitches them together, complexity compounds: one system for storage, one for compute, one for feature management, one for search, one for governance. Data moves between every layer. Permissions need alignment. Lineage needs patching. Failures need retries. Consistency needs reconciliation.

OceanBase DataStudio's differentiation comes from OceanBase Lakebase's unified architecture — the core pipeline of AI data production sits on one foundation.

Compared to Feature Store Products (Tecton / Feast / Hopsworks)

These products specialize in feature management — but they're feature-layer middleware. They don't manage underlying multimodal storage or compute. Using Tecton, you still need to configure storage and Spark/Ray underneath, then wire them together.

With OceanBase DataStudio, storage (wide table, columnar, Volumes), computation (Daft on Ray), hybrid search, and row-level updates are built in. Building a feature platform is simpler, more usable, and doesn't require repeatedly moving data between the feature layer and storage or reconciling consistency.

Compared to Open Lakehouse Formats (Lance / LanceDB / Databricks)

Lance is an AI-native open lakehouse format — strong for AI training data and vector retrieval. But its foundation is a file format on object storage: append-biased, with weak row-level updates and transaction capabilities. The most common operation in training data backfill — "update one column of one row by UUID" — often means rewriting a file. On a database foundation, it's a standard row-level UPDATE.

Databricks is a mature lakehouse platform, but its foundation is similarly file format plus Spark, with transactions and updates handled through table-format semantics.

OceanBase DataStudio uses a database as the foundation: row locks, UPSERT/MERGE, strong consistency, add-column-on-demand, and native hybrid search are all built in. This is the value of a database over a pure lake — database capabilities where they matter.

Compared to Compute Engines (Spark / Ray Data / Daft)

Many platforms still push multimodal processing onto Spark, where model loading is expensive and CPU/GPU utilization stays low. OceanBase DataStudio chose Daft on Ray — a technical path better suited for multimodal workloads (Daft has also shown faster performance than Spark on official TPC-H benchmarks, though this is vendor-reported and should be verified independently) — and integrated it into the platform's development and scheduling workflow, rather than requiring users to set up their own Ray cluster.

Beyond Training Data

AI training data production is one scenario. The underlying foundation — combining the lake's multimodal capabilities with the database's transaction and retrieval capabilities — applies equally to any business where structured and unstructured data converge with strong transaction requirements and frequent updates.

Autonomous driving generates daily volumes of video, images, sensor data, and GPS traces. Teams need to find edge cases — extreme weather, unusual road layouts, collision risks — across massive driving datasets. Data needs splitting, frame extraction, scene recognition, and vectorization, then combined with geographic location, time, vehicle model, and scene tags for hybrid search. This requires not point tools but a unified foundation for managing and processing multimodal data.

Securities industry has structured data (market quotes, trades, financials, customer records) alongside unstructured data (research reports, disclosures, policy documents, news sentiment). Real business value comes from combining both — research report parsing, compliance Q&A, investment knowledge retrieval all need structured business data and unstructured text in one data service system.

As agents become data consumers, this becomes even more critical. Agents need not just a one-time query result but continuous, trustworthy, traceable context — access to business data, documents, vectors, state, memory, and execution records, within permission and governance boundaries.

This is the combined value of DataStudio and Lakebase. Lakebase provides the lakebase-integrated data foundation. DataStudio organizes data production, governance, and service workflows. The former solves where data lives and how it's unified and computed; the latter solves how data is produced, governed, delivered, and reused.

The Architecture Shift

Compressing a training data filtering request from two days to a single traceable SQL query doesn't come from any single feature. It comes from a change in the underlying data architecture.

In the past, data scattered across multiple systems — moved by people, stitched by scripts, backed by experience. In the AI era, this approach can no longer support the high-frequency calls from models and agents.

OceanBase DataStudio puts data ingestion, processing, governance, filtering, tracing, and serving into one pipeline. OceanBase Lakebase makes that pipeline run on a data foundation with multimodal capabilities, hybrid search, transactional consistency, and open compute.

Training data production is just the first typical scenario. More broadly, for AI applications to truly enter production systems, enterprises must first manage, organize, and serve their data well. That's the problem OceanBase AI Database is built to solve — not adding a few AI features to a database, but making enterprise data a production-grade asset that models and agents can continuously understand, retrieve, and call.

To learn more about how OceanBase DataStudio fits into your AI data pipeline, explore the OceanBase AI Database documentation.