One of many key challenges that organizations face when adopting the open information lakehouse is deciding on the optimum format for his or her information. Among the many out there choices, Linux Basis Delta Lake, Apache Iceberg, and Apache Hudi are all wonderful storage codecs that allow information democratization and interoperability. Any of those codecs is best than placing your information right into a proprietary format. Nonetheless, selecting a single storage format to standardize on could be a daunting process, which can lead to choice fatigue and concern of irreversible penalties.

Delta UniForm (quick for Delta Lake Common Format) gives a easy, straightforward to implement, seamless unification of desk codecs with out creating extra information copies or silos. On this weblog, we’ll cowl the next:

A number of codecs, single copy of knowledge

Delta UniForm takes benefit of the truth that Delta Lake, Iceberg, and Hudi are all constructed on Apache Parquet information information. The primary distinction among the many codecs is within the metadata layer, and even then, the variations are delicate. The metadata for all three codecs serves the identical goal and comprises overlapping units of data.

Previous to the discharge of Delta UniForm, the methods to change between open desk codecs had been copy- or conversion-based and solely supplied a point-in-time view of the information. In distinction, Delta UniForm solves interoperability wants extra elegantly by offering a stay view of the information for all readers, no matter format.

Below the hood, Delta UniForm works by robotically producing the metadata for Iceberg and Hudi alongside Delta Lake – all towards a single copy of the Parquet information. Because of this, groups can use probably the most appropriate software for every information workload and all function on a single information supply, with good interoperability throughout the three totally different ecosystems.

Quick setup, minimal overhead

Delta UniForm is extraordinarily straightforward to arrange, and as soon as it is enabled it really works seamlessly and robotically.

To start out, let’s create a Delta UniForm desk to generate Iceberg metadata:

CREATE TABLE most important.default.UniForm_demo_table (msg STRING)
TBLPROPERTIES('delta.universalFormat.enabledFormats' = 'iceberg');

With Delta UniForm tables, the metadata for the extra codecs is robotically created upon desk creation and up to date every time the desk is modified. This implies there is no such thing as a want for handbook refresh instructions or operating pointless compute to translate desk codecs. For instance, let’s write a row to this desk:

INSERT INTO most important.default.UniForm_demo_table (msg) VALUES ("hey UniForm!");

This command triggers a Delta Lake commit, which then robotically and asynchronously generates the Iceberg metadata for this desk. By doing this, Delta UniForm ensures information pipelines are uninterrupted, enabling seamless entry to probably the most up-to-date info for all readers.

Delta UniForm has negligible efficiency and useful resource overhead, making certain optimum utilization of computational sources. Even for petabyte-scale tables, the metadata is often a tiny fraction of the information file dimension. As well as, Delta UniForm is ready to incrementally generate metadata scoped to solely the adjustments for the reason that earlier commit.

Studying Delta UniForm as Iceberg

Delta UniForm generates Iceberg metadata in accordance with the Apache Iceberg specification, which suggests when information is written to a Delta UniForm desk, the desk might be learn as Iceberg by any shopper within the Iceberg ecosystem that adheres to the open supply Iceberg specification.

Per the Iceberg specification, reader purchasers should work out which Iceberg metadata represents the most recent, newest model of the Iceberg desk. Throughout the Iceberg ecosystem, we have seen purchasers take two totally different approaches to this, each of that are supported by UniForm. We’ll clarify the variations right here after which present examples within the subsequent part.

Some Iceberg readers require customers to offer the trail to a metadata file representing the most recent snapshot of the Iceberg desk. This method might be cumbersome for patrons because it requires customers to offer up to date metadata file paths each time the desk adjustments.

As a substitute, the Iceberg group recommends utilizing the REST catalog API. The shopper talks to the catalog to get the most recent state of the desk, permitting customers to learn the most recent state of an Iceberg desk with out handbook refreshes or worrying about metadata paths.

Unity Catalog now implements the open Iceberg Catalog REST API in accordance with the Apache Iceberg specification. That is aligned with Unity Catalog’s dedication to supporting open APIs, and builds on the momentum of Unity Catalog’s HMS API assist. The Unity Catalog Iceberg REST API gives open entry to UniForm tables within the Iceberg format with none costs for Databricks compute, whereas permitting interoperability and auto-refresh assist for accessing the most recent information. As a byproduct, this could allow different catalogs to federate to Unity Catalog and assist Delta UniForm tables.

Within the subsequent part, we’ll stroll via examples of Delta UniForm’s assist for each the metadata path and Iceberg REST Catalog API approaches.

Instance: learn Delta Lake as Iceberg in BigQuery by supplying metadata location

When studying Iceberg in an current catalog, BigQuery requires you to offer a pointer to the JSON file representing the most recent Iceberg snapshot (BigQuery documentation), like the next:

In BigQuery:

CREATE EXTERNAL TABLE myexternal-desk
  WITH CONNECTION `myproject.us.myconnection`
  OPTIONS (
         format = 'ICEBERG',
         uris = ["gs://mybucket/mydata/mytable/metadata/iceberg.metadata.json"]
   )

Delta UniForm with Unity Catalog makes it straightforward so that you can discover the required Iceberg metadata file path. Unity Catalog exposes plenty of Delta Lake desk properties, together with this path. You possibly can retrieve metadata location in your Delta UniForm desk through UI or API.

Retrieving Delta UniForm Iceberg metadata path through UI:

Navigate to your Delta UniForm desk within the Databricks Knowledge Explorer, then click on on the Particulars tab. Right here, you will discover the Delta UniForm Iceberg row containing the metadata path.

In Databricks:

Retrieving Delta UniForm Iceberg metadata location through API:

From a software of your selecting, submit the next GET request to retrieve your Delta UniForm desk’s Iceberg metadata location.

GET api/2.1/unity-catalog/tables/<catalog-identify>.<schema-identify>.<desk-identify>

The delta_uniform_iceberg.metadata_location discipline within the response comprises the metadata location for the most recent Iceberg snapshot.

Merely paste the situation from both the UI or API strategies outlined above into the aforementioned BigQuery command, and BigQuery will learn the snapshot as Iceberg.

In case your desk will get up to date, you’ll have to present BigQuery with the up to date metadata location to learn the most recent information. For manufacturing use circumstances, you need to add a step in your ingestion pipeline that updates BigQuery with the most recent Iceberg metadata path(s) each time you write to the Delta UniForm desk. Be aware that the necessity for metadata path updates is a normal limitation with this method, and isn’t particular to UniForm.

Instance: Learn Delta Lake as Iceberg in Trino through REST Catalog API

Let’s now learn the identical Delta UniForm desk we created earlier via Trino utilizing Unity Catalog’s Iceberg REST Catalog API.

Be aware: Uniform just isn’t crucial for studying Delta tables with Trino as Trino immediately helps Delta tables. That is simply for example how Uniform additional expands the interoperability within the open supply ecosystem.

After organising Trino, you’ll be able to modify Iceberg properties by updating the and so forth/catalog/iceberg.properties file to configure Trino to make use of Unity Catalog’s Iceberg REST API Catalog endpoint:

connector.identify=iceberg
iceberg.catalog.sort=relaxation
iceberg.rest-catalog.uri={UNITY_CATALOG_ICEBERG_URL}
iceberg.rest-catalog.safety=OAUTH2
iceberg.rest-catalog.oauth2.token={PERSONAL_ACCESS_TOKEN}

The place:

As soon as your properties file is configured, you’ll be able to run the Trino CLI and challenge an Iceberg question to the Delta UniForm desk:

SELECT * FROM iceberg."most important.default".UniForm_demo_table

Since Trino implements the Apache Iceberg REST Catalog API, we did not create any exterior desk, nor did we have to provide the trail to the most recent Iceberg metadata information. Trino robotically fetches the most recent Iceberg metadata from UC after which reads the most recent information within the Delta UniForm desk.

You will need to observe that, from Trino’s perspective, there’s nothing Delta UniForm-specific taking place right here. It’s studying an Iceberg desk, whose metadata has been generated to spec, and retrieving that metadata with a regular REST API name to an Iceberg catalog.

That is the simplicity of Delta UniForm. To Delta Lake writers and readers, the Delta UniForm desk is a Delta Lake desk. To Iceberg readers, the Delta UniForm desk is an Iceberg desk – all on a single set of knowledge information with out pointless copies of knowledge and tables.

Delta UniForm Influence

All through its Preview, we have already helped many purchasers speed up in direction of the open information lakehouse interoperability with Delta UniForm. Organizations can write as soon as to Delta Lake, after which entry this information any approach, attaining optimum efficiency, cost-effectiveness, and information flexibility throughout numerous workloads comparable to ETL, BI, and AI – all with out the burden of expensive and complicated migrations.

“At Instacart, our imaginative and prescient is to have an open information lakehouse with a single copy of knowledge that’s interoperable with all compute platforms. Delta UniForm is instrumental to that purpose. With Delta UniForm, we will shortly and simply generate tables that may be learn as both Delta Lake or Iceberg, unlocking interoperability with all of the instruments in our ecosystem.”

— Doug Hyde, a Sr. Workers Software program Engineer at Instacart, shared his expertise with Delta UniForm

Databricks’ mission is to assist information groups resolve the world’s hardest issues, and that begins with with the ability to use the fitting software for the fitting job with out having to make copies of your information. We’re excited concerning the enhancements in interoperability that Delta UniForm brings and can proceed to take a position on this space for years to return.

Delta UniForm is obtainable as a part of the preview launch candidate for Delta Lake 3.0. Databricks prospects also can preview Delta UniForm with Databricks Runtime model 13.2 or the Databricks SQL 2023.35 preview channel.