How I Discovered Apache Iceberg: Understanding Modern Lakehouses

While browsing through Cloudflare’s platform offerings recently, a product name caught my eye: R2 Data Catalog.

Its description read:

“Managed Apache Iceberg catalog with automated table maintenance — Turn your R2 bucket into a managed data lake.”

I wasn’t actively shopping for a data lake solution, but that line made me pause. What exactly is Apache Iceberg? Why does an object storage bucket need a dedicated “table catalog” and “table maintenance” to serve as a data lake? And why was an open-source table format gaining so much traction across modern data engineering?

That product description sent me down a rabbit hole. As I started researching, I realized how much the data lake landscape has evolved—and how many painful problems of the past Apache Iceberg was designed to solve.

In this post, I want to share what I learned: why legacy data lakes were notoriously fragile, how Apache Iceberg fundamentally redesigns table management, and what running it actually looks like in practice.

---

The “Data Lake” Era (and Why Hive Tables Broke Down)

For years, the standard recipe for building an analytical “data lake” looked like this:

1. Dump gigabytes or terabytes of raw logs and event streams into an S3 bucket.
2. Convert those files into columnar formats like Apache Parquet or ORC.
3. Partition the files into directory hierarchies based on date or category:

   s3://my-lakehouse/events/year=2026/month=09/day=11/part-0001.parquet
   s3://my-lakehouse/events/year=2026/month=09/day=11/part-0002.parquet

4. Register the directory in an Apache Hive Metastore (HMS) or AWS Glue Data Catalog so engines like Presto, Trino, Spark, or Athena could query it using SQL.

This architecture, originally popularized by Apache Hive in the early Hadoop days, served the industry for over a decade. But as datasets grew to petabyte scales and write frequencies increased from daily batch jobs to near-real-time streaming, the cracks in the Hive model became impossible to ignore.

1. The S3 LIST Operation Bottleneck

Hive-style tables don’t track files explicitly; they track directories. When a query engine needed to plan a query over a partition, it had to issue an object storage LIST request across the bucket.

On object stores like S3, LIST operations are notoriously slow (paginated at 1,000 keys per call) and computationally expensive. If a partition had tens of thousands of files, query planning alone could take several minutes before a single row of data was actually read.

2. No ACID Guarantees (The Broken Ingestion Problem)

Object stores are key-value systems; they do not have transactional multi-file commits. If an ETL pipeline was halfway through writing 50 Parquet files to a partition and crashed:

• The destination partition was left in a corrupted, half-written state.
• Readers querying the table concurrently saw partial, inconsistent data.
• Cleaning up required manual, risky intervention.

3. Schema Evolution Headaches

Need to drop a column, rename one, or reorder fields? In Hive tables, schema was tied to column positions or names inside individual Parquet files. Renaming or reordering columns often broke historical files or produced silent nulls in downstream dashboards.

4. Partitioning Was Rigid and Fragile

If you initially partitioned by day and later wanted to repartition by hour (or by customer_id), you had to rewrite every single existing file in your lakehouse. Furthermore, users had to remember the explicit physical partitioning scheme in every query predicate (WHERE year=2026 AND month=09 AND day=11), or risk triggering a full bucket scan.

---

Enter Apache Iceberg: The Open Table Format

To fix these problems, Netflix created Apache Iceberg (later open-sourced through the Apache Software Foundation).

The single most important concept to grasp about Apache Iceberg is this:

Apache Iceberg is not a database engine, nor is it a file format. It is an open table format specification.

Where Parquet defines how bytes are laid out inside a single file, Iceberg defines how hundreds or millions of files are organized and tracked as a single, consistent table.

The Core Shift: State in Metadata, Not Directories

Instead of inferring table state by scanning directory paths, Iceberg tracks every single data file explicitly in a hierarchical tree of immutable metadata files.

graph LR
    Catalog["<b>Catalog Pointer</b><br/>R2 Data Catalog"]:::catalog
    Meta["<b>Table Metadata</b><br/><code>v3.metadata.json</code>"]:::meta
    Snap["<b>Manifest List</b><br/><code>snap-1092834.avro</code>"]:::manifestList
    
    subgraph Manifests["Manifest Layer"]
        M1["<code>m1.avro</code><br/>Stats & Bounds"]:::manifest
        M2["<code>m2.avro</code><br/>Stats & Bounds"]:::manifest
    end

    subgraph DataFiles["Data Layer"]
        F1[("f1.parquet")]:::dataFile
        F2[("f2.parquet")]:::dataFile
        F3[("f3.parquet")]:::dataFile
        F4[("f4.parquet")]:::dataFile
    end

    Catalog --> Meta --> Snap
    Snap --> M1
    Snap --> M2
    M1 --> F1 & F2
    M2 --> F3 & F4

    classDef catalog fill:#0284c7,stroke:#38bdf8,color:#ffffff,font-weight:600
    classDef meta fill:#1e293b,stroke:#38bdf8,color:#f8fafc
    classDef manifestList fill:#1e293b,stroke:#94a3b8,color:#f8fafc
    classDef manifest fill:#0f172a,stroke:#64748b,color:#e2e8f0
    classDef dataFile fill:#161b26,stroke:#334155,color:#cbd5e1

This tree structure provides several key capabilities:

1. Snapshot Isolation and True ACID Transactions

Every write, append, update, or delete in Iceberg creates a new Snapshot. Reads always query a specific snapshot. If a write fails midway, the new snapshot is never committed—readers only see clean, validated snapshots. Writers commit via an atomic compare-and-swap (CAS) operation on the catalog pointer.

2. High-Performance Metadata Pruning

Each Manifest File stores column-level statistics (min/max values, null counts) for every Parquet data file it tracks.

When you run a query like:

SELECT * FROM events 
WHERE user_id = 42 AND event_time >= '2026-09-01';

The query engine doesn’t even need to touch object storage data files or list directories. It simply inspects the manifest metadata, prunes out 99% of data files that cannot possibly contain matching values, and only downloads the exact Parquet files needed.

3. Safe Schema Evolution

Iceberg assigns every column a permanent, unique integer ID that never changes. You can add, rename, reorder, or drop columns freely. Older Parquet files written with previous column names continue to read seamlessly without rewriting data.

4. Hidden Partitioning

Iceberg abstracts partition transforms (such as month(event_time) or bucket(16, user_id)). Users query the logical column directly (WHERE event_time >= '2026-09-01'), and Iceberg automatically derives which partitions to scan. If you change the partition scheme later, Iceberg supports Partition Evolution—new data is written with the new spec while historical data remains untouched.

5. Time Travel and Rollbacks

Because snapshots are immutable, you can query historical versions of your table out of the box:

SELECT * FROM events FOR SYSTEM_TIME AS OF '2026-09-01 12:00:00 UTC';

If a faulty deployment inserts corrupted rows, rolling back is as simple as repointing the table to the previous snapshot.

---

The Catch: Catalogs & “Day-2” Table Maintenance

While Iceberg solved the table format dilemma, adopting it in production historically introduced two operational burdens:

1. You Need a Catalog

To make atomic commits, multiple engines (Spark, Trino, DuckDB) need a shared central authority that holds the current metadata.json pointer.

While AWS Glue or Hive Metastore were commonly used, the community increasingly rallied around the Apache Iceberg REST Catalog Specification—an open standard allowing any HTTP service to act as a catalog. Still, self-hosting a REST catalog (like Apache Polaris or Project Nessie) means provisioning servers, managing databases, and securing credentials.

2. The Maintenance Overhead (Compaction & Cleanup)

When you write streaming data or micro-batches into Iceberg, you inevitably create the small file problem—thousands of small Parquet files and manifest entries.

To keep query performance fast and storage costs low, someone has to run table maintenance jobs:

• Compaction: Combining hundreds of small Parquet files into optimal 128 MB or 512 MB files.
• Snapshot Expiration: Deleting old snapshots and manifests that are past retention limits.
• Orphan File Cleanup: Purging abandoned files from failed writes.

In most organizations, this meant setting up scheduled Apache Spark clusters just to run housekeeping jobs like rewrite_data_files().

---

Connecting the Dots: How Cloudflare’s Catalog Fits In

Seeing Cloudflare introduce R2 Data Catalog is what originally prompted this exploration, and looking closely at their implementation is a good real-world example of how modern lakehouses are structured.

Rather than building a proprietary database engine or locking data into custom formats, Cloudflare implemented an open Iceberg architecture:

1. The Open REST Catalog Standard

Cloudflare’s catalog implements the open Apache Iceberg REST Catalog specification. Because it follows an open standard, any query engine or tool that supports Iceberg REST can interact with it directly:

• DuckDB
• PyIceberg
• Apache Spark
• Trino
• ClickHouse
• Snowflake
• Databricks

There is no proprietary vendor lock-in here. The underlying data remains standard Parquet files in object storage, and the table metadata adheres strictly to the open Iceberg specification.

2. Offloading Table Maintenance

As mentioned earlier, running compaction and snapshot expiration is typically a major operational hurdle that requires dedicated Spark clusters or scheduled jobs. Cloudflare handles background compaction and snapshot cleanup directly at the storage level, which removes the need to maintain separate maintenance infrastructure when testing or running smaller workloads.

3. Cross-Tool Exploration Without Egress Costs

In typical cloud environments like AWS S3, data egress charges make it costly to query data across regions or from local developer machines. Because R2 doesn’t charge egress fees, it serves as a convenient environment to experiment with Iceberg across different tools—such as running quick local queries from a laptop using DuckDB, running scripts via PyIceberg, or piping data into other analytics engines without worrying about data transfer costs.

---

Hands-On: Testing Iceberg with Python and DuckDB

To see how an Iceberg REST catalog works in practice, here is a walkthrough of setting up a table and querying it using both PyIceberg and DuckDB.

1. Enabling the Catalog via Wrangler

You can enable the Iceberg catalog directly on an existing R2 bucket using the Cloudflare wrangler CLI:

# Enable the Iceberg catalog on your R2 bucket
npx wrangler r2 bucket catalog enable my-lakehouse-bucket

This returns your Catalog URI (https://catalog.cloudflarestorage.com/<ACCOUNT_ID>/my-lakehouse-bucket) and Warehouse Name (<ACCOUNT_ID>_my-lakehouse-bucket).

[!NOTE] Connecting requires two sets of credentials:

1. Cloudflare API Token (Bearer token with Workers R2 Storage or R2 Data Catalog permissions) to communicate with the Iceberg REST Catalog.
2. R2 S3 Access Keys (Access Key ID and Secret Access Key) to read and write underlying Parquet data files via R2’s S3-compatible API.

2. Querying with PyIceberg

Because Cloudflare exposes the open REST catalog protocol, you can interact with your lakehouse using Python and pyiceberg (installed with pyiceberg[pyarrow,s3fs]):

import os
from pyiceberg.catalog import load_catalog

account_id = os.getenv("CF_ACCOUNT_ID")
bucket_name = "my-lakehouse-bucket"

# Initialize the Iceberg REST Catalog pointing to Cloudflare R2
catalog = load_catalog(
    "cloudflare_r2",
    **{
        "type": "rest",
        "uri": f"https://catalog.cloudflarestorage.com/{account_id}/{bucket_name}",
        "warehouse": f"{account_id}_{bucket_name}",
        "token": os.getenv("CF_API_TOKEN"),
        "s3.endpoint": f"https://{account_id}.r2.cloudflarestorage.com",
        "s3.access-key-id": os.getenv("R2_ACCESS_KEY_ID"),
        "s3.secret-access-key": os.getenv("R2_SECRET_ACCESS_KEY"),
        "s3.region": "auto",
        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
    }
)

# List namespaces and load the table
print("Namespaces:", catalog.list_namespaces())
table = catalog.load_table("analytics.user_events")

# Scan the table using predicate pushdown
df = table.scan(
    row_filter="event_type == 'purchase' and amount > 30.0",
    selected_fields=("event_id", "user_id", "event_type", "amount", "event_timestamp")
).to_arrow().to_pandas()

print(df.head())

3. Querying with DuckDB

Prefer lightweight, lightning-fast SQL on your local machine? DuckDB has first-class native support for the Iceberg REST catalog:

-- Install and load DuckDB extensions
INSTALL iceberg;
INSTALL httpfs;
LOAD iceberg;
LOAD httpfs;

-- Configure Cloudflare API token for the Iceberg REST catalog
CREATE SECRET r2_catalog (
    TYPE ICEBERG,
    TOKEN '<CF_API_TOKEN>'
);

-- Attach the Cloudflare Iceberg REST catalog directly
ATTACH '<ACCOUNT_ID>_my-lakehouse-bucket' AS lakehouse (
    TYPE ICEBERG,
    ENDPOINT 'https://catalog.cloudflarestorage.com/<ACCOUNT_ID>/my-lakehouse-bucket'
);

-- Query using standard SQL with predicate pushdown
SELECT event_id, user_id, event_type, amount, event_timestamp
FROM lakehouse.analytics.user_events
WHERE amount > 30.0
LIMIT 10;

---

Final Thoughts: What I Took Away

What started as curiosity over a brief product blurb in a Cloudflare announcement turned into a much deeper appreciation for where the data engineering ecosystem is moving.

The key takeaway for me was realizing how data architectures are progressively decoupling:

1. Decoupling Compute from Storage: Moving away from monolithic Hadoop clusters to object storage.
2. Decoupling Storage from Proprietary Formats: Storing raw data in open columnar formats like Parquet.
3. Decoupling Table Management from Engines: Using open table specifications like Apache Iceberg so no single query engine owns the table definition.
4. Decoupling Catalogs via Open Standards: Using the Iceberg REST Catalog specification so any engine can interact with any catalog implementation.

Before digging into this, “data lakehouse” often felt like another marketing buzzword. But seeing how Iceberg solves tangible engineering problems—eliminating slow directory scans, providing real ACID transactions, and enabling fearless schema changes—made its value clear.

Whether you test Iceberg using a managed REST catalog, AWS Glue, or an open-source catalog like Apache Polaris locally, understanding open table formats is well worth the time for anyone interested in modern data architecture.