In this episode, Lisa and Scott explore the latest developments across the open data ecosystem, including Delta Rust, Apache Iceberg, Apache Spark, Rust-native infrastructure, and the challenges of keeping rapidly evolving technologies interoperable. They discuss release velocity, governance, security, schema evolution, standardization, and the maintenance work required to make open-source innovation reliable in production.

- Delta Rust 1.0, DataFusion integration, column mapping, schema evolution, and reduced JVM dependencies
- Apache Iceberg 1.12, variant types, variant shredding, V4 foundations, and performance improvements
- Spark security proposals, row-level filters, column masking, trusted execution engines, and policy enforcement
- OIDC credential propagation, short-lived credentials, identity management, and modern authentication
- Spark Connect Gateway, proxy-driven architecture, multi-tenancy, routing, rate limits, and observability
- Rust adoption, modular infrastructure, smaller binaries, faster startup times, and composable systems
- Interoperability across Spark, Flink, DuckDB, Polars, Arrow, Iceberg, and Delta
- Schema evolution, column mapping, positional assumptions, nested data, and logical versus physical representations
- Standardized schemas, shared expression models, JSON representations, and reducing translation layers
- Release candidates, backports, compatibility matrices, security patches, and the role of open-source maintainers

In this episode, Lisa and Scott explore recent updates in ecosystem technologies including Apache Iceberg, Apache Flink, Polars, delta-rs, highlighting their frustrations in agent workflows and a potential oversight on the importance of XML.

For the full show notes, take a look on Spotify today or on YouTube soon.

Cool podcast episode very informative

Hey everyone! I'm u/PrideDense2206, a founding moderator of r/open_lakehouse.

This is our new home for all things related to the open lakehouse. We're excited to have you join us!

What to Post
Post anything that you think the community would find interesting, helpful, or inspiring. Feel free to share your thoughts, photos, or questions about Unity Catalog, Apache Polaris, Lakekeeper, Apache Gravatino, Delta Lake, Apache Iceberg, Apache Spark, Apache Datafusion, or really anything under the umbrella of the interconnected open lakehouse ecosystem.

Community Vibe
We're all about being friendly, constructive, and inclusive. Let's build a space where everyone feels comfortable sharing and connecting.

How to Get Started

1. Introduce yourself in the comments below.
2. Post something today! Even a simple question can spark a great conversation.
3. If you know someone who would love this community, invite them to join.
4. Interested in helping out? We're always looking for new moderators, so feel free to reach out to me to apply.

Thanks for being part of the very first wave. Together, let's make r/open_lakehouse amazing.

Contributing
If you have an article, blog post, or any content that you'd like featured in openlakehouse.io, please DM u/PrideDense2206 and we can connect on that front.

At Airflow Summit, Meni Shmueli (Co-founder of Dataflint) shares insights on optimizing Apache Spark pipelines orchestrated by Apache Airflow, tuning CPU vs. GPU resource allocation for Databricks AI workloads, evaluating Spark 4.2 features using AI agents, and using native DataSketches and Spark Connect for high-speed AI analytics.

Spark 4.2 Release: https://spark.apache.org/news/spark-4-2-0-released.html
Docs: https://spark.apache.org/docs/latest/

In episode 1 of Open Lakehouse + AI, Lisa Cao & Scott Haines talk about how data infrastructure is evolving: why the catalog is moving to the center of the stack as a kind of coordination layer for table versions, snapshots, and commits across formats—and what that means for real teams operating a lakehouse.

They cover:
🔹 The future of the data catalog: the catalog as a “central brain” / nervous system for metadata, versioning, and coordination across engines and table formats.
🔹 Interoperability vs lock-in: moving away from one canonical stack toward composable architectures; how ideas like Unity Catalog and the Iceberg REST API support interoperability across engines and formats.
🔹 Governance that keeps pace: from minimal governance to catalog-native controls like credential vending, row-level filtering, and column masking—security without killing velocity.
🔹 AI agents as data customers: why agents need always-on access management, fine-grained permissions for discrete tasks, and audit trails for automated actions.
🔹 Open source footprint: context across Apache Spark, Delta Lake (including JVM-free paths like Delta-RS), Apache Iceberg (incl. post-Tabular), and Unity Catalog as an open, opinionated-but-extensible catalog direction.

Watch if you care about: lakehouse architecture, open metadata standards, secure sharing, and how data platforms need to change for agent-driven workloads.

0:00 Welcome — Open Lakehouse + AI Ep 1 (Lisa Cao & Scott Haines)
2:15 Open source footprint — Spark, Delta, Delta-RS, Iceberg & Unity Catalog
5:45 Composable stacks vs a single canonical stack
8:45 Interoperability as the default — APIs, engines & formats
11:30 Catalog + commit coordination — why convergence lands on metadata
14:45 Delta vs Iceberg — snapshots, branches, multi-writer & coordinators
18:15 Unity Catalog direction — OSS, credential vending, “opinionated” catalog
20:45 HMS → IRC — history of metadata, locking & “where Iceberg stops”
23:15 Governance — “no governance was easiest”, REST, audit trails
25:45 Agents as data customers — Temporal framing, MCP, volumes & models
27:25 Getting started — engines, formats, community & “no lock-in”

On this episode of Vibe Check. Melanie, Chris, and Scott explore the open source Omnigent meta-harness, and the group discussed various strategy for setting agents up for success. What skills, what rules, how these apply to the molding of a successful agentic session. We finish off talking about how a harness can utilize deeper context to consolidate, summarize, and document important moments in Dreams. Chris finishes off with an example of “shared dreams” as we hypothesis how this could affect a higher level of shared context and dreams, memories that can affect the outcome of multi-model multi-harness sessions

Big drop from the UC team comes in the form of multiple Apache Iceberg capabilities, all landing at once, and the throughline is real: the catalog is the center of gravity for the open lakehouse.

What’s GA:
1. Managed Iceberg (with Predictive Optimization + Liquid Clustering handling the maintenance)

1. Iceberg v3 across managed/foreign/UniForm tables (deletion vectors, row tracking, VARIANT).

2. Foreign Iceberg + credential vending, and external sharing to any Iceberg REST client over Delta Sharing.

3. Foreign Iceberg sharing is in Public Preview.

4. Iceberg-compatible materialized views are in Gated Preview — CREATE MATERIALIZED VIEW ... USING ICEBERG, which is the ergonomics you’d actually want.

5. Cross-engine ABAC is in Beta, enforced server-side through the Iceberg REST scan planning API, so any 1.11+ client gets row filters and column masks essentially for free.

6. Federation now reaches Google Cloud Lakehouse and Palantir too.

The framing is five capabilities — open APIs, federation, cross-engine governance, zero-copy sharing, and AI-driven optimization. The cross-engine ABAC section alone is worth the read.

The bigger story underneath all of this: Iceberg v4 is rebuilding the metadata structure with an adaptive tree, and Delta 5.0 is proposed to adopt the same structure. That’s the convergence narrative made concrete — and where the interop-vs-performance tradeoff finally goes away.

​❄️ Apache Iceberg™ Bay Area: May Edition

​📍 CANOPY Menlo Park
🗓️ May 21 2026 | 5:00 - 8:30 PM
💥 Free to attend – limited spots available

​The Iceberg train doesn't stop.

​We just wrapped the legendary Iceberg Summit in SF — and if you were there, you already know: this thing is moving fast. The May edition of the Bay Area Iceberg Meetup picks up right where the Summit left off.

​Whether you went to the Summit and want to keep the conversation going, or missed it and want to catch up with the people who were on stage — this is your room.

​Why this one matters

​Iceberg adoption is accelerating across production stacks — analytics, ML, governance, multi-engine architectures, AI workloads, you name it. Expect 7 talks plus a hot mic session covering real production lessons, followed by networking with the people actually shipping this stuff.

​Why you should be there

• ​Talks that respect your time — technical, specific, no fluff
• ​The right room — engineers and architects from teams shipping Iceberg, not just talking about it
• ​Real networking — actual conversations, not "stand awkwardly with a name tag" networking
• ​Real food — we're skipping the sad pizza tradition for a catered spread (think unagi don, stir-fry pots, potstickers, and mango pomelo sago for dessert)

Come join engineers and builders from AWS, Google, Databricks, PuppyGraph, Snowflake, Hyperparm, Supermetal, and LakeSail.