opensource.google.com

Menu

Bringing the lake back to the lakehouse: Introducing the FILE type in Apache Iceberg

Thursday, November 19, 2026

The original promise of the data lake was simple: a single, universal home for all an organization’s data. Whether structured database exports, raw text documents, audio recordings, or high-resolution imagery, everything landed in object storage. But as data lakes grew into sprawling swamps lacking transactional guarantees and consistency, the industry pioneered the lakehouse.

Open table formats like Apache Iceberg brought order to the chaos, introducing data warehouse rigor—ACID transactions, snapshot isolation, partition evolution, and fast columnar pruning—directly to cloud storage.

Yet, in our collective rush to build the “house”, we quietly left the “lake” behind.

Wooden stilt house floating on calm grassy water at Inle Lake, illustrating bringing the lake back to the lakehouse for multimodal data
Photo by mydaydream89 (CC0) via Wikimedia Commons

With the rapid rise of multimodal Large Language Models (LLMs), interest in processing unstructured data directly within the lakehouse has grown rapidly. Multimodal LLMs enable modern data warehouses and lakehouses to analyze unstructured inputs—such as raw images, audio, and documents—much like traditional structured data, without requiring complex, bespoke machine learning staging pipelines.

Because table formats were designed around structured rows and columns, unstructured media was left behind. Raw files were relegated back to external storage buckets, loosely connected to analytical tables through plain, unvalidated string URIs.

The Apache Iceberg community recently took a major step toward reuniting these two worlds by merging support for the new FILE data type into the Iceberg specification.

Limitations of using bare string URIs

Historically, platforms referenced external unstructured assets by storing simple string paths inside tabular columns. While straightforward, relying on plain string URIs introduces several operational challenges:

  • No file metadata in the table: A string path doesn't say what type a file is or how big it is. Engines guess types from file extensions, or make a separate call to object storage for each file.
  • Inconsistent engine behavior: Nothing marks a string column as a file reference, so every engine and team invents its own way of handling one, and what one writes, another can't reliably interpret.
  • No way to detect changes: If a file is replaced at the same path, the string stays the same, so nothing in the table shows that the bytes behind it have changed.

Addressing these operational bottlenecks required a structural shift in how table formats model unstructured assets.

Introducing the FILE type

In the specification slated for Iceberg v4, FILE is defined as a specialized nested type. Rather than treating external assets as opaque strings, it bundles the essential dimensional, transport, and integrity properties of unstructured data into a single schema element:

Apache Iceberg v4 FILE nested type definition showing implicit uri, offset, size, content_type, checksum, and inline binary fields

The native FILE type resolves these operational bottlenecks by storing essential file metadata alongside the reference in every value. Instead of storing an opaque path string, the column stores a structured reference containing standard attributes like uri, offset, size, content_type, checksum, and inline payload. Standardizing these properties gives query engines a single, predictable contract to handle common file management challenges:

  • Pay metadata fetch costs once: Writers capture file metadata such as size and content type once, in a standard format that any engine supporting the FILE type can read. Engines no longer have to make a separate call to object storage for each file every time they plan a query.
  • Detect stale metadata with checksums: Storing typed digests or ETags as checksum explicitly allows readers to detect when underlying files are modified or out of date, preventing silent asset drift.
  • Supply required metadata for LLM processing: The content_type attribute delivers explicit MIME metadata directly to multimodal LLMs and inference APIs, eliminating reliance on fragile file extension guessing.
  • Embedded micro-asset storage: Small binary payloads can be stored directly inside the inline field. The resolution rule allows engines to stream these co-located locally during columnar scans, bypassing the need to store an additional object entirely.
  • Govern structured and unstructured data side by side: The primary win lies in bringing structured business attributes and raw media assets into the exact same transactional table snapshot under unified governance.

Now that we know how and why the FILE type is structured as such, let’s take a look at it in action.

The FILE type in action: Native multimodal AI

Historically, tables pointed at unstructured files with plain string paths. Everything an engine needed to know about a file (what type it is, how big it is, whether it has changed) lived somewhere else: inside tables each team designed for itself, or only in object storage. Let’s look at an example where this breaks down. A claims application keeps policy data in the lakehouse, photos and documents in object storage, and file details in a side table that one team built and others had to find and join by path.

When multimodal AI processing happens (reading the bytes, calling a model, writing results back) the process breaks because of everything around it. Every engine has to guess file types from extensions, track down someone else's side table, or call object storage once per file. And when a file is replaced at the same path, nothing in the table changed, so nobody could tell.

Consider an auto insurer that uses this structure. Each claim pairs structured data (policy, amount, status, and the customer's written account of the accident) with evidence files: damage photos, repair-estimate PDFs, and recorded phone statements. They want to answer this question: "Which open claims have videos that aren’t zero bytes, and has any of that evidence changed since an adjuster reviewed it?"

Finding the videos becomes a filter on real metadata:

Example SQL query against a claim_evidence table filtering open claims by evidence.content_type video MIME type, evidence.size greater than zero, and changed checksums

Consider how this simplifies an end-to-end multimodal workflow:

  • Every engine reads the same column: Teams using different engines can read the same data.
  • Filter on the real type: The job filters on content_type instead of guessing from extensions, and statistics lets engines skip data files that contain no data.
  • Changed evidence gets flagged: Comparing the checksum recorded at intake with the current object shows when a photo has been replaced.
  • Go straight to the bytes: Engines read the exact byte range from uri, offset, and size, and pass content_type to the multimodal model, with no side table and no extra metadata calls.

By elevating files from unmanaged external pointers to governed schema types, Iceberg transforms multimodal AI from a brittle multi-system coordination problem into a standard analytical query.

Ecosystem impact and looking ahead

The integration of the FILE type into the Apache Iceberg specification marks a major step forward for open lakehouse architectures. Rather than forcing organizations to rely on proprietary wrappers or fragmented storage systems to handle unstructured media, Iceberg establishes a portable, open standard.

With the specification finalized and targeted for adoption in the upcoming Iceberg v4 release, momentum now turns to native reader and writer implementations across the open source compute ecosystem—including Apache Spark, Trino, and DuckDB. As multimodal AI becomes an integral part of everyday enterprise workflows, standardizing the FILE type ensures that the open lakehouse remains the definitive, vendor-neutral foundation for structured and unstructured data alike.

Two decades of The Apache Way: Celebrating 20 years of Google and the Apache Software Foundation

Wednesday, November 18, 2026

In 2006, Google formalized its relationship with the Apache Software Foundation (ASF) as one of its first corporate sponsors and twenty years later, that partnership continues through ongoing sponsorship and upstream contributions.

A graphic featuring the Google logo, a plus sign, and the Apache Software Foundation (THE ASF) oak leaf logo.

At the core of our collaboration is a shared belief in "The Apache Way" and its principles of open, vendor-neutral, and community-led development. From the early days of corporate open source to modern machine learning and open data systems, Google and ASF communities have collaborated on widely used open source software.

Shaping how Google does open source

When we established Google’s open source contribution workflow, we drew on established ASF practices. Both our Individual and Corporate Contributor License Agreements (CLAs), which external developers sign before contributing to a Google-owned open source project, are based on the ASF’s CLAs.

The Apache 2.0 license also remains Google’s preferred open source license for code. Alongside our traditional software releases, we also use Apache 2.0 for open hardware designs and open models like Gemma 4.

Co-engineering upstream and in the cloud

Across Google Cloud, our engineering teams contribute code upstream alongside Apache project communities and build managed services directly around those open standards:

  • Google Cloud Dataflow & Apache Beam: Drawing on Google’s internal Flume and Dataflow systems, Google donated the original Dataflow SDK and runners to the Apache Incubator in 2016 to help establish Apache Beam as a vendor-neutral open standard for unified batch and stream processing.
  • Google Cloud Managed Service for Apache Airflow & Apache Airflow: Managed workflow orchestration built on Apache Airflow, helping engineering and data teams author, schedule, and monitor complex pipelines.
  • Google Cloud Managed Service for Apache Spark & Apache Spark: Managed big data and analytics processing built on Apache Spark, running directly on Google’s cloud infrastructure.
  • Google BigQuery & Apache Iceberg: Support for the Apache Iceberg open table format to power large-scale analytics and machine learning datasets.

Setting the Platinum standard

Google has invested in the long-term health of open source foundations from the start. When the ASF first established its corporate sponsorship tiers, Google became the first Platinum Sponsor.

Early Google engineers and open source advocates worked closely with ASF maintainers to build the organizational and technical bridge between Google and the foundation. Today, Google maintains Platinum sponsorship through targeted funding and Google Cloud infrastructure contributions that support the ASF’s day-to-day technical operations.

That sponsorship pairs with everyday participation in ASF projects. Many Googlers serve upstream as ASF committers and Project Management Committee (PMC) members, maintaining code and stewarding projects alongside contributors across ASF communities. The ASF has also been a mentoring organization in Google Summer of Code (GSoC) for over 20 years, working with us to welcome and mentor new open source contributors around the world.

The next 20 years: AI, open data, and beyond

Looking ahead to the next two decades, much of our shared work sits at the intersection of open data, cloud infrastructure, and artificial intelligence.

Projects like Apache Iceberg are widely used in modern analytics and machine learning workflows, while using the Apache 2.0 license for open models helps bring permissive licensing to a new generation of tools.

Thank you to the developers, committers, PMC members, and maintainers who keep the Apache Software Foundation strong. Here’s to 20 more years of building together.

Fluent Interface in C - Look ma, no this!

Tuesday, November 17, 2026

Can C be Object Oriented? Yes!

A stylized code editor window showing C code that chains method calls to build a nested hash map, with comments questioning how it manages state and memory.

Object-oriented languages like Python, Java, Rust, or C++ offer native object methods and automatic object lifetime cleanup, enabling chain operations on data structures, so you can call obj->a()->b()->c() and each method knows what it's working on. These fluent interfaces are common to many libraries that streamline configuration and reduce boilerplate. Let's take a look at an example:

ChSlice *mid = ch_new_String("One_Two_Three_Four")
    ->append("_Five_Six")
    ->split("_")
    ->sub_slice(1, 4);

puts(mid->generic.to_chars()); /* ["Two","Three","Four","Five"] */
ch_release(mid);

This chain of functions takes a string, appends it to another string, splits it into an array, and slices the array in one chain of function calls. It's a pretty basic demonstration of using fluent interfaces for building a data structure with method chaining. But if I told you this was plain ANSI-C, you would think I committed a preprocessor felony and leaked three heap allocations along the way. And there are two big reasons why:

  • There are no objects in C. No member variables or methods. No this or self to reference your object's data. The function append("_Five_Six") has no attached state, seeing only a single string literal and whatever is global.
  • There is no garbage collector. In standard C, chaining heap allocations together without saving each intermediate pointer to a variable and freeing it by hand inevitably results in a memory leak.

Manipulating strings and collections in C usually means threading state through standalone functions (str_append(s, ...)), holding onto every intermediate pointer, and freeing each temporary buffer by hand. We built iso-c-chaining to explore how far we could push fluent method chaining and automatic temporary cleanup using only standard ISO C. Here is how the library appends to a string, splits it on a delimiter, and takes a sub-slice of the resulting array in a single expression.

This code does exactly like you would expect in a modern language. It allocates a string and some kind of array, manipulates them both without a byte leaked, and cc -E shows the preprocessor never touched it. In fact, every C feature powering that chain has been part of the language since 1989 ANSI C (C89).

Party like it's 1989

Let's kick this up a notch. Take a kind of map, put another map inside it, and see what happens. That would surely break this no-leaking-or-preprocessor-tricks little C89 exercise. Well, it's time to party:

ChHash *user = ch_new_Hash(3)
    ->put("name", ch_new_String("Ada"))
    ->put("role", ch_new_Hash(2)
        ->put("title", ch_new_String("Engineer"))
        ->put("level", ch_new_Number(5)))
    ->put("active", ch_new_Boolean(1));

puts(user->generic.to_chars());
/* {"name":"Ada","role":{"title":"Engineer","level":5},"active":true} */
ch_release(user);

These put() calls are being asked to unravel the chain inside the chain of a map inside a map. Where is it going to find the right container? And now there are Strings and Numbers and Booleans, oh my!

How? You might remember: functions in C only see their arguments and whatever is global.

1. Scoping with static

C's static keyword does two different jobs depending on where you put it, and self() uses both at once:

static String* self(String *s) {
  static String *slf = NULL;
  if (s != NULL) {
    slf = s;
  }
  return slf;
}

The outer static scopes self() to the file (the type's .c implementation), but the inner static keeps slf around in every call. We cleverly use NULL to just retrieve the existing value, since we should never be setting our self pointer to NULL.

2. Casting generics

Since struct pointers in C can be cast as their first field, we take advantage of that to give us a baseline of functionality for our types:

struct ch_generic_el {
  int refs;
  void* (*bind)(ChGenericEl *);
  void* (*save)(void);
  void (*del)(void *);
  char* (*to_chars)(void);
};

Putting ChGenericEl first in every struct lets put(char *key, void *val) accept a String*, Number*, Boolean*, or Hash*, cast it to ChGenericEl*, and call to_chars() or del() without caring which type it is.

3. Stacking chains

A single slf pointer works until you build a Hash inside another Hash, where the inner Hash would overwrite the outer one. Adding prev_self and next_self pointers to ChGenericEl turns each type's self() into an intrusive stack:

if (s == (ChGenericEl*)-1) {
  *top_ptr = top->prev_self;
} else if (s != NULL && s != top) {
  s->prev_self = top;
  *top_ptr = s;
}
return *top_ptr;

Constructing the inner Hash pushes it on top of the outer Hash so .put("title", ...) and .put("level", ...) target the inner map, while passing -1 pops it back off. (And in C11, adding _Thread_local to top gives every thread its own independent stack.)

4. Counting garbage

And what about all those potential memory leaks? We're making elements mid-chain that get consumed before the chain is finished. Here we turn to the old classic: reference counting, but with a twist: Every new object starts with refs = 0, meaning nothing owns it yet. If a method like split() or sub_slice() consumes an object whose refs is still 0, it frees that temporary on the spot before returning the next link in the chain.

And when a container method like put() receives an element, it claims ownership:

if (elem == container_slf || el->refs == 0) {
  el->bind((ChGenericEl*)-1);
}
el->refs++;

Note that this means ephemeral objects are automatically released, so the caller needs to bind them manually if the ephemeral object is needed after the chain is complete.

Because the inline Hash enters put() with refs == 0:

  • el->bind((ChGenericEl*)-1) pops the finished inner Hash off the Hash self stack, restoring the outer Hash to the top so .put("active", ...) targets the outer map.
  • el->refs++ transitions the element from an unowned temporary (refs == 0) to a container-owned value (refs == 1). That keeps consuming methods from freeing it mid-chain if you later retrieve it with get(), and lets an element be shared across multiple containers or variables—each owner increments refs, and ch_release() decrements refs on every child in the tree, freeing each object only when its last reference drops to 0.

Now what can we do with this?

You might be thinking, "Wow, we could use this for all kinds of wonderful projects!" We chose a webserver instead:

ch_new_Server(8080)
    ->get("/users", list_users)
    ->get("/users/:id", get_user)
    ->post("/users", create_user)
    ->put("/users/:id", update_user)
    ->delete_route("/users/:id", delete_user)
    ->listen();

Each callback receives the parsed HTTP request: URL :id parameters, query string, and parsed JSON body. This is packaged as a Hash*, and whatever object the callback returns gets serialized to JSON (to_chars()) and freed automatically after the response is sent.

Here is what list_users looks like when wired up to a SQLite Db object:

static GenericEl* list_users(Hash *req) {
  return (GenericEl*)$(db)
      ->params($(req)->get("query"))
      ->query("SELECT id, name, role, score FROM users "
              "WHERE (:role IS NULL OR role = :role) "
              "  AND (:min_score IS NULL OR score >= :min_score) "
              "ORDER BY score DESC");
}

What comes back from ->query(...) is the rare combination of everything we built above: a Slice of Hashes holding Numbers and Strings as database result rows. Because every one of those types shares ChGenericEl, the server just calls to_chars() on the returned Slice to stream out [{"id":1,"name":"Alice","role":"admin","score":98.25}, ...] and releases the entire result tree in a single call once the response is sent.

Wait, what's with the $?

In our earlier examples, every chain started with a ch_new_* constructor. Inside list_users, however, db and req are already-existing objects, so before calling ->params(...) or ->get("query"), we need to reference our previously created elements by calling their bind() function pointer: db->bind((GenericEl*)db).

Nobody wants to type db->bind((GenericEl*)db) at the start of every statement, so the library provides four tiny one-line convenience macros:

  • $(obj) expands to ((obj)->bind((ChGenericEl*)(obj))) which binds a previously created variable so you can start chaining on it.
  • ->$$ expands to ->generic.save() which bumps refs (refs++) at the end of a chain when you want to keep an object in a variable (db = create(Db, ":memory:")->...->$$;) instead of letting the chain consume it as a temporary.
  • create(Type, ...) expands to ch_new_Type(...).
  • ->chars() expands to ->generic.to_chars().

Pulling back a little performance

You might look at this and think, "Wow, this is wrapping every single int in an object like it's a horrible java.lang.Integer." And you would be right, but we can take a page from JavaScript's TypedArrays and not have a bunch of function pointers attached to every element, giving us back the performance we expect from C:

TypedIntArray *scores = create(TypedIntArray, 0)
    ->push_n(6, 42, 15, 88, 23, 16, 4)
    ->sort()
    ->$$; /* [4, 15, 16, 23, 42, 88] */

Now a single TypedIntArray object wraps a raw contiguous int* buffer, so a massive array of integers isn't attached to an even more massive number of function pointers. TypedIntArray, TypedCharArray, and TypedFloatArray are included by default (and can be sorted, mapped, filtered, sliced, and dropped straight into a Hash for JSON serialization), or you can declare one for any fixed-width type with CH_DECLARE_TYPED_ARRAY.

In examples/chain_vector_db.c, TypedFloatArray stores 256-dimensional embedding vectors for over 4,000 Linux whatis man pages in SQLite—computing each vector's norm in a single map + reduce chain:

double norm = sqrt($(vec)
    ->subarray(0, vec->length)
    ->map(square_double)
    ->reduce(add_doubles, 0.0));

Try it out

We've released iso-c-chaining as an open source demonstration library under the Apache 2.0 license for C developers to explore, tinker with, or drop into side projects via its single-header build (chain_single.h). The full source, including TypedArrays, the JSON parser/serializer, and the SQLite/HTTP server examples, is available on GitHub at google/iso-c-chaining. Contributions and feedback may be given on the GitHub project directly.

This Week in Open Source for November 13, 2026

Friday, November 13, 2026

This Week in Open Source banner

A look around the world of open source

Open source communities are navigating a structural tension: automated tools are accelerating code generation and security probing faster than human maintainers can review pull requests, build consensus, or fund shared infrastructure. Across this edition, we look at how maintainers, foundations, and protocol builders are responding—from neutral governance for agent sandboxes and empirical supply-chain security to inspectable agent protocols, independent code forges, and permissioned spaces on the AT Protocol.

Upcoming events

🗓️ December 2026

  • Yocto Project Virtual Summit 2026.12 (December 1–3, 2026) – Virtual. Community summit for embedded Linux engineers and maintainers building custom, reproducible Linux distributions across edge and Internet of Things (IoT) hardware.
  • Open Source Summit Japan 2026 (December 7–9, 2026) – Tokyo, Japan. The Linux Foundation's premier Asia-Pacific conference connecting open source developers, kernel maintainers, and enterprise leaders across cloud, embedded Linux, and open AI—featuring a keynote by Google's Kaz Sato on building agents on open standards alongside sessions from Googlers Tanya Sharma, Tarun Sahu, and Anurag Sinha.
  • Open Compliance Summit 2026 (December 10–11, 2026) – Tokyo, Japan. Specialized gathering uniting legal, security, and Open Source Programs Office (OSPO) practitioners around OpenChain, Software Package Data Exchange (SPDX), software bills of materials (SBOMs), and global open source supply-chain compliance.

🗓️ January 2027

  • Free and Open Source Software Developers' European Meeting (FOSDEM) 2027 (January 30–31, 2027) – Brussels, Belgium. Europe's largest free, volunteer-organized gathering for free and open source software developers at the Université libre de Bruxelles—with calls for participation now open across 77 community developer rooms and main-track proposals open through November 16.

Open calls for proposals (CFPs)

  • Free and Open Source Software Developers' European Meeting (FOSDEM) 2027 (Event: January 30–31, 2027 – Brussels, Belgium) – Main-track CFP deadline: November 16, 2026 (individual community developer room CFPs run on rolling deadlines from November 20 through December 3, with full devroom schedules published by December 7, 2026).
  • AtmosphereConf 2027 (Event: April 29 – May 2, 2027 – Amsterdam, The Netherlands) – Round 1 CFP deadline: November 30, 2026 (Round 2: January 11 – February 14, 2027). Community conference uniting developers, researchers, and creators building on the Authenticated Transfer Protocol (AT Protocol), open social web standards, and decentralized agent substrates across Build, Create, Inform, and Govern tracks.
  • NeoCloudCon Europe 2027 (KubeCon + CloudNativeCon Europe Co-Located Event) (Event: March 15, 2027 – Barcelona, Spain) – CFP deadline: November 30, 2026 (23:59 CET). Co-located CNCF summit bringing cloud-native infrastructure builders together around emerging AI cloud platforms, GPU orchestration, and open distributed infrastructure.
  • CNCF Maintainer Summit Europe 2027 (Event: March 14, 2027 – Barcelona, Spain) – CFP deadline: December 1, 2026 (23:59 CET). Exclusive co-located gathering for CNCF Technical Oversight Committee (TOC) representatives, Technical Advisory Group (TAG) leads, and Graduated or Incubating project maintainers to collaborate on governance, API design, and project health.

We've already submitted talks to FOSDEM and AtmosphereConf—consider submitting a proposal to one of these open calls too, as we'd love to see your work on the schedule!

Open source reads and links

  • [Article] Are good intentions hurting open source? - Code generation is no longer the bottleneck when Ingress NGINX runs across half of Kubernetes clusters on two maintainers flooded with AI pull requests. In this summary and keynote, Akamai Cloud's Ruckus Voxi highlights documentation, triage, and stewardship as the real gaps—raising an honest question: how do we recruit contributors beyond code when open source still feels like a "coding club"?
  • [Article] Kubernetes, mark II? Why Google's giving away gVisor, a code sandbox used by Anthropic and OpenAI - Anthropic and OpenAI already use gVisor to isolate untrusted code, yet upstream Linux kernel maintainers resisted performance patches while Google kept sole control. Paired with the Cloud Native Computing Foundation (CNCF) donation of Agent Substrate and OpenRL (Google Cloud blog), can neutral governance turn agent sandboxes into shared cloud infrastructure the way Kubernetes did for containers?
  • [Recap] OpenSSF Community Day Europe 2026: Driving Systemic Security and Operational Resilience in Open Source - Drawing 200+ practitioners in Prague—with session recordings now online and a Google-sponsored academic security track—speakers unveiled Model Context Protocol (MCP) audit tools like SandScope, skill-registry attack detection, empirical Software Bill of Materials (SBOM) studies, and European Cyber Resilience Act due diligence. As "cheap trust signals" collapse under automated noise, what empirical verification belongs in release pipelines?
  • [Article] GitHub is experiencing an open-source exodus. Here’s where the developers are going instead - "Exodus" might be too strong of a word, but continuous integration (CI) outages and opt-out AI training have pushed projects like Zig and Ghostty off GitHub toward Codeberg, self-hosted Forgejo, and SourceHut—though as an AT Protocol watcher, decentralized forge Tangled should likely have earned a mention too. When communities weigh contributor network effects against hosting independence, could protocol-level identity help smaller forges bridge that gap?
  • [Report] Navigating Open Source Software Development in the Generative AI Era - Surveying 234 contributors and maintainers (companion blog post), Linux Foundation Research finds generative AI cuts task time by up to 52%, yet maintainers estimate 25% of incoming pull requests are low-quality "AI slop"—with 64% reporting slower reviews, 63% citing burnout, and only 35% of projects holding a formal AI policy. Rather than blanket bans or unchecked noise, what lightweight verification norms actually scale across smaller maintainer teams?
  • [Article] Munich dumps proprietary for open source (again) - As Munich renews its open-source push beneath the desktop layer, Matrix.org co-founder Amandine Le Pape highlights the other half of European public-sector sustainability: while civil servants across Europe now rely on Matrix, only one government agency funds the underlying specification. Can new bodies like the Digital Commons European Digital Infrastructure Consortium (EDIC) align government procurement with upstream maintenance?
  • [Article] Why science keeps choosing open source - Enterprise IT expects software upgrades every few years, but research labs often pair modern Graphics Processing Unit (GPU) clusters with 15-year-old instruments and validated scripts that cannot change without breaking reproducible results. Highlighting how Linux, containers, and open source outlive commercial vendor roadmaps, how else does source access protect long-term knowledge from planned obsolescence?
  • [Blog] Fast, reliable A2UI, with swappable inference formats - Pairing Google’s open Agent-to-User Interface (A2UI) protocol with Genkit for Go, Daniela Petruzalek demonstrates how agents can stream rich generative user interfaces over compact, swappable wire formats without locking frontend layouts to a single model provider. As agent workflows move beyond plain-text chat boxes, how can open UI protocols keep human interfaces fast and inspectable across runtimes?
  • [Blog] Testing and Serving A2A Agents from the Terminal with A2A CLI - In a hands-on terminal walkthrough, Mete Atamel shows how the open source Agent2Agent (A2A) command-line interface (CLI) lets developers inspect agent cards, stream multi-turn tasks, and serve local A2A endpoints directly from the command line without spinning up a full web UI. Just as curl made HTTP debugging universal, what terminal primitives do builders need to make multi-agent interoperability easy to test locally?
  • [Blog] Behind the scenes, two months later: How we ship Google Agent Skills like products - Treating natural-language SKILL.md packages on GitHub as living software releases rather than static docs, maintainers on Google’s open-source skills repository enforce strict semantic versioning, pre-review CI schema linters, and continuous evaluation suites against a no-skill baseline. When open-source repositories distribute natural-language instructions instead of deterministic code, how must release engineering evolve to prevent silent behavioral regressions?

Spotlight: ATProto Spaces and permissioned data

With a release specification and beta for permissioned spaces on the Authenticated Transfer Protocol (AT Protocol) targeted for late November, what should we be building with shared spaces in mind:

  • [Video] FediForum Talk: Introduction to Atproto Spaces & Reintroducing Spaces - Bluesky’s Daniel Holmgren introduces ATProto Spaces as lightweight "mini AT proto networks" built around locality rather than per-record Google+-style Circles (it still lives in our hearts)—proving via "Daniel’s Circles Postulate" why binding each space to one owner and one modality keeps OAuth consent screens legible in two sentences while preserving cross-app remixability for private groups, newsletters, and 100,000-member forums.
  • [Blog] Modeling communities on permissioned data & Permissioned Data Diary 5: What’s in a Name? - Rather than cramming forums, chat, and events into one "universal space" that over-grants OAuth read permissions, Holmgren models a community as a single transferable Decentralized Identifier (DID) owning separate modality-specific spaces (at://<space_owner_did>/space/<space_type>/<space_key>/<author_did>/<collection>/<rkey>) so community ownership can change hands without breaking URI backlinks.
  • [Blog] Permissioned Data Diary 6: Boring Auth & Permissioned Data Diary 7: Off the Record - Explaining why Spaces passed on User Controlled Authorization Networks (UCAN) capability chains in favor of a "narrow waist" OAuth Space Credential with reader-enforced writes, Holmgren details how space repositories replace Merkle Search Trees (MSTs), asymmetric signatures, and global Relays with LtHash (an ~80-line quantum-resistant lattice multiset hash), deniable Hash-based Message Authentication Code (HMAC) signatures, and direct HTTP/2 pull sync.
  • [Forum] STAR format stabilization (space repo archives!) & STAR: settling the details - Because space repositories use order-independent LtHash sets rather than Merkle trees, ATProto Spaces is adopting STreaming ARchives (STAR on Tangled) instead of Content Addressable aRchives (CARs) for repository exports—sparking an active community design discussion around canonical varint strictness, $type Namespace Identifier (NSID) collisions, and truncation checks.
  • [Blog] Building on ATProto - Offering a local-first developer counterpoint, Puppet founder Luke Kanies examines the friction of promoting records between private spaces and public broadcast repositories and explains why offline-capable applications still need local working copies alongside remote Personal Data Servers (PDSes). Once open protocols support private and group spaces natively, what kinds of community tools would you want to build first?

Which of these stories will you be chatting about at your next meetup or conference? Let us know! Share with us on our @GoogleOSS X account or our @opensource.google Bluesky account.

Kubernetes Python client gets a modern overhaul for the AI/ML era

Thursday, November 12, 2026

Google Open Source circuit-board banner featuring the blue Kubernetes helm wheel surrounded by colorful icons representing code, cloud infrastructure, configuration, and community collaboration.

When you train a large language model with Kubeflow, orchestrate distributed compute with Ray on Google Kubernetes Engine (GKE), serve inference traffic with KServe, or schedule thousands of ML pipeline pods with Apache Airflow, two technologies almost always meet under the hood: Python and Kubernetes.

Python is the undisputed lingua franca of artificial intelligence and machine learning. Kubernetes is the de facto operating system for AI infrastructure—scheduling GPUs and TPUs, orchestrating distributed training jobs, and scaling model serving endpoints. Connecting these two worlds is the official Kubernetes Python client (kubernetes-client/python).

Second only to Kubernetes's in-tree Go client (client-go) in popularity, the Kubernetes Python client is downloaded more than 73 million times per month on PyPI (~1.7 million times a day) and has earned over 7,500 GitHub stars.

Yet for the past several years, this critical bridge between Python and Kubernetes has operated in low-maintenance mode. As AI/ML workloads have exploded in scale and complexity, the gap between what modern AI infrastructure demands and what the legacy Python client was built to deliver has become impossible to ignore.

Today, we're sharing how Google Cloud engineers are partnering with open source contributors in Kubernetes SIG API Machinery to modernize the official Kubernetes Python client for the AI/ML era.

The historical problem: Legacy machinery meets hyperscale AI

As AI/ML clusters grew from dozens of nodes to 50,000-node hyperscale environments, the limitations of the Python client's aging architecture began surfacing as real-world production bottlenecks:

  1. Frozen on a decommissioned code generator: Up through Kubernetes 1.35, the Python client generated its ~400,000 lines of API and model code using OpenAPI Generator v4.3.0 (python-legacy)—a generator version that was deprecated in 2020 and completely removed from the upstream OpenAPI Generator project in May 2022. Staying pinned to v4.3.0 meant the client accumulated DeprecationWarning debt across newer Python releases, struggled with modern dependency resolution alongside PyTorch and TensorFlow, and couldn't easily receive upstream generator bug fixes.
  2. Synchronous-only execution in an asyncio world: Modern Python web frameworks, inference servers, and orchestrators have overwhelmingly shifted to asyncio to handle high-concurrency I/O without being bottlenecked by Python's Global Interpreter Lock (GIL). Because the official client lacked native asyncio support from the start, users were forced to turn to community clients like kubernetes-asyncio or third-party clients (kr8s, lightkube).
  3. Missing controller primitives (SharedInformer): In Go, controllers use SharedInformer to maintain a local, event-driven cache of cluster resources without repeatedly polling kube-apiserver. Without an equivalent primitive in Python, applications watching thousands of training pods or custom resources often hammered the API server or ran into watch disconnects.
  4. Slow model deserialization and HTTP/1.1 transport limits: In large clusters, listing pods or nodes in Python could take 30+ seconds due to excessive model reconfiguration during JSON deserialization (issue #1921). Combined with an underlying urllib3 transport layer restricted to HTTP/1.1, Python workloads missed out on the HTTP/2 multiplexing, flow control, and binary framing that make client-go fast at scale.

A principled approach: Built for AI/ML, not 1:1 parity

When we set out to modernize the Kubernetes Python client, we established two guiding principles to ensure our engineering investments deliver the highest possible return for the community:

  1. Target high-impact gaps, not 1:1 client-go parity: client-go is the in-tree Kubernetes client used by the control plane itself (kube-controller-manager, kube-scheduler, and in-tree controllers). Attempting to replicate every long-tail client-go feature in Python would yield diminishing returns. Instead, we focus strictly on the primitives Python developers and operators actually need in production.
  2. Prioritize AI/ML workload requirements: Because AI/ML training, inference, and batch frameworks are the primary drivers of Python on Kubernetes today, every architectural improvement—from async concurrency and CRD ergonomics to large-cluster list performance—is prioritized through the lens of AI/ML scale.

How we're modernizing the Kubernetes Python client

Our roadmap focuses on three technical pillars: Machinery, Features, and Performance.

1. Machinery: Upgrading to OpenAPI Generator v7.20+

Every improvement to the Python client starts with its code generation pipeline (kubernetes-client/gen). Over the past few years, multiple community attempts to migrate off the decommissioned v4.3.0 python-legacy generator stalled due to the sheer scale of template and schema changes across hundreds of Kubernetes API models.

We are actively driving the migration to OpenAPI Generator v7.20.0+ and establishing continuous tracking with upstream generator releases. Completing this foundational upgrade:

  • Replaces dead python-legacy templates with actively maintained v7+ Python code generation.
  • Modernizes type annotations and packaging metadata, easing pip dependency resolution alongside heavy ML frameworks.
  • Unlocks the ability to contribute performance and serialization fixes directly to upstream OpenAPI Generator templates.

2. Features: Native asyncio, Shared Informers, and WebSocket v5

To prevent ecosystem fragmentation and give Python developers first-class Kubernetes primitives out of the box, we are bringing three major capabilities to the official client:

  • Official asyncio support: Rather than forcing developers to choose between the official synchronous client and third-party async libraries, we are introducing official asyncio support directly into kubernetes-client/python. Async-first ML orchestrators, FastAPI-based model control planes, and notebooks will be able to issue concurrent, non-blocking Kubernetes API calls natively.
  • Shared Informers (kubernetes-client/python#2515): We are bringing the SharedInformer pattern to the Python client. With built-in list-watch caching, automatic reconnection, and resource event handlers (on_add, on_update, on_delete), Python controllers and AI job managers can monitor cluster state and Custom Resources (TrainJob, RayCluster, InferenceService) with a fraction of the memory footprint and API server load.
  • WebSocket v5 streaming (KEP-4006, kubernetes-client/python#2486): As upstream Kubernetes transitions remote command streaming (exec, attach, port-forward) away from legacy SPDY to standard WebSockets via the v5.channel.k8s.io subprotocol, we are adding native WebSocket v5 channel support so interactive debugging and remote job execution work seamlessly against modern control planes.

3. Performance: Slashing deserialization overhead

For AI/ML platforms operating on multi-thousand-node clusters, client performance is an availability requirement. Our performance track targets the largest bottlenecks in Python client execution:

  • Eliminating model reconfiguration overhead: When deserializing large JSON responses (such as listing tens of thousands of Pods across a training cluster), the legacy generated model layer repeatedly instantiates and reconfigures schema validation objects (issue #1921). With the move to OpenAPI Generator v7+, we are revamping model initialization so large cluster queries deserialize in a fraction of the time.

What's next and how to get involved

The intersection of Python and Kubernetes is where modern AI infrastructure is built. By modernizing the code generation machinery, adding native asyncio and SharedInformer primitives, and closing the deserialization performance gap, we aim to give every ML engineer and platform builder a fast, reliable, and Pythonic official client—whether they are running a single notebook experiment or orchestrating a 50,000-node GKE training fleet.

This effort is happening in the open under Kubernetes SIG API Machinery, and we welcome feedback, use cases, and contributions from the community:

.