Kubernetes Python client gets a modern overhaul for the AI/ML era

by Yu Liao, Haowei Cai, and Federico Bongiovanni, Google Cloud Kubernetes Engineering & SIG API Machinery

When you train a large language model with Kubeflow, orchestrate distributed compute with Ray on Google Kubernetes Engine (GKE), serve inference traffic with KServe, or schedule thousands of ML pipeline pods with Apache Airflow, two technologies almost always meet under the hood: Python and Kubernetes.

Python is the undisputed lingua franca of artificial intelligence and machine learning. Kubernetes is the de facto operating system for AI infrastructure—scheduling GPUs and TPUs, orchestrating distributed training jobs, and scaling model serving endpoints. Connecting these two worlds is the official Kubernetes Python client (kubernetes-client/python).

Second only to Kubernetes's in-tree Go client (client-go) in popularity, the Kubernetes Python client is downloaded more than 73 million times per month on PyPI (~1.7 million times a day) and has earned over 7,500 GitHub stars.

Yet for the past several years, this critical bridge between Python and Kubernetes has operated in low-maintenance mode. As AI/ML workloads have exploded in scale and complexity, the gap between what modern AI infrastructure demands and what the legacy Python client was built to deliver has become impossible to ignore.

Today, we're sharing how Google Cloud engineers are partnering with open source contributors in Kubernetes SIG API Machinery to modernize the official Kubernetes Python client for the AI/ML era.

The historical problem: Legacy machinery meets hyperscale AI

As AI/ML clusters grew from dozens of nodes to 50,000-node hyperscale environments, the limitations of the Python client's aging architecture began surfacing as real-world production bottlenecks:

1. Frozen on a decommissioned code generator: Up through Kubernetes 1.35, the Python client generated its ~400,000 lines of API and model code using OpenAPI Generator v4.3.0 (python-legacy)—a generator version that was deprecated in 2020 and completely removed from the upstream OpenAPI Generator project in May 2022. Staying pinned to v4.3.0 meant the client accumulated DeprecationWarning debt across newer Python releases, struggled with modern dependency resolution alongside PyTorch and TensorFlow, and couldn't easily receive upstream generator bug fixes.
2. Synchronous-only execution in an asyncio world: Modern Python web frameworks, inference servers, and orchestrators have overwhelmingly shifted to asyncio to handle high-concurrency I/O without being bottlenecked by Python's Global Interpreter Lock (GIL). Because the official client lacked native asyncio support from the start, users were forced to turn to community clients like kubernetes-asyncio or third-party clients (kr8s, lightkube).
3. Missing controller primitives (SharedInformer): In Go, controllers use SharedInformer to maintain a local, event-driven cache of cluster resources without repeatedly polling kube-apiserver. Without an equivalent primitive in Python, applications watching thousands of training pods or custom resources often hammered the API server or ran into watch disconnects.
4. Slow model deserialization and HTTP/1.1 transport limits: In large clusters, listing pods or nodes in Python could take 30+ seconds due to excessive model reconfiguration during JSON deserialization (issue #1921). Combined with an underlying urllib3 transport layer restricted to HTTP/1.1, Python workloads missed out on the HTTP/2 multiplexing, flow control, and binary framing that make client-go fast at scale.

A principled approach: Built for AI/ML, not 1:1 parity

When we set out to modernize the Kubernetes Python client, we established two guiding principles to ensure our engineering investments deliver the highest possible return for the community:

1. Target high-impact gaps, not 1:1 client-go parity: client-go is the in-tree Kubernetes client used by the control plane itself (kube-controller-manager, kube-scheduler, and in-tree controllers). Attempting to replicate every long-tail client-go feature in Python would yield diminishing returns. Instead, we focus strictly on the primitives Python developers and operators actually need in production.
2. Prioritize AI/ML workload requirements: Because AI/ML training, inference, and batch frameworks are the primary drivers of Python on Kubernetes today, every architectural improvement—from async concurrency and CRD ergonomics to large-cluster list performance—is prioritized through the lens of AI/ML scale.

How we're modernizing the Kubernetes Python client

Our roadmap focuses on three technical pillars: Machinery, Features, and Performance.

1. Machinery: Upgrading to OpenAPI Generator v7.20+

Every improvement to the Python client starts with its code generation pipeline (kubernetes-client/gen). Over the past few years, multiple community attempts to migrate off the decommissioned v4.3.0 python-legacy generator stalled due to the sheer scale of template and schema changes across hundreds of Kubernetes API models.

We are actively driving the migration to OpenAPI Generator v7.20.0+ and establishing continuous tracking with upstream generator releases. Completing this foundational upgrade:

• Replaces dead python-legacy templates with actively maintained v7+ Python code generation.
• Modernizes type annotations and packaging metadata, easing pip dependency resolution alongside heavy ML frameworks.
• Unlocks the ability to contribute performance and serialization fixes directly to upstream OpenAPI Generator templates.

2. Features: Native asyncio, Shared Informers, and WebSocket v5

To prevent ecosystem fragmentation and give Python developers first-class Kubernetes primitives out of the box, we are bringing three major capabilities to the official client:

• Official asyncio support: Rather than forcing developers to choose between the official synchronous client and third-party async libraries, we are introducing official asyncio support directly into kubernetes-client/python. Async-first ML orchestrators, FastAPI-based model control planes, and notebooks will be able to issue concurrent, non-blocking Kubernetes API calls natively.
• Shared Informers (kubernetes-client/python#2515): We are bringing the SharedInformer pattern to the Python client. With built-in list-watch caching, automatic reconnection, and resource event handlers (on_add, on_update, on_delete), Python controllers and AI job managers can monitor cluster state and Custom Resources (TrainJob, RayCluster, InferenceService) with a fraction of the memory footprint and API server load.
• WebSocket v5 streaming (KEP-4006, kubernetes-client/python#2486): As upstream Kubernetes transitions remote command streaming (exec, attach, port-forward) away from legacy SPDY to standard WebSockets via the v5.channel.k8s.io subprotocol, we are adding native WebSocket v5 channel support so interactive debugging and remote job execution work seamlessly against modern control planes.

3. Performance: Slashing deserialization overhead

For AI/ML platforms operating on multi-thousand-node clusters, client performance is an availability requirement. Our performance track targets the largest bottlenecks in Python client execution:

• Eliminating model reconfiguration overhead: When deserializing large JSON responses (such as listing tens of thousands of Pods across a training cluster), the legacy generated model layer repeatedly instantiates and reconfigures schema validation objects (issue #1921). With the move to OpenAPI Generator v7+, we are revamping model initialization so large cluster queries deserialize in a fraction of the time.

What's next and how to get involved

The intersection of Python and Kubernetes is where modern AI infrastructure is built. By modernizing the code generation machinery, adding native asyncio and SharedInformer primitives, and closing the deserialization performance gap, we aim to give every ML engineer and platform builder a fast, reliable, and Pythonic official client—whether they are running a single notebook experiment or orchestrating a 50,000-node GKE training fleet.

This effort is happening in the open under Kubernetes SIG API Machinery, and we welcome feedback, use cases, and contributions from the community:

• Explore the repository and roadmap: Check out kubernetes-client/python and kubernetes-client/gen on GitHub.
• Join the conversation: Connect with us in the #sig-api-machinery channel on Kubernetes Slack or at the bi-weekly Kubernetes Python client triage meetings.

Scaling AI agents with open standards: Kurtis Van Gent on MCP and Agent Skills joining the AAIF

by Kurtis Van Gent, Google Cloud, and Daryl Ducharme, Google Open Source Programs Office

Building production AI agents requires more than a capable reasoning model. Agents need a reliable way to connect to external systems and data, and they need procedural domain knowledge so they know how to use those tools safely without exhausting their context window.

Over the past year, open source communities have rallied around two complementary open standards to solve these challenges: the Model Context Protocol (MCP) for connecting agents to live tools, and Agent Skills for loading procedural instructions and scripts on demand. Following Anthropic's donation of the Agent Skills specification to the Linux Foundation's Agentic AI Foundation (AAIF) alongside MCP, engineers from Google, Anthropic, OpenAI, and the wider ecosystem are now co-maintaining the standard under vendor-neutral governance.

To explore how open standards prevent ecosystem fragmentation, and why the best solutions don't always have to be homegrown, we sat down with Kurtis Van Gent, co-creator of the open source MCP Toolbox for Databases and a core maintainer on both MCP and Agent Skills.

From the N-by-M database problem to open source collaboration

Daryl Ducharme: How did your work on AI ecosystems for Google Cloud Databases and co-creating the open source MCP Toolbox for Databases lead to your role as a core maintainer on both the Model Context Protocol (MCP) and Agent Skills?

Kurtis Van Gent: When I started my current role around three years ago, it was early in the generative AI cycle. My mission was to make it easy for developers to use cloud databases for their AI applications. As a strong proponent of developer experience, I loved that charter. As model reasoning improved, context and data were clearly the keys to making effective use of it, and there's no better place to store data than a database.

At the time, most database conversations focused on retrieval-augmented generation (RAG) using vector search. However, our team believed that agents and tools held the most interesting potential for large language models (LLMs) going forward. We immediately hit two scaling challenges: Google Cloud supports more than 15 different databases across its portfolio, and new open source agent frameworks (starting with LangChain and LlamaIndex) were emerging every week.

That created a classic N-by-M integration problem between many clients and many data hosts. To solve it, we started GenAI Toolbox for Databases (now MCP Toolbox for Databases) as an open source framework for building and serving tools. We built our own server, clients, and protocol to standardize how agents look up and invoke database tools. Back then, agent terminology wasn't common vernacular yet, so every talk I gave had to start with three to five slides explaining what agents and tools actually were.

When Anthropic announced MCP, it was exciting validation that the broader ecosystem was arriving at the same architectural conclusions we were seeing across our database integrations. Rather than insisting on a homegrown protocol or spinning off a competing specification, we reached out to the MCP co-creators, Justin Spahr-Summers and David Soria Parra, and began meeting regularly. Because I believe strongly in open source communities and shared standards, I advocated for adopting MCP directly. We evolved our project into MCP Toolbox for Databases so it worked out of the box with any MCP client, and I focused on contributing capabilities from our original protocol back into the upstream MCP specification.

Leading the transition to a stateless MCP specification

Kurtis Van Gent: One of my primary upstream focuses was transport scalability. I began leading the MCP Transports Working Group with the goal of evolving the specification toward a stateless, horizontally scalable architecture. At first, it wasn't a crowded room. Our early working group meetings had around five participants, three of whom were from Google.

As MCP adoption accelerated across the industry, however, it became clear that early stateful assumptions around persistent sessions and initialization were holding MCP back at production scale. That effort culminated in a two-day Transports Summit where Google and fellow working group members hosted the MCP core maintainers at our San Francisco offices to work through the scaling bottlenecks and align on a path forward. Shortly afterward, I was invited to become an MCP core maintainer, both in recognition of our work identifying those transport challenges and to keep the Transports Working Group closely aligned with the core specification. The result of that collaboration was the July 28, 2026 MCP specification, which transitioned MCP to a stateless model.

That experience as an MCP core maintainer naturally positioned me to help co-maintain Agent Skills. Having already built working relationships with contributors across Google, Anthropic, and OpenAI, stepping in to help steward Agent Skills inside the AAIF felt like a natural continuation of the same collaborative model.

Why vendor-neutral governance in the AAIF matters for developers

Daryl Ducharme: From your perspective as a core maintainer representing Google, why does bringing Agent Skills into the Linux Foundation's Agentic AI Foundation (AAIF) alongside MCP matter for developers and open source communities?

Kurtis Van Gent: We've seen open source technologies like Kubernetes achieve widespread adoption and sustained cross-company contributions, but organizations are understandably hesitant to build critical workflows on a specification controlled by a single vendor. Just as the Cloud Native Computing Foundation (CNCF) provided a vendor-neutral home for container orchestration, the AAIF provides a neutral forum where organizations and independent maintainers can collaborate on shared agent standards.

Bringing Agent Skills into the AAIF signals how foundational this format has become across the AI ecosystem. Following in the footsteps of MCP, the organizations and open source communities adopting Agent Skills now have a shared seat at the table to shape how the specification evolves. For developers, this means you can author a skill once and trust that it will remain portable across different agents, IDEs, and harnesses without vendor lock-in.

Solving tool bloat with progressive disclosure: how MCP and Agent Skills fit together

Daryl Ducharme: You have spoken about the tension between "tool bloat" and context saturation when building production agents. How should developers think about the division of labor between MCP (connecting agents to live tools and data) and Agent Skills (loading procedural domain knowledge on demand)?

Kurtis Van Gent: One of the most impactful things MCP accomplished was commoditizing tools: it became straightforward for anyone to wrap a set of actions in an MCP server and share them. The unintended consequence was tool and context bloat. As agents grew more capable and connected to dozens of MCP servers, deciding which tool schemas and operational instructions to load into the context window became a major bottleneck.

Agent Skills introduced a remarkably simple, effective pattern for solving that problem through progressive disclosure. By loading only a lightweight YAML frontmatter name and description at startup, and pulling the full SKILL.md instructions, reference files, or scripts into context only when a task requires them, Agent Skills commoditized context the same way MCP commoditized tools.

While some observers initially viewed MCP and Agent Skills as competing approaches, they have always been complementary: Agent Skills package domain expertise into modular chunks that teach an agent how to approach a problem, while MCP provides the live execution interface to carry it out. Today, the broader ecosystem has converged on combining them, including efforts around Agent Plugins and proposals for serving Agent Skills over MCP so platform maintainers can distribute domain instructions and live tools together.

Putting MCP and Agent Skills to work in database and developer workflows

Daryl Ducharme: What is a concrete example of how developers are using Agent Skills and MCP together today—whether in Antigravity, database workflows, or across the broader ecosystem?

Kurtis Van Gent: Both MCP and Agent Skills have become ubiquitous across everyday developer workflows. In my own daily work, I write custom skills whenever I notice an agent struggling to get a repeatable workflow right on the first try. For example, I write skills for how I like to structure GitHub repositories or format commit messages so I never have to re-explain those conventions in a prompt.

At the platform level, our Google Cloud Databases team pairs MCP and Agent Skills together inside our Agent Plugins. When a developer using Antigravity connects to a PostgreSQL database on Cloud SQL or AlloyDB, they can install the corresponding Cloud SQL or AlloyDB plugin. From MCP, the agent gets a list of safe actions it can take against the database. From the bundled Agent Skills, it gets step-by-step instructions on how to use those tools for specific tasks, like provisioning an instance or configuring Identity and Access Management (IAM) database authentication securely—and because both specifications are open standards, the same SKILL.md and MCP pattern works across any open source database or agent harness.

That combination is both faster and safer than pointing an agent at raw documentation written for human readers. In our agent workflow benchmarks, we see developers complete complex database tasks much more quickly and reliably when MCP tools are paired with targeted Agent Skills.

What comes next for the Agent Skills specification

Daryl Ducharme: Working alongside maintainers from Anthropic, OpenAI, and the wider community inside the AAIF, what technical or governance areas are you most focused on next?

Kurtis Van Gent: Two major areas I'm focused on evolving next are packaging and discovery. Today, the Agent Skills specification defines how an individual skill is structured, but it doesn't yet standardize how collections of skills are bundled, how harnesses discover them, or how to resolve naming conflicts across skill sets. Closing those gaps will help address conformance differences across harnesses.

Versioning is another priority. As the Agent Skills specification matures under the AAIF, we need a clear, portable way for skill authors and agent harnesses to declare which version of the specification they support. I'm looking forward to working with the other maintainers and open source communities to standardize packaging, discovery, and versioning in upcoming releases.

How to get involved

Daryl Ducharme: For open source developers and maintainers who want to author portable skills or contribute to the specification and tooling at github.com/agentskills, where should they start?

Kurtis Van Gent: If you want to start authoring portable skills today, the documentation at agentskills.io walks through creating your first SKILL.md package in just a few minutes, and you can explore Google's open source catalog of developer skills at github.com/google/skills (along with our engineering retrospective on how we ship Google Agent Skills like products).

If you want to help shape the specification itself, visit the GitHub organization at github.com/agentskills to review open pull requests, file issues, and join the community Discord server where maintainers and contributors discuss upcoming specification proposals.

Sandboxing Untrusted AI Agents at Scale: Google Donates Agent Substrate and gVisor to the CNCF

by Mofi Rahman, Developer Relations Engineer, and Brandon Royal, Product Manager, Google Cloud

When Google introduced Kubernetes twelve years ago, the Pod became the default unit of compute for the cloud. The design assumed a straightforward lifecycle: a program boots, binds to a port, serves traffic, and runs until you stop it.

AI agents work fundamentally differently.

An AI agent that’s running on a machine executes untrusted code, commands, and tool calls, and must be isolated from its host and its network resources. At the same time, these agents, whether they’re an interactive coding assistant, an SRE troubleshooter, or a multi-turn evaluation loop, do not run continuously. They execute a step, call a language model, and wait. Most of the time, the agent does nothing at all while it waits for a model to stream tokens or a human to review a pull request. In the same way that Kubernetes was created to address the needs of containerized applications, we introduced Agent Substrate as a new orchestrator for the emerging agentic applications. Secure by default, highly efficient and built for multi-million agent scale.

Agent Substrate was designed from the ground up for AI agent security and density and brings the best parts of Kubernetes together with a new control and data plane for fully-isolated, ultra-low-latency sandboxed compute and storage. The growing Agent Substrate community has been rapidly building core capabilities and stabilizing APIs, culminating in the latest 1.0 release.

This week at KubeCon North America in Salt Lake City, Google Cloud is donating Agent Substrate and gVisor to the Cloud Native Computing Foundation (CNCF) in Sandbox status, alongside OpenRL.

This post explains why traditional approaches struggle with stateful agents, how Agent Substrate solves the problem, how it connects to enterprise governance on Google Kubernetes Engine (GKE), and where to see it in action at KubeCon NA 2026.

Challenges of running AI agents at scale

AI agents introduce two infrastructure bottlenecks that standard Kubernetes was never built to handle: idle resource waste and control plane overload.

1. The idle resource problem

During a typical interactive session, an agent can spend 90% to 95% of its time waiting for model inference, external APIs, or human reviews.

During that wait, the agent consumes zero CPU. However, you cannot simply kill the container. The agent holds critical state in memory and on disk: active language runtimes, local Git repositories, uncommitted file edits, and tool sessions.

With standard Kubernetes, platform engineers face two poor options:

1. Keep a dedicated Pod running all day: The agent responds quickly because it is already warm, but you pay for idle compute around the clock. At ten thousand active users, cluster costs become unsustainable.
2. Scale to zero with serverless containers: You save compute costs while the agent sits idle, but resuming from cold storage forces the container to clone repositories, reinstall packages, and rebuild memory. Wakeup latency climbs past 20 seconds, breaking the interactive experience.

Platform teams need agents that wake up in under 500 milliseconds with full memory and disk state intact—without paying for running Pods while the agent sits idle.

2. Control plane overload

The opposite issue occurs when training or evaluating agents. Reinforcement learning loops and benchmark suites need to run untrusted agent code across thousands of tasks at the exact same time.

If every sandbox requires a full Kubernetes Pod, the control plane quickly grinds to a halt. Scheduling, creating, and destroying thousands of short-lived Pods every minute overwhelms etcd, saturates kube-apiserver, and exhausts subnet IP addresses. Kubernetes was built to manage stable application topologies, not to act as a microsecond sandbox allocator.

How Agent Substrate solves the problem

Agent Substrate adds an actor-based control and data plane built on top of Kubernetes. It solves the idle resource problem by decoupling Actors from Workers:

• Actors hold the state: An Actor represents an individual agent session, complete with its own filesystem, environment variables, memory state, and network rules.
• Workers provide the compute: A Worker is a pre-warmed Kubernetes Pod running in a shared pool (WorkerPool), equipped with container runtimes and sandbox boundaries.

The lifecycle of an agent turn

1. Request parking: An incoming call hits the atenet-router L7 proxy. If the target Actor is suspended, the proxy holds the request in memory and alerts the Substrate controller.
2. Fast placement: The controller assigns the Actor to an available slot in the warm Worker pool.
3. Sub-second restore: The Actor’s memory and disk state restore from tiered storage, the process wakes up, and the proxy delivers the request. The agent resumes in under 500 milliseconds right where it paused.
4. Suspension on idle: When the turn finishes, Substrate snapshots the Actor’s state to tiered storage and frees the Worker slot for other workloads.

Because Substrate manages Actor state directly, thousands of agents can activate, suspend, and resume every second without creating new Pod objects or stressing etcd.

Safe execution with gVisor and microVMs

Agents frequently execute model-generated, untrusted code. Standard Linux containers share the host kernel, which presents an unacceptable security risk on multi-tenant nodes.

Agent Substrate pairs with gVisor (runsc), Google's open source application kernel written in Go. gVisor intercepts system calls in user space, isolating untrusted code from the host kernel. Because gVisor virtualizes its own memory and process tables, Substrate can freeze a running environment, write memory pages to storage, and restore them on another node in under 500 milliseconds.

For workloads requiring hardware virtualization or custom kernel modules, Substrate also supports Cloud Hypervisor microVMs under the same unified API.

Why Google donated Agent Substrate to the CNCF

Over the past two years, engineering teams across the industry have run into the same agent scaling boundaries. Without a shared open standard, every organization ends up building custom schedulers on top of VMs or working around Kubernetes.

Donating Agent Substrate and gVisor to the CNCF gives the cloud-native community a neutral, vendor-independent foundation that works across any conformant Kubernetes cluster.

Together with community projects, they form an open stack for agentic infrastructure:

• Layer 1: Secure Isolation (gVisor): Sandboxes untrusted code execution with user-space kernel isolation and fast memory checkpointing.
• Layer 2: Agent Runtime (Agent Substrate): Multiplexes stateful actors onto warm worker pools with sub-second wakeups.
• Layer 3: Training Orchestration (OpenRL): Coordinates Kubernetes-native reinforcement learning loops and evaluation sandboxes.

Where to find us at KubeCon NA 2026

If you are heading to Salt Lake City for KubeCon + CloudNativeCon North America from November 9 to 12, visit our sessions, workshops, and booth demos:

When	Session or Event	Speakers & Location
Monday, Nov 9	Agent Substrate Day (ATEcon)	Welcome to ATECon, Tim Hockin, Eitan Yarmush, Ron Lev Pleading the Agent: Proof of Agent identity, Greg Castle, Taahir Ahmed Evolving The Substrate Dataplane, Benjamin Elder Truly Sandboxed Networking: Moving beyond MVP, Bowei Du, Lior Lieberman Using Substrate for anything but Agents, Mofi Rahman Agent Substrate Environment, Jaana Dogan, Lior Lieberman Closing Remarks, Tim Hockin, Eitan Yarmush, Ron Lev Salt Palace
Monday, Nov 9	Detonation and Isolation: Hardening Kubernetes Runtimes for Agents Multi-agent orchestration and runtime governance	Syeda Anjum, Kaslin Fields, and Kanchana Rajendran Agentics Day
Monday, Nov 9	GKE Day Announcements and hands-on agent infrastructure labs	Google Cloud GKE Team
Tuesday, Nov 10	Flagship Fireside Chat: Kubernetes Act II Why AI Agents Change Everything Under the Hood	Tim Hockin and Kelsey Hightower Moderated by Savannah Peterson
Wednesday, Nov 11	Hands-on Workshop: Building and Governing Agent Substrate on GKE Hands-on lab deploying warm worker pools, testing snapshot restores, and configuring egress secrets	Mofi Rahman, Drew Brown, Casey West, Kaslin Fields, and Olivier Bourgeois Schedule ID: 1247804
Wednesday, Nov 11	Expo Hall Demo Theater: Inside the Show-Floor Agent Swarm Live fleet multiplexing on GKE	Brandon Royal and Mofi Rahman
Nov 10 to 12	Google Cloud Booth #122 • Agent Swarm demo: See agent substrate in action. • Partner demos with NVIDIA on OpenShell and GPU Actor infrastructure • Filestore Agent Volumes in action	Booth #122 Expo Hall

Get started

Agent Substrate is open source and ready for testing today:

• Documentation: Read the guides and architecture specs at ate.dev.
• Code: Explore the runtime on GitHub at github.com/agent-substrate/substrate.
• Community: Join #substrate-dev #substrate-user on the CNCF Slack.

See you in Salt Lake City!

Supercharging XLA-based inference: Announcing the launch of XLA Serving Engine

by Oleg Shaldybin and Monica Song, Google TPU Software

Today we are excited to announce the launch of XLA Serving Engine (XSE), a general-purpose model export and inference runtime for the OpenXLA ecosystem, open-sourced under the Apache 2.0 license.

We created XLA Serving Engine to enable a “Train in Python, Serve in C++” workflow. Most ML engineers develop models using Python (with frameworks like JAX and PyTorch); however, some production serving environments do not allow Python due to operational and deployment complexity. XLA Serving Engine offers a solution by providing an API for engineers to export their ML models to an “XLA SavedModel” containing the StableHLO representation of their model’s function, which can then be loaded and executed by the XLA Serving Runtime.

This approach allows users to serve models (typically non-LLMs) directly in a C++ runtime, which enables high throughput at any scale and optimizes for low latency by removing the overhead of the Python interpreter. In our studies, we found that for short requests, XSE delivers 2–5x the throughput of the best Python server we could build, and 8–36x that of a straightforward one, while using 30–70% less CPU time per prediction and a quarter of the memory.

Last year, we began deploying core components of XLA Serving Engine as part of Google’s internal load-bearing, production serving infrastructure. Today, XSE serves over 6 million queries per second and powers some of Google's highest-volume online inference pipelines, including YouTube Shorts, Google Ads, Search, and parts of Gemini Omni, with plans to onboard more customers internally.

Having successfully scaled XSE to support the high-throughput workloads of our internal teams, we are now ready to bring this core piece of infrastructure to the wider ecosystem of hardware partners and Cloud TPU users. We designed XLA Serving Engine to be self-contained and modular, so users can deploy it as a standalone inference engine or plug it into their more complex serving solutions.

Core production-ready features

• Python-Free Serving: XSE enables a “Train in Python, Serve in C++” workflow via the intermediary compiled XLA SavedModel serialization format. In our studies, we found that XSE delivers 2–5x the throughput of a pure Python server.
• Low-latency execution: The XLA Serving Runtime wraps PJRT directly and optimizes the model using the XLA compiler. This eliminates Python runtime overhead and the global interpreter lock. XSE is C++ native, which means one process is capable of using all eight cores.

Figure 1: Throughput under latency target

• Orbax Checkpointing Integration: XLA SavedModel stores JAX model computations as StableHLO and uses Orbax checkpoint to store the weights.
• Flexible Scale: Supports both single-host and multi-host serving by leveraging PJRT.
• Performance Parity: XSE preserves the performance of the model’s forward function (including kernel optimizations). Hand off your model to XLA Serving Engine with confidence that performance will be identical to its training environment.
• Docker Packages: We provide a suite of Docker images that are tested on CPU, GPU, and TPU and expose simple gRPC APIs for calling your model.

Customer testimonials and partner use cases

Our Private Preview partners have been actively leveraging XLA Serving Engine to bring their deep learning stacks to production. Here is a selection of private preview customers (alphabetized) to showcase the diversity of uses:

Arm: Performance Parity on Arm Neoverse platforms

Our work with Arm has proven the production capabilities of JAX and XLA for training and serving on Arm-based CPUs:

“As AI inference scales, software optimization is critical to delivering performance and efficiency across diverse workloads. Through our collaboration with Google and contributions to OpenXLA, we’ve optimized the XLA CPU backend and serving runtime for vision, embedding and recommendation workloads on Arm Neoverse-based platforms, including Google Cloud Axion.”

— Ian Bratt, Vice President of ML Technology and Fellow, Arm

Intel: Native JAX on Xeon server CPUs

We have also been partnering with Intel to showcase the power of Intel’s optimized compute:

“Our partnership with Google showcases the XLA Serving Engine (XSE) on Intel® Xeon® processors, delivering native, high-performance JAX inference accelerated by Intel® AMX. With XSE, enterprises get out-of-the-box performance across Vision, Recommendation, and LLM workloads — and the flexibility to serve them efficiently wherever their business runs, from cloud to edge to on-prem.”

— Bill Pearson, Vice President of Data Center and AI software, Intel

Moloco: Scaling advertising and recommenders

“To support our demand-side platform (DSP) processing over 650 billion daily inferences and peaking at 10.5 million QPS, we needed an inference runtime that could match our massive scale. XLA Serving Engine is the high-performance, C++-based JAX serving solution we’ve been waiting for that keeps us framework agnostic and complements our Cloud TPU Trillium investments.”

— Kunal Kukreja, Staff Machine Learning Engineer, Moloco

What’s next

• Torch Support: We plan to integrate with TorchTPU, allowing PyTorch users to export to an XLA SavedModel to serve their models on TPUs in C++.
• Ahead-of-Time (AOT) Compilation: We are working on rolling out AOT compilation to our internal users before bringing this feature to external users. AOT compilation will reduce cold-start time by offering users the ability to compile their models offline (on CPU) for their target serving hardware.
• C++ Data Preprocessing: We are offering a C++ based preprocessor for users who want to combine their data processing with their model’s graph in C++.

Limitations

XSE is a lightweight, one-shot inference engine. For large autoregressive language models with long context lengths, we recommend popular projects such as vLLM or SGLang, which benefit from LLM-specific memory management techniques.

Get started today

Ready to streamline your JAX production pipeline?

• Check out the XLA Serving Engine quick-start instructions at github.com/openxla/xla-serving. You can find the list of available Docker images here.
• File an issue or contribute pull requests at github.com/openxla/xla-serving.

Acknowledgements

Thank you to everyone who contributed to XLA Serving Engine, including but not limited to the following individuals:

Ashish Agarwal, MD Faijul Amin, Carlos Araya, Renato Arantes, Isha Arkatkar, Sagun B, Ashok Bhat, Xevi Miró Bruix, Chris Chan, Ray Chen, Chen Cheng, Hyun Min Choi, Andrea Covre, Gauri Deshpande, Tiffany Dong, Sayce Falk, Happy Gao, Vaibhav Gurunathan, Michael Hudgins, Kritanjali Jain, He Jiang, Leon Kennedy, Young Seok (Tony) Kim, Penporn Koanantakool, Kunal Kukreja, Younghee Kwon, Shawn Lee, Shawn Lu, Siddhartha Menon, Marek Michałowski, Sachin Muradi, César Naranjo, Pat Notz, Adam O’Brien, Olivia Qin, AG Ramesh, Shubha Rao, Mayank Raunak, Crefeda Rodrigues, Gassan Salama, Oleg Shaldybin, Brian Seung, Juanli Shen, Alex Shires, Jeyeon Si, Prairie Sims, Monica Song, Anne Stern, Quoc Truong, Armando Ugalde Velasco, Jess Wiser, Siqiao (James) Wu, Qinglan Xiang, Yunjie Xu, Jeffrey Yoo, Hyungsuk Yoon, and Wangyuan Zhang.

GKE Workspaces: Your notebook, on Google's AI infrastructure

by Siyuan Zhang & Antonio Ojea, Google Cloud

If you're an ML engineer or researcher, the GPUs and TPUs you need increasingly live on Kubernetes, but Kubernetes should be an invisible layer. While your platform team sets up the GKE cluster and manages accelerator capacity and reservations, your job is to train and ship models—not to manage Kubernetes infrastructure.

Today we're introducing GKE Workspaces, a self-hosted AI/ML development platform for Google Kubernetes Engine, built on the open source Kubeflow Workspaces project. Deployed on your team's GKE cluster, it provides a dedicated web dashboard where you can pick an IDE and a machine, and start coding on cloud accelerators in minutes.

Why Kubeflow Workspaces

Kubeflow Workspaces gives every researcher a personal, persistent development environment on shared cluster hardware.

• Self-service, in a few clicks. Sign in to your team's Workspaces web UI, choose JupyterLab or in-browser VS Code, pick an environment, and select a hardware size such as "Small CPU", "GPU", or "TPU". Researchers don't need Google Cloud Console access, YAML, or command-line setup.
• Your work persists. Each workspace mounts its own home directory on a dedicated, encrypted Persistent Disk in your Google Cloud project—isolated to that workspace, so your files, notebooks, and installed packages stay safe across restarts, pauses, and machine changes.
• Guardrails set by your platform team. Admins curate the approved images and hardware options, and decide which machines, reservations, and capacity types (such as on-demand or Spot) back them. Researchers only see choices that work.
• Open source and portable. Workspaces is part of the Kubeflow community, empowering you with the freedom and flexibility of a fully open source notebook ecosystem.
• Upstream integration and open licensing. GKE Workspaces builds directly on Kubeflow Workspaces with key enhancements for GKE environments—including standalone deployment without Istio dependencies, integrated authentication token flows, and stateful GPU/CPU pause and resume capabilities. Both GKE Workspaces and upstream Kubeflow Workspaces are released under the Apache 2.0 open source license, and we are actively working with the Kubeflow community to upstream these features.

What's unique about GKE Workspaces

GKE Workspaces brings Kubeflow Workspaces together with capabilities that are only available on Google Cloud.

1. Pause and resume—without losing your state

You've spent an hour loading a large model into GPU memory and building up results across notebook cells. Now it's time to go home. Normally you have two bad options: shut down and lose everything, or leave it running and pay for an idle GPU overnight.

With GKE Workspaces, you just pause by clicking a button from the Workspaces Dashboard. Built on Pod Snapshots, GKE takes a snapshot of your running notebook—including variables in memory and models loaded on the GPU—saves it to Cloud Storage, and releases the hardware so you stop paying. When you resume, you pick up exactly where you left off. Workspaces can also be paused automatically after a period of inactivity.

2. An easy on-ramp to Cloud TPUs

Moving from a quick experiment to large-scale training on TPUs used to mean switching tools at every step. GKE Workspaces keeps you in the same notebook the whole way, with both JAX and PyTorch:

1. Prototype on a single TPU chip, just like you would on a GPU.
2. Scale to multiple chips on one machine by choosing a bigger hardware option.
3. Scale out to multi-host TPU slices by sending the same training function from your notebook to a larger slice. Logs stream back to your notebook, and the TPUs are released as soon as the job finishes.

3. Use your local coding agents with cloud accelerators

Prefer using your own AI coding agents and editors locally? You can easily bridge them to cloud compute. In the Workspaces web UI, generate a short-lived, revocable link to run your notebooks and agent-generated code on a remote GPU or TPU. Traffic is secured over HTTPS via Google Identity-Aware Proxy and the cluster's access proxy, checking permissions on every request—so no SSH keys, local tunnels, or cluster credentials ever sit on your laptop.

4. A launchpad for distributed work

Your workspace doesn't have to be big to drive big jobs. From a small, inexpensive CPU notebook you can process data with Spark, launch multi-host training, serve your model, and scale Ray clusters—all from Python. Built-in Cloud Storage access, with no keys to manage, makes it easy to pass data between each step.

5. A portal to safe elastic sandboxes for coding subagents

Running a fleet of coding agents directly on your local machine can drain your battery, expose sensitive files or credentials to agent-executed code, and halt progress as soon as you close your laptop. From a standard GKE Workspace, you can safely task your favorite coding agent to hand off parallel work—such as resolving failing tests across a repository—to background subagents. Crucially, while your primary workspace operates as a full-featured environment, these subagents are isolated within lightweight, in-cluster sandboxes using providers like Agent Substrate, requiring zero local setup. Each subagent executes independently in its own sandbox on shared cluster nodes, continues running even when you log off, snapshots to Cloud Storage at zero compute cost while idle, and outputs a clean patch for your review and merge.

6. Lightweight and secure by default

GKE Workspaces runs on its own on GKE Standard or Autopilot—you don't need to install the full Kubeflow platform. Users sign in with their Google accounts, traffic is encrypted with Google-managed certificates, and each team's work is isolated from the others.

Get started

GKE Workspaces lets ML teams focus on models, not infrastructure: one-click development environments, stateful pause and resume, a smooth path to Cloud TPUs, and the power of GKE behind every notebook.

Ask your platform team to deploy it using the GKE deployment guide, and learn more about the upstream project at Kubeflow Workspaces. We welcome feedback and contributions for both Apache 2.0-licensed projects.

This Week in Open Source for October 30, 2026

by Daryl Ducharme & Shane Glass, Open Source Programs Office

A look around the world of open source

Right before Halloween weekend, it feels fitting to ask what actually haunts open source communities right now—from supply-chain ghosts lurking in commit histories and the hidden cost of maintaining isolated "zombie" forks to models wearing open source costumes without sharing what is under the mask. Across this week's reads, maintainers, researchers, and engineers share how they are keeping the lights on: setting clear community boundaries for AI contributions, wiring human sign-off and cost guardrails directly into automated workflows, and co-building open infrastructure across hardware stacks. Technology only serves people when the humans doing the work stay in control of the machinery—and transparent, inspectable tools remain our best defense against things that go bump in the night.

Upcoming events

🗓️ November 2026

• Open Source in Finance Forum (OSFF) New York (November 4–5, 2026) – New York, New York, USA. Dedicated industry conference examining open source compliance, supply-chain security, and collaborative innovation across regulated financial institutions.
• SeaGL (Seattle GNU/Linux Conference) 2026 (November 6–7, 2026) – Seattle, Washington, USA & Virtual. Free, grassroots technical summit dedicated to spreading awareness and knowledge of free and open source software, hardware, and culture.
• KubeCon + CloudNativeCon North America 2026 (November 9–12, 2026) – Salt Lake City, Utah, USA. The Cloud Native Computing Foundation's (CNCF) flagship conference uniting adopters and maintainers around Kubernetes and containerized distributed infrastructure—featuring over 35 sessions and keynotes from Googlers including Abdel Sghiouar, Jago Macleod, Tim Hockin, Kaslin Fields, John Belamaric, Dawn Chen, Janet Kuo, Benjamin Elder, Antonio Ojea, Lior Lieberman, Michelle Au, and Braydon Kains across Kubernetes Special Interest Groups (SIGs), etcd, containerd, Gateway API, Common Expression Language (CEL) admission policies, and agent sandboxes, plus the Google-sponsored Community Hub.
• SFSCON (South Tyrol Free Software Conference) 2026 (November 13–14, 2026) – Bolzano, Italy. One of Europe's longest-running free software conferences bringing together public-sector decision-makers and developers to advance digital sovereignty and open infrastructure.

🗓️ December 2026

• Yocto Project Virtual Summit 2026.12 (December 1–3, 2026) – Virtual. Community summit for embedded Linux engineers and maintainers building custom, reproducible Linux distributions across edge and Internet of Things (IoT) hardware.
• Open Source Summit Japan 2026 (December 7–9, 2026) – Tokyo, Japan. The Linux Foundation's premier Asia-Pacific conference connecting open source developers, kernel maintainers, and enterprise leaders across cloud, embedded Linux, and open AI—featuring a keynote by Google's Kaz Sato on building agents on open standards alongside sessions from Googlers Tanya Sharma, Tarun Sahu, and Anurag Sinha.
• Open Compliance Summit 2026 (December 10–11, 2026) – Tokyo, Japan. Specialized gathering uniting legal, security, and Open Source Programs Office (OSPO) practitioners around OpenChain, Software Package Data Exchange (SPDX), software bills of materials (SBOMs), and global open source supply-chain compliance.

🗓️ January 2027

• Free and Open Source Software Developers' European Meeting (FOSDEM) 2027 (January 30–31, 2027) – Brussels, Belgium. Europe's largest free, volunteer-organized gathering for free and open source software developers at the Université libre de Bruxelles—with calls for participation now open across 77 community developer rooms and main-track proposals open through November 16.

Open calls for proposals (CFPs)

• Free and Open Source Software Developers' European Meeting (FOSDEM) 2027 (Event: January 30–31, 2027 – Brussels, Belgium) – Main-track CFP deadline: November 16, 2026 (developer room calls for participation open across 77 accepted community developer rooms, with devroom schedules published by December 7, 2026).
• AtmosphereConf 2027 (Event: April 29 – May 2, 2027 – Amsterdam, The Netherlands) – Round 1 CFP deadline: November 30, 2026 (Round 2: January 11 – February 14, 2027). Community conference uniting developers, researchers, and creators building on the Authenticated Transfer Protocol (AT Protocol), open social web standards, and decentralized agent substrates across Build, Create, Inform, and Govern tracks.

Open source reads and links

• [Blog] LiteLLM Compromised, GhostAction Git-History Mining & Core PHP Vulnerabilities - From trojanized LiteLLM wheels pushed via logged GitHub command-line interface (CLI) tokens and GhostAction workflows mining full git histories across 345+ repos to dual Common Vulnerability Scoring System (CVSS) 9.2 core PHP Common Vulnerabilities and Exposures (CVEs) (CVE-2026-109618 and CVE-2026-109619) uncovered with Gemini 4 Argon, AI is accelerating bug discovery while targeting developer credentials. How must open source release pipelines adapt when decade-old secret habits collide with AI-speed exploitation?
• [Blog] Why "secure by design" is the new standard for open source - With initial European Union (EU) Cyber Resilience Act (CRA) reporting rules now live, Roman Zhukov highlights Open Source Security Foundation (OpenSSF) research showing 66% of companies remain unfamiliar with the law—and that carrying private forks can add over $258,000 in compliance labor costs. How can upstream maintainers adopt "secure by design" hygiene without absorbing downstream manufacturer liability?
• [Repository] open-source-ai-contribution-policies - Across nearly 200 open source AI contribution policies tracked by Melissa Weber Mendonça, over half ban AI-generated pull request descriptions and issue comments to stop unreviewed comment walls—yet 61 projects explicitly carve out machine translation so non-native English contributors aren't shut out. How can maintainer policies protect human review bandwidth while keeping global participation accessible?
• [Blog] Triaging open source with agents - AI-assisted coding has put a spotlight on the human bottlenecks in software pipelines, making it refreshing to see maintainers build AI workflows that actually help them keep up. Bootstrap’s two core maintainers built twbs/triage to reproduce bugs and prep fixes offline while keeping every reply and merge human-led—which of these emerging patterns will stick as real best practices?
• [Blog] Genkit Go 1.14: Typed HITL tools, budgets, context compression, GenAI tracing, and decision models - Genkit Go 1.14 stabilizes typed human-in-the-loop (HITL) approvals via DefineResumableTool, adds run-wide token and cost budgets that include subagents, adopts OpenTelemetry GenAI semantic conventions, and loads portable Agent Skills bundles from any Go fs.FS. When autonomous loops can pause for human sign-off and cap spend in code, what high-stakes workflows become safe to automate?
• [Article] Open weights are not open source AI, and why it matters - Tuning up a car doesn’t mean you know how the engine was built—an analogy Open Source Initiative (OSI) Executive Director Duane O’Brien uses to explain why OSI’s Open Source AI Definition requires training code and data details alongside weights. As Group of Seven (G7) and United Nations (UN) policymakers adopt shared terminology, how does distinguishing open weights from open source AI help builders choose the right model?
• [Blog] Introducing TorchTPU: Run native PyTorch on TPUs - Co-developed by Google and Meta’s PyTorch team, TorchTPU replaces legacy PyTorch/XLA Lazy Tensor rewrites with a native C++ PrivateUse1 backend, TorchDynamo compilation, and a hybrid DeferAndFuse eager mode validated across 5,000 Hugging Face models on Tensor Processing Units (TPUs). When switching accelerators is as simple as changing .to("cuda") to .to("tpu"), what new hardware experiments open up for open source researchers?
• [Blog] llm-d v1.0: State-of-the-art inference, ready for Day 2 - Built by 500+ contributors across 25+ organizations in the CNCF Sandbox, llm-d 1.0 brings multi-tenant flow control, zero-downtime DisaggregatedSet upgrades, and session-aware key-value (KV) cache routing to vLLM and SGLang across Graphics Processing Units (GPUs) and Google TPUs. Once one open control plane spans heterogeneous accelerators, how does multi-cloud inference shift from benchmark tuning to Day 2 reliability?
• [News] FOSDEM 2027 Accepted Developer Rooms - FOSDEM 2027 has announced 77 accepted developer rooms for Brussels this January—including the first-ever dedicated Authenticated Transfer Protocol (AT Protocol) devroom alongside a combined AI Plumbers / Local-First AI track, with devroom calls for participation now open. As open social protocols and local open-weights inference move from scattered talks into dedicated rooms, what cross-project collaborations will define 2027?

Which of these stories will you be chatting about at your next meetup or conference? Let us know! Share with us on our @GoogleOSS X account or our @opensource.google Bluesky account.