Announcing Apache Iceberg 1.11.0

by Alex Stephen & Talat Uyarer, Lakehouse

Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version.

Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets

The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before.

Let's take a look at some of the biggest changes.

Spark & Flink Updates

As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both.

• Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming.
• Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink started supporting nanosecond, variant and unknown types from V3 Spec.

Server-side scan planning

In previous versions of Iceberg, the client handled the heavy lifting of scan orchestration. The driver of engine would traverse the table's metadata tree, retrieving manifest lists and files from object storage to filter data against specific partition requirements. Iceberg 1.11.0 shifts this computational burden into the catalog through server-side scan planning.

Instead of manually traversing manifests, the engine submits a single POST …/plan request detailing the scan allowing the REST catalog to return optimized FileScanTasks.

The API is designed to handle data at any scale: smaller scans return immediate results, extensive operations return a plan-id for polling, and massive datasets are retrieved via parallel plan-tasks through POST …/tasks.

Planning moves off the query engine and into the catalog — the driver no longer touches metadata in object storage.

Built-in table encryption

As data lakes increasingly serve as the central hub for sensitive PII and financial data, relying solely on bucket-level storage encryption is no longer enough. Iceberg 1.11.0 introduces built-in table encryption, bringing fine-grained, KMS-backed security directly to the table level.

This provides data platform teams with robust capabilities for security and compliance:

• Zero-Trust Storage Security: Even if a malicious actor gains direct access to your underlying object storage bucket, the data remains completely unreadable.
• Total Index Protection: It isn't just the raw data that is protected; Iceberg encrypts the manifest lists as well, preventing attackers from inferring sensitive information from table statistics.
• Tamper-Proof Data: Built-in authentication tags guard against unauthorized modifications, ensuring data integrity.
• Effortless Key Rotation: Keys are rotated automatically as they age, satisfying strict compliance mandates without requiring you to rewrite massive datasets.

Iceberg achieves this using envelope encryption with a three-tier key hierarchy. A table master key lives securely in your KMS and never touches Iceberg storage. This master key wraps key-encryption keys (KEKs), which are stored safely inside the table metadata. Finally, each KEK wraps a unique, per-file data-encryption key (DEK).

Every data file and manifest list is then encrypted with AES-GCM under its own unique DEK. This decoupled architecture ensures maximum security while maintaining the high performance expected of Iceberg workloads.

File Format API

Historically, Iceberg's format-handling code was tightly coupled, growing organically around Parquet, Avro, and ORC. Adding a new format or enforcing consistent feature support (like V3 default values or new column types) across all formats meant duplicating complex engine-specific switch/case code paths.

Iceberg 1.11.0 introduces the finalized File Format API, bringing a consistent API to reading and writing all of these file formats.

Instead of hardcoded engines handling binary extraction, the architecture introduces:

• FormatModel: A standardized implementation defining how a file format handles reader/writer construction and its specific capabilities.
• FormatModelRegistry: A central directory where query engines fetch appropriate read and write builders.

This API (which is already seeing adoption around other Apache Iceberg implementations) provides a significant code cleanup for the future of the project. It also opens the door for more file formats as time goes on.

Moreover, this new interface facilitates the implementation of Column Families, enabling vertical partitioning of storage. This advancement lets teams perform targeted updates or rewrites on isolated columns—such as recalculating vector embeddings—while leaving the remaining table data undisturbed.

SQL UDF Specification

1.11.0 includes the SQL UDF specification, which adds a brand new metadata format for both Scalar and Table Functions:

• Immutable Versioning and Rollback: UDF metadata is written as self-contained, versioned JSON files stored right in the object store. If a data engineer deploys a buggy UDF update, administrators can execute an atomic rollback to a previous version log state
• Standardized Schema Typings: Parameters and return types map cleanly to Iceberg Type JSON representations, directly accommodating complex nested maps, structs, and the upcoming Iceberg V3 variant type.
• Engine Specific Execution: Each SQL UDF has a function implementation for each engine, allowing users to leverage engine-specific functionality in their UDFs.

Google Analytics Library Integration

For Google Cloud customers, version 1.11.0 delivers substantial throughput gains by embedding the GCS Analytics Core library into GCSFileIO (Issue #14326, PR #14333).

This integration introduces Footer Prefetching, which optimizes Parquet length checks by caching object suffixes to remove network overhead. Combined with threaded VectoredIO for concurrent multi-range operations and specialized small object caching for sub-1MB files, these enhancements eliminate persistent I/O bottlenecks. Initial benchmarks indicate that these architectural improvements can reduce Parquet metadata parsing latency and boost total record processing speeds, empowering high-scale Spark, Flink, and Trino workloads to run with improved efficiency on Google Cloud Storage.

Getting Started with 1.11.0

We are excited to be part of the Apache Iceberg community and innovating together. As a compliant Iceberg REST Catalog, Lakehouse for Apache Iceberg (formerly BigLake) already has support for version 1.11.0.

To upgrade your environment, update your build dependencies to version 1.11.0. Remember to review your deployment runtimes to ensure compatibility with the new JDK 17 baseline, and test your workloads if you are transitioning from Spark 3.4.

For a full breakdown of every bug fix, contributor attribution, and dependency bump, check out the official Apache Iceberg Releases Page

TestParameterInjector introduces an idiomatic Kotlin API

by Jens Nyman, TestParameterInjector Team

In March 2021, we announced the open source release of TestParameterInjector: a simple but powerful parameterized test runner for JUnit4. In September 2022, we followed up with JUnit5 support, bringing our framework to developers who had moved on to the Jupiter API.

We're excited to announce our biggest update yet for our Kotlin users: KotlinTestParameters.

The de facto standard for parameterized testing

When we first introduced TestParameterInjector, we shared a graph showing its rapid adoption within Google. Over the past few years, that trajectory has continued to a point where TestParameterInjector is the de facto parameterized test framework.

Usage of all other alternative frameworks continues to steadily decline, while TestParameterInjector's adoption keeps growing rapidly. It has fundamentally lowered the barrier to writing data-driven unit tests, empowering Googlers and open source developers alike to maximize test coverage with minimal boilerplate. We believe its ubiquity internally is a strong testament to its reliability and utility for the broader developer communities.

The Kotlin challenge

As Kotlin's popularity has surged, developers have naturally been writing more of their TestParameterInjector tests in Kotlin. However, specifying explicit test values in Kotlin historically meant falling back to Java-centric paradigms.

If you wanted to provide specific values to a test, you typically had three options, none of which felt truly idiomatic in Kotlin:

1. @TestParameter({"123", "456"}): This relies on string arrays, limiting you to a subset of types that the string parsing supports.
2. @TestParameters: This allows for more complex sets of data, but relies on YAML strings (e.g.,"{age: 17, expectIsAdult: false}"). These strings however are not type-safe, and are completely ignored by IDE refactoring tools.
3. Provider classes: For complex types that couldn't be easily represented in strings, you have to write Provider classes, adding a bit of boilerplate code and indirection.

Enter KotlinTestParameters

To bring a seamless experience to Kotlin, we are introducing a significant new Kotlin-only feature: KotlinTestParameters.

By leveraging Kotlin's default function arguments, you can now define parameterized tests in a fully type-safe, concise, and refactor-friendly way using the testValues() function (and friends).

Here is what it looks like in practice:

import com.google.testing.junit.testparameterinjector.TestParameterInjector
import com.google.testing.junit.testparameterinjector.TestParameter
import com.google.testing.junit.testparameterinjector.KotlinTestParameters.testValues
import com.google.testing.junit.testparameterinjector.KotlinTestParameters.namedTestValues
import org.junit.Test
import org.junit.runner.RunWith

@RunWith(TestParameterInjector::class)
class MyTest {

  // Testing simple types directly
  @Test
  fun simpleTest(@TestParameter limit: Int = testValues(20, 100)) {
    // This test method is run twice: once for limit=20 and once for limit=100
  }

  // Testing complex types without YAML strings or Provider classes!
  data class TestCase(val age: Int, val expectIsAdult: Boolean)

  @Test
  fun complexTest(
    @TestParameter testCase: TestCase = namedTestValues(
      "teenager" to TestCase(age = 17, expectIsAdult = false),
      "young adult" to TestCase(age = 22, expectIsAdult = true)
    )
  ) {
    // This test method is run twice with fully typed data class instances
  }
}

Why we recommend making the switch

Because testValues() seamlessly integrates with Kotlin's language features, any type is supported. This completely eliminates the need for stringly-typed YAML maps or verbose Provider classes.

We firmly believe that KotlinTestParameters is a massive leap forward in readability and maintainability. Moving forward, this should be the default way of specifying test values for all new Kotlin tests, replacing the older @TestParameters, @TestParameter({"..."}), and Provider class patterns.

Try it out!

You can read more and start using KotlinTestParameters today over on our GitHub repository.

Let us know what you think on GitHub if you have any questions, comments, or feature requests!

Disrupting the presentation layer using autonomous workflows

by Adrian Chung, Abdelfettah Sghiouar & Matt Larkin, Google Kubernetes Engine

Empowering every engineer to do more with Kubernetes

Kubernetes is the gold standard for container orchestration. Its power, flexibility, and rich API surface are exactly why it has become the foundation of modern cloud-first infrastructure. Today, engineers express that power through the K8s API, declarative YAML manifests, and cloud consoles, a remarkably expressive toolbox.
We believe the next step is to expand how engineers interact with that toolbox. A Kubernetes expert should be able to converse with a deep-domain peer that speaks fluent control-plane and can reason about cluster state in real time. An engineer who isn't a Kubernetes specialist should be able to express higher-order intent, such as "deploy my application," "rebalance this workload" and have it carried out safely against the same powerful APIs. Both audiences get more leverage out of the platform they already trust.
This is the vision behind Kube-Agents: a system of intelligent, autonomous, and human-in-the-loop agents that act as a new, intent-driven presentation layer for Kubernetes. We are moving from declarative intent via API to higher-order, human intent-driven operations while preserving everything that makes Kubernetes great underneath.

The Vision: Expanding the Presentation Layer

Today, engineers do impressive work stitching together metrics, alerts, and multi-step commands to keep clusters healthy. Agents extend that work, not replace it. By complementing existing interfaces with autonomous agents, engineers can choose the level of abstraction that fits the task: drop down to kubectl and YAML when precision matters, or describe intent in plain language when speed and clarity matter more. The agents continuously observe system state and can execute complex operations in real time on the engineer's behalf. This isn't about hiding Kubernetes. It's about giving every engineer a more capable collaborator on top of it.

Meet the Agents: A Specialized Team

Our architecture is currently built upon three core specialized agents, each acting as a new kind of intent-driven collaborator for different stakeholders:

1. The Platform Agent
   • Role: A partner for the central governance layer, your management plane.
   • Focus: Codifying best practices and keeping platform blueprints evergreen and synchronized across the entire fleet.
   • Example: When a new egress policy is defined at the org level, the Platform Agent propagates it to the Dev Team agents and confirms enforcement, giving platform teams confidence in compliance while letting developers stay focused on their applications.
2. The Cluster Operator Agent
   • Role: A trusted teammate for your infrastructure operators.
   • Focus: Global concerns like multi-cluster balancing, automated provisioning, security patching, and zero-downtime version upgrades.
   • Example: It can detect a degrading node and proactively migrate workloads before application latency spikes, expanding what a single operator can safely manage at scale.
3. The Development Team Agent
   • Role: A production-savvy peer for developers.
   • Focus: The primary collaborator for developers. It supports the full workload lifecycle — reconciling manifest drift, right-sizing resources, and assisting with real-time debugging.
   • Example: When a developer asks "Why is my service failing?" in chat, the agent responds with relevant logs, correlated metrics, and a diagnosis of recent config changes — meeting a Kubernetes expert at depth and meeting a less specialized developer at intent.

Leveraging Industry Benchmarks

DevOps Bench is a comprehensive suite of benchmarks. These specialized agents learn from those results so they are equipped with the context to make well-reasoned decisions when autonomously supporting infrastructure work.

The First Demo

To be truly useful at the presentation layer, these agents can't be short-lived request/response scripts. They need to be persistent, long-running "team members" capable of continuous learning and collaboration.
As a first step, we've launched a set of workspaces compatible with OpenClaw for a demo, installable into your OpenClaw environment, leveraging existing out-of-the-box capabilities around identity, storage persistence, and memory. The agents included are: the Platform Agent, the Cluster Operator Agent, and the Development Team Agent.

• Autonomous GitOps & JIT Probing (Dev Team Agent): Demonstrates prompt-driven staging deployments and dynamically generated, context-aware probers. The agent adheres strictly to GitOps workflows by opening PRs for infrastructure updates (such as node failure tolerance) and actively prevents configuration drift by reconciling manual manifest edits upon merge.
• Self-Healing Infrastructure (Dev Team Agent): Showcases automated troubleshooting when a manifest is deployed with an image name typo. The agent executes a complete, autonomous 5-step remediation loop—Notification, Learning, Recommendation, Mutation, and Validation—to detect, fix, and verify the deployment without human intervention.
• Multi-Agent Governance & Policy Coordination (Cluster Operator & Dev Team Agents): Highlights cross-agent negotiation when the Cluster Operator attempts to downscale underutilized resources for cost savings. The Dev Team Agent steps in to enforce minimum capacity policies, successfully prioritizing application reliability and governance over financial savings.

Going forward, we will further productize this pattern, building on open standards to define agents and their capabilities (AGENTS.md, skills, MCP), and provide an out-of-the-box harness to orchestrate these agents.

Redefining the Stack

An intent-driven presentation layer is just the beginning. With an agentic interface in place, we can keep evolving the underlying infrastructure — adopting new components or integrating directly with additional infrastructure APIs — while engineers continue to interact with the system the way they already do. The interface stays intent-driven and stable; the agents adapt to the evolving stack underneath, so investment in how teams work today carries forward.

Call to Action

We're building Kube-Agents in the open because we believe the best infrastructure solutions are built collaboratively. Our goal is to use our expertise to give back to the open source community, while also actively learning from the ecosystem's real-world challenges. By working together, we can define best practices that benefit everyone.
If you're interested in helping shape the future of Kubernetes management, check out the Kube Agents repo.

We are seeking engagement on two fronts:

• Share your use cases: What would you most like an autonomous teammate to help with? We want to learn from your unique operational needs—whether it's multi-cluster balancing, specific debugging scenarios, or policy enforcement—to ensure we're building tools that provide real leverage.
• Define future roles: What new specialized agents should exist? We value your input on the roles these agents should fulfill to best serve diverse team structures and operational requirements.

Join the conversation, contribute your ideas, and help us build a self-driving cloud that works for everyone. Check out the Kube Agents project to open an issue or start a discussion.

The Journey Begins: Meet the 2026 GSoC Contributors!

by Stephanie Taylor, Mary Radomile & Lucila Ortíz, Google Summer of Code

A warm welcome to the 1,141 Contributors of Google Summer of Code (GSoC) 2026! We are excited to start this new edition alongside our 184 mentoring orgs. Organizations reviewed a record-breaking 23,371 proposals to find the best matches for their communities.

2026 Application Statistics:

• 15,245 applicants from 131 countries submitting a total of 23,371 proposals
• Over 2,000 mentors and org admins

What's Next?

Before the first line of code is written, there is Community Bonding. This 3.5-week GSoC tradition is about more than just tool configuration; it's about immersion. It's a dedicated space for Contributors to master the codebase, align with community standards, and understand the 'why' behind their projects. By the time the coding period begins, every Contributor is ready to turn project fundamentals into real-world impact.

The official coding period begins on May 25. For our contributors, this period represents a deep dive into collaborative development, offering the chance to learn new tools and contribute to the heartbeat of open source projects.

Thank you, Mentors!

Finally, we want to express our deepest gratitude to our phenomenal Mentors and Org Admins. As AI profoundly shifts the landscape of open source communities, GSoC is no exception. Your patience, grit, and tireless volunteer efforts are the heartbeat of this program, ensuring its continued success as we welcome a new generation of contributors into the open source ecosystem.

Introducing AMS: Activation-based model scanner for open-weight LLM safety verification

by Glen Messenger, Google Kubernetes Engine (GKE)

The open-weight model ecosystem is thriving—and so is its shadow. A 2025 study identified over 8,000 safety-modified model repositories on Hugging Face alone, with modified models complying with unsafe requests at rates of 74% compared to 19% for their original instruction-tuned counterparts.

For organizations deploying open-weight models, a critical question emerges: how do you know the model you downloaded is safe to run?

We believe defensive security tools should be widely available. AMS represents our contribution to a safer AI ecosystem—one where developers everywhere can verify model integrity before deployment.

Today we're releasing AMS (Activation-based Model Scanner), an open source tool that answers this question in 10–40 seconds—without sending a single prompt.

The Problem with Behavioral Testing

Traditional safety verification relies on behavioral testing: send harmful prompts, check if the model refuses. This approach has three fundamental limitations.

It's slow. Comprehensive benchmarks like HarmBench require hundreds of queries. For organizations running continuous integration pipelines or screening large model registries, this can be impractical.

It's incomplete. No benchmark covers every harmful behavior. Models can exhibit safe behavior on known test sets while remaining unsafe on novel or out-of-distribution prompts.

It's gameable. Models can be fine-tuned to refuse benchmark prompts while complying with novel attacks—a known limitation of purely behavioral evaluation approaches.

A Structural Approach

Clean vs Tampered Models

AMS takes a different approach entirely. Instead of testing what a model says, it measures how a model thinks.

Safety training creates measurable geometric structure in a model's activation space. Instruction-tuned models develop internal "direction vectors"—representations that separate harmful content from benign content with high statistical confidence (4–8σ separation). When safety training is removed—through fine-tuning, abliteration, or training on unfiltered data—this geometric structure collapses.

AMS measures this collapse directly. The approach is grounded in recent research on representation engineering, which demonstrates that high-level concepts are encoded linearly in LLM activation space and can be reliably extracted via simple linear probes on intermediate-layer hidden states.

git clone https://github.com/GoogleCloudPlatform/activation-model-scanner.git
cd activation-model-scanner && pip install -e .

# Standard scan (3 concepts: harmful_content, injection_resistance, refusal_capability)
ams scan ./my-model

# Quick scan (2 concepts, ~40% faster)
ams scan ./my-model --mode quick

# Full scan (4 concepts including truthfulness)
ams scan ./my-model --mode full

# JSON output for CI/CD pipelines
ams scan ./my-model --json

What AMS Detects

AMS operates as a two-tier scanner. Tier 1 measures whether safety-relevant activation structure exists at all—no baseline required. Tier 2 compares a model's activation fingerprint against a verified baseline to detect subtle modifications, including supply chain substitution.

In our validation across 14 model configurations:

• Instruction-tuned models (Llama, Gemma, Qwen) show 3.8–8.4σ separation—consistent with strong safety training
• Uncensored variants (Dolphin, Lexi) show collapsed separation at 1.1–1.3σ—flagged as CRITICAL
• Abliterated models show partial degradation at 3.3σ—flagged as WARNING
• Base models (no safety training) show 0.69σ—confirming the absence of safety structure
• Quantized models (INT4/INT8) show less than 5% separation drift—safe to scan production deployments

Use Cases

Threat Landscape

CI/CD Safety Gates

Integrate AMS into your model deployment pipeline to block unsafe models before they reach production. An example Github Actions workflow:

jobs:
model-safety-check:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3

    - name: Install AMS
      run: pip install ams-scanner[cli]

    - name: Scan model
      run: |
        ams scan ./model \
          --verify meta-llama/Llama-3-8B-Instruct \
          --json > scan-results.json

    - name: Upload results
      uses: actions/upload-artifact@v3
      with:
        name: ams-scan-results
        path: scan-results.json

Supply Chain Verification

Confirm that downloaded weights match their claimed identity using Tier 2 fingerprint comparison.

# First, create a baseline from the official model
ams baseline create ./my-model

# Then verify an unknown model against it
ams scan ./suspicious-model --verify ./my-model

Registry Screening

Automatically screen models at upload or download time to flag degraded safety structure before deployment.

# Standard scan (3 concepts: harmful_content, injection_resistance, refusal_capability)
ams scan ./my-model

# Quick scan (2 concepts, ~40% faster)
ams scan ./my-model --mode quick

# Full scan (4 concepts including truthfulness)
ams scan ./my-model --mode full

# JSON output for CI/CD pipelines
ams scan ./my-model --json

How It Works

AMS processes a set of contrastive prompt pairs—examples that differ only in whether they contain harmful content—through the model under inspection. It extracts hidden states at an intermediate layer (typically 35–40% depth), computes a direction vector that separates the two classes, and measures class separation as a σ score.

How it Works

The key insight is that this measurement requires no generation, no benchmark queries, and no ground-truth labels. The entire scan completes in a single forward pass per prompt pair, typically 10–40 seconds on GPU hardware.

The probe consists of a single direction vector (~16KB for standard 4096-dimensional models). No model weights are modified. The tool works with any Hugging Face-compatible model.

Get Started

AMS is available now under Apache 2.0:

• GitHub: github.com/GoogleCloudPlatform/activation-model-scanner/

We welcome contributions, baseline additions for new model families, and feedback from the communities. See the contributing guide in the repository for details.