opensource.google.com

Menu

Meet the Mentoring organizations of GSoC 2025!

Thursday, February 27, 2025

We are thrilled to share that we have selected 185 open source projects for the 21st year of Google Summer of Code (GSoC)

Get to know more about each organization via their individual GSoC program page. There you will find the best way to engage with each community, view project ideas, and read their contributor guidance for applying to their organization.


Applications for the GSoC Contributors are open March 24 - April 8, 2025

The 2025 GSoC program is open to students and to beginners in open source software development. If you are eager to enhance your chances of becoming a GSoC contributor this year, we highly recommend following these steps:

  • Get inspired by watching the 'Introduction to GSoC' video for a quick overview and the Community Talks to learn more about past projects that contributors completed.
  • Review the Contributor Guide and Advice for Applying to GSoC.
  • Review the list of accepted organizations.
    • We recommend finding two to three Orgs that interest you and reading through their project ideas. Use the filters on the site to help you narrow down based on the programming languages you are familiar with or categories that interest you.
  • Once you find an idea that excites you, reach out to the organization right away via their preferred communication methods. Communicating early and often will increase your chances of being accepted.
    • Introduce yourself to the mentors and community and ask questions to determine if this project idea is a good fit for your skill set and interests.
    • Use the information you received to write up your proposal.

Join us in our upcoming Info session!


Finally, you can find more information about the program on our website which includes the full 2025 timeline. You’ll also find the FAQ, Program Rules and some videos with more details about GSoC for both contributors and mentors.

Welcome aboard 2025 Mentoring Organizations! We are looking forward to an amazing year!

By Stephanie Taylor, Mary Radomile & Lucila Ortíz – GSoC Program Admins

Tag-Based Protection Made Easy

Tuesday, February 18, 2025


Scalable, Customizable, and Automated Backup Management

Managing backups across cloud environments can be challenging, particularly when dealing with large-scale infrastructure and constantly evolving workloads. With tag-based protection, organizations can automate backup assignments, ensure broad resource protection, and tailor policies to fit their needs – all through an open-source approach designed for flexibility and scalability leveraging Google Cloud Backup and DR.


Why Open Source? Flexibility and Customization

Traditional backup management often requires manual configurations, making it difficult to scale. By leveraging open-source automation, this solution allows users to:

  • Customize backup policies using VM tags that align with business needs (e.g., application, environment, or criticality).
  • Eliminate manual effort with automated backup assignments and removals.
  • Ensure bulk resource protection, dynamically adjusting backup coverage as infrastructure scales.
  • Integrate seamlessly with existing Google Cloud workflows, APIs, and automation tools.

With open-source flexibility, users can tailor backup strategies to fit their exact needs – automating, scaling, and adapting in real-time.


Scalable and Dynamic Backup Management

This approach provides:

  • Bulk inclusion/exclusion of projects and folders, simplifying administration.
  • Dynamic adjustments based on real-time tag updates.
  • Cloud Run automation to execute backups at scheduled intervals (hourly, daily, weekly, etc.).
  • Comprehensive protection reports, ensuring visibility into backup coverage.

Seamless Google Cloud Integration

To maximize efficiency, this open-source backup automation ensures:

  • Role-based access through predefined Google Cloud permissions (Tag Viewer, Backup, and DR Backup User).
  • Enhanced security by ensuring only authorized VMs are included in backup plans.

Get Started with the Open-Source Script

The backup automation script is available on GitHub, allowing users to customize and contribute to its development:

🔗 Explore the repository

By leveraging Google Cloud’s open-source backup automation, teams can effortlessly scale, automate, and customize their backup strategies – reducing operational overhead while ensuring critical resources remain protected.

By Ashika Ganesh – Product Manager, Google Cloud

Fabrication begins for production OpenTitan silicon

Thursday, February 6, 2025

With malicious software on the rise, how can you be certain that a computer, server, or mobile device is running the code (and provisioning data) that was intended? You can't just ask the code itself, so where do you start? The answer is deceptively simple – start where you have certainty and build up a chain of trust. For communication on the web, we rely on Certificate Authorities (CAs) to ensure the security of web content before it reaches the user. In products composed of an interconnected jungle of hardware and software, like Chromebooks and our Cloud infrastructure, we rely on a small dedicated secure microcontroller called a Root of Trust (RoT). And, some devices even have several RoTs for specialized needs.

Over the past six years, Google has been working with the open source community to build OpenTitan, the first open source silicon RoT. Today, we are excited to announce that we have started fabrication of the first production-ready OpenTitan silicon by Nuvoton. This silicon will be the first broadly used RoT chip at Google with a fully transparent design and origin. We have production OpenTitan chips available for lab testing and evaluation with larger volumes available from Nuvoton starting in Spring 2025.

ALT TEXT

History of RoTs and OpenTitan at Google

In 2009, Google began shipping devices with dedicated off-the-shelf RoTs. By 2014, it became clear that higher levels of assurance would only be attainable by investing in a first party RoT solution. A first party solution enabled Google to have full visibility and control over the security of its products throughout their life cycles. Previous off-the-shelf parts were black- or gray-box solutions where vendors are responsible for designing their own hardware and software – all with limited or no access to the source. Without full transparency, it is impossible to completely understand the security assurances for products using these proprietary parts. In addition, it was becoming harder to meet product needs with off-the-shelf RoT solutions, from footprint to function to cost – we needed a better solution for Chromebooks, Cloud, and later, Pixel.

Today, open source software powers nearly every consumer experience, from open source operating systems like Linux, to web browsers like Chromium. Open source is often the most economically efficient solution for developing foundational technology: it enables companies to work together and pool resources to build common, compatible products. Until now, this development approach has not been demonstrated in a commercially relevant setting for silicon.

OpenTitan is the first open-source silicon project to reach commercial availability based on the engineering samples we released last year. The OpenTitan project started from scratch in 2018 with a coalition of commercial, academic, and not-for-profit partners. The OpenTitan project is hosted by lowRISC CIC in Cambridge, UK. Google and project partners – Nuvoton, ETH Zurich, G+D Mobile Security, lowRISC, Rivos, Seagate, Western Digital, Winbond, zeroRISC, and a number of independent contributors – provide open source hardware register-transfer level (RTL) and design verification (DV) code, along with integration guidelines, and reference firmware to drive adoption throughout industry.


The Future

With the introduction of production-ready OpenTitan chips, we are excited to welcome an era where security is based on transparency from the very beginning of the stack. OpenTitan is the first commercially available open source RoT to support PQC secure boot based on SLH-DSA (formerly known as SPHINCS+). Our vision is that these chips will help drive broader industry adoption not only of open designs and their security properties, but also of this innovative method of open source collaboration between organizations.

Samples of production OpenTitan silicon are now available, with reference provisioning and application-level firmware releases coming soon. Product integrations have begun to intercept Chromebooks shipping later this year, with datacenter integrations following shortly after.


Getting Involved

With OpenTitan, we’ve introduced brand new methodologies for how commodity chips get designed that are increasingly economical moving forward. OpenTitan provides Google with a high-quality, low-cost, commoditized hardware RoT that can be used across the Google ecosystem. This will also facilitate the broader adoption of Google-endorsed security features across the industry.

The fabrication of production OpenTitan silicon is the realization of many years of dedication and hard work from our team. It is a significant moment for us and all contributors to the project. OpenTitan’s broad community has been critical to its success. As the following metrics show (baselined from the project’s public launch in 2019), the OpenTitan community is rapidly growing:

  • Almost nine times the number of commits at launch: from 2,500 to over 24,200.
  • 176 contributors to the code base
  • 17k+ merged pull requests
  • 1.5M+ LoC, including 500k LoC of HDL
  • 2.5k Github stars

If you are interested in learning more or contributing to OpenTitan, visit the open source GitHub repository or reach out to the OpenTitan team.

By Cyrus Stoller and Miguel Osorio – OpenTitan

The New Frontier of Security: Creating Safe and Secure AI Models

Wednesday, January 29, 2025


Are you looking to safely create the next state-of-the-art AI model? Today we’re sharing a list of recommendations on how to create and distribute your models securely.


Choosing the Right Foundation: Safe Model Formats

Before you start building your model, consider using a safe file format, as it can influence your development tool options. However, if you've already created a model, you can also convert it to a safe format before sharing it.

Once trained, models are saved and distributed as binary files. Common formats include PyTorch (Pickles usually with .pt, .pth extensions), TensorFlow SavedModel (.pb), GGUF (.gguf), and Safetensors (.safetensors). However, binary files are dangerous because it's hard to verify if their content is safe. This is especially true with formats such as Pickles and SavedModels, which are designed to include arbitrary code, raising the risk of remote code execution (RCE) on users' machines.

To mitigate these risks:

  • If sharing only model weights: Consider formats such as Safetensors. These formats only contain model weights, and are therefore safe from RCE.
  • If sharing weights and metadata: Consider formats like GGUF, which include weights and additional metadata but not executable code configurations.
  • For any format, but especially if your model requires custom code: Keep reading to see how to help users verify that they're getting the correct model.

Secure Releases and Verification Methods

To ensure your users are getting the model you originally deployed, consider automating your releases, making them transparent and auditable. Instead of training your model on your local machine, consider using a predefined script to train your model within an isolated environment. When building smaller models, using GitHub Actions can be a good option. However, for larger models, GitHub Actions might not have the necessary hardware capabilities or availability. In that case, and if budget allows, consider using other platforms with proper security safeguards, such as Google Cloud Platform (GCP).

If building your model on a cloud platform is not an option, you can sign the model on your local machine to give your users confidence it was created by you.

However, if building your model on a cloud platform, sign the model and generate a provenance attestation for the release. This allows users to not only confirm that the model was created by you, but that it came from your approved infrastructure, was trained following the specific instructions defined in your training script, and wasn't tampered with by a malicious actor.

While signatures and provenance do not guarantee the absence of malicious intent from the developer, they provide users with a means to verify the integrity of the model they downloaded.

On GitHub, signing and provenance can be easily achieved using GitHub Artifact Attestations. For the general use case, tools like Sigstore and SLSA are also available to sign and attest provenance to your deployments.

For a few examples, check out this workflow to build a model with SLSA on GitHub and on GCP, and an example of how to sign models.


Educate Your Users

After sharing your model with the world, it is essential to educate users on the safety and security concerns surrounding model consumption. You should therefore:

  • Document potential biases in your models and datasets.
  • Clearly display all licenses associated with the model and datasets.
  • Benchmark your model, assessing and disclosing metrics around hallucinations, prompt injection risk and fairness.

With this information users can make informed decisions and abide by the ethical and technical guidelines associated with your model. For instance, they might choose to implement an input sanitization layer to enhance their software security.


Establish a Security Policy

Your model’s privacy, safety, and security guarantees can be documented in separate files or in a security policy. A security policy helps you address users' safety and security concerns regarding your models. It is a dedicated file that instructs users on how to privately report vulnerabilities, such as prompt injection strategies or potential out-of-memory (OOM) errors, allowing you to investigate and address the potential vulnerabilities before they become public knowledge. It is also a good place to define the scope of what your project considers a vulnerability.

In summary, considering model security from the outset of development is crucial. Additionally, ensuring safe distribution and informing users of potential risks is essential. It's important to remember that security is a continuous process – more like a marathon than a sprint – and constant vigilance is necessary to mitigate potential threats.


Keep Improving

The steps above will put your models’ security on a solid footing, but there is always more you can learn and do. Please take a look at Google's Secure AI Framework for a deeper dive in this subject, and take its risk self assessment to better understand which risks are most important for you.

By Gabriela Gutierrez and Pedro Nacht – GOSST Upstream Team

Producer java library for Data Lineage is now open source

Tuesday, January 28, 2025

Integrating OpenLineage producers with GCP Lineage just got a lot easier


What is Data Lineage

Data Lineage is a GCP feature that allows tracking data movement. This tool helps data owners and analysts detect anomalies in data flows, find connections between data sources and verify the potential consequences of planned changes in data pipelines.

Lineage is injected automatically for some Google Cloud products (BigQuery, Cloud Data Fusion, Cloud Composer, Dataproc, Vertex AI). That means, if Lineage integration with any of those products is enabled in the projects, data movements coming from executing jobs by these products will be reported to GCP Lineage.

For custom integrations, the API can be used to report and fetch lineage.

After injecting, lineage can be viewed in the Google Cloud console (available from DataCatalog UI, BigQuery UI, Vertex UI). There are two representations: graph view, with data sources as nodes and data movements as edges, and list view, a tabular representation. Lineage information can also be fetched from the API.

More information is available in the documentation.


GCP Lineage information model

We describe data flows using the following concepts:

  • Process is a definition of some data transformation. For example, a SQL or Spark script.
  • Run is an execution of a Process.
  • Lineage Event is a data transformation event. It is reported in context of a Run.
  • A Link represents a connection between two data sources, when data in the link’s Target depends on its Source. A Lineage Event contains a list of Links.

OpenLineage support

OpenLineage is an open standard for reporting lineage information. It unifies lineage reporting between systems, which means the events generated in this format can be consumed by any product supporting it. This leads to more flexibility: adding or replacing a lineage producer does not imply changing the consumer, and vice versa.

OpenLineage format is adopted by a number of lineage producers and consumers, meaning there is already tooling available to report lineage from/to those systems. GCP Lineage is one of those consumers: users can report events in OpenLineage format, see the resulting lineage on the UI, and query it via the API.

OpenLineage is the preferred method for reporting lineage in GCP Lineage. It is used by the Dataproc lineage integration. To find out more about sending OpenLineage events to GCP Lineage refer to the documentation.

After injecting lineage in OpenLineage format, it can be accessed in the same way as if it was injected via other API methods or automatically: from the Google Cloud console or the API.


Why producer library

The GCP Lineage producer library is an extension of the client library. Client libraries are recommended for calling Cloud APIs programmatically. They handle low level API call details, leaving the necessary user code simpler and shorter.

The producer library further simplifies integration by providing ready to use code needed to call the API from Java. It adds additional functionality such as synchronous and asynchronous clients, translating OpenLineage JSON messages to the API friendly format, error handling etc.

Using the producer library, all the code needed to send a request to GCP Lineage API is:

SyncLineageProducerClient client = SyncLineageProducerClient.create();
ProcessOpenLineageRunEventRequest request =
        ProcessOpenLineageRunEventRequest.newBuilder()
            .setParent(parent)
            .setOpenLineage(openLineageMessage)
            .build();
client.processOpenLineageRunEvent(request);

The field openLineageMessage here is a protobuf Struct that includes information about job execution, inputs and outputs and other metadata. The object model is described in the documentation. An example message is:

{
  "eventType": "START",
  "eventTime": "2023-04-04T13:21:16.098Z",
  "run": {
    "runId": "502483d6-3e3d-474f-9380-da565eaa7516",
    "facets": {
       "spark_properties": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.22.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet",
        "properties": {
          "spark.master": "yarn",
          "spark.app.name": "sparkJobTest.py"
        }
      }
    }
  },
  "job": {
    "namespace": "project-name",
    "name": "cluster-name",
    "facets": {
    "jobType": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.22.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json#/$defs/JobTypeJobFacet",
        "processingType": "BATCH",
        "integration": "SPARK",
        "jobType": "SQL_JOB"
      },

    }
  },
  "inputs": [
    {
      "namespace": "bigquery",
      "name": "project.dataset.input_table",
    }],
  "outputs": [
   {
      "namespace": "bigquery",
      "name": "project.dataset.output_table",
    }],
  "producer": "https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark",
  "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunEvent"
}

Learn more about building an OpenLineage message.


Best Practices for Constructing OpenLineage Messages

The openLineageMessage should follow the OpenLineage format. The fields that are required for correct parsing by the GCP Lineage API are:

job

mapped to Process

job.namespace

used to construct Process name

job.name

used to construct Process name

run

mapped to Run

run.runId

used to construct Run name

producer

URI identifying the producer of this metadata

eventTime

time of the data movement

schemaURL

URL pointing to the schema definition for this message

In addition to those, the fields used to create lineage are:

eventType

corresponds to the status of the Run

inputs

mapped to sources of links. Must be specified according to the naming conventions

outputs

mapped to targets of links. Must be specified according to the naming conventions

The GCP Lineage API supports OpenLineage major versions 1 and 2. For more information please refer to the documentation.


How to access GCP Lineage?

The code is now publicly available on GitHub. The library is also published to Maven.


GcpLineageTransport

To simplify integration with GCP Lineage, we offer GcpLineageTransport. It is available on the OpenLineage GitHub repository and is built to a separate maven artifact. It is built on top of the producer library mentioned above.

Using the transport minimises the code for sending events to GCP Lineage. The GcpLineageTransport can be configured as the event sink for any existing OpenLineage producer such as Airflow, Spark, and Flink. Find more information and examples on GCP Lineage.

By Mary Idamkina – Data Lineage

.