Introducing Abseil, a new common libraries project

Today we are open sourcing Abseil, a collection of libraries drawn from the most fundamental pieces of Google’s internal codebase. These libraries are the nuts-and-bolts that underpin almost everything that Google runs. Bits and pieces of these APIs are embedded in most of our open source projects, and now we have brought them together into one comprehensive project. Abseil encompasses the most basic building blocks of Google’s codebase: code that is production tested and will be fully maintained for years to come.

Our C++ code repository is available at: https://github.com/abseil/abseil-cpp

By adopting these new Apache-licensed libraries, you can reap the benefit of years (over a decade in many cases) of our design and optimization work in this space. Our past experience is baked in.

Just as interesting, we’ve also prepared for the future: several types in Abseil’s C++ libraries are “pre-adoption” versions of C++17 types like string_view and optional - implemented in C++11 to the greatest extent possible. We look forward to moving more and more of our code to match the current standard, and using these new vocabulary types helps us make that transition. Importantly, in C++17 mode these types are merely aliases to the standard, ensuring that you only ever have one type for optional or string_view in a project at a time. Put another way: Abseil is focused on the engineering task of providing APIs that remain stable over time.

Consisting of the foundational C++ and Python code at Google, Abseil includes libraries that will grow to underpin other Google-backed open source projects like gRPC, Protobuf and TensorFlow. We love those projects, and we love the users of those projects - we want to ensure smooth usage for these things over time. In the next few months we’ll introduce new distribution methods to incorporate these projects as a collection into your project.

Continuing with the “over time” theme, Abseil aims for compatibility with major compilers, platforms and standard libraries for approximately 5 years. Our 5-year target also applies to language version: we assume everyone builds with C++11 at this point. (In 2019 we’ll start talking about requiring C++14 as our base language version.) This 5-year horizon is part of our balance between “support everything” and “provide modern implementations and APIs.”

Highlights of the initial release include:

• Zero configuration: most platforms (OS, compiler, architecture) should just work.
• Pre-adoption for C++17 types: string_view, optional, any. We’ll follow up with variant soon.
• Our primary synchronization type, absl::Mutex, has an elegant interface and has been extensively optimized.
• Efficient support for handling time: absl::Time and absl::Duration are conceptually similar to std::chrono types, but are concrete (not class templates) and have defined behavior in all cases. Additionally, our clock-sampling API absl::Now() is more heavily optimized than most standard library calls for std::chrono::system_clock::now().
• String handling routines: among internal users, we’ve been told that releasing absl::StrCat(), absl::StrJoin(), and absl::StrSplit() would itself be a big improvement for the open source C++ world.

The project has support for C++ and some Python. Over time we’ll tie those two projects together more closely with shared logging and command-line flag infrastructure. To start contributing, please see our contribution guidelines and fork us on GitHub. Check out our documentation and community page for information on how to contact us, ask questions or contribute to Abseil.

By Titus Winters, Abseil Lead

Google Summer of Code turns 14

Google Open Source is proud to announce the 14th year of Google Summer of Code (GSoC)! Yes, GSoC is officially well into its teenage years - hopefully without that painful awkward stage - and we are excited to introduce more new student developers to the world of open source software development.

Over the last 13 years GSoC has provided over 13,000 university students from around the world with an opportunity to hone their skills by contributing to open source projects during their summer break. Participants gain invaluable experience working directly with mentors on open source projects, and earn a stipend upon successful completion of their project.

We’re excited to keep the tradition going! Applications for interested open source organizations open on January 4, 2018 and student applications open in March*.

Are you an open source project interesting in learning more? Visit the program site to learn about what it means to be a mentor organization and how to submit a good application. We welcome all types of organizations - both large and small - and each year about 20% of the organizations we accept are completely new to GSoC.

Students, it’s never too early to start thinking about your proposal. You can check out the organizations that participated in Google Summer of Code 2017 as well as the projects students worked on. We also encourage you to explore other resources like the student and mentor guides and frequently asked questions.

You can always learn more on the program website. Please stay tuned for more details!

By Mary Radomile, Google Open Source

* Exact dates will be announced later this year.

Authenticating to Hashicorp Vault using GCE Signed Metadata

Applications often require access to small pieces of sensitive data at build or run time, referred to as secrets. Secrets are generally more sensitive than other environment variables or parts of your repository as they may grant access to additional data, such as user information.

HashiCorp Vault is a popular open source tool for secret management, which allows a developer to store, manage and control access to tokens, passwords, certificates, API keys and other secrets. Vault has authentication backends, which allow developers to use many kinds of identities to access Vault, including tokens, or usernames and passwords.

Today, we’re announcing a new Google Compute Engine signed metadata authentication backend for Vault. This allows a developer to use an existing running instance to authenticate directly to Vault, using a JWT of this instance’s signed metadata as its identity. Learn more about the latest release of Vault.

The following example shows how a user can enable and configure the GCP auth backend and then authenticate an instance with Vault. See the docs to learn more.

In the following example, an admin mounts the backend at auth/gcp and adds:

• Credentials that the backend can use to verify the instance (requiring some read-only permissions)
• A Vault-specific role (roles determine what Vault secrets the authenticating GCE VM instance can access, and can restrict login to a set of instances by zone or region, instance group, or GCP labels)

# Mount the backend at ‘auth/gcp’.
$ vault auth-enable ‘gcp’
...

# Add configuration. This can include credentials Vault will 
# use to make calls to GCP APIs to verify authenticating instances. 
$ vault write auth/gcp/config credentials=@path/to/creds.json
...


# Add configuration. This can include credentials Vault will 
# use to make calls to GCP APIs to verify authenticating instances. 
$ vault write auth/gcp/role/my-gce-role \
type='gce' \
policies=’dev,prod’ \
project_id=”project-123456” \
zone="us-central1-c"
instance_group=”my-instance-group” \
labels=”foo:bar,prod:true,...” \
...

Then, a Compute Engine instance can authenticate to the Vault server using the following script:

#! /bin/bash

# [START PARAMS]
# Subtitute real vault address or env variable.
VAULT_ADDR="https://vault.rocks"

# Substitute another service account for the VM instance or use the
# built-in default.
SERVICE_ACCOUNT="default"

# The instance will attempt login under this role.
ROLE="my-gce-role"
# [END PARAMS]


# Generate a metadata identity token JWT with the expected audience.
# The GCP backend expect a JWT 'aud' claim ending in “vault/$ROLE”.
AUD="$VAULT_ADDR/vault/$ROLE"
TOKEN="$(curl -H "Metadata-Flavor: Google" \
http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=$AUD&format=full)"


# Attempt login against VAULT. 
# Using API calls:

cat > payload.json <<-END 
{ 
  "role": "$ROLE", 
  "jwt": "$TOKEN" 
}
END

$ RESP="$(curl \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/auth/gcp/login)"

# OR

# Use CL vault tool if downloaded
vault write auth/gcp/login role=”$ROLE" jwt="$TOKEN" > response

# Get auth token from JSON
# response (response) and get
# your secrets from Vault!
# ...

A few weeks ago, we also announced a Google Cloud Platform IAM authentication backend for HashiCorp Vault, allowing authentication using an existing IAM service account identity. For further guidance on using HashiCorp Vault with Google Cloud, take a look at today’s Google Cloud blog post.

By Emily Ye, Software Engineer