Posts from 2017

TensorFlow Lattice: Flexibility Empowered by Prior Knowledge

Wednesday, October 11, 2017

Crossposted on the Google Research Blog

Machine learning has made huge advances in many applications including natural language processing, computer vision and recommendation systems by capturing complex input/output relationships using highly flexible models. However, a remaining challenge is problems with semantically meaningful inputs that obey known global relationships, like “the estimated time to drive a road goes up if traffic is heavier, and all else is the same.” Flexible models like DNNs and random forests may not learn these relationships, and then may fail to generalize well to examples drawn from a different sampling distribution than the examples the model was trained on.

Today we present TensorFlow Lattice, a set of prebuilt TensorFlow Estimators that are easy to use, and TensorFlow operators to build your own lattice models. Lattices are multi-dimensional interpolated look-up tables (for more details, see [1--5]), similar to the look-up tables in the back of a geometry textbook that approximate a sine function.  We take advantage of the look-up table’s structure, which can be keyed by multiple inputs to approximate an arbitrarily flexible relationship, to satisfy monotonic relationships that you specify in order to generalize better. That is, the look-up table values are trained to minimize the loss on the training examples, but in addition, adjacent values in the look-up table are constrained to increase along given directions of the input space, which makes the model outputs increase in those directions. Importantly, because they interpolate between the look-up table values, the lattice models are smooth and the predictions are bounded, which helps to avoid spurious large or small predictions in the testing time.

How Lattice Models Help You

Suppose you are designing a system to recommend nearby coffee shops to a user. You would like the model to learn, “if two cafes are the same, prefer the closer one.”  Below we show a flexible model (pink) that accurately fits some training data for users in Tokyo (purple), where there are many coffee shops nearby.  The pink flexible model overfits the noisy training examples, and misses the overall trend that a closer cafe is better. If you used this pink model to rank test examples from Texas (blue), where businesses are spread farther out, you would find it acted strangely, sometimes preferring farther cafes!
Slice through a model’s feature space where all the other inputs stay the same and only distance changes. A flexible function (pink) that is accurate on training examples from Tokyo (purple) predicts that a cafe 10km-away is better than the same cafe if it was 5km-away. This problem becomes more evident at test-time if the data distribution has shifted, as shown here with blue examples from Texas where cafes are spread out more.
A monotonic flexible function (green) is both accurate on training examples and can generalize for Texas examples compared to non-monotonic flexible function (pink) from the previous figure.

In contrast, a lattice model, trained over the same example from Tokyo, can be constrained to satisfy such a monotonic relationship and result in a monotonic flexible function (green). The green line also accurately fits the Tokyo training examples, but also generalizes well to Texas, never preferring farther cafes.

In general, you might have many inputs about each cafe, e.g., coffee quality, price, etc. Flexible models have a hard time capturing global relationships of the form, “if all other inputs are equal, nearer is better, ” especially in parts of the feature space where your training data is sparse and noisy. Machine learning models that capture prior knowledge (e.g.  how inputs should impact the prediction) work better in practice, and are easier to debug and more interpretable.

Pre-built Estimators

We provide a range of lattice model architectures as TensorFlow Estimators. The simplest estimator we provide is the calibrated linear model, which learns the best 1-d transformation of each feature (using 1-d lattices), and then combines all the calibrated features linearly. This works well if the training dataset is very small, or there are no complex nonlinear input interactions. Another estimator is a calibrated lattice model. This model combines the calibrated features nonlinearly using a two-layer single lattice model, which can represent complex nonlinear interactions in your dataset. The calibrated lattice model is usually a good choice if you have 2-10 features, but for 10 or more features, we expect you will get the best results with an ensemble of calibrated lattices, which you can train using the pre-built ensemble architectures. Monotonic lattice ensembles can achieve 0.3% -- 0.5% accuracy gain compared to Random Forests [4], and these new TensorFlow lattice estimators can achieve 0.1 -- 0.4% accuracy gain compared to the prior state-of-the-art in learning models with monotonicity [5].

Build Your Own

You may want to experiment with deeper lattice networks or research using partial monotonic functions as part of a deep neural network or other TensorFlow architecture. We provide the building blocks: TensorFlow operators for calibrators, lattice interpolation, and monotonicity projections. For example, the figure below shows a 9-layer deep lattice network [5].

Example of a 9-layer deep lattice network architecture [5], alternating layers of linear embeddings and ensembles of lattices with calibrators layers (which act like a sum of ReLU’s in Neural Networks). The blue lines correspond to monotonic inputs, which is preserved layer-by-layer, and hence for the entire model. This and other arbitrary architectures can be constructed with TensorFlow Lattice because each layer is differentiable.

In addition to the choice of model flexibility and standard L1 and L2 regularization, we offer new regularizers with TensorFlow Lattice:
  • Monotonicity constraints [3] on your choice of inputs as described above.
  • Laplacian regularization [3] on the lattices to make the learned function flatter.
  • Torsion regularization [3] to suppress un-necessary nonlinear feature interactions.
We hope TensorFlow Lattice will be useful to the larger community working with meaningful semantic inputs. This is part of a larger research effort on interpretability and controlling machine learning models to satisfy policy goals, and enable practitioners to take advantage of their prior knowledge. We’re excited to share this with all of you. To get started, please check out our GitHub repository and our tutorials, and let us know what you think!

By Maya Gupta, Research Scientist, Jan Pfeifer, Software Engineer and Seungil You, Software Engineer


Developing and open sourcing TensorFlow Lattice was a huge team effort. We’d like to thank all the people involved: Andrew Cotter, Kevin Canini, David Ding, Mahdi Milani Fard, Yifei Feng, Josh Gordon, Kiril Gorovoy, Clemens Mewald, Taman Narayan, Alexandre Passos, Christine Robson, Serena Wang, Martin Wicke, Jarek Wilkiewicz, Sen Zhao, Tao Zhu


[1] Lattice Regression, Eric Garcia, Maya Gupta, Advances in Neural Information Processing Systems (NIPS), 2009
[2] Optimized Regression for Efficient Function Evaluation, Eric Garcia, Raman Arora, Maya R. Gupta, IEEE Transactions on Image Processing, 2012
[3] Monotonic Calibrated Interpolated Look-Up Tables, Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, Alexander van Esbroeck, Journal of Machine Learning Research (JMLR), 2016
[4] Fast and Flexible Monotonic Functions with Ensembles of Lattices, Mahdi Milani Fard, Kevin Canini, Andrew Cotter, Jan Pfeifer, Maya Gupta, Advances in Neural Information Processing Systems (NIPS), 2016
[5] Deep Lattice Networks and Partial Monotonic Functions, Seungil You, David Ding, Kevin Canini, Jan Pfeifer, Maya R. Gupta, Advances in Neural Information Processing Systems (NIPS), 2017

Google Code-in 2017 is seeking organization applications

Monday, October 9, 2017

We are now accepting applications for open source organizations who want to participate in Google Code-in 2017. Google Code-in, a global online contest for pre-university students ages 13-17, invites students to learn by contributing to open source software.

Working with young students is a special responsibility and each year we hear inspiring stories from mentors who participate. To ensure these new, young contributors have a great support system, we select organizations that have gained experience in mentoring students by previously taking part in Google Summer of Code.

Organizations must apply before Tuesday, October 24 at 16:00 UTC.

17 organizations were accepted last year, and over the last 7 years, 4,553 students from 99 different countries have completed more than 23,651 tasks for participating open source projects. Tasks fall into 5 categories:

  • Code: writing or refactoring 
  • Documentation/Training: creating/editing documents and helping others learn more
  • Outreach/Research: community management, outreach/marketing, or studying problems and recommending solutions
  • Quality Assurance: testing and ensuring code is of high quality
  • User Interface: user experience research or user interface design and interaction

Once an organization is selected for Google Code-in 2017 they will define these tasks and recruit mentors who are interested in providing online support for students.

You can find a timeline, FAQ and other information about Google Code-in on our website. If you’re an educator interested in sharing Google Code-in with your students, you can find resources here.

By Josh Simmons, Google Open Source

Announcing more Open Source Peer Bonus winners

Tuesday, October 3, 2017

We’re excited to announce 2017’s second round of Open Source Peer Bonus winners. Google Open Source established this program six years ago to encourage Googlers to recognize and celebrate external contributors to the open source ecosystem Google depends on.

The Open Source Peer Bonus program works like this: Googlers nominate open source contributors outside of the company who deserve recognition for their contributions to open source projects, including those used by Google. Nominees are reviewed by a volunteer team of engineers and the winners receive our heartfelt thanks and a small token of our appreciation.

To date, we’ve recognized nearly 600 open source contributors from dozens of countries who have contributed their time and talent to more than 400 open source projects. You can find past winners in recent blog posts: Fall 2016, Spring 2017.

Without further ado, we’d like to recognize the latest round of winners and the projects they worked on. Here are the individuals who gave us permission to thank them publicly:

Name Project Name Project
Mo JangdaAMP ProjectEric TangMaterial Motion
Osvaldo LopezAMP ProjectNicholas Tollerveymicro:bit, Mu
Jason JeanAngular CLIDamien GeorgeMicroPython
Henry ZhuBabelTom SpilmanMonoGame
Oscar BoykinBazel Scala rulesArthur EdgeNARKOZ/gitlab
Francesc AltedBloscSebastian BergNumPy
Matt HoltCaddyBogdan-Andrei IancuOpenSIPS
Martijn CroonenChromiumAmit AmbastaOR-tools
Raphael CostaChromiumMichael PowellOR-tools
Mariatta WijayaCPythonWestbrook JohnsonPolymer
Victor StinnerCPythonMarten Seemannquic-go
Derek ParkerDelveFabian HennekeSecure Shell
Thibaut CouroubledevdocsChris FillmoreShaka Player
David Lechnerev3devTakeshi KomiyaSphinx
Michael NiedermayerFFmpegDan KennedySQLite
Mathew HuuskoFirebaseJoe MistachkinSQLite
Armin RonacherFlaskRichard HippSQLite
Nenad StojanovskiForseti SecurityYuxin WuTensorpack
Solly RossHeapsterMichael Herzogthree.js
Bjørn Erik PedersenHugoTakahiro Aoyagithree.js
Brion VibberJS-InterpreterJelle ZijlstraTypeshed
Xiaoyu ZhangKubernetesVerónica LópezWomen Who Go
Anton KachurinMaterial Components for the Web

Thank you all so much for your contributions to the open source community and congratulations on being selected!

By Maria Webb, Google Open Source

Talk to Google at Node.js Interactive

Monday, October 2, 2017

We’re headed to Vancouver this week, with about 25 Googlers who are incredibly excited to attend Node.js Interactive. With a mix of folks working on Cloud, Chrome, and V8, we’re going to be giving demos and answering questions at the Google booth. 

A few of us are also going to be giving talks. Here’s a list of the talks Googlers will be giving at the conference, ranging from serverless Slack bots to JavaScript performance tuning.

Wednesday, October 4th

Keynote: Franzi Hinkelmann
9:40am - 10:00am, West Ballroom A
Franzi Hinkelmann, Software Engineer @ Google

Franzi is located in Munich, Germany where she works at Google on Chrome V8. Franzi, like James and Anna, is a member of the Node.js Core Technical Committee. She speaks across the globe on the topic of JavaScript virtual machines. She has a PhD in mathematics, but left academia to follow her true passion: writing code.

Franzi will discuss her perspective on Chrome V8 in Node.js, and what the Chrome V8 team is doing to continue to support Node.js. Want to know what the future of browser development looks like? This is a must-attend keynote.

Functionality Abuse: The Forgotten Class of Attacks
12:20pm - 12:50pm, West Ballroom A
Nwokedi Idika, Software Engineer @ Google

If you were given a magic wand that would remove all implementation flaws from your web application, would it be free of security problems? If it took you more five seconds to say “No!” (or if, worse, you said “Yes!”), then you’re the target audience for this talk. If you’re in the target audience, don’t fret, much of the security community is there with you. After this talk, attendees will understand why the answer to the aforementioned question is an emphatic “No!” and they will learn an approach to decrease their chance of failing to consider an important vector of attack for their current and future web applications.

High Performance JS in V8
5:20pm - 5:50pm, West Ballroom A
Peter Marshall, Software Engineer @ Google

This year, V8 launched Ignition and Turbofan, the new compiler pipeline that handles all JavaScript code generation. Previously, achieving high performance in Node.js meant catering to the oddities of our now-deprecated Crankshaft compiler. This talk covers our new code generation architecture - what makes it special, a bit about how it works, and how to write high performance code for the new V8 pipeline.

Thursday, October 5th

New DevTools Features for JavaScript
11:40am - 12:10pm, West Meeting Room 122
Yang Guo, Software Engineer @ Google

Ever since v8-inspector was moved to V8's repository, we have been working on a number of new features for DevTools, usable for both Chrome and Node.js. The talk will demonstrate code coverage, type profiling, and give a deep dive into how evaluating a code snippet in DevTools console works in V8.

Understanding and Debugging Memory Leaks in Your Node.js Applications
12:20pm - 12:50pm, West Meeting Room 122
Ali Sheikh, Software Engineer @ Google

Memory leaks are hard. This talk with introduce developers to what memory leaks are, how they can exist in a garbage collected language, the available tooling that can help them understand and isolate memory leaks in their code. Specifically it will talk about heap snapshots, the new sampling heap profiler in V8, and other various other tools available in the ecosystem.

Workshop: Serverless Bots with Node.js
2:20pm - 4:10pm, West Meeting Room 117
Bret McGowen, Developer Advocate @ Google
Amir Shevat, DevRel @ Slack

This talk will show you how to build both voice and chat bots using serverless technologies. Amir Shevat, Head of Developer Relations at Slack, has overseen 17K+ bots deployed on the platform. He will present a maturing model, as best practices, for enterprise bots covering all sorts of use cases ranging for devops, HR, and marketing. Alan Ho from Google Cloud will then show you how to use various serverless technologies to build these bots. He’ll give you a demo of Slack and Google Assistant bots incorporating Google’s latest serverless technology including Edge (API Management), CloudFunctions (Serverless Compute), Cloud Datastore, and

Modules Modules Modules
3:00pm - 3:30pm, West Meeting Room 120
Myles Borins, Developer Advocate @ Google

ES Modules and Common JS go together like old bay seasoning and vanilla ice cream. This talk will dig into the inconsistencies of the two patterns, and how the Node.js project is dealing with reconciling the problem. The talk will look at the history of modules in the JavaScript ecosystem and the subtle difference between them. It will also skim over how ECMA-262 is standardized by the TC39, and how ES Modules were developed.

Keynote: The case for Node.js
4:50pm - 5:05pm, West Ballroom A
Justin Beckwith, Product Manager @ Google

Node.js has had a transformational effect on the way we build software. However, convincing your organization to take a bet on Node.js can be difficult. My personal journey with Node.js has included convincing a few teams to take a bet on this technology, and this community. Let’s take a look at the case for Node.js we made at Google, and how you can make the case to bring it to your organization.

We can’t wait to see everyone and have some great conversations. Feel free to reach out to us on Twitter @googlecloud, or request an invite to the Google Cloud Slack community and join the #nodejs channel.

By Justin Beckwith, Languages and Runtimes Team

Introducing Abseil, a new common libraries project

Tuesday, September 26, 2017

Today we are open sourcing Abseil, a collection of libraries drawn from the most fundamental pieces of Google’s internal codebase. These libraries are the nuts-and-bolts that underpin almost everything that Google runs. Bits and pieces of these APIs are embedded in most of our open source projects, and now we have brought them together into one comprehensive project. Abseil encompasses the most basic building blocks of Google’s codebase: code that is production tested and will be fully maintained for years to come.

Our C++ code repository is available at:

By adopting these new Apache-licensed libraries, you can reap the benefit of years (over a decade in many cases) of our design and optimization work in this space. Our past experience is baked in.

Just as interesting, we’ve also prepared for the future: several types in Abseil’s C++ libraries are “pre-adoption” versions of C++17 types like string_view and optional - implemented in C++11 to the greatest extent possible. We look forward to moving more and more of our code to match the current standard, and using these new vocabulary types helps us make that transition. Importantly, in C++17 mode these types are merely aliases to the standard, ensuring that you only ever have one type for optional or string_view in a project at a time. Put another way: Abseil is focused on the engineering task of providing APIs that remain stable over time.

Consisting of the foundational C++ and Python code at Google, Abseil includes libraries that will grow to underpin other Google-backed open source projects like gRPC, Protobuf and TensorFlow. We love those projects, and we love the users of those projects - we want to ensure smooth usage for these things over time. In the next few months we’ll introduce new distribution methods to incorporate these projects as a collection into your project.

Continuing with the “over time” theme, Abseil aims for compatibility with major compilers, platforms and standard libraries for approximately 5 years. Our 5-year target also applies to language version: we assume everyone builds with C++11 at this point. (In 2019 we’ll start talking about requiring C++14 as our base language version.) This 5-year horizon is part of our balance between “support everything” and “provide modern implementations and APIs.”

Highlights of the initial release include:
  • Zero configuration: most platforms (OS, compiler, architecture) should just work.
  • Pre-adoption for C++17 types: string_view, optional, any. We’ll follow up with variant soon.
  • Our primary synchronization type, absl::Mutex, has an elegant interface and has been extensively optimized.
  • Efficient support for handling time: absl::Time and absl::Duration are conceptually similar to std::chrono types, but are concrete (not class templates) and have defined behavior in all cases. Additionally, our clock-sampling API absl::Now() is more heavily optimized than most standard library calls for std::chrono::system_clock::now().
  • String handling routines: among internal users, we’ve been told that releasing absl::StrCat(), absl::StrJoin(), and absl::StrSplit() would itself be a big improvement for the open source C++ world.
The project has support for C++ and some Python. Over time we’ll tie those two projects together more closely with shared logging and command-line flag infrastructure. To start contributing, please see our contribution guidelines and fork us on GitHub. Check out our documentation and community page for information on how to contact us, ask questions or contribute to Abseil.

By Titus Winters, Abseil Lead

Google Summer of Code turns 14

Monday, September 25, 2017

Google Open Source is proud to announce the 14th year of Google Summer of Code (GSoC)! Yes, GSoC is officially well into its teenage years - hopefully without that painful awkward stage - and we are excited to introduce more new student developers to the world of open source software development.

Over the last 13 years GSoC has provided over 13,000 university students from around the world with an opportunity to hone their skills by contributing to open source projects during their summer break. Participants gain invaluable experience working directly with mentors on open source projects, and earn a stipend upon successful completion of their project.

We’re excited to keep the tradition going! Applications for interested open source organizations open on January 4, 2018 and student applications open in March*.

Are you an open source project interesting in learning more? Visit the program site to learn about what it means to be a mentor organization and how to submit a good application. We welcome all types of organizations - both large and small - and each year about 20% of the organizations we accept are completely new to GSoC.

Students, it’s never too early to start thinking about your proposal. You can check out the organizations that participated in Google Summer of Code 2017 as well as the projects students worked on. We also encourage you to explore other resources like the student and mentor guides and frequently asked questions.

You can always learn more on the program website. Please stay tuned for more details!

By Mary Radomile, Google Open Source

* Exact dates will be announced later this year.

Authenticating to Hashicorp Vault using GCE Signed Metadata

Tuesday, September 19, 2017

Applications often require access to small pieces of sensitive data at build or run time, referred to as secrets. Secrets are generally more sensitive than other environment variables or parts of your repository as they may grant access to additional data, such as user information.

HashiCorp Vault is a popular open source tool for secret management, which allows a developer to store, manage and control access to tokens, passwords, certificates, API keys and other secrets. Vault has authentication backends, which allow developers to use many kinds of identities to access Vault, including tokens, or usernames and passwords.

Today, we’re announcing a new Google Compute Engine signed metadata authentication backend for Vault. This allows a developer to use an existing running instance to authenticate directly to Vault, using a JWT of this instance’s signed metadata as its identity. Learn more about the latest release of Vault.

The following example shows how a user can enable and configure the GCP auth backend and then authenticate an instance with Vault. See the docs to learn more.

In the following example, an admin mounts the backend at auth/gcp and adds:
  • Credentials that the backend can use to verify the instance (requiring some read-only permissions)
  • A Vault-specific role (roles determine what Vault secrets the authenticating GCE VM instance can access, and can restrict login to a set of instances by zone or region, instance group, or GCP labels)
# Mount the backend at ‘auth/gcp’.
$ vault auth-enable ‘gcp’

# Add configuration. This can include credentials Vault will 
# use to make calls to GCP APIs to verify authenticating instances. 
$ vault write auth/gcp/config credentials=@path/to/creds.json

# Add configuration. This can include credentials Vault will 
# use to make calls to GCP APIs to verify authenticating instances. 
$ vault write auth/gcp/role/my-gce-role \
type='gce' \
policies=’dev,prod’ \
project_id=”project-123456” \
instance_group=”my-instance-group” \
labels=”foo:bar,prod:true,...” \

Then, a Compute Engine instance can authenticate to the Vault server using the following script:

#! /bin/bash

# Subtitute real vault address or env variable.

# Substitute another service account for the VM instance or use the
# built-in default.

# The instance will attempt login under this role.

# Generate a metadata identity token JWT with the expected audience.
# The GCP backend expect a JWT 'aud' claim ending in “vault/$ROLE”.
TOKEN="$(curl -H "Metadata-Flavor: Google" \

# Attempt login against VAULT. 
# Using API calls:

cat > payload.json <<-END 
  "role": "$ROLE", 
  "jwt": "$TOKEN" 

$ RESP="$(curl \
    --request POST \
    --data @payload.json \

# OR

# Use CL vault tool if downloaded
vault write auth/gcp/login role=”$ROLE" jwt="$TOKEN" > response

# Get auth token from JSON
# response (response) and get
# your secrets from Vault!
# ...

A few weeks ago, we also announced a Google Cloud Platform IAM authentication backend for HashiCorp Vault, allowing authentication using an existing IAM service account identity. For further guidance on using HashiCorp Vault with Google Cloud, take a look at today’s Google Cloud blog post.

By Emily Ye, Software Engineer

Announcing Google Code-in 2017: The Latest and Greatest for Year Eight

Monday, September 18, 2017

We are excited to announce the 8th consecutive year of the Google Code-in (GCI) contest! Students ages 13 through 17 from around the world can learn about open source development working on real open source projects, with mentorship from active developers. GCI begins on Tuesday, November 28, 2017 and runs for seven weeks through to Wednesday, January 17, 2018.

Google Code-in is unique because not only do the students choose what they want to work on from the 2,000+ tasks created by open source organizations, but they have mentors available to help answer their questions as they work on each of their tasks.

Starting to work on open source software can be a daunting task in and of itself. How do I get started? Does the organization want my help? Am I too inexperienced? These are all questions that developers (of all ages) might consider before contributing to an open source organization.

The beauty of GCI is that participating open source organizations realize teens are often first time contributors, and the volunteer mentors are equipped with the patience and the experience to help these young minds become part of the open source community.

Open source communities thrive when there is a steady flow of new contributors who bring new perspectives, ideas, and enthusiasm. Over the last 7 years, GCI open source organizations have helped over 4,500 students from 99 countries become contributors. Many of these students are still contributing to open source years later. Dozens have gone on to become Google Summer of Code (GSoC) students and even mentors for other students.

The tasks open source organizations create vary in skill set and level, including beginner tasks any student can take on, such as “setup your development environment.” With tasks in five different categories, there’s something to fit almost any student’s skills:

  • Code: writing or refactoring 
  • Documentation/Training: creating/editing documents and helping others learn more
  • Outreach/Research: community management, marketing, or studying problems and recommending solutions
  • Quality Assurance: testing and ensuring code is of high quality
  • User Interface: user experience research, user interface design, or graphic design

Open source organizations can apply to participate in Google Code-in starting on Monday, October 9, 2017. Google Code-in starts for students November 28th!

Visit the contest site to learn more about the contest and find flyers, slide decks, timelines and more.

By Stephanie Taylor, Google Open Source

Making the Google Developers Documentation Style Guide Public

Monday, September 11, 2017

Cross-posted on the Google Developers Blog

You can now use our developer-documentation style guide for open source documentation projects.

For some years now, our technical writers at Google have used an internal-only editorial style guide for most of our developer documentation. In order to better support external contributors to our open source projects, such as Kubernetes, AMP, or Dart, and to allow for more consistency across developer documentation, we're now making that style guide public.

If you contribute documentation to projects like those, you now have direct access to useful guidance about voice, tone, word choice, and other style considerations. It can be useful for general issues, like reminders to use second person, present tense, active voice, and the serial comma; it can also be great for checking very specific issues, like whether to write "app" or "application" when you want to be consistent with the Google Developers style.

The style guide is a reference document, so instead of reading through it in linear order, you can use it to look things up as needed. For matters of punctuation, grammar, and formatting, you can do a search-in-page to find items like "Commas," "Lists," and "Link text" in the left nav. For specific terms and phrases, you can look at the word list.

Keep an eye on the guide's release notes page for updates and developments, and send us your comments and suggestions via the Send Feedback link on each page of the guide—we want to hear from you as we continue to evolve the style guide.

Posted by Jed Hartman, Technical Writer

Google Summer of Code 2017 Student Curtain Call

Wednesday, September 6, 2017

Back in early May we announced the students accepted into the 13th edition of the Google Summer of Code (GSoC) program, our largest program ever! Today we are pleased to announce the 1,128 (86.2%*) students from 68 countries that successfully completed the 2017 GSoC. Great job, students!

Students worked diligently with 201 open source organizations and over 1,600 mentors to learn to work with internationally distributed online teams, write great code and help their mentoring org enhance, extend and refine their codebases. Students have also become an important part of these communities. We feel strongly that to keep open source organizations thriving and evolving, they need new ideas - GSoC students help to bring fresh perspectives to these important projects.

We look forward to seeing even more from the 2017 students. Many will go on to become GSoC mentors in future programs and many more will become committers to these and other open source organizations. Some may even create their own open source projects! These students have a bright future ahead of them in technology and open source.

Interested in what the students worked on this summer? Check out their work as well as statistics on past programs.

A big thank you to our mentors and organization administrators who make this program possible. Their dedication to welcoming new student contributors into their communities and teaching them the fundamentals of open source is awesome and inspiring. Thank you all!

Congratulations to all of the GSoC 2017 students and the mentors who made this our biggest and best Google Summer of Code yet.

By Stephanie Taylor, Google Open Source

* 1,309 students started the coding period on May 30th, stats are based upon that number.

The Mentors of Google Summer of Code 2017

Tuesday, September 5, 2017

Mentors are the bread and butter of our program - without their hard work and dedication, there would be no Google Summer of Code (GSoC). These volunteers spend 12 weeks (plus a month of community bonding) tirelessly guiding their students to create the best quality project possible and welcoming them into their communities - answering questions and providing help at all hours.

Each year we pore over oodles of data to extract the most interesting and relevant statistics about the GSoC mentors. Here’s a quick snapshot of our 2017 group:
  • Total mentors: 3,439
  • Mentors assigned to an active project: 1,647
  • Mentors who have participated in GSoC over 10 years: 22
  • Percentage of new mentors: 49%
GSoC 2017 mentors are a worldly group, hailing from 69 countries on 6 continents - we’re still waiting on a mentor from Antarctica… Anyone?

Interested in the data? Check out the full list of countries.
Some interesting factoids about our mentors:
  • Average age: 39
  • Youngest: 15*
  • Oldest: 68
  • Most common first name: Michael (there are 40!)
GSoC mentors help to introduce the next generation to the world of open source software development — for that we are very grateful. To show our appreciation, we invite two mentors from each of the 201 participating organizations to attend the annual mentor summit at the Google campus in Sunnyvale, California. It’s three days of food, community building, lively debate and lots of fun.

Thank you to everyone involved in Google Summer of Code. Cheers to yet another great year!

By Mary Radomile, Google Open Source

* Say what? 15 years old!? Yep! We had 12 GSoC mentors under the age of 18. This group of enthusiastic teens started their journey in our sister program, Google Code-in, an open source coding competition for 13-17 year olds. You can read more about it at

Bringing Real-time Spatial Audio to the Web with Songbird

Monday, August 28, 2017

For a virtual scene to be truly immersive, stunning visuals need to be accompanied by true spatial audio to create a realistic and believable experience. Spatial audio tools allow developers to include sounds that can come from any direction, and that are associated in 3D space with audio sources, thus completely enveloping the user in 360-degree sound.

Spatial audio helps draw the user into a scene and creates the illusion of entering an entirely new world. To make this possible, the Chrome Media team has created Songbird, an open source, spatial audio encoding engine that works in any web browser by using the Web Audio API.

The Songbird library takes in any number of mono audio streams and allows developers to programmatically place them in 3D space around the user. Songbird allows you to create immersive soundscapes, realistically reproducing reflection and reverb for the space you describe. Sounds bounce off walls and reflect off materials just as they would in real-life, capturing truly 360-degree sound. Songbird creates an ambisonic soundfield that can then be rendered in real-time for use in your application. We’ve partnered with the Omnitone project, which we blogged about last year, to add higher-order ambisonic support to Omnitone’s binaural renderer to produce far more accurate sounding audio than ever before.

Songbird encapsulates Omnitone and with it, developers can now add interactive, full-sphere audio to any web based application. Songbird can scale to any order ambisonics, thereby creating a more realistic sound and higher performance than what is achievable through standard Web Audio API.
Songbird Audio Processing Diagram
The implementation of Songbird is based on the Google spatial media specification. It expects mono input and outputs ambisonic (multichannel) ACN channel layout with SN3D normalization. Detailed documentation may be found here.

As the web emerges as an important VR platform for delivering content, spatial audio will play a vital role in users’ embrace of this new medium. Songbird and Omnitone are key tools in enabling spatial audio on the web platform and establishing it as a preeminent platform for compelling VR experiences. Combining these audio experiences with 3D JavaScript libraries like three.js gives a glimpse into the future on the web.
Demo combining spatial sound in 3D environment
This project was made possible through close collaboration with Google’s Daydream and Web Audio teams. This collaboration allowed us to deliver similar audio capabilities to the web as are available to developers creating Daydream applications.

We look forward to seeing what people do with Songbird now that it's open source. Check out the code on GitHub and let us know what you think. Also available are a number of demos on creating full spherical audio with Songbird.

By Jamieson Brettle and Drew Allen, Chrome Media Team

Authenticating to HashiCorp Vault using Google Cloud IAM

Wednesday, August 16, 2017

Applications often require access to small pieces of sensitive data at build or run time, referred to as secrets. Secrets are generally more sensitive than other environment variables or parts of your repository as they may grant access to additional data, such as user data.

HashiCorp Vault is a popular open source tool for secret management, which allows a developer to store, manage and control access to tokens, passwords, certificates, API keys and other secrets. Vault has many options for authentication, called authentication backends. These allow developers to use many kinds of identities to access Vault, including tokens, or usernames and passwords. As the number of developers on a team grows, these kinds of authentication options become impractical; and in enterprise scenarios, managing and auditing these identities becomes burdensome.

Today, we are pleased to announce a Google Cloud Platform IAM authentication backend for Vault. This allows a developer to use an existing IAM identity to authenticate to Vault. Using a service account, you can sign a JWT to show it came from a particular account, and use that to authenticate to Vault. Learn more in the documentation.

The following example in Go shows how a user can authenticate with Vault using this backend. This example assumes the Vault server has already been mounted at auth/gcp and configured.
package main

import (
 vaultapi ""

func main() {
 // Start [PARAMS]
 project := "project-123456"
 serviceAccount := ""
 credsPath := "path/to/creds.json"

 os.Setenv("VAULT_ADDR", "")
 defer os.Setenv("VAULT_ADDR", "")
 // End [PARAMS]

 // Start [GCP IAM Setup]
 jsonBytes, err := ioutil.ReadFile(credsPath)
 if err != nil {
 config, err := google.JWTConfigFromJSON(jsonBytes, iam.CloudPlatformScope)
 if err != nil {

 httpClient := config.Client(oauth2.NoContext)
 iamClient, err := iam.New(httpClient)
 if err != nil {
 // End [GCP IAM Setup]
 // 1. Generate signed JWT using IAM.
 resourceName := fmt.Sprintf("projects/%s/serviceAccounts/%s", project, serviceAccount)
 jwtPayload := map[string]interface{}{
  "aud": "auth/gcp/login",
  "sub": serviceAccount,
  "exp": time.Now().Add(time.Minute * 10).Unix(),

 payloadBytes, err := json.Marshal(jwtPayload)
 if err != nil {
 signJwtReq := &iam.SignJwtRequest{
  Payload: string(payloadBytes),

 resp, err := iamClient.Projects.ServiceAccounts.SignJwt(
resourceName, signJwtReq).Do()
 if err != nil {

 // 2. Send signed JWT in login request to Vault.
 vaultClient, err := vaultapi.NewClient(vaultapi.DefaultConfig())
 if err != nil {

 vaultResp, err := vaultClient.Logical().Write(
   "role": "test",
   "jwt":  resp.SignedJwt,

 if err != nil {

 // 3. Use auth token from response.
 log.Println("Access token %s", vaultResp.Auth.ClientToken)
 // ...

Vault is just one way of managing secrets in development. For further reading on choosing a solution that’s right for you, see Google Cloud Platform’s documentation on Secret Management.

By Emily Ye, Software Engineer

Making Great Mobile Games with Firebase

Tuesday, August 15, 2017

So much goes into building and maintaining a mobile game. Let’s say you want to ship it with a level builder for sharing content with other players and, looking forward, you want to roll out new content and unlockables linked with player behavior. Of course, you also need players to be able to easily sign into your soon-to-be hit game.

With a DIY approach, you’d be faced with having to build user management, data storage, server side logic, and more. This will take a lot of your time, and importantly, it would take critical resources away from what you really want to do: build that amazing new mobile game!

Our Firebase SDKs for Unity and C++ provide you with the tools you need to add these features and more to your game with ease. Plus, to help you better understand how Firebase can help you build your next chart-topper, we’ve built a sample game in Unity and open sourced it: MechaHamster. Check it out on Google Play or download the project from GitHub to see how easy it is to integrate Firebase into your game.
Before you dive into the code for Mecha Hamster, here’s a rundown of the Firebase products that can help your game be successful.


One of the best tools you have to maintain a high-performing game is your analytics. With Google Analytics for Firebase, you can see where your players might be struggling and make adjustments as needed. Analytics also integrates with Adwords and other major ad networks to maximize your campaign performance. If you monetize your game using AdMob, you can link your two accounts and see the lifetime value (LTV) of your players, from in-game purchases and AdMob, right from your Analytics console. And with Streamview, you can see how players are interacting with your game in realtime.

Test Lab for Android - Game Loop Test

Before releasing updates to your game, you’ll want to make sure it works correctly. However, manual testing can be time consuming when faced with a large variety of target devices. To help solve this, we recently launched Firebase Test Lab for Android Game Loop Test at Google I/O. If you add a demo mode to your game, Test Lab will automatically verify your game is working on a wide range of devices. You can read more in our deep dive blog post here.


Another thing you’ll want to be sure to take care of before launch is easy sign-in, so your users can start playing as quickly as possible. Firebase Authentication can help by handling all sign-in and authentication, from simple email + password logins to support for common identity providers like Google, Facebook, Twitter, and Github. Just announced recently at I/O, Firebase also now supports phone number authentication. And Firebase Authentication shares state cross-device, so your users can pick up where they left off, no matter what platforms they’re using.

Remote Config

As more players start using your game, you realize that there are few spots that are frustrating for your audience. You may even see churn rates start to rise, so you decide that you need to push some adjustments. With Firebase Remote Config, you can change values in the console and push them out to players. Some players having trouble navigating levels? You can adjust the difficulty and update remotely. Remote Config can even benefit your development cycle; team members can tweak and test parameters without having to make new builds.

Realtime Database

Now that you have a robust player community, you’re probably starting to see a bunch of great player-built levels. With Firebase Realtime Database, you can store player data and sync it in real-time, meaning that the level builder you’ve built can store and share data easily with other players. You don't need your own server and it’s optimized for offline use. Plus, Realtime Database integrates with Firebase Auth for secure access to user specific data.

Cloud Messaging & Dynamic Links

A few months go by and your game is thriving, with high engagement and an active community. You’re ready to release your next wave of new content, but how can you efficiently get the word out to your users? Firebase Cloud Messaging lets you target messages to player segments, without any coding required. And Firebase Dynamic Links allow your users to share this new content — or an invitation to your game — with other players. Dynamic Links survive the app install process, so a new player can install your app and then dive right into the piece of content that was shared with him or her.

At Firebase, our mission is to help mobile developers build better apps and grow successful businesses. When it comes to games, that means taking care of the boring stuff, so you can focus on what matters — making a great game. Our mobile SDKs for C++ and Unity are available now at

By Darin Hilton, Art Director

Professors from Around the World Get Their Students into HFOSS

Friday, July 21, 2017

Over the last four years instructors from around the world have gathered for the Professors’ Open Source Software Experience (POSSE) workshop to integrate open source concepts into their curriculum. At each event, professors make more progress toward providing students with hands on experience via contributions to humanitarian free and open source software (HFOSS).

This year Google was proud to not only host a workshop at our San Francisco office in April, but also to collaborate with the organizers to bring a POSSE workshop to Europe for the first time.
POSSE workshop leaders, from left to right: Clif Kussmaul (Muhlenburg College), Lori Postner (Nassau Community College), Stoney Jackson (Western New England University),  Heidi Ellis (Western New England University), Greg Hislop (Drexel University), and Darci Burdge (Nassau Community College).
The workshop in Italy was led by Dr. Gregory Hislop from Drexel University, and Drs. Heidi Ellis and Stoney Jackson from Western New England University, and brought together 20 instructors from Germany, Hungary, India, Italy, Macedonia, Qatar, Spain, Swaziland, the United Kingdom, and the United States. This was the most geographically diverse workshop to date!
Group photos in San Francisco, USA on April 22, 2017 (left) and Bologna, Italy on July 1, 2017 (right).
What’s next for POSSE? University instructors from institutions in the US can apply now to participate in the next workshop, November 16-18 in Raleigh, NC and join their peers in the community of instructors weaving HFOSS into their curriculum.

By Helen Hu, Google Open Source

Facets: An Open Source Visualization Tool for Machine Learning Training Data

Monday, July 17, 2017

Cross-posted on the Google Research Blog

Getting the best results out of a machine learning (ML) model requires that you truly understand your data. However, ML datasets can contain hundreds of millions of data points, each consisting of hundreds (or even thousands) of features, making it nearly impossible to understand an entire dataset in an intuitive fashion. Visualization can help unlock nuances and insights in large datasets. A picture may be worth a thousand words, but an interactive visualization can be worth even more.

Working with the PAIR initiative, we’ve released Facets, an open source visualization tool to aid in understanding and analyzing ML datasets. Facets consists of two visualizations that allow users to see a holistic picture of their data at different granularities. Get a sense of the shape of each feature of the data using Facets Overview, or explore a set of individual observations using Facets Dive. These visualizations allow you to debug your data which, in machine learning, is as important as debugging your model. They can easily be used inside of Jupyter notebooks or embedded into webpages. In addition to the open source code, we've also created a Facets demo website. This website allows anyone to visualize their own datasets directly in the browser without the need for any software installation or setup, without the data ever leaving your computer.

Facets Overview

Facets Overview automatically gives users a quick understanding of the distribution of values across the features of their datasets. Multiple datasets, such as a training set and a test set, can be compared on the same visualization. Common data issues that can hamper machine learning are pushed to the forefront, such as: unexpected feature values, features with high percentages of missing values, features with unbalanced distributions, and feature distribution skew between datasets.
Facets Overview visualization of the six numeric features of the UCI Census datasets[1]. The features are sorted by non-uniformity, with the feature with the most non-uniform distribution at the top. Numbers in red indicate possible trouble spots, in this case numeric features with a high percentage of values set to 0. The histograms at right allow you to compare the distributions between the training data (blue) and test data (orange).

Facets Overview visualization showing two of the nine categorical features of the UCI Census datasets[1]. The features are sorted by distribution distance, with the feature with the biggest skew between the training (blue) and test (orange) datasets at the top. Notice in the “Target” feature that the label values differ between the training and test datasets, due to a trailing period in the test set (“<=50K” vs “<=50K.”). This can be seen in the chart for the feature and also in the entries in the “top” column of the table. This label mismatch would cause a model trained and tested on this data to not be evaluated correctly.

Facets Dive

Facets Dive provides an easy-to-customize, intuitive interface for exploring the relationship between the data points across the different features of a dataset. With Facets Dive, you control the position, color and visual representation of each data point based on its feature values. If the data points have images associated with them, the images can be used as the visual representations.
Facets Dive visualization showing all 16281 data points in the UCI Census test dataset[1]. The animation shows a user coloring the data points by one feature (“Relationship”), faceting in one dimension by a continuous feature (“Age”) and then faceting in another dimension by a discrete feature (“Marital Status”).
Facets Dive visualization of a large number of face drawings from the “Quick, Draw!” Dataset, showing the relationship between the number of strokes and points in the drawings and the ability for the “Quick, Draw!” classifier to correctly categorize them as faces.

Fun Fact: In large datasets, such as the CIFAR-10 dataset[2], a small human labelling error can easily go unnoticed. We inspected the CIFAR-10 dataset with Dive and were able to catch a frog-cat – an image of a frog that had been incorrectly labelled as a cat!
Exploration of the CIFAR-10 dataset using Facets Dive. Here we facet the ground truth labels by row and the predicted labels by column. This produces a confusion matrix view, allowing us to drill into particular kinds of misclassifications. In this particular case, the ML model incorrectly labels some small percentage of true cats as frogs. The interesting thing we find by putting the real images in the confusion matrix is that one of these "true cats" that the model predicted was a frog is actually a frog from visual inspection. With Facets Dive, we can determine that this one misclassification wasn't a true misclassification of the model, but instead incorrectly labeled data in the dataset.
Screen Shot 2017-07-14 at 2.59.13 PM.png
Can you spot the frog-cat?
We’ve gotten great value out of Facets inside of Google and are excited to share the visualizations with the world. We hope they can help you discover new and interesting things about your data that lead you to create more powerful and accurate machine learning models. And since they are open source, you can customize the visualizations for your specific needs or contribute to the project to help us all better understand our data. If you have feedback about your experience with Facets, please let us know what you think.

By James Wexler, Senior Software Engineer, Google Big Picture Team


This work is a collaboration between Mahima Pushkarna, James Wexler and Jimbo Wilson, with input from the entire Big Picture team. We would also like to thank Justine Tunney for providing us with the build tooling.


[1] Lichman, M. (2013). UCI Machine Learning Repository []. Irvine, CA: University of California, School of Information and Computer Science

[2] Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky (2009).

After a "close call," a coding champion

Thursday, July 13, 2017

Cross-posted on The Keyword

Eighteen-year-old Cameroon resident Nji Collins had just put the finishing touches on his final submission for the Google Code-In competition when his entire town lost internet access. It stayed dark for two months.

“That was a really, really close call,” Nji, who prefers to be called Collins, tells the Keyword, adding that he traveled to a neighboring town every day to check his email and the status of the contest. “It was stressful.”

Google’s annual Code-In contest, an effort to introduce teenagers to the world of open source, invites high school students from around the world to compete. It’s part of our mission to encourage and inspire the next generation of computer scientists, and in turn, the contest allows these young people to play a role in building real technologies.

Over the course of the competition, participants complete open-source coding and design “tasks” administered by an array of tech companies like Wikimedia and OpenMRS. Tasks range from editing webpages to updating databases to making videos; one of Collins’ favorites, for example, was making the OpenMRS home page sensitive to keystrokes. This year, more than 1,300 entrants from 62 countries completed nearly 6,400 assignments.

While Google sponsors and runs the contest, the participating tech organizations, who work most closely with the students, choose the winners. Those who finish the most tasks are named finalists, and the companies each select two winners from that group. Those winners are then flown to San Francisco, CA for an action-packed week involving talks at the Googleplex in Mountain View, office tours, segway journeys through the city, and a sunset cruise on the SF Bay.
Collins with some of the other winners from Google Code-in 2016
“It’s really fun to watch these kids come together and thrive,” says Stephanie Taylor, Code-In’s program manager. “Bringing together students from, say, Thailand and Poland because they have something in common: a shared love of computer science. Lifelong friendships are formed on these trips.”

Indeed, many Code-In winners say the community is their main motivator for joining the competition. “The people are what brought me here and keep me here,” says Sushain Cherivirala, a Carnegie Mellon computer science major and former Code-In winner who now serves as a program mentor. Mentors work with Code-In participants throughout the course of the competition to help them complete tasks and interface with the tech companies.
Google Code-in winners on the Google campus
Code-In also acts as an accessible introduction to computer science and the open source world. Mira Yang, a 17-year-old from New Jersey, learned how to code for the first time this year. She says she never would have even considered studying computer science further before she dabbled in a few Code-In tasks. Now, she plans to major in it.
Google Code-in winners Nji Collins and Mira Yang

“Code-In changed my view on computer sciences,” she says. “I was able to learn that I can do this. There’s definitely a stigma for girls in CS. But I found out that people will support you, and there’s a huge network out there.”

That network extended to Cameroon, where Collins’ patience and persistence paid off as he waited out his town’s internet blackout. One afternoon, while checking his email a few towns away, he discovered he’d been named a Code-In winner. He had been a finalist the year prior, when he was the only student from his school to compete. This year, he’d convinced a handful of classmates to join in.

“It wasn’t fun doing it alone; I like competition,” Collins, who learned how to code by doing his older sister’s computer science homework assignments alongside her, says. “It pushes me to work harder.”

Learn more about the annual Code-In competition.

By Carly Schwartz, Editor-in-Chief, Google Internal News

Supercharge your Computer Vision models with the TensorFlow Object Detection API

Thursday, June 15, 2017

Crossposted on the Google Research Blog

At Google, we develop flexible state-of-the-art machine learning (ML) systems for computer vision that not only can be used to improve our products and services, but also spur progress in the research community. Creating accurate ML models capable of localizing and identifying multiple objects in a single image remains a core challenge in the field, and we invest a significant amount of time training and experimenting with these systems.
Detected objects in a sample image (from the COCO dataset) made by one of our models.
Image credit: Michael Miley, original image
Last October, our in-house object detection system achieved new state-of-the-art results, and placed first in the COCO detection challenge. Since then, this system has generated results for a number of research publications1,2,3,4,5,6,7 and has been put to work in Google products such as NestCam, the similar items and style ideas feature in Image Search and street number and name detection in Street View.

Today we are happy to make this system available to the broader research community via the TensorFlow Object Detection API. This codebase is an open source framework built on top of TensorFlow that makes it easy to construct, train and deploy object detection models.  Our goals in designing this system was to support state-of-the-art models while allowing for rapid exploration and research.  Our first release contains the following:
The SSD models that use MobileNet are lightweight, so that they can be comfortably run in real time on mobile devices. Our winning COCO submission in 2016 used an ensemble of the Faster RCNN models, which are are more computationally intensive but significantly more accurate.  For more details on the performance of these models, see our CVPR 2017 paper.

Are you ready to get started?
We’ve certainly found this code to be useful for our computer vision needs, and we hope that you will as well.  Contributions to the codebase are welcome and please stay tuned for our own further updates to the framework. To get started, download the code here and try detecting objects in some of your own images using the Jupyter notebook, or training your own pet detector on Cloud ML engine!

By Jonathan Huang, Research Scientist and Vivek Rathod, Software Engineer

The release of the Tensorflow Object Detection API and the pre-trained model zoo has been the result of widespread collaboration among Google researchers with feedback and testing from product groups. In particular we want to highlight the contributions of the following individuals:

Core Contributors: Derek Chow, Chen Sun, Menglong Zhu, Matthew Tang, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, Jasper Uijlings, Viacheslav Kovalevskyi, Kevin Murphy

Also special thanks to: Andrew Howard, Rahul Sukthankar, Vittorio Ferrari, Tom Duerig, Chuck Rosenberg, Hartwig Adam, Jing Jing Long, Victor Gomes, George Papandreou, Tyler Zhu

  1. Speed/accuracy trade-offs for modern convolutional object detectors, Huang et al., CVPR 2017 (paper describing this framework)
  2. Towards Accurate Multi-person Pose Estimation in the Wild, Papandreou et al., CVPR 2017
  3. YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video, Real et al., CVPR 2017 (see also our blog post)
  4. Beyond Skip Connections: Top-Down Modulation for Object Detection, Shrivastava et al., arXiv preprint arXiv:1612.06851, 2016
  5. Spatially Adaptive Computation Time for Residual Networks, Figurnov et al., CVPR 2017
  6. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions, Gu et al., arXiv preprint arXiv:1705.08421, 2017
  7. MobileNets: Efficient convolutional neural networks for mobile vision applications, Howard et al., arXiv preprint arXiv:1704.04861, 2017