Expanding our Differential Privacy Library

All developers have a responsibility to treat data with care and respect. Differential privacy helps organizations derive insights from data while simultaneously ensuring that those results do not allow any individual's data to be distinguished or re-identified. This principled approach supports data computation and analysis across many of Google’s core products and features.

Last summer, Google open sourced our foundational differential privacy library so developers and organizations around the world can benefit from this technology. Today, we’re announcing the addition of Go and Java to our library, an end-to-end solution for differential privacy: Privacy on Beam, and new tools to help developers implement this technology effectively.

We’ve listened to feedback from our developer community and, as of today, developers can now perform differentially private analysis in Java and Go. We’re working to bring these two libraries to full feature parity with C++.

We want all developers to have access to differential privacy, regardless of their level of expertise. Our new Privacy on Beam framework captures years of Googler developer experience and efficiency improvements in a comprehensive and easy-to-use solution that handles computation end-to-end. Built on Apache Beam, Privacy on Beam can reduce implementation mistakes, and take care of all the steps that are essential to differential privacy, including noise addition, partition selection, and contribution bounding. If you’re new to Apache Beam or differential privacy, our codelab can get you started.

Tracking privacy budgets is another challenge developers face when implementing differential privacy. So, we’re also releasing a new Privacy Loss Distribution tool for tracking privacy budgets. With this tool, developers can maintain an accurate estimate of the total cost to user privacy for collections of differentially private queries, and better evaluate the overall impact of their pipelines. Privacy Loss Distribution supports widely used mechanisms (such as Laplace, Gaussian, and Randomized response) and can scale to hundreds of compositions.

We hope these new languages, tools, and features unlock differential privacy for even more developers. Continue to share your stories and suggestions with us at dp-open-source@google.com—your feedback will help inform our future differential privacy launches and updates.

Acknowledgements

Software Engineers: Yurii Sushko, Daniel Simmons-Marengo, Christoph Dibak, Damien Desfontaines, Maria Telyatnikova, Dennis Kraft, Jimmy Ross, Vadym Doroshenko
Research Scientists: Pasin Manurangsi, Ravi Kumar, Sergei Vassilvitskii, Alex Kulesza, Jenny Gillenwater, Kareem Amin

By: Miguel Guevara, Mirac Vuslat Basaran, Sasha Kulankhina, and Badih Ghazi – Google Privacy Team and Google Research

Apache Beam presents its mascot to the world

Four years after it graduated from The Apache Software Foundation’s incubator, Apache Beam welcomes a new addition to the family: its mascot, the Beam firefly! Apache Beam is an open source data streaming programming model that runs on the back end of some of your favorite apps. It is the technology behind many popular apps that need to process big data in real time. And the reason it has come this far is the community of developers that contribute to this open source project every week. In the last few months, we worked with Apache Beam contributors to collaboratively design a mascot for the project—a creative asset that can represent the values of the project and attract new users and contributors—to keep growing the project and expanding its reach.

In these four years, the Apache Beam project has been a busy… firefly. According to the Apache Software Foundation’s 2019 Annual Report, Beam ranks fourth in the top five most active projects by commits, and it ranks first in the top five most active dev@ mailing list, showing a strong and transparent communication exchange within the community of developers. On top of that, 53 Meetup groups across the globe are directly or indirectly connected to sharing knowledge about Apache Beam use cases, applications and functionalities.

With this much momentum and enthusiasm for this project, it was a good opportunity to cement some of Apache Beam’s most valued characteristics, to help raise awareness of this rapidly growing project, and convert more users to contributors. “Projects with a mascot are more relatable. They signal that there is more to the project than its technical vision. It signals that there is more to the project than its code,” said project contributor Maximilian Michel. In November of last year, the community discussed adopting a mascot and as a result, 11 animal ideas emerged for a possible mascot: beaver, hedgehog, lemur, owl, salmon, trout, robot dinosaur, firefly, cuttlefish, dumbo octopus, and angler fish. After 48 contributors expressed their vote, the collective decision was a firefly. In January of this year, artist Julián Bruno set out to bring the mascot to life. There were four rounds of feedback on different iterations of the mascot, plus a final vote, where 18 people participated; Engagement increased with every new round. In the end, this process produced an original mascot, a model sheet (so that anyone may reproduce it), and two adaptations of the Beam firefly: one where it is learning, and one where it is doing what it does best… stream data! You can read more about this process on the Apache Software Foundation’s blog.

In a year that has presented a lot of challenges to bring people together, working on the mascot project with the Apache Beam community was refreshing, and felt like a medium for contributors to connect beyond code and technical questions. It is our wish that Apache Beam continues to grow as a project, and we hope to continue to support its community to: support newcomers, share what works, and collaborate with others to build great solutions.

By María Cruz, Google Open Source

The Apache Beam Community in 2019

2019 has already been a busy time for the Apache Beam. The ASF blog featured our way of community building and we've had more Beam meetups around the world. Apache Beam also received the Technology of the Year Award from InfoWorld.

Success at Apache: What You Need to Know by Maximilian Michels https://t.co/XjtVYgPAHX #Apache #Open #Innovation #Community #people #processes #JustWorks @stadtlegende pic.twitter.com/xSibnyWAMe

— Apache - The ASF (@TheASF) 26 maart 2019

As these events happened, we were building up to the 20th anniversary of the Apache Software Foundation. The contributions of the Beam community were a part of Maximilian Michels blog post on the success of the ASF's open source development model:

As the founder of the first Beam meetup in London back in 2017, seeing the community flourish on a larger and worldwide scale is something that makes me happy. And we have come quite a long way since 2017, both in terms of geographical spread:



As well as in numbers:



All of this culminates in two Beam Summits this year—one we already had a few weeks ago in Berlin, and the other which will take place in a few weeks in Las Vegas, where we worked together with Apache and the ApacheCon team.

In that spirit, let's have a more detailed overview of the things that have happened, what the next few months look like, and how we can foster even more community growth.

Meetups

We've had a flurry of activity, with several meetups in the planning process and more popping up globally over time. As diversity of contributors is a core ASF value, this geographic spread is exciting for the community. Here's a picture from the latest Apache Beam meetup organized at Lyft in San Francisco:



We have more Bay Area meetups coming soon, and the community is looking into kicking off a meetup in Toronto and New York! In Europe, London had its first meetup of 2019 at the start of April, as did Stockholm at the start of May:

Stockholm had its second @ApacheBeam meetup at the start of this week, with @kanterov from @SpotifyEng and @while from @eqt.

Slides can be found here: https://t.co/85LZMw7pDA
and the recording here: https://t.co/CutirIgu9V

Check out @BeamSummit as well - speakers announced now! pic.twitter.com/4veoeQeLgg

— Matthias Baetens 🌆 (@matthiasbaetens) 12 mei 2019

Meetup groups are becoming active in Berlin and New York also, so stay tuned for events there and more meetups internationally! If you are interested in starting your own meetup, feel free to reach out! Good places to start include our Slack channel, the dev and user mailing lists, or the Apache Beam Twitter. Even if you can’t travel to these meetups, you can stay informed on the happenings of the community. The talks and sessions from previous conferences and meetups are archived on the Apache Beam YouTube channel. If you want your session added to the channel, don’t hesitate to get in touch!

Summits

The first summit of the year was held in Berlin this past June. You can read about the inaugural edition of the Beam Summit Europe here. At these summits, you have the opportunity to meet with other Apache Beam creators and users, get expert advice, learn from the speaker sessions, and participate in workshops. We are proud to say that the Summit doubled in size this year with attendees from 24 countries across 4 continents.

You can find resources from this year’s Summit here:

• 📹 the recordings can be found on our YouTube channel.
• 📊 presentations of the Summit are made available via the website and in this folder.
• We strongly encourage you to get involved again this year! You can still sign up for the upcoming summit in North America.
• 🎫 If you want to secure your ticket to attend the Beam Summit North America 2019, check our the ApacheCon website.
• 🔈 In case you want to get involved in speaking at events, do not hesitate to contact us via email or Twitter.

Why community engagement matters

Why we need a strong Apache Beam community:

• We’re gaining lots of code contributions and need committers to review them
• We want people to feel a sense of ownership to the project. By fostering this level of engagement, the work becomes even more exciting.
• A healthy community has a further reach and leads to more growth. More hours can be contributed to the project as we can spread the work and ownership.

Why are we organizing these summits:

• We’d like to give folks a place to meet and share ideas.
• We know that offline interactions often changes the nature of the online ones in a positive manner.
• Building an active and diverse community is part of the Apache Way. These summits provide an opportunity for us to engage people from different locations, companies, and backgrounds.

By Matthias Baetens, Google Developer Expert for Cloud, Apache Beam committer and community organiser