Google Summer of Code wrap-up: Apache Flink (previously Stratosphere)

We continue our Friday Google Summer of Code wrap-up series with Apache Flink (previously Stratosphere) who was a first time participant in the program. Organization Administrator Robert Metzger talks below about their two successful student participants as well as their project’s transition to the Apache Software Foundation incubator program. 

Apache Flink is a system for expressive, declarative, fast, and efficient data analysis. Flink combines the scalability and programming flexibility of distributed MapReduce-like platforms with the efficiency, out-of-core execution, and query optimization capabilities found in parallel databases.

We were accepted to this year’s Google Summer of Code (GSoC) under our former project name “Stratosphere”. But during the summer our project entered the incubator of the Apache Software Foundation (ASF). Incubation is a process for new projects to enter the umbrella of the ASF. As part of the process our project name was subsequently changed from Stratosphere to Flink.

Our move to the ASF also meant quite a few changes for us and our students during the course of their projects. Both mentors and students were able to learn together about the new processes required by the ASF and in the end this transition worked out quite well for everyone involved.

The acceptance of our project into GSoC was a huge, exciting accomplishment for all of the Flink / Stratosphere developers and especially thrilling to a new, first time organization. We had two students this summer: Artem Tsikiridis and Frank Wu.

Artem worked on a full Hadoop MapReduce compatibility layer for Flink. Both Hadoop and Flink are distributed systems for processing huge amounts of data. Hadoop is an open source implementation of the MapReduce algorithm published by Google. It is widely used for a broad range of data intensive computing applications. Flink offers a broad range of operators and can be used to execute MapReduce-style applications.

Artem’s summer project concerned the implementation of a compatibility layer that exposes exactly the same APIs as Apache Hadoop. This feature allows existing Hadoop users to run their Hadoop jobs with Flink. Consequently, users are now able to utilize a faster execution engine for their existing code! Artem worked closely with the community and succeeded in bringing his changes into our main code line. His work will be available with the 0.7-incubating release of Apache Flink.

Frank Wu, our second GSoC student, worked on a large sub-project of Flink called Support for Streaming (Stratosphere Streaming). Frank initiated the development of the mini-batch processing API of Stratosphere Streaming, enabling operations on windows of tuples. Additionally, he contributed to both the iterative and stateful streaming solutions, two of the most challenging applications of streaming. Frank also provided numerous code examples for the topics he was working on. Like Artem, his work will be available with the 0.7-incubating release of Apache Flink.

I would like to thank the mentors, Fabian Hüske and Marton Belassi, as well as our second organization administrator, Ufuk Celebi, for their help with Stratosphere/Flink’s GSoC participation in the summer of 2014.

By Robert Metzger, Organization Administrator for Apache Flink

Google Summer of Code 2014 by the numbers: Part three

In our third statistics post for Google Summer of Code (GSoC) 2014 we have a list of schools with the highest number of student participants. For the first time in seven years a new school has claimed the top spot—congratulations to International Institute of Information Technology - Hyderabad.

Rank	School	Country	# of Accepted Students 2014	# of Accepted Students 2013
1	International Institute of Information Technology - Hyderabad	India	69	29
2	University of Moratuwa	Sri Lanka	44	39
3	Birla Institute of Technology and Science Pilani	India	26	19
4	Birla Institute of Technology & Science - Pilani, K K Birla Goa Campus	India	25	16
5	University POLITEHNICA Of Bucharest	Romania	17	25
6	Indian Institute of Technology, Bombay	India	15	9
7	National University of Singapore	Singapore	14	15
8	Indian Institute of Technology (BHU) Varanasi	India	13	9
9	Budapest University of Technology and Economics	Hungary	12	7
9	Dhirubhai Ambani Institute of Information and Communication Technology	India	12	17
10	Indian Institute of Technology, Roorkee	India	11	10
10	International Institute of Information Technology, Bhubaneswar	India	11	0
10	Vienna University of Technology	Austria	11	13
10	University Of Waterloo	Canada	11	2

Student majors in 2014 were predominantly in Computer Science and other technical fields (as expected).  But this year we also had students studying Anthropology, Cartography, Evolutionary Biology, Linguistics and even Metallurgy. GSoC certainly attracts a diverse set of students year after year! For more stats from 2014, check out the other posts in this series.

We’d like to thank the schools and professors that help spread the word about GSoC to their students. But don’t forget that students from any university are encouraged to participate! Reviewing statistics each year from GSoC is exciting, but being in “first place” is certainly not the most important part of the program. Our goal since the inception of GSoC is to get students involved in the creation of free and open source software, and to encourage contributions to projects that have the potential to make a difference worldwide — no matter what university the student attends.

By Mary Radomile, Open Source Programs

Google Summer of Code wrap-up: Twitter

Google Summer of Code 2014 has come to a close and news of the great work completed by our 1300+ student participants is starting to pour in. Our first student “wrap-up” post is from Twitter, a three time Summer of Code participant. We’ll be featuring these stories on Fridays this fall.

For the third time, Twitter had the opportunity to participate in Google Summer of Code (GSoC), and we wanted to share news on the resulting open source activities. Unlike many GSoC participating organizations that focus on a single ecosystem, Twitter has a variety of projects that span multiple programming languages and communities. They include:  

Use zero-copy read path in @ApacheParquet

Sunyu Duan worked with mentors Julien Le Dem (@J_) and Gera Shegalov (@gerashegalov) on improving performance in Parquet by using the new ByteBuffer based APIs in Hadoop. As a result of the work over the summer, performance has improved up to 40% based on initial testing and the work will make its way into the next Parquet release.

A pluggable algorithm to choose next EventLoop in Netty

Jakob Buchgraber worked with mentor Norman Maurer (@normanmaurer) to add pluggable algorithm support to Netty’s event loop (see pull request). At the start of the summer when a new EventLoop was needed to register a Channel, EventLoopGroup implementations used a round-robin like algorithm to choose the next EventLoop. This was challenging because different events may become more busy than others over time, hence the need for Jakob’s project to support pluggable algorithms to increase performance.  

Various compression codecs for Netty

Idel Pivnitskiy (@pivnitskiy) worked with mentor Trustin Lee (@trustin) to add multiple compression codes (LZ4, FastLZ and BZip2) to the Netty project. Compression codecs will allow cutting traffic and creating applications, which are able to transfer large amounts of data even more effectively and quickly.

Android Support For Pants

Mateo Rodriguez (@mateornaut) added Android support to the Pants build system (see commits) so Pants can build Android applications (APKs) on top of the many other languages and tools it supports.

A pure ZooKeeper client for Finagle 

Pierre-Antoine Ganaye (@pa-ganaye) was mentored by Evan Meagher (@evanm) to add a pure Apache ZooKeeper client to Finagle to improve performance (see project).

An SMTP client for Finagle

Lera Dymbitska (@suncelesta) worked with mentors Selvin George (@selvin) and Travis Brown (@travisbrown) to add SMTP protocol support to Finagle to improve performance (see pull request). Finagle strives to provide fully asynchronous protocol support so baking in SMTP support was required versus using third party libraries such as javamail and commons-email which are synchronous by design.

Analyze Wikipedia using Cassovary

Szymon Matejczyk (@szymonmatejczyk) worked with mentors Pankaj Gupta (@pankaj) and Ajeet Grewal (@ajeet) to enable Cassovary to analyze Wikipedia data. The result of this work improved the performance of Cassovary when dealing with large graphs. See the commits associated with the project to see how it was done.

We really enjoyed the opportunity to take part in GSoC again this year. Thanks again to our seven students, mentors and Google for the program. We hope to participate again next summer.

Chris Aniszczyk, Organization Administrator, Twitter

Security for the people

A recent Pew study found that 86% of people surveyed had taken steps to protect their security online. This is great—more security is always good. However, if people are indeed working to protect themselves, why are we still seeing incidents, breaches, and confusion? In many cases these problems recur because the technology that allows people to secure their communications, content and online activity is too hard to use.

In other words, the tools for the job exist. But while many of these tools work technically, they don’t always work in ways that users expect. They introduce extra steps or are simply confusing and cumbersome. (“Is this a software bug, or am I doing something wrong?”) However elegant and intelligent the underlying technology (and much of it is truly miraculous), the results are in: if people can’t use it easily, many of them won’t.

We believe that people shouldn’t have to make a trade-off between security and ease of use. This is why we’re happy to support Simply Secure, a new organization dedicated to improving the usability and safety of open-source tools that help people secure their online lives.

Over the coming months, Simply Secure will be collaborating with open-source developers, designers, researchers, and others to take what’s there—groundbreaking work from efforts like Open Whisper Systems, The Guardian Project, Off-the-Record Messaging, and more—and work to make them easier to understand and use.

We’re excited for a future where people won’t have to choose between ease and security, and where tools that allow people to secure their communications, content, and online activity are as easy as choosing to use them.

By Meredith Whittaker, Open Source Research Lead and Ben Laurie, Senior Staff Security Engineer

gcloud-node - a Google Cloud Platform Client Library for Node.js

Today we are announcing a new category of client libraries that has been built specifically for Google Cloud Platform. The very first library, gcloud-node, is idiomatic and intuitive for Node.js developers. With today’s release, you can begin integrating Cloud Datastore and Cloud Storage into your Node.js applications, with more Cloud Platform APIs and programming languages planned.

 The easiest way to get started is by installing the gcloud package using npm:

$ npm install gcloud

With gcloud installed, your Node.js code is simpler to write, easier to read, and cleaner to integrate with your existing Node.js codebase. Take a look at the code required to retrieve entities from Datastore:

var gcloud = require('gcloud');

var dataset = new gcloud.datastore.Dataset({
projectId: 'my-project',
keyFilename: '/path/to/keyfile.json' // Details at 
              //https://github.com/googlecloudplatform/gcloud-node#README
});

dataset.get(dataset.key('Product', 123), function(err, entity) {
console.log(err, entity);
});

gcloud is open-sourced on Github; check out the code, file issues and contribute a PR - contributors are welcome. Got questions? Post them on StackOverflow with the [gcloud-node] tag. Learn more about the Client Library for Node.js at http://googlecloudplatform.github.io/gcloud-node/ and try gcloud-node today.

 -Posted by JJ Geewax, Software Engineer

Node.js is a trademark of Joyent, Inc. and npm is a trademark of npm, Inc.