opensource.google.com

Menu

Posts from March 2015

How to format Python code without really trying

Monday, March 30, 2015

Years of writing and maintaining Python code have taught us the value of automated tools for code formatting, but the existing ones didn’t quite do what we wanted. In the best traditions of the open source community, it was time to write yet another Python formatter.

YAPF takes a different approach to formatting Python code: it reformats the entire program, not just individual lines or constructs that violate a style guide rule. The ultimate goal is to let  engineers focus on the bigger picture and not worry about the formatting. The end result should look the same as if an engineer had worried about the formatting.

You can run YAPF on the entire program or just a part of the program. It’s also possible to flag certain parts of a program which YAPF shouldn’t alter, which is useful for generated files or sections with large literals.

Consider this horribly-formatted code:

x = {  'a':37,'b':42,

'c':927}

y = 'hello ''world'
z = 'hello '+'world'
a = 'hello {}'.format('world')
class foo  (     object  ):
 def f    (self   ):
   return       \
37*-+2
 def g(self, x,y=42):
     return y
def f  (   a ) :
 return      37+-+a[42-x :  y**3]

YAPF reformats this into something much more consistent and readable:

x = {'a': 37, 'b': 42, 'c': 927}

y = 'hello ' 'world'
z = 'hello ' + 'world'
a = 'hello {}'.format('world')


class foo(object):
   def f(self):
       return 37 * -+2

   def g(self, x, y=42):
       return y


def f(a):
   return 37 + -+a[42 - x:y ** 3]

Head to YAPF's GitHub page for more information on how to use it, and take a look at YAPF’s own source code to see a much larger example of the output it produces.

by Bill Wendling, YouTube Code Health Team

Google Code-in 2014 wrap up with OpenMRS

Friday, March 27, 2015

OpenMRS is a medical records system used around the world, especially in places where resources are scarce. It’s also being used with Google’s chlorine-submersible tablets designed for Médecins Sans Frontières to use while treating ebola patients. The OpenMRS community recently participated in Google Code-in, providing young students with an opportunity to get involved with real open source projects and learn about contributing to them. Chaitya Shah, one of OpenMRS’ two grand prize winners, shared this story with us about his participation in the contest.

GCI_2014_logo_small.png

For 7 weeks in December 2014 and January 2015, I worked with OpenMRS in the Google Code-in (GCI) competition. GCI introduces highschool aged kids to open source software development by providing a wide variety of tasks we can complete. For me, it has worked wonders. I’d been interested in the concept of open source software for about a year and even participated in GCI 2013, but this year, the experience turned my interest into a passion. I worked on many new things, met lots of new people, and learned several important skills along the way.

A few days before the competition started, I decided to see how OpenMRS’s software worked. I went through the GitHub repositories and tried to get openmrs-core, the main application, running. After a few tries and the help of several contributors on IRC, I was finally able to do so. Their help showed me what the OpenMRS community was truly about: everyone was very helpful throughout the contest and there was always someone online to help me out at any time of the day.

Several of the tasks I worked on this year were much more complex than the ones I worked on last year, giving me more of a challenge and motivating me to put forth my best effort! The early tasks, however, involved getting acquainted with the OpenMRS community and learning how things work in the organization. Several of these tasks taught some key aspects of open source software or of programming in general. One of the simplest but most important tasks was introducing myself to the community. If the communication between a developer and an organization is weak, the code produced will suffer. It was also inspiring to see so many other people interested in contributing to OpenMRS through GCI.

After learning the basics of OpenMRS, I started to explore tasks in the UI Revamp epic. With guidance from a mentor, I worked on making the OpenMRS ID site look more like the redesigned wireframes provided. These tasks really taught me a lot about design, one of my weak points. I used to know very little about HTML/CSS in general. The revamp tasks taught me about good practices in UI Design and I loved every minute of it.

In the last two weeks of the competition, I decided that I was ready to contribute something brand new to the organization. While deploying OpenMRS on the OpenShift cloud platform as part of a task, I found the developer guide was vague in some areas and difficult to follow. It took me a few days and some experimentation to get it working. To ensure that others wouldn’t have the same troubles, I made two videos showing the exact steps to follow: one for Windows and one for Unix-based systems.

After that, I decided to take on a Docker task. Docker is a system that lets you build, ship, and run distributable applications. This task directed me to create an image that downloads, sets up, and runs OpenMRS automatically. I was slightly overwhelmed at first, but Docker proved to be quite useful because it uses a system of containers rather than virtual machines, making it much faster and easier to deploy applications. I felt a big sense of accomplishment once I had finished publishing my work, writing up documentation, and making a quick video tutorial on how to set it up.

I learned a lot from OpenMRS and GCI this year. I was especially impacted by the weight that community interaction has in open source work. Previously, I’d always had the notion that being a programmer is very lonesome, sitting in a room with nothing but a computer for many hours at a time. However, I now know that everything in open source software development is collaborative; everyone works together to accomplish a single goal. I hope to someday find a job with a company that embraces this collaborative nature. Thank you to OpenMRS and GCI for an awesome experience this year!


By Chaitya Shah, GCI grand prize winner

Classp: a “classier” way to parse

Tuesday, March 24, 2015

If you’ve worked on compilers, translators, or other tools that need to read a programming language, chances are you’ve spent many painful hours detailing that language’s grammar. Recently, we opened up the source code for Classp, a side-project a few of us have been working on that demonstrates it’s possible to have an automatic parser generator that is not based on formal grammars. Instead of grammar productions, you write classes similar to C++ or Java and you write class patterns to define the syntax. Although there are libraries like Boost.Spirit and Scala Parsers that give you a nice way to write a grammar in the programming language itself, in the end you are still writing a grammar. Even though Classp looks a lot like C++ or Java, it is not just a C-like way to write a grammar. It’s an entirely different way to specify syntax.

Grammars are great for diagramming a complex syntactic structure for human readers, but as a computer specification, they leave a lot to be desired. Four key problems with grammars inspired us to work on Classp as an alternative.

First, a grammar is intended to represent the actual syntactic structure of the language: all of the little details like what goes first and what goes second, where to put your commas and semicolons, where can you substitute one thing for another, etc... But this surface structure doesn’t really matter to the programmer who wants to process the language. It just gets in the way. What you really care about are the logical parts of the declaration: what is the type? What is the name? Is there an initializer and what is it?

Second, many common parser generators don’t actually specify any tree at all. They let the programmer write actions to build up a parse tree. But the actions in most systems tend to form an awkward fit with the grammar.

Next, when you write a grammar, you have to worry not just about the surface structure of the language, but also about how the language will be parsed. You have to write the grammar around ambiguities in the language and sometimes around other features. You can’t just write the rules as you would write them for a human reader:

Expr ::= Int | ( Expr ) | Expr + Expr | Expr - Expr | Expr * Expr | Expr / Expr

instead, you have to write them in a way that avoids ambiguity:

Expr ::= Expr + Term | Expr - Term;
Term ::= Term * Factor | Term / Factor
Factor ::=  Int | ( Expr )

Finally, grammar-based parsers are extremely verbose. For serious parsing tasks, it is common to write a grammar, design a parse tree, write actions in the grammar to create the parse tree, design an abstract syntax tree, and write code to translate the parse tree into an abstract syntax tree. There are many dependencies among these parts that all have to be kept consistent over the life of the program. It’s a complex and error-prone process.

Classp attempts to avoid these problems. The abstract syntax tree is what programmers typically want to work with. With class patterns, you only have two jobs: design the abstract syntax tree and write a formatter for it. (A formatter is the function that writes out the abstract syntax tree in the target language.)

Here’s an example class declaration for an abstract syntax tree. The class pattern is the part inside the parentheses of the syntax statement: “arg1 '+' arg2”.

class Plus: Expression {
 Expression arg1;
 Expression arg2;
 syntax(arg1 '+' arg2);
}

This class pattern says that to print a Plus node, you first print arg1, then you print a plus sign, then you print arg2. So it looks like a nice formatting language, but where do we specify the parser? The answer is that we don’t specify a parser; Classp will invert the formatter to generate a parser for us. Since formatters are typically much easier to write and maintain than parsers, it almost feels like magic.

Classp is still a work in progress. We still have to deal with ambiguity in languages, features that are only output in one way but may be input in several ways, and a few other issues. But it’s ready to play with now and we’d love to hear from others interested in this subject. To learn more, visit http://google.github.io/classp.


By David Gudeman, Classp team

GSoC project Sambamba published in scientific journal

Friday, March 20, 2015

Student applications for this year’s Google Summer of Code (GSoC) are still open until March 27th. One of our goals with GSoC is to inspire young developers to participate in open source development, hopefully continuing well beyond the summer. Pjotr Prins from the Open Bioinformatics Foundation shared this story with us about a GSoC 2012 student who has continued leading the development of a software tool used in laboratories around the world. That tool, Sambamba, was recently featured in an Oxford University Press scientific journal.


The Open Bioinformatics Foundation (OBF) participated in Google Summer of Code (GSoC) in 2012 and again in 2014. One of our projects, Sambamba, enables users to rapidly process large sequence alignment files in the SAM, BAM and CRAM formats using parallel processing. Sambamba, which means “parallel” in Swahili, was recently the subject of a paper published in Bioinformatics Journal by GSoC alumnus Artem Tarasov. Since the tool is now used in DNA sequencing centres around the world, Artem has become well known in the bioinformatics community as Sambamba’s creator.

When we participated in GSoC 2012, we accepted five students, one of whom was Artem. His project was to “write the fastest parallelized BAM parser in D” as an alternative to the existing SAMtools software written in single-threaded C. I consider the D language to be particularly well-suited to bioinformatics given its modern hybrid OOP/functional syntax with close-to-the-metal performance optimizations.

Even before GSoC started that year, Artem was doing research and cranking out code. In his blog, he wrote about learning the D language, dealing with parallel executing code, and the sometimes-buggy compiler and garbage collector. The file formats he was working with are complicated and contain many assumptions, but he made wise choices which led to a very effective piece of software: people tend to rave about Sambamba when they use it the first time. Artem and I continued working on Sambamba after GSoC and before long, I found that he was the one mentoring me!

Since then, Artem has been invited to visit the Cuppen sequencing lab in the Netherlands where he added depth analysis to Sambamba. This is also when we started work on the manuscript for the Bioinformatics Journal. Later, the OBF was able to sponsor a second trip to the European Bioinformatics Institute in Cambridge, UK where he and I took part in a Codefest and met with other bioinformatics researchers and developers, including some OBF contributors.

Artem isn’t our only GSoC student who has continued making a difference in open source. Four of our five GSoC 2012 students are still active FOSS committers on GitHub, with three of them continuing in the bioinformatics space. Although GSoC can be competitive and we haven’t been accepted into the program every year, we’re grateful for the opportunities it has given us. Organizations like OBF and SciRuby are proof that GSoC and scientific projects work really well together. Without GSoC, Artem and I would probably not have ever met. He and I both hope to introduce more students to scientific open source projects in the future.


By Pjotr Prins, Sambamba GSoC Mentor

Google Summer of Code: Meet-up Round-up!

Wednesday, March 18, 2015

Over the past ten years, Google Summer of Code (GSoC) has given over 8,500 students a bridge into open source communities. GSoC alumni have played a large role in the program’s success by encouraging their fellow students to take part. Around the world, energetic students who have already participated in GSoC lead the way by hosting meet-up events that help others learn about the opportunities GSoC provides and get answers to their questions about the program. Who better than GSoC alumni to tell students what it’s really like spending the summer coding for open source projects?


Student applications for GSoC 2015 are open until March 27th, so the community has been spreading the word at school and beyond during the past few months. Here are just a few of the student-initiated events we’ve heard about recently.

Buea, Cameroon: 17 December 2014

Chris Nuvagda wanted to spread the word about GSoC after finishing his project, so he reached out to other GSoC alumni at the University of Buea to make it happen. Over 150 students attended the meet-up, with about 30% being women -- a number we’re happy to hear since only 10% of GSoC 2014 students were women and we’re trying to improve on that. The 5 presenters spoke about their experiences in GSoC and the benefits of it, gave advice on writing a great proposal, and shared what it’s like for women working in open source. Read more...



Delhi, India: 31 January 2015

Heena Mahour and Ayush Gupta hosted a meet-up for about 200 prospective students and open source enthusiasts. With help from others in their community, they organized a full-day event featuring 8 speakers. Students who previously participated with organizations like KDE, OWASP, and Mifos shared their experiences and answered questions from the audience. Technical topics were also discussed, including memory leaks and an introduction to version control systems. Read more...



Udine, Italy: 24 February 2015

Claudio Desideri spent GSoC 2014 coding for KDE and organized a meet-up for students at the University of Udine. With 80 attendees, there were just barely enough seats for everyone. Claudio shared his personal experiences but also learned that many students who might want to apply to GSoC fear being judged by potential mentors or failing out of the program. He helped reassure students that GSoC is an opportunity to grow and that they aren’t expected to already know everything. Read more...



Bangalore, India: 2 March 2015

With a desire to increase the participation of nearby colleges, especially among female students, Tejas Dharamsi and Rajath Kumar organized a GSoC meet-up at the Google Bangalore office. 102 students, including 44 women, from 11 colleges gathered to hear from the 6 speakers who shared details about the coding projects they worked on and gave practical advice for students applying to GSoC for the first time. The ending panel discussion gave attendees a chance to ask questions and offered students encouragement to apply for the program. Read more...



Kampala, Uganda: 12 March 2015

Organized by Kaweesi Joseph with help from OpenMRS, the organization he worked with during GSoC 2014, more than 110 students gathered for this meet-up. The 5 presenters discussed the benefits of contributing to open source, how students can participate in GSoC, and how OpenMRS is being used in Uganda. Read more...



We thank these GSoC alumni and many more who are spreading their enthusiasm for open source in their local communities. It’s because of local networking events like these that GSoC is able to have rich representation from all around the world. We look forward to welcoming some of these meet-up attendees into the upcoming Google Summer of Code 2015 -- don’t forget to apply before March 27th!


By Ashleigh Rentz, Open Source team, with thanks to everyone who allowed us to share their stories and photos here.

Google Summer of Code now open for student applications

Monday, March 16, 2015

If you’re a university student looking to earn real-world experience this summer, consider writing code for a cool open source project with the Google Summer of Code program.


GoogleSummer_2015logo_horizontal.jpg


Students who are accepted into the program will put the skills they have learned in university to good use by working on an actual software project over the summer. Students receive a stipend and are paired with mentors to help address technical questions and concerns throughout the course of the project. With the knowledge and hands-on experience students gain during the summer, they strengthen their future employment opportunities. Best of all, more source code is created and released for the use and benefit of all.


Interested students can submit proposals on the website starting now through Friday, March 27 at 19:00 UTC. Get started by reviewing the ideas pages of the 137 open source projects in this year’s program and decide which projects you’re interested in. Because Google Summer of Code has a limited number of spots for students, writing a great project proposal is essential to being selected to the program — be sure to check out the Student Manual for advice.


For ongoing information throughout the application period and beyond, see the Google Open Source Blog, join our Summer of Code mailing lists or join us on Internet relay chat at #gsoc on Freenode.


Good luck to all the open source coders out there, and remember to submit your proposals early — you only have until March 27 to apply!

By Carol Smith, Google Open Source team

Google Code-in 2014 wrap up with Haiku

Friday, March 13, 2015

The Haiku community is developing a new operating system for personal computing. They were one of the twelve organizations who mentored students during Google Code-in 2014, our contest introducing 13 to 17 year old students to working in open source communities. Today, Scott McCreary reflects on the achievements of students who participated with Haiku.


GCI 2014.png

2014 was the fifth year for Google Code-in (GCI) and also Haiku’s fifth year participating. This was the first year where we offered beginner tasks designed to give students an easy way to get started. These tasks introduced students to booting and using Haiku, compiling Haiku, and using Haikuporter to build a package from a recipe file.

149 students completed at least one task with us this year and 36 students completed at least three tasks with us to earn a GCI t-shirt. Six students completed over 20 tasks each in a bid to become grand prize winners. Each mentoring organization selected two grand prize winners and three finalists. Our finalists were Augustin Cavalier, Markus Himmel and Chirayu Desai. (Chirayu was also a GCI 2013 winner with RTEMS.) Our winners were Josef Gajdusek and Puck Meerburg.

After getting started with beginner tasks, many students moved on to tasks which added improvements to Haiku or fixed bugs. Students contributed to HaikuWeather, SuperFreeCell, BookmarkConvert, and our CatKey editor. Many HaikuPorter recipes were added and updated, making it easier to bring existing software into Haiku. Several tickets were resolved, new GUI controls were created, new screensavers were added, and several apps were updated to make use of Layout Manager. In total, students completed a staggering 435 tasks for Haiku during the contest.

GCI grand prize winners are invited to San Francisco for an award trip. Last year, I had the pleasure of attending as a mentor along with our 2013 winners. This year, the winners selected Adrien Destugues (a former Google Summer of Code student, now one of our most active contributors) as the mentor they’d like to meet on the trip. I’m excited for the great time they’ll have!

I'd like to thank the 17 Haiku mentors and all 149 students who took part in GCI with Haiku this year. Also, a special thanks to our community members who were on IRC to help students during the contest. Finally, a shout-out to Stephanie Taylor and the rest of the team at Google for running Google Code-in and selecting Haiku to participate.

By Scott McCreary, Haiku GCI mentor

Bidding farewell to Google Code

Thursday, March 12, 2015

When we started the Google Code project hosting service in 2006, the world of project hosting was limited. We were worried about reliability and stagnation, so we took action by giving the open source community another option to choose from. Since then, we’ve seen a wide variety of better project hosting services such as GitHub and Bitbucket bloom. Many projects moved away from Google Code to those other systems. To meet developers where they are, we ourselves migrated nearly a thousand of our own open source projects from Google Code to GitHub.

As developers migrated away from Google Code, a growing share of the remaining projects were spam or abuse. Lately, the administrative load has consisted almost exclusively of abuse management. After profiling non-abusive activity on Google Code, it has become clear to us that the service simply isn’t needed anymore.

Beginning today, we have disabled new project creation on Google Code. We will be shutting down the service about 10 months from now on January 25th, 2016. Below, we provide links to migration tools designed to help you move your projects off of Google Code. We will also make ourselves available over the next three months to those projects that need help migrating from Google Code to other hosts.

  • March 12, 2015 - New project creation disabled.
  • August 24, 2015 - The site goes read-only. You can still checkout/view project source, issues, and wikis.
  • January 25, 2016 - The project hosting service is closed. You will be able to download a tarball of project source, issues, and wikis. These tarballs will be available throughout the rest of 2016.

Google will continue to provide Git and Gerrit hosting for certain projects like Android and Chrome. We will also continue maintaining our mirrors of projects like Eclipse, kernel.org and others.

How To Migrate Your Data Off Google Code

The simplest way to migrate off of Google Code is to use the Google Code to GitHub exporter tool, which provides an automated way to migrate a project’s source, issues, and wikis to a new GitHub repo. Please note: GitHub’s importer will convert any Subversion or Mercurial Google Code projects to use Git in the process.

We also offer stand-alone tools for migrating to GitHub and Bitbucket, and SourceForge offers a Google Code project importer service.

If you encounter any problems using these tools, please log issues with us, contact google-code-shutdown@google.com, or email me directly (cdibona@google.com). We’ll also be closely tracking Hacker News, Reddit, and other popular forums to answer questions in public. We know this decision will cause some pain for those of you still using Google Code and we're sorry for that. We'll continue to do our best to make the migration process easy for you.

GitHub and Bitbucket are both looking forward to working with developers moving off of Google Code. They’ve been great to work with leading up to this announcement, so we’d like to thank those sites for their continued support of the community. There are some great options for people today that didn’t exist in 2006, and we look forward to helping you find the one that works for your project.

Chris DiBona, Director of Open Source

CritiqueBrainz and rocking the Google Summer of Code

Friday, March 6, 2015

Student applications for Google Summer of Code 2015 will be opening on March 16th. University students interested in applying for this year’s program can get ready now by checking out the organizations taking part in this year’s program. Below, Robert Kaye shares the story of an outstanding student from last summer who made a big impact with the MetaBrainz Foundation, maintainers of the MusicBrainz service.


At MusicBrainz, we’re proud to collect and freely share information about all kinds of music. We strive to keep that information objective and factual, but being music fans, we can’t help but want to share our opinions sometimes, too. To do that, we developed a concept called CritiqueBrainz where we’d welcome people to write Creative Commons licensed, non-neutral point of view music reviews. Unfortunately, we didn’t have the resources to actually build it, nor was there any off-the-shelf software that would suit our needs.

When we took part in Google Summer of Code 2013, we put the CritiqueBrainz concept on our ideas page. It was too much to ask of one student to build a complete website with all the custom features we wanted in one summer. We accepted one student who wrote all of the features we requested as part of his successful GSoC project. However, over the summer it became clear that we needed to put a lot more work into the project before we could deploy the new site.

In 2014, we put CritiqueBrainz on our ideas page again and this time we asked for someone to finish the site in the first half of the summer, then to deploy, maintain and debug the site during the latter half of the summer. When the student application period opened, things got interesting. Several years ago, we added a requirement that applying students needed to submit a pull request to the project in order for us to consider their application. One student in particular submitted a pull request as part of his application. And then he submitted another. And another.

That one particular student submitted some truly excellent work as part of his application. So we accepted Roman Tsukanov, better known to us as “Gentlecat”, to finish the CritiqueBrainz project. At almost the instant we announced our accepted student proposals, his stream of pull requests started up again and never let up. As we were approaching the official start date of GSoC, the number of tasks left to finish the coding portion of the project was dwindling. Gentlecat even wrote a tool that parsed, cleaned up and imported 9,000 reviews that the BBC had given us. Things were looking excellent!

Given all that progress, we decided that the beginning of GSoC would be a perfect opportunity to release the first beta version of the CritiqueBrainz to the world. We gave Gentlecat access to one of our servers and let him deploy the code onto the server himself. With some guidance on best practices from us, he quickly got things up and running. Many people from the MusicBrainz community began writing reviews on the site, which naturally meant they found bugs and made tons of suggestions for improving the site.

Gentlecat plowed through the new tickets opened during the beta period and created pull request after pull request. My summer mornings consisted of reviewing his pull requests from the previous day. Gentlecat knocked out a long list of bugs while refactoring the code to have a better layout and improve readability. He added HTTPS support, found and fixed problems from the GSoC 2013 project, and even added integration with Spotify to support a hack I was creating at San Francisco Music Hack Day.

In the end, Gentlecat finished at least twice what he had promised to do during the span of GSoC. And the best part? He continues to pound out code and fixes, and he’s now a contributor to three of our projects.

He plans to apply for Google Summer of Code again this year, but he also feels that more people ought to be involved with CritiqueBrainz so he’s been contributing additional project ideas for other students to consider. If you’re a university student and CritiqueBrainz sounds interesting to you, please check out our GSoC 2015 ideas page.

Gentlecat rocked GSoC so hard that his nickname has become a term in our community: if you Gentlecat something, it means that you finished faster than anyone expected while going above and beyond what you promised to do. In other words: you rocked it. Hard!


by Robert Kaye, MusicBrainz GSoC mentor
.