Posts from May 2012

BSDCan 2012 - "The technical BSD conference"

Thursday, May 31, 2012

As opposed to what? The *non*-technical conference for people who write operating systems in their free time? :-) Anyway, it turned out mostly true.  It was my first BSDCan and it was very focused on actually developing BSD systems, as opposed to other conferences that have more of an emphasis on deploying BSD.

It was a very full several days: too much to cover here, but I want to call out several great talks.

Locking mechanisms
My first talk was Kirk McKusick's overview of locking in the FreeBSD kernel.  It wasn't about what to lock when, it was an overview of the different locking mechanisms in FreeBSD, all 7 of them.  I really liked what I heard, finally some place where spinning is still considered cool under the right circumstances. The talk covered lending out higher priorities that a waiting thread might have to a thread holding a lock the waiter wants, lock order and detecting violations thereof and went off to describe the various more and much more featureful kinds of locks in the kernel today. Highly relevant even for those not working on a BSD, but it makes us who are stuck in userland sad.

Now I know where my RAM goes. Holy cow. Big webs of buffers. Lots of data structures.  Lots of aggressive readahead and other speculative reads (prime example is for contiguous writes which are interrupted by synchronous reads of metadata for block allocation information, so why not preemptively read all this stuff?).  Buffer duplication, some for good reasons (new buffers for zRAID splits), some for bad reasons (underdeveloped). I’m still not sure how out of sync the view of a mmap'ed file wrt the view that read/write/seek presents is (reminds me of the early Mach VM). The whole session pointed out repeatedly how much emphasis they put on keeping "all parts of the system as saturated as possible". Interesting that much of this work and other work presented was happening in some form of Hypervisor (in this case XEN).

Consumer router products
My friend Stefan Bethke gave a talk about his work to make FreeBSD work on various consumer router products. Think OpenWRT but BSD based. Much of the talk described how the devices in these routers are mostly in a tree but the vendors made some things very odd which led to a bunch of circular dependencies in the device structure, preventing regular driver startup. Stefan was forced to experiment with various schemes of proxy device drivers that get all the ducks in a row before the actual drivers find everything they need. Much cheering from the embedded crowd ensued.

Recent IPv6 security 
The main takeaway was that there is a serious problem in IPv6 resulting from packet fragmentation potentially dividing a packet within the header which then of course makes firewalls and other packet inspection mechanisms unable to find out what's going on.

My main takeaways:
- Hypervisors of some form are making it into all kinds of systems and environments.
- Don't buy systems that can't take registered RAM in a bazillion sockets.
- Huge amounts of data are "in flight" these days, both network and disk.
- Testing on systems with single memory banks will be useless soon.
- Flash memory and methods of dealing with its problems are a hot topic. It looks like people decided to tank through the problems and not wait for better storage cells.

By Martin Cracauer, Programmer, QPX, ITA Software by Google

Gerrit Hackathon #3, May 7-11, 2012

Wednesday, May 30, 2012

A great thanks goes out to Google for hosting the 3rd Gerrit Code Review Hackathon earlier this month at their Mountain View campus, it was well attended and very productive.

After quickly deciding to focus on adding a plugin framework to Gerrit during the hackathon, we discussed a few areas which could really benefit from plugins (authentication, hook, events, UI) and we identified new SSH commands as a possible quick prototype objective. After discovering some minor pitfalls of intense collaboration using Gerrit, we realized that we could make good use of the new Gerrit builtin rebase button. The rebase button did indeed come in handy for many of us to avoid overwriting each other’s changes with rebases, but we quickly noticed that simply having the owner and submitters allowed to rebase would not be adequate. This led to the proposal to upload a new rebase permission, which has been backported to be in the 2.4 release since that is the first release with the new rebase feature.

There were other minor Gerrit UI quirks discussed and fixes proposed for many of these during the following days:
user links now link to the owner:user search instead of to the user’s dashboard, most of us found  that this is what we really want most of the time.
Fixing the lack of reviewer visibility in the case of draft reviews
set-account ssh command was also merged.

Now back to plugins; we completed our first prototype for SSH commands early Tuesday and we were on our way to a plugin framework.  We decided to break plugin concepts into 2 main types: plugins with full access to Gerrit’s internals (and bound to a specific Gerrit version), and extensions which would only be bound to light stable interfaces. Plugins and extensions both got their own classloaders, but extensions were isolated from Gerrit classes. Helloworld sample plugins and repositories were created on for several of these allowing users to host plugins alongside Gerrit as distinct projects.

Shawn Pearce created two jar files to compile plugins and extensions against, the plugin jar is one massive jar with Gerrit and all its dependencies in it. This large jar file should make it possible for plugin developers to create plugins quickly without having to download the many dependencies independently using maven. The extensions jar file consists mainly of the new fairly sparse top level gerrit-extensions-api package.

Other plugin accomplishments:
plugins got their own Guice injectors
HttpServlet plugin support in Gerrit
pegdown can be used by plugin developers to create lightweight documentation for their plugins
plugin command to be able to operate on and view plugins in a live Gerrit instance

With an initial plugin infrastructure in place, initial rework of the Gerrit core started by expressing part of it in terms of plugin: refactoring the replication as plug-in and introducing the ability to create aliases in order to use plugins commands as surrogates of existing commands. We added the ability to capture JSON/RPC and SSH actions to generate audit-trails and, later in the week, the ability to add extra custom commit validation through plugins.

Additional plugins have been created to provide long awaited functionalities in Gerrit, such as the delete project command and an online server viewer accessible through a web link. As the number of plugins is expected to grow quickly it was proposed that we define a simple but published process to allow people to search through existing plugins and host in Gerrit contributed plugins and their descriptive wikis.

The final day we focused on how topic branches might be approached in Gerrit. We decided we did not want to add support for change labels to the database since we are attempting to migrate away from the db. To put this data into git notes, it is necessary to have an indexing system and we concluded it would need to be done (or attempted) with Lucene. This could then pave the path for greater use of Lucene in Gerrit.

Scalability was naturally on many of our minds. Git maintainer Junio C Hamano joined our discussions of the many hacks being implemented to deal with the git protocol's ref advertisement performance impacts.  Android Gerrit users seem to push many of the original git use case assumptions beyond their original intents, especially when it comes to ref counts.  We were delighted to get Junio's approval to propose non backwards compatible changes to the git protocol to make our use cases more efficient.

We also spent some time outlining an incremental approach to making open source Gerrit clusterable (the version running for android-review and gerrit-review is already clustered but uses much of Google's proprietary technologies such as GFS and BigTable). Several incremental steps were outlined on how to move Gerrit in that direction. The first step which is needed is to support a very simple master/master installation of Gerrit where both (or all if more than 2) masters share a common filesystem backend (likely a high end NFS server) and a common db. Two issues were identified here which need to be resolved before this is possible: Cache coherency and Submit conflict resolution.

So while this is by no means a full run down of the week, it should give you some insights into the progress which we made during this week and also into some of the plans for future Gerrit directions. With the beginnings of a solid plugin infrastructure now in place, we hope that the Gerrit community will pick up the task of providing shared plugins for common tasks and host these new plugins alongside the gerrit project at the gerrit-review site.

Some fun stats (external contributions counted also):
New changes created during the 5 days of the Hackathon: 115
New patchsets uploaded during the 5 days of the Hackathon: 353
Changes touched during the 5 days of the Hackathon:143
Changes submitted during the 5 days of the Hackathon: 63
Changes submitted during the week after the Hackathon (many carryovers): 43

By Martin Fick, Qualcomm Innovation Center, Gerrit project submitter

This has been cross posted from the Repo and Gerrit Discussion List

A Correction to Google Summer of Code 2012 Stats - Part 2

Friday, May 25, 2012

When we published the annual list of schools with the largest number of accepted students for Google Summer of Code 2012, we received an email pointing out that we were missing one school that belongs in the Top 10. PESIT - Peoples Education Society is now shown below in the corrected chart.

RankSchoolCountry# of Accepted Students# in 2011
1University of MoratuwaSri Lanka2927
2Polytechnic University Of BucharestRomania2123
2National University of SingaporeSingapore2111
3Dhirubhai Ambani Institute of Information and Communication TechnologyIndia178
4International Institute of Information Technology - HyderabadIndia169
5Birla Institute of Technology and Science Pilani, Goa campus / BITS-Pilani - K.K.Birla Goa CampusIndia141
6Institute of Technology, Banaras Hindu University (IT-BHU)India136
7Technical University Of GdanskPoland129
7Birla Institute of Technology and Science, Pilani (BITS Pilani)India1210
8Technische Universität Wien, (TU Wien)Austria1013
8Indian Institute of Technology, KharagpurIndia1014
8University of LjubljanaSlovenia107
9Indian Institute of Technology, DelhiIndia96
9Chernihiv State Technological UniversityUkraine96
10PESIT - Peoples Education SocietyIndia83
10UNICAMP - Universidade Estadual de CampinasBrazil814

By Cat Allman, Google Open Source Programs

Google Summer of Code coding starts today

Monday, May 21, 2012

Today is the first day of coding for our 8th year of the Google Summer of Code program. This year 1,208 students will spend the next 12 weeks writing code for 180 different open source organizations. Students are working on a diverse group of projects ranging from developing game systems to Spotify integration for existing tools to modules for group theory.

We are excited to see the contributions this year’s students will make to the open source community.

For more information on important dates for the program please visit our timeline.

Have a great summer!

By Carol Smith, Open Source Team

Google Summer of Code 2012 Stats - Part 2

Friday, May 18, 2012

And now the news that some of you have been waiting for: the annual list of schools with the largest number of accepted students for Google Summer of Code 2012!

RankSchoolCountry# of Accepted Students# in 2011
1University of MoratuwaSri Lanka2927
2Polytechnic University Of BucharestRomania2123
2National University of SingaporeSingapore2111
3Dhirubhai Ambani Institute of Information and Communication TechnologyIndia178
4International Institute of Information Technology - HyderabadIndia169
5Birla Institute of Technology and Science Pilani, Goa campus / BITS-Pilani - K.K.Birla Goa CampusIndia141
6Institute of Technology, Banaras Hindu University (IT-BHU)India136
7Technical University Of GdanskPoland129
7Birla Institute of Technology and Science, Pilani (BITS Pilani)India1210
8Technische Universität Wien, (TU Wien)Austria1013
8Indian Institute of Technology, KharagpurIndia1014
8University of LjubljanaSlovenia107
9Indian Institute of Technology, DelhiIndia96
9Chernihiv State Technological UniversityUkraine96
10UNICAMP - Universidade Estadual de CampinasBrazil814

As you can see, some schools have a big jump in participants over 2011 - yes, I’m talking about you, BITS-Pilani - while a few schools show a drop off in students.  To compare, let’s look at the overall “Top 10” schools for the past 8 years.

RankSchoolCountry# of Accepted Students: 2005-2012
1University of MoratuwaSri Lanka164
2Polytechnic University Of BucharestRomania75
3National University of SingaporeSingapore58
3Technische Universität Wien, (TU Wien)Austria58
3Universidade Estadual de Campinas - UNICAMP Brazil58
4University of TorontoCanada57
5Indian Institute of Technology, KharagpurIndia51
6Technical University Of GdanskPoland49
7Graduate University of Chinese Academy of SciencesChina48
8International Institute of Information Technology - HyderabadIndia42
8Politechnika WroclawskaPoland42
9University of Illinois at Urbana-ChampaignUnited States41
10Institute of Technology, Banaras Hindu University (IT-BHU)India38

Charts like these highlight how international Google Summer of Code truly is.  

It’s important to remember that the number of students from each school is not the point of the program. Over 13% of accepted students are from schools whose students are participating for the first time this year. What matters is participation - getting more students involved with creating free and open source software. 

By Cat Allman, Google Open Source Programs

Macedonia Google Summer of Code meetups are a success

Wednesday, May 16, 2012

Congratulations to everyone who was accepted for Google Summer of Code 2012! I hope we have a great summer, and write some free, clean, reusable code.

Free Software Macedonia is a local organization from Macedonia that again helped spread the word about this year’s Google Summer of Code. Last year I was introduced to free software and eventually participated in the program after reading a blog post by one of the Free Software Macedonia members. The right information and the right timing can do wonders, especially for students that are enthusiastic about doing new things and getting some practical coding experience. Three students from Macedonia participated in last year’s Google Summer of Code, including me. This number and all the others I will present may seem small to you, but Macedonia is a very small country with approximately 2,000,000 citizens.

This year we wrote blog posts, organized a lecture at the biggest technical university in the country and filled the amphitheater, and held follow up sessions to introduce possible future participants in Google Summer of Code with the tools that are used by free software communities and share some knowledge with them. We also provided support for anyone who needed it along the way as well as a space for the students to work.

At the lecture at the university, the three students that participated last year all shared their experiences as well as tips and tricks for picking an organization and writing proposals. One of the senior members of Free Software Macedonia explained the concept of free software to those audience members who were unfamiliar with the term.

We have a hackerspace in Skopje, Macedonia called KIKA that we fund and manage ourselves, and it plays a big role in promoting these kind of programs. We had 20 students attend the follow up session where we showed the prospective students the basic tools needed to be successful in free software development such as mailing lists, bug tracking software, coding concepts, git etc. We installed Linux on the many machines and discussed projects with the students. KIKA is also a public space so it was open for students to come and work there during the application period and throughout the summer. The older and more experienced members are always a great support and can help the struggling coder to see the code from a different angle and eventually achieve success.

In the end, ten people from our sessions ended up applying for Google Summer of Code, and six, all of whom worked together to achieve this at KIKA, got in. You can see their smiling faces at the moment the emails arrived on the image (I am missing from the photo).

So, from just three people last year, we doubled that number this year and we hope to double our student participation again next year. We will continue spreading the word about the program, giving support to interested students and a place for the students to work. Many thanks to Google for giving us this great chance, and helping free software. I wish you a successful summer!

Editor’s note:  There was an additional student from Macedonia who was also accepted into the program making the total number of Macedonian Google Summer of Code students for 2012 seven.

By Tamara Atanasoska, 2011 and 2012 Google Summer of Code student 

Googlers at Large

Monday, May 14, 2012

With May comes the annual ramp up of conference season, and Googlers who are part of the open source community will be giving talks and participating in events around the world over the next two months.

Last week, Jeremy Allison attended Samba XP, the 11th international Samba conference for users and developers in Göttingen, Germany speaking on "The Evolution of IO in Samba".

Our Chicago Google office will host past and present Google Summer of Code students and mentors along with students from DePaul, IIT, Northwestern, UChicago, and UIC Universities on May 23rd (Wednesday) for an evening of lightning talks delivered by accepted Google Summer of Code students and Google engineers. You can find all the details here, including how to register (pre-registration is required).

Chris DiBona will be speaking to the Open Cyber Security Roundtable in Menlo Park on May 30. The roundtable is part of a larger program to get more open source tools and apps into the security arsenal of federal, state and local government.

On June 12th, Cat Allman will be facilitating a session at the inaugural USENIX Women in Advanced Computing (WiAC) Summit. WiAC will take place as part of USENIX Federated Conferences Week (FCW) in Boston, MA.

Attendees and viewers of Google I/O coming June 27-29, 2012 in San Francisco will hear about a variety of Google’s open source projects, stay tuned to for more information on sessions. There will also be a dinner for GSoC Mentors hosted by Carol Smith in late June.

We hope to see you at one of these spring events!

By Cat Allman, Open Source Programs

Google Summer of Code 2012 by the Numbers: Part 1.1

Thursday, May 10, 2012

So many of you have asked about the number of students from your particular country that we’ve decided to simply post the whole list.

Number of Accepted Students by Country
India = 227
United States = 173
Germany = 72
Russian Federation = 56
China = 45
Poland = 42
Sri Lanka = 42
Romania = 40
France = 38
Canada = 36
Brazil = 33
United Kingdom = 31
Italy = 30
Austria = 25
Ukraine = 25
Spain = 23
Singapore = 20
Greece = 18
Czech Republic = 15
Turkey = 15
Hungary = 14
Sweden = 14
Slovenia = 11
Belgium = 9
Croatia = 8
Finland = 8
Australia = 7
Belarus = 7
Ireland = 7
Macedonia = 7
Netherlands = 7
Switzerland = 7
Vietnam = 7
Slovak Republic = 6
Bulgaria = 5
Japan = 5
Serbia = 5
Argentina = 4
Egypt = 4
Latvia = 4
Lithuania = 4
Pakistan = 4
Portugal = 4
Bangladesh = 3
Chile = 3
Ecuador = 3
Israel = 3
Moldova = 3
Norway = 3
Peru = 3
South Korea = 3
Colombia = 2
Estonia = 2
Indonesia = 2
Taiwan = 2
Algeria = 1
Bosnia-Herzegovina = 1
Denmark = 1
Hong Kong = 1
Mauritius = 1
Mexico = 1
Nepal = 1
New Zealand = 1
Saudi Arabia = 1
South Africa = 1
Uruguay = 1
Uzbekistan = 1
Venezuela = 1

By Cat Allman, Open Source Programs

Google Summer of Code 2012 by the Numbers: Part 1 of 2

Tuesday, May 8, 2012

Now that the mentoring organizations have made their selections, it’s time for the first of two posts breaking down the numbers about the students accepted into the 2012 instance of Google Summer of Code.

This 8th year of Google Summer of Code is the largest yet. More mentoring organizations received more applications from more students than ever before. We received a record number of applications - 6685 - from 4258 students from 98 countries to work with the 180 selected mentoring organizations.

We also accepted more students this year: 1212 from 69 countries. This year India supplied the largest number of students, 227.

The 10 countries with the highest number of accepted students are:

Country# of 2012 Accepted Students
United States172
Russian Federation56
Sri Lanka42

This year we are excited to have a student from the African nation of Mauritius, ( which hasn’t previously been represented in Google Summer of Code.

The program set another record this year: the highest percentage yet of women accepted. 8.3% of our accepted students identified themselves as female, versus 7.1% percent last year. Overall the percentage of women accepted has slowly but surely increased every year since 2006.

We’re very excited to see what our record crop of students come up with this year! Currently the students are in the community bonding period where they familiarize themselves with their projects, mentors and communities, before the coding period starts on Monday, May 21st.  Good luck to them all!

Please visit our program site and timeline for more information, and watch this space for part 2 of “Google Summer of Code 2012 by the Numbers” about the universities represented this year.

By Cat Allman, Open Source Programs

Introducing Ceres Solver - A Nonlinear Least Squares Solver

Tuesday, May 1, 2012

Solving non-linear least squares problems comes up in a broad range of areas across science and engineering - from fitting complicated curves in statistics, to constructing 3D models from photographs in computer vision. Today we're happy to announce the release of a solver we use at Google.

Ceres Solver is a portable C++ library that allows for modeling and solving large complex nonlinear least squares problems. The notable features are:
A simple, expressive API
Automatic differentiation
Robust loss functions
Local parameterizations
A threaded Jacobian evaluators and linear solvers
Dense QR factorization (using Eigen) for small problems
Sparse Cholesky factorization (using SuiteSparse) for large sparse problems
Specialized solvers for problems in 3D computer vision
A liberal license (New BSD)
Scales from servers to cell phones.

We use Ceres Solver at Google to estimate the pose of Street View cars, aircrafts, and satellites; to build 3D models for PhotoTours; to estimate satellite image sensor characteristics, and more. For example, the video below shows three instances of Ceres Solver fusing data from the sensors mounted on a Street View car to estimate its correct position and orientation on the surface of earth.

We hope you will download the code, solve some problems, and share your experiences on the mailing list.

By Sameer Agarwal & Keir Mierle, Geo Software Engineers