Bringing 64-bit data to R

The R programming language has become one of the standard tools for statistical data analysis and visualization, and is widely used by Google and many others. The language includes extensive support for working with vectors of integers, numerics (doubles), and many other types, but has lacked support for 64-bit integers. Romain Francois has recently uploaded the int64 package to CRAN as well as updated versions of the Rcpp and RProtobuf packages to make use of this package. Inside Google, this is important when interacting with other engineering systems such as Dremel and Protocol Buffers, where our engineers and quantitative analysts often need to read in 64-bit quantities from a datastore and perform statistical analysis inside R.

Romain has taken the approach of storing int64 vectors as S4 objects with a pair of R’s default 32-bit integers to store the high and low-order bits. Almost all of the standard arithmetic operations built into the R language have been extended to work with this new class. The design is such that the necessary bit-artihmetic is done behind the scenes in high-performance C++ code, but the higher-level R functions work transparently. This means, for example, that you can:

• Perform arithmetic operations between 64-bit operands or between int64 objects and integer or numeric types in R.
• Read and write CSV files including 64-bit values by specifying int64 as a colClasses argument to read.csv and write.csv (with int64 version 1.1).
• Load and save 64-bit types with the built-in serialization methods of R.
• Compute summary statistics of int64 vectors, such as max, min, range, sum, and the other standard R functions in the Summary Group Generic.

For even higher levels of precision, there is also the venerable and powerful GNU Multiple Precision Arithmetic Library and the R GMP package on CRAN, although Romain’s new int64 package is a better fit for the 64-bit case.

We’ve had to work around the lack of 64-bit integers in R for several years at Google. And after several discussions with Romain, we were very happy to be able to fund his development of this package to solve the problem not just for us, but for the broader open-source community as well. Enjoy!

By Murray Stokely, Software Engineer, Infrastructure Quantitative Team

Gathering of the Mentor Minds

The sixth annual Google Summer of Code Mentor Summit was held Saturday and Sunday, October 22rd and 23th, 2011 at Google headquarters in Mountain View, California.

Over 285 mentors and organization admins from over 145 Open Source organizations attended, again demonstrated how vibrant, creative, and productive Free and Open Source (FOSS) community members are.

Picture taken by Arthur Liu

This was one of our largest gatherings of mentors yet and it produced 62 diverse sessions over the course of a two day “unconference.” Topics included:

• Community building and successful Google Summer of Code how to’s
• Open Source gaming
• Git and submodules for large projects
• Humanitarian FOSS
• Teaching with Open Source
• Recruiting long term contributors
• … and many more

Conference attendees actively participated in the event — organizing sessions, solving logistical issues, cleaning up rooms, and generally pitching in to keep things running smoothly.

Friday Night Meet and Greet

• Mentors began arriving on Friday and in the early evening attended a meet-and-greet party by the hotel pool, where old friendships were renewed and new ones were formed.
• Large quantities of Thai food and refreshing beverages were consumed by mentors amid sounds of laughter and some decidedly technical conversations.
• Mentors turned in relatively early, as two very full days of excitement lay ahead, and considering the distances many had travelled to attend, sleep was very welcome

Chocolate!

With the success of the chocolate session at last year’s Mentor Summit this year an entire room was reserved for participants to bring their favorite chocolates from their country and share with the group.

The response was staggering! Hundreds of pounds of chocolate from all over the world were brought for everyone to sample. From dark, milk, and white chocolate and lots of chocolate hazelnut goodness to unusual offerings such as fish chocolate (it must be an acquired taste), anything your heart desired resided somewhere in that glorious mound of chocolate.

Picture taken by Robert Kaye

Saturday Highlights

• Saturday morning breakfast and lunch at Charlie's Cafe at the Googleplex. Great fuel!
• Saturday welcome and unconference session selection process - A surprisingly efficient way to schedule conference sessions, with good humor.
• Sessions were interesting and well-attended. Notes from many sessions are on the Mentor Summit wiki.
• We received our orange 2011 Mentor Summit t-shirts!
• Saturday night back at the hotel, massive amounts of pizza (and quite a bit of beer) was consumed while equally impressive amounts of social and technical networking was going on in the lounge, in the pool and hot tub, and at tables all around the pool.
• Live musical entertainment was provided by Bart Massey, David Percy, myself, and lots of people who sat in on guitar, bass, and keyboards. Thank you all!

Sunday Highlights

• Many more great sessions!
• The Google on-site store was open for business both days and many summit attendees loaded up with top quality swag!
• The summit wrap-up session was built around the question "How can we make the Mentor Summit better?" Many interesting suggestions were discussed during this introspective, conversational talk, and will no doubt be incorporated into future Mentor Summits.

Some things we learned about ourselves:

• We are part of an incredibly creative, energetic and self-motivated community.
• Given minimal structure we are able to create a productive and enjoyable event.
• Magic happens when so many passionate, productive FOSS community members meet in one place with the space, time, and infrastructure to network and create together.
• We can squeeze a lot of practical work and good fun into just two days.
• One attendee described the Mentor Summit as "a rare place where pretty much everybody understands what you're talking about when you discuss code and community."

Major Thanks:

• To the Google Open Source Programs Office for providing the venue, meals, travel reimbursements, and local transportation for the weekend.
• Special thanks to Carol Smith, Cat Allman, Stephanie Taylor, and Wolf Bergenheim, for all the time, energy and care they put into this event.
• To the OSUOSL for ongoing hosting of the Google Summer of Code Wiki.
• To all the organization Mentors and Admins who attended, for making this another productive and fun Mentor Summit!
• To everyone who has supported Google Summer of Code through the years -- you have made a big difference in countless ways!

We’re already looking forward to next year!

By Marty Connor, Mentor, Etherboot Project

And we’re off! Google Code-in contest opens for students today

And we’re off! Starting today, 13-17 year old students enrolled in a pre-university program (high school, etc.) as of November 21, 2011 can begin working on tasks for Google Code-in. The contest introduces students to the world of open source software development by having students claim tasks involving things like translating documents, writing pieces of code, developing materials for outreach efforts, working on quality assurance issues and user interface projects among many other types of tasks. 18 open source organizations will provide mentors for the students as they work through tasks over the next two months.

Students will learn more about the many different activities involved in open source software development while also earning fun prizes during the contest. For successfully completing one task students will receive a certificate of participation and a tee shirt. For completing three tasks students will receive the tee shirt, certificate and 100 USD. Students can receive up to 500 USD when they complete 15 tasks. Each task is assigned a point value and students will receive the prize money, tee shirts and certificates based on the number of tasks they complete, regardless of the point values. Point values will be used to select the grand prize winners. The 10 students with the most points at the end of the contest period will win a grand prize trip to Google’s Mountain View, California campus for themselves and a parent or legal guardian late next spring.

For more information please visit our program site and read the contest rules and our Frequently Asked Questions. Join our discussion list to ask any additional questions. If you meet the eligibility requirements you can create your account and start claiming tasks today!

The contest will end on January 16, 2012 so get busy claiming those tasks and have fun!

By Stephanie Taylor, Open Source Programs

Doc Summit Wrap up: 4 Books written in 3 days!

In mid October a Document Summit was held at Google headquarters in Mountain View, California where documentation teams from 4 open source projects, KDE, OpenStreetMap, OpenMRS and Sahana Eden as well as a few documentation ‘free agents’ gathered to a write 4 books in the course of three days and take part in a two day unconference. Below, one of the dedicated documentation volunteers and the FLOSS Manuals founder/organizer recount their experiences over the course of the week.

This past month, Google took a big step toward giving documentation its due with a five-day Google Summer of Code Document Sprint. The event was inspired and driven by FLOSS Manuals, an organization of volunteers (of which I'm one) that has received increasing recognition for its documentation projects and the related community-building they stimulate.

It was predicated on the realization at Google (and at least among pockets of open source developers) that free software needed more than good coders to be successful. It needs communities of people caring for each other and guiding each other through the best use of the software, and part of this community effort is good documentation.

To understand the five-day conference itself (which I wrote about extensively on Radar), you have to know something about FLOSS Manuals and its intense "book sprint" process. FLOSS Manuals was started by artist Adam Hyde several years ago to fill the gap in free software's documentation. From the start it focused not on small articles or wikis but on full manuals. Adam developed the book sprint as a way to pull together a community and get something done quickly that everybody could point to as an achievement for their community.

Five to ten developers, power users, and core supporters meet in a workplace for three to five days and write, sharing their work. Remote contributions are encouraged, and outsiders often weigh in with key points. It's a chaotic process that converges suddenly on the last day into a 80-page to 150-page book, and it leaves a high-endorphin sensation among the participants that propels them toward other community-related activities. Books are frequently translated into other languages, are available both on web sites running FLOSS Manuals software and in print, and are kept "live" so that people can contribute to them later.

My Radar articles contain my own lessons from the Google/FLOSS Manuals sprint. The four projects that participated took back not only a book but guidelines for keeping it alive and capitalizing on the educational and promotional activities that a book permits.

By Andy Oram, O’Reilly Editor

-----

The Google Summer of Code Document Summit was the first of its kind - a special mix of formats with an unconference and book sprint tied together. This promised to be not only intensive and productive but exhausting!

We kicked off the first day of the summit with a one day unconference facilitated by Allen Gunn. It was a great way to get started, we covered many interesting topics related to free documentation. At the end of the day everyone was tired yet inspired. We also started to really come together as a group quite quickly under Allen’s guidance and there were many smiling faces and intensive discussions on the bus back to the hotel.

Day two - start sprinting! Well, the start of an ambitious process - 4 parallel book sprints. Zero to book in 3 days with 4 concurrent projects. I had a pretty good feeling it was going to work, having now done 30 or so sprints, but facilitating 4 sprints concurrently is extra tricky. Thankfully Anne Goldenberg (on the board of the French FLOSS Manuals) was there as I am training her to facilitate Book Sprints. I briefed Anne and she started working through the sprint with the OpenStreetMap team and I began facilitating Sahana and KDE while Allen helped here and there a lot for the first day especially with the generation of the table of contents and oiling the engine for OpenMRS. We also divided the "free agents" (people not affiliated to projects) to the groups.

Well, the rest is more or less the Book Sprint process. Writing, reviewing, discussing, workshopping and using the various tricks and methods developed over the last 3 or so years with this methodology. All went pretty smoothly. We finished 4 great books in 3 days. I think the final word counts were something like 25,000 words or so for each of OpenStreetMap, OpenMRS, and Sahana Eden; 10,000 or so words for KDE.

Laleh Torabi designed some wonderful covers for the books and Tuukka Hastrup was there working on a special new development for Booki (the platform we use for Book Sprints). Tuukka finished the beta and implemented it about 35 seconds before we were planning to use it and he didn’t even sweat!

After dinner on the 3rd sprint day we invited Sahana Eden up to the front of the group and they used the new Booki feature to export the book directly to lulu.com (a print on demand service). Thats right, one push of the button and their book was IMMEDIATELY for sale as a paper book online - it was magic!

The last day was feedback and a debrief unconference facilitated by Allen and then... you thought it had ended? No! The Google Open Source Programs Office team had agreed to get paper books printed so we distributed 20 each of the four *beautiful* books to the mentor summit the next day. All bound and shiny...they looked amazing and set off quite a buzz.

Many thanks to everyone involved. Especially the fabulous Google Open Source Program team.

By Adam Hyde, FLOSS Manuals

With this Document Summit, Google had the opportunity to support 4 important projects and the overarching need for good documentation of free and open source software.

By Stephanie Taylor, Open Source Programs

Build great font tools and services with sfntly

Earlier this month we released the sfntly font programming library as open source. Created by the Google Internationalization Engineering team, the sfntly Java and C++ library makes it easy for programmers to build high performance font manipulation applications and services. sfntly is really, really fast: Raph Levien, Google Web Fonts Engineer, says, "Using sfntly we can subset a large font in a millisecond. It’s faster than gzip'ing the result."

Now, both Java and C++ programmers can use sfntly to quickly and easily develop code to read, edit, and subset OpenType and TrueType fonts. The Google Web Fonts team uses the Java version to dynamically subset fonts, and the Chrome/Chromium browser uses the C++ version to subset fonts for PDF printing.

sfntly (\s-’font-lē\) was built from the ground up to provide high performance, an easy to use API, and both high-level and low-level access to font data. Font objects are both thread safe and high performance while still providing access for editing. After about a year of internal development sfntly is stable enough to move it into open source and share with others.

Currently, sfntly has editing support for most core TrueType and OpenType tables, with support for more tables being added. Using sfntly’s basic sfnt table read and write capability, programmers can do basic manipulation of any of the many font formats that use the sfnt container, including TrueType, OpenType, AAT/GX, and Graphite. Tables that aren’t specifically supported can still be handled and round-tripped by the library without risk of corruption.

sfntly is already capable of allowing many really exciting things to be done with fonts, but there is much more planned: expanding support for the rest of the OpenType spec and other sfnt-container font formats, other serialization forms, better higher level abstractions, and more.

I encourage you to you join us on our journey as a user or a contributor.

By Stuart Gill, sfntly Architect

- Cross posted from the official Google Code blog