Posts from September 2008
Explore Your Hardware: iotools and prettyprint
Tuesday, September 30, 2008
One of the most difficult and tedious things we do here in the Platforms Team is figure out what is going wrong with a new piece of hardware. Generally that means we need to examine a bunch of hardware registers and see how the device is configured. If you've never had to do this, let us assure you - it isn't much fun, especially when you have a few hundred bit fields to decode.
We're software guys, so the solution, of course, was to write some code! We thought some of you out there might find our work useful, so we're happy to release these tools.
The first piece is iotools - a suite of simple command line tools which allow you to read and write hardware registers. With iotools you can write simple shell scripts to read the device information you need, process it, and write it back to the hardware. Currently iotools includes support for PCI, IO, memory mapped IO, MSR, CPUID, TSC, SMBus, and CMOS. It also includes a bunch of simple arithmetic and logical tools to make your scripting even easier.
The second piece is prettyprint - a library and set of tools which allow you extract hardware data in human-readable form. prettyprint includes a simple device description "language". Once a device has been described, prettyprint can find any instances of the device and produce a tree of objects which can be read from and written to symbolically. Never again will you have to remember that bits [8:6] of the wibble register contain the frobnicator amplification factor, encoded in 1/2 dB units. prettyprint will just tell you.
These tools have proven to be useful to us in the platforms team at Google. We hope some of you find them useful too. Patches are welcome!
To find out more, please visit the iotools home page or iotools discussion group and the prettyprint home page or prettyprint discussion group.
Open Sourcing Update Engine
Monday, September 29, 2008
By Greg Miller, Mac Engineering Team
Google's Macintosh Engineering team has just open sourced the foundation on which their software updater is built. You can learn more on the Google Mac Blog.
Google's Macintosh Engineering team has just open sourced the foundation on which their software updater is built. You can learn more on the Google Mac Blog.
Plone and Google Summer of Code
By Matthew Wilkes, Plone Google Summer of Code Organization Adminstrator
Following our wonderfully successful participation in the Google Highly Open Participation Contest, the Plone Foundation was looking forward to some larger projects that would benefit the community. We certainly weren't disappointed. After months of dedicated mentoring, the Plone project is very happy to report that our four Summer of Code projects are now complete.
Javier Mansilla was raring to go with his Plone Batch editing project, providing screenshots of his initial work along with his request for commit rights. This culminated in him presenting his work at the Jornadas Regionales de Software Libre conference in Argentina. The level of community involvement of the students was a particular highlight with both Florian Friesdorf (working on new authentication systems) and Martin Lundwall (zope.component introspector for Plone), both participating in multiple sprints for some face to face collaboration with their mentor and other students. On the deployment side of things, Kenneth Miller has been working on making Plone installations easier to customise with his Buildout Builder project. Buildout Builder allows non-technical users to manipulate zc.buildout based Python environments through a slick graphical user interface.
We're quite excited to bring you this report. Many thanks to all of our mentors and students for making Summer of Code 2008 a success for the Plone project!
Following our wonderfully successful participation in the Google Highly Open Participation Contest, the Plone Foundation was looking forward to some larger projects that would benefit the community. We certainly weren't disappointed. After months of dedicated mentoring, the Plone project is very happy to report that our four Summer of Code projects are now complete.
Javier Mansilla was raring to go with his Plone Batch editing project, providing screenshots of his initial work along with his request for commit rights. This culminated in him presenting his work at the Jornadas Regionales de Software Libre conference in Argentina. The level of community involvement of the students was a particular highlight with both Florian Friesdorf (working on new authentication systems) and Martin Lundwall (zope.component introspector for Plone), both participating in multiple sprints for some face to face collaboration with their mentor and other students. On the deployment side of things, Kenneth Miller has been working on making Plone installations easier to customise with his Buildout Builder project. Buildout Builder allows non-technical users to manipulate zc.buildout based Python environments through a slick graphical user interface.
We're quite excited to bring you this report. Many thanks to all of our mentors and students for making Summer of Code 2008 a success for the Plone project!
KOffice Projects Rock KDE's Google Summer of Code 2008
Friday, September 26, 2008
By Boudewijn Rempt, Krita Maintainer and KDE Summer of Code Mentor
This year's Summer of Code has been very succesful for the KOffice project. Working under KDE's auspices, KOffice had the privilege of working with seven students.
Lorenzo Villani has been adding new features to Kexi web forms, while Carlos Licea has worked on making it possible to load KPresenter 1.x (.kpr) documents in KPresenter 2.0. Fredy Yanardi, two time Summer of Code student working with KDE, also tackled improvements to KPresenter, adding notes and presenter views. Lukáš Tvrdý has worked on a Chinese brush simulation for Krita and Fela Winkelmolen on a calligraphy brush for Karbon. Another of our returning students, Pierre Ducroquet, worked on adding ODF support to KWord. And speaking of ODF, Benjamin Cail wrote code to improve the graphics support in libwv2 and to allow direct importing of .doc files into KWord in ODF format.
Our choice of projects in 2008 was dictated less by a desire for flashiness; instead, we were determined to choose those projects most likely to add solid worth to KOffice. Compared to last year, life was easier for our students, when both KOffice and KDE were still under heavy development and every Monday was basically spent on getting the latest binary and source incompatible changes incorporated. This year, only KOffice was a rapidly moving target!
Most importantly, all the students worked directly in the KOffice trunk Subversion repository and their work has been released as part of the tenth alpha release of KOffice 2.0. You may also be interested in our end of term report, which also includes screencasts.
Many congratulations to all of our students and mentors for their tremendous efforts!
This year's Summer of Code has been very succesful for the KOffice project. Working under KDE's auspices, KOffice had the privilege of working with seven students.
Lorenzo Villani has been adding new features to Kexi web forms, while Carlos Licea has worked on making it possible to load KPresenter 1.x (.kpr) documents in KPresenter 2.0. Fredy Yanardi, two time Summer of Code student working with KDE, also tackled improvements to KPresenter, adding notes and presenter views. Lukáš Tvrdý has worked on a Chinese brush simulation for Krita and Fela Winkelmolen on a calligraphy brush for Karbon. Another of our returning students, Pierre Ducroquet, worked on adding ODF support to KWord. And speaking of ODF, Benjamin Cail wrote code to improve the graphics support in libwv2 and to allow direct importing of .doc files into KWord in ODF format.
Our choice of projects in 2008 was dictated less by a desire for flashiness; instead, we were determined to choose those projects most likely to add solid worth to KOffice. Compared to last year, life was easier for our students, when both KOffice and KDE were still under heavy development and every Monday was basically spent on getting the latest binary and source incompatible changes incorporated. This year, only KOffice was a rapidly moving target!
Most importantly, all the students worked directly in the KOffice trunk Subversion repository and their work has been released as part of the tenth alpha release of KOffice 2.0. You may also be interested in our end of term report, which also includes screencasts.
Many congratulations to all of our students and mentors for their tremendous efforts!
Updates from the Free Software Foundation Europe
Thursday, September 25, 2008
By Shane Coughlan, FTF Coordinator, Free Software Foundation Europe
My name is Shane Coughlan and and I work for Free Software Foundation Europe (FSFE), a non-profit, non-governmental organisation with offices in Germany, Switzerland, Belgium and Sweden. You may remember me from my last FSFE update on this blog on the Freedom Task Force, which is the FSFE's legal training project which I coordinate. Google's Open Source Team invited me to send an update about some of the cool stuff we are doing, so I'd like to start by giving you a little background about who we are and what we do.
The FSFE is one of four sister organisations, the other three being the Free Software Foundation (FSF) in the United States, the Free Software Foundation India and the Free Software Foundation Latin America. The FSFE works on European Union and United Nations software policy and we engage with issues like the European software patents debate. We also engage the community on these issues at many European software conferences. The Freedom Task Force focuses on providing education, training, consultancy and legal assistance in answering questions about Free Software licenses. On a day-to-day level this involves attending conferences and local meetings as well as dealing with incoming requests via email from both commercial and non-commercial community members.
One of our main activities during the last two years has been facilitation for the
European Legal Network. This is a group of Free Software legal and technical experts who share knowledge with the community. In the last year and a half we have scaled to cover nineteen European countries (a total of twenty-seven) and to have over one hundred and twenty members. During the month of September alone, we will have two physical meetings to discuss Free Software business processes and software patents. It's quite high level stuff, but we hope to provide participants insights into the resolution of long-term deployment questions and builds on the excellent work of organisations like gpl-violations.org.
Other wonderful news comes from the recent the Akademy 2008 conference, where one of our long-term infrastructure projects came to fruition. KDE e.V. — the organisation representing the KDE project in legal and financial matters — voted to adopt the Fiduciary Licence Agreement (FLA) as an option for its project contributors. The FLA is a legal document that addresses the issue of copyright fragmentation by providing consolidation in common and civil legal systems, and acts as a tool to assist with re-licensing and license enforcement. It's a baby of the FSFE legal project, drafted some years ago for FSFE's own fiduciary programme, and is available for other organisations to adapt for their own use.
That brings me to our biggest news for this week. The FSFE has just announced that we are translating the Fiduciary License Agreement (FLA) into ten languages to provide copyright consolidation options to local projects across Europe. We're looking forward to providing tools to help people manage their code and we believe it's important to at least have reference documents in as many native languages as possible. Such tools are likely to become more important as Free Software grows, and FSFE is proud to make a contribution in this area. Kudos are also due to the kind folks at Google, who made a donation to support both our outreach and translation work earlier in the year.
If you would like to learn more about FSFE's work you can visit our website at www.fsfeurope.org. You might also want to learn more about the legal project specifiically or information about contributing to our efforts. If you want to become part of the FSFE family you might also want to join the Fellowship.
Thanks for reading, Happy Hacking, and I'll see you at the next conference....
My name is Shane Coughlan and and I work for Free Software Foundation Europe (FSFE), a non-profit, non-governmental organisation with offices in Germany, Switzerland, Belgium and Sweden. You may remember me from my last FSFE update on this blog on the Freedom Task Force, which is the FSFE's legal training project which I coordinate. Google's Open Source Team invited me to send an update about some of the cool stuff we are doing, so I'd like to start by giving you a little background about who we are and what we do.
The FSFE is one of four sister organisations, the other three being the Free Software Foundation (FSF) in the United States, the Free Software Foundation India and the Free Software Foundation Latin America. The FSFE works on European Union and United Nations software policy and we engage with issues like the European software patents debate. We also engage the community on these issues at many European software conferences. The Freedom Task Force focuses on providing education, training, consultancy and legal assistance in answering questions about Free Software licenses. On a day-to-day level this involves attending conferences and local meetings as well as dealing with incoming requests via email from both commercial and non-commercial community members.
One of our main activities during the last two years has been facilitation for the
European Legal Network. This is a group of Free Software legal and technical experts who share knowledge with the community. In the last year and a half we have scaled to cover nineteen European countries (a total of twenty-seven) and to have over one hundred and twenty members. During the month of September alone, we will have two physical meetings to discuss Free Software business processes and software patents. It's quite high level stuff, but we hope to provide participants insights into the resolution of long-term deployment questions and builds on the excellent work of organisations like gpl-violations.org.
Other wonderful news comes from the recent the Akademy 2008 conference, where one of our long-term infrastructure projects came to fruition. KDE e.V. — the organisation representing the KDE project in legal and financial matters — voted to adopt the Fiduciary Licence Agreement (FLA) as an option for its project contributors. The FLA is a legal document that addresses the issue of copyright fragmentation by providing consolidation in common and civil legal systems, and acts as a tool to assist with re-licensing and license enforcement. It's a baby of the FSFE legal project, drafted some years ago for FSFE's own fiduciary programme, and is available for other organisations to adapt for their own use.
That brings me to our biggest news for this week. The FSFE has just announced that we are translating the Fiduciary License Agreement (FLA) into ten languages to provide copyright consolidation options to local projects across Europe. We're looking forward to providing tools to help people manage their code and we believe it's important to at least have reference documents in as many native languages as possible. Such tools are likely to become more important as Free Software grows, and FSFE is proud to make a contribution in this area. Kudos are also due to the kind folks at Google, who made a donation to support both our outreach and translation work earlier in the year.
If you would like to learn more about FSFE's work you can visit our website at www.fsfeurope.org. You might also want to learn more about the legal project specifiically or information about contributing to our efforts. If you want to become part of the FSFE family you might also want to join the Fellowship.
Thanks for reading, Happy Hacking, and I'll see you at the next conference....
Hacking Zurich
Monday, September 22, 2008
By Leslie Hawthorn, Open Source Team
Many things wonderful and Open Source are happening in Zurich, Switzerland these days. Most recently, Joël Fisler of the OLAT (Online Learning and Training) project wrote in to let us know about the meetup held at the University of Zurich to celebrate the conclusion of Google Summer of Code™ 2008. Joel writes:
It's not all Summer of Code and barbecue, though. The OpenExpo Conference kicks off this week on Wednesday, September 24th, and our very own Brian "Fitz" Fitzpatrick will be delivering Do You Believe in the Users? on Thursday morning. Finally, some of the organizers of OpenExpo are also spearheading Hackontest, a 24 hour programming competition between teams developing for three Open Source projects as selected by community participants via online voting. Developer teams will work furiously to add features to OpenLieroX, sK1 and TYPO3, with the results of the 24 hour sprint reviewed by a panel of celebrity judges from the Open Source community. Fitz will be on hand to weigh in, as will Google Open Sourcerers Jeremy Allison of the Samba project, Alexander Limi, co-creator of Plone and Bram Moolenaar, author of Vim.
Should you find yourself in the area, please stop in to say hello. If not, Happy Hacking wherever you might find yourself!
Many things wonderful and Open Source are happening in Zurich, Switzerland these days. Most recently, Joël Fisler of the OLAT (Online Learning and Training) project wrote in to let us know about the meetup held at the University of Zurich to celebrate the conclusion of Google Summer of Code™ 2008. Joel writes:
2008 was the first year that the OLAT Learning Management System (LMS) and e-Learning Markup Language (eLML) were accepted into the Summer of Code program. All the students successfully finished their projects and presented their work at our program closing party. After two hours of presentations and discussions — one as a conference call from Germany and two students via video from India — the student and mentors celebrated with barbecue and beer until late at the Irchel campus. The presentations are available online as video streams.
OLAT's 6.1.x release will contain the results of Udit Sajjanhar's iCal Calendar Feed project, as well as Maurus Rohrer's work on an OLAT LDAP authentication module. The code from OLAT's two eLML student projects, Thomas Linowsky's eLML template generator and Alberto Sanz's work on DocBoook conversion for eLML lessons, will be released later this month. We're anticipating that the work done by our other four students, Thomas Bernhart, Sergio Trentini, Marcel Karras and Mayank Jain, will be published in our 6.2 release, and we're already prototyping against their code.
It's not all Summer of Code and barbecue, though. The OpenExpo Conference kicks off this week on Wednesday, September 24th, and our very own Brian "Fitz" Fitzpatrick will be delivering Do You Believe in the Users? on Thursday morning. Finally, some of the organizers of OpenExpo are also spearheading Hackontest, a 24 hour programming competition between teams developing for three Open Source projects as selected by community participants via online voting. Developer teams will work furiously to add features to OpenLieroX, sK1 and TYPO3, with the results of the 24 hour sprint reviewed by a panel of celebrity judges from the Open Source community. Fitz will be on hand to weigh in, as will Google Open Sourcerers Jeremy Allison of the Samba project, Alexander Limi, co-creator of Plone and Bram Moolenaar, author of Vim.
Should you find yourself in the area, please stop in to say hello. If not, Happy Hacking wherever you might find yourself!
Highlights from DjangoCon 2008
Tuesday, September 16, 2008
By Leslie Hawthorn, Open Source Team
Google's Open Source Team recently hosted DjangoCon 2008, the first ever international conference for developers and users of this popular web framework. While we're always pleased to host Open Source events at the Googleplex, we were particularly excited to welcome the Django folks once again. We love Django, making use of it extensively in products like Google App Engine, so it was a pleasure and privilege to give back to this community. More than 250 attendees in Mountain View, California, USA enjoyed two days of technical sessions and celebrated the recent release of Django 1.0, and thanks to the efforts of one of our stalwart Site Reliability Engineers, Rodolpho Eckhardt, we were able to welcome more than 50 additional guests via videoconference with Sao Paulo, Brazil. Better still, attendees raised several hundred dollars for the Django Software Foundation through a charity auction, their donations netting them swag like conference tees signed by Keynote speaker Guido van Rossum and notebooks (the legacy interface variety - paper only) signed by the Django Core Developer Team.
Many thanks to Conference Chair Robert Lofthouse for making DjangoCon 2008 a reality in just a few short months, our attendees for joining us and the entire Django community for both their amazing spirit and great code.
Wish you could have made it? We've got you covered; the conference sessions are now available on YouTube for your perusal.
Ed. Note: Due to technical difficulties, Jacob Kaplan-Moss' Saturday Keynote, The State of Django, was not recorded. Fortunately we've heard from Jacob that he'll be back in the San Francisco Bay Area later this year and will be happy to reprise his talk then. We'll post to this blog when that talk happens and the video is make available.
Google's Open Source Team recently hosted DjangoCon 2008, the first ever international conference for developers and users of this popular web framework. While we're always pleased to host Open Source events at the Googleplex, we were particularly excited to welcome the Django folks once again. We love Django, making use of it extensively in products like Google App Engine, so it was a pleasure and privilege to give back to this community. More than 250 attendees in Mountain View, California, USA enjoyed two days of technical sessions and celebrated the recent release of Django 1.0, and thanks to the efforts of one of our stalwart Site Reliability Engineers, Rodolpho Eckhardt, we were able to welcome more than 50 additional guests via videoconference with Sao Paulo, Brazil. Better still, attendees raised several hundred dollars for the Django Software Foundation through a charity auction, their donations netting them swag like conference tees signed by Keynote speaker Guido van Rossum and notebooks (the legacy interface variety - paper only) signed by the Django Core Developer Team.
Many thanks to Conference Chair Robert Lofthouse for making DjangoCon 2008 a reality in just a few short months, our attendees for joining us and the entire Django community for both their amazing spirit and great code.
Wish you could have made it? We've got you covered; the conference sessions are now available on YouTube for your perusal.
Ed. Note: Due to technical difficulties, Jacob Kaplan-Moss' Saturday Keynote, The State of Django, was not recorded. Fortunately we've heard from Jacob that he'll be back in the San Francisco Bay Area later this year and will be happy to reprise his talk then. We'll post to this blog when that talk happens and the video is make available.
Cmockery: Easier Unit Tests for C Applications
Monday, September 15, 2008
By Justin TerAvest and Stewart Miles, Platforms Engineering Team
We're pleased to announce the initial release of the Open Source project Cmockery, a lightweight library to simplify and generalize the process of writing unit tests for C applications. This testing framework tries to keep testing quick and simple by avoiding complex compiler features such as variadic macros or C99 features that may not be available in older compilers. Secondly, Cmockery doesn't assume the availability of the standard C library in code modules being tested, which makes it useful in code for embedded systems that isn't executed within the snug environment of a typical operating system.
To find out more, check out the code and start writing unit tests for your C applications, please visit the project homepage. We also welcome your feedback in our discussion group.
We're pleased to announce the initial release of the Open Source project Cmockery, a lightweight library to simplify and generalize the process of writing unit tests for C applications. This testing framework tries to keep testing quick and simple by avoiding complex compiler features such as variadic macros or C99 features that may not be available in older compilers. Secondly, Cmockery doesn't assume the availability of the standard C library in code modules being tested, which makes it useful in code for embedded systems that isn't executed within the snug environment of a typical operating system.
To find out more, check out the code and start writing unit tests for your C applications, please visit the project homepage. We also welcome your feedback in our discussion group.
Wrapping our Fourth Summer of Code
Friday, September 12, 2008
By Leslie Hawthorn, Open Source Team
Google Summer of Code™ 2008 concluded on September 2, 2008 and, as usual, we had another resoundingly successful year. Eighty-three percent of our students received passing final evaluations from their mentors and communities, our highest rate of success in all four years running the program. We're excited to bring the community this update, but we're even more excited to invite you to check out the actual source code created by these burgeoning developers; to learn more about each effort, visit each FOSS project's page linked from the program website.
We'll be bringing you more program statistics and wrap up reports from our mentoring organizations in the coming weeks, so stay tuned for more news. We'd like to congratulate all of our student participants and encourage them to continue working with their project communities now that the program has concluded. However, from what we have heard many of them don't need this extra encouragement and are actively continuing development on their Summer of Code work. Finally, our sincerest thanks to all of our mentors for making 2008 the best Summer yet!
Google Summer of Code™ 2008 concluded on September 2, 2008 and, as usual, we had another resoundingly successful year. Eighty-three percent of our students received passing final evaluations from their mentors and communities, our highest rate of success in all four years running the program. We're excited to bring the community this update, but we're even more excited to invite you to check out the actual source code created by these burgeoning developers; to learn more about each effort, visit each FOSS project's page linked from the program website.
We'll be bringing you more program statistics and wrap up reports from our mentoring organizations in the coming weeks, so stay tuned for more news. We'd like to congratulate all of our student participants and encourage them to continue working with their project communities now that the program has concluded. However, from what we have heard many of them don't need this extra encouragement and are actively continuing development on their Summer of Code work. Finally, our sincerest thanks to all of our mentors for making 2008 the best Summer yet!
Highlights from useR!
Thursday, September 11, 2008
By Pete Meyer, Ads Quality Statistics Team
Google was a sponsor of this year's international user conference in Dortmund, Germany for the R statistical programming environment. In addition to sessions focusing on statistical modeling in various areas of applications, the conference included a number of sessions focusing on user interfaces for R and producing high quality output for reports and browsers. There were sessions on high performance computing, including parallelized approaches to R, and new routines for handling massive objects in response to the increasing size and complexity of datasets. Google was among a number of companies who showed examples of how R can be used in production environments as a statistical tool to better understand collected data. We presented some examples (PDF, 348 KB) of how HTML and email output and database access can be integrated into R's statistical environment. The goal is to provide new insights in a familiar format for those who oversee the human evaluation component of Google's advertising quality efforts, without having to generate intermediate datasets.
We're a member of the R community as both users and contributors via Google Summer of Code™. We see the R community as an ally in our mission to organize the world's information and make it universally accessible and useful, and we are glad to be able to support their efforts as a sponsor.
Google was a sponsor of this year's international user conference in Dortmund, Germany for the R statistical programming environment. In addition to sessions focusing on statistical modeling in various areas of applications, the conference included a number of sessions focusing on user interfaces for R and producing high quality output for reports and browsers. There were sessions on high performance computing, including parallelized approaches to R, and new routines for handling massive objects in response to the increasing size and complexity of datasets. Google was among a number of companies who showed examples of how R can be used in production environments as a statistical tool to better understand collected data. We presented some examples (PDF, 348 KB) of how HTML and email output and database access can be integrated into R's statistical environment. The goal is to provide new insights in a familiar format for those who oversee the human evaluation component of Google's advertising quality efforts, without having to generate intermediate datasets.
We're a member of the R community as both users and contributors via Google Summer of Code™. We see the R community as an ally in our mission to organize the world's information and make it universally accessible and useful, and we are glad to be able to support their efforts as a sponsor.
DepAn: A Dependency Analysis Tool
Wednesday, September 10, 2008
By Lee Carver, Software Engineering Team
Has your elegant little application turned into a crushing behemoth, with spaghetti code and a hairball of interdependencies? You know it has got to be simpler, but where to start?
DepAn is an experiment to better visualize and control dependencies in large applications. Once a project grows beyond a few dozen components, simple diagrams rarely capture the essential complexities of the project. Any analysis needs to carefully select the appropriate entities and the different dependencies that exist between them. Worse, these analysis sets change dramatically for different types of analyzes.
Traditional analysis tools tend to focus on a few specific forms of analysis: reference and call dependencies are used to find unused variables, method packaging defines different complexity measures. Other tools, such asUML, are appropriate for design work but present challenges for understanding existing work.
For DepAn, we based the application on a simple, generic model that can feed data into many different kinds of analysis. Projects are modeled as generic dependency graphs, with many different node and relation types. For example, the Java module uses 6 different node types and 24 different relations. These entities provide class-member level details for both packaging and reference analyzes. And since these interactions can be visualized simultaneously, you can see how to clean up your code based on both types of information.
On top of this basic model, we've added a variety of direct manipulation tools to select and display this dependency information in powerful and productive forms. The application's UI framework is built on the Eclipse Rich Client Platform (RCP), providing a powerful and familiar look-and-feel for engineers. Selecting the nodes and redrawing the dependancy graph is done with just a few clicks of the mouse.
Still coming are more supported languages, more powerful analysis metrics, and better tools for restructuring applications. Today,DepAn supports dependency graphs for Java and File-system projects. Soon, we should have additional modules for other popular languages, including C/C++, JavaScript, and Python. Today, the analyzes tab provides simple node level details. Soon, it should provide a variety of graph-level metrics. Today, the restructuring component allows some nodes to be called out. Soon, it should provide full re-allocation of components, and historical comparison of evolving architectures.
We welcome contributions for these and other features! Keep tabs on our progress and let us know what you think on the DepAn mailing list.
Has your elegant little application turned into a crushing behemoth, with spaghetti code and a hairball of interdependencies? You know it has got to be simpler, but where to start?
DepAn is an experiment to better visualize and control dependencies in large applications. Once a project grows beyond a few dozen components, simple diagrams rarely capture the essential complexities of the project. Any analysis needs to carefully select the appropriate entities and the different dependencies that exist between them. Worse, these analysis sets change dramatically for different types of analyzes.
Traditional analysis tools tend to focus on a few specific forms of analysis: reference and call dependencies are used to find unused variables, method packaging defines different complexity measures. Other tools, such asUML, are appropriate for design work but present challenges for understanding existing work.
For DepAn, we based the application on a simple, generic model that can feed data into many different kinds of analysis. Projects are modeled as generic dependency graphs, with many different node and relation types. For example, the Java module uses 6 different node types and 24 different relations. These entities provide class-member level details for both packaging and reference analyzes. And since these interactions can be visualized simultaneously, you can see how to clean up your code based on both types of information.
On top of this basic model, we've added a variety of direct manipulation tools to select and display this dependency information in powerful and productive forms. The application's UI framework is built on the Eclipse Rich Client Platform (RCP), providing a powerful and familiar look-and-feel for engineers. Selecting the nodes and redrawing the dependancy graph is done with just a few clicks of the mouse.
Still coming are more supported languages, more powerful analysis metrics, and better tools for restructuring applications. Today,DepAn supports dependency graphs for Java and File-system projects. Soon, we should have additional modules for other popular languages, including C/C++, JavaScript, and Python. Today, the analyzes tab provides simple node level details. Soon, it should provide a variety of graph-level metrics. Today, the restructuring component allows some nodes to be called out. Soon, it should provide full re-allocation of components, and historical comparison of evolving architectures.
We welcome contributions for these and other features! Keep tabs on our progress and let us know what you think on the DepAn mailing list.
Google Releases More Patches for MySQL
Monday, September 8, 2008
By Mark Callaghan, Software Engineering Team
Did you know that Google uses MySQL as part of its Ads system? As you can imagine, we demand a lot from this Open Source code base and so we have spent a fair amount of time enhancing it to work better in our massively scaled environment. In the past, we have published several patches and today we have a few more to offer. We expect several of these features to be merged into a future official MySQL release, and one of them, semi-synchronous replication, is already available as a MySQL feature preview.
All of the features in the patch are described on our project wiki. The features include:
Did you know that Google uses MySQL as part of its Ads system? As you can imagine, we demand a lot from this Open Source code base and so we have spent a fair amount of time enhancing it to work better in our massively scaled environment. In the past, we have published several patches and today we have a few more to offer. We expect several of these features to be merged into a future official MySQL release, and one of them, semi-synchronous replication, is already available as a MySQL feature preview.
All of the features in the patch are described on our project wiki. The features include:
- enhancements and bug fixes for features from the previous patch
- changes to make InnoDB run faster on multi-core servers
- changes to display mutex contention statistics
- changes to monitor and rate-limit activity by database account and client IP
- a patch for MySQL 5.0.37 with all of our changes
- a patch for MySQL 5.1.26 with the changes for mutex contention statistics
- a patch for MySQL 5.0.67 to make InnoDB run faster on multi-core servers
Second Annual LLVM Developers' Meeting
Friday, September 5, 2008
By Ted Kremenek, LLVM Developer
At the beginning of August 2008 we had the second annual Low Level Virtual Machine (LLVM) Developers' Meeting at Apple's main campus in Cupertino, California, USA. LLVM is a collection of libraries and tools that make it easy to build compilers, optimizers, Just-In-Time code generators, and many other compiler-related programs. LLVM is currently being used for both a wide variety of academic research projects and commercial applications.
More than 100 participants attended 12 technical presentations (slides and videos available) on LLVM and its applications.
Highlights from the meeting included talks on Clang, a new, multipurpose frontend for C-based languages, the LLVM optimizer, code generator, and JIT component, and various applications of LLVM.
This event would not been possible without the generous support of Google, who funded the travel expenses for several students (Nicolas Geoffray, Nuno Lopes, Anton Korobeynikov, Tim Sander) who all presented their LLVM-related work at the meeting. All of them were kind enough to send us some notes about their experience at the developer meeting!
Nicolas Geoffray writes:
Nuno Lopes writes:
Anton Korobeynikov writes:
Tim Sander writes:
Many thanks to Anton, Nicolas, Nuno, and Tim for sharing their experiences, and for the generous support of Google's Open Source Programs Office in making this event possible!
At the beginning of August 2008 we had the second annual Low Level Virtual Machine (LLVM) Developers' Meeting at Apple's main campus in Cupertino, California, USA. LLVM is a collection of libraries and tools that make it easy to build compilers, optimizers, Just-In-Time code generators, and many other compiler-related programs. LLVM is currently being used for both a wide variety of academic research projects and commercial applications.
More than 100 participants attended 12 technical presentations (slides and videos available) on LLVM and its applications.
Highlights from the meeting included talks on Clang, a new, multipurpose frontend for C-based languages, the LLVM optimizer, code generator, and JIT component, and various applications of LLVM.
This event would not been possible without the generous support of Google, who funded the travel expenses for several students (Nicolas Geoffray, Nuno Lopes, Anton Korobeynikov, Tim Sander) who all presented their LLVM-related work at the meeting. All of them were kind enough to send us some notes about their experience at the developer meeting!
Nicolas Geoffray writes:
At the LLVM Developers' Meeting I presented VMKit, an implementation of both the JVM and a CLI Virtual Machines (Microsoft .NET is an implementation of the CLI) using the Just-In-Time compiler of LLVM. It runs regular Java and C# applications by translating instructions to LLVM's internal representation and executing the code LLVM generates dynamically. (slides, video)
Besides LLVM, VMKit uses existing projects such as the Boehm garbage collector or the GNU Classpath libraries to create complete instances of virtual machines. In essence, VMKit is a runtime engine which dynamically loads and lookups classes, methods, and fields.
My talk described how VMKit combines components from various systems to create complete virtual machines, and how the translation from bytecode to LLVM instructions works. After giving the current performance status of the system, it discusses areas for future work in VMKit as well as in LLVM.
About Nicolas: Nicolas is a Ph.D. student in computer science at the University of Pierre et Marie Curie, France. Nicolas was a Google Summer of Code™ (GSoC) mentor this year for both LLVM and VMKit.
Nuno Lopes writes:
In August, I attended the 2nd LLVM conference, where I gave a talk on "Building a JIT compiler for PHP in 2 days." This was made possible by Google, who sponsored the trip. Apple (in particular Ted & Tanya) organized the conference. A big thank to you all for making this possible!
Attending the conference was such a great experience: I was finally able to meet all those guys that I usually discuss with via e-mail and IRC! This is also good to strengthen the relationship between the developers. I learned a lot from the talks I've attended and also from the nice discussions I've joined. As a student it's also very interesting to meet and chat with people from corporations, to get some feeling of what is being done in the "real world" and how they are doing it. I was amazed with the big amount of interest in LLVM from big companies.
My talk's room was near full, which was a nice surprise! I was delighted to share our experience in making a simple JIT compiler with LLVM in a matter of days. The project was started just this year with the participation of a Summer of Code student and it is already able to pass the whole PHP test suite. It also performs good on benchmarks, but in the next months I'm sure we will see huge improvements.
I hope others can follow the idea and do a JIT compiler for their own interpreter of choice. What are you waiting for? Go do your new JIT compiler! :-)
You can take a look at the source code of the PHP JIT compiler (currently in CVS only).
Finally I would like to thank all the LLVM developers for producing such a great piece of software. It's the easiest, though very powerful, compiler I've ever hacked.
About Nuno: Nuno is pursuing his Masters degree in Information Systems and Computer Engineering at the Technical University of Lisbon, Portugal. He was a GSoC mentor for the PHP project in 2008.
Anton Korobeynikov writes:
The LLVM Compiler Infrastructure project grew up rapidly during last few years. And I think an idea of developers' meeting was really nice and appeared in proper time. It was second LLVM Developer's Meeting had been held in the August, 1. It seems, that number of people attended the meeting was as twice as comparable with the previous year. I think it's really good indicator for the LLVM Project.
The schedule of the meeting was rich. It was divided into two parts: talks about project itself and talks about cool and crazy ways to use LLVM in applications. Project-related talks covered huge variety of different themes, starting from new C and Objective-C frontend and how different interesting tools can be built on top of it (for example - static code checker). There were some talks describing internal structure of the LLVM target-independent code generator, LLVM's JIT compiler component, etc. Secondly, there were talks about cool uses of LLVM. We started from Adobe After Affects, passed through Java/.NET machine on top of LLVM, code generation for FPGAs, secure memory architecture built on top of LLVM, and ended with Quake running in the web browser! Sometimes applications are really crazy. :)
I doubt I would have been able to attend the meeting without generous support from the meeting sponsors. So, just 16 hours in planes and I'm in the Pacific Time with talk ready to show. I prefer to work on different backend-related bits in LLVM, but despite of that, my talk was about new LLVM component added to recent 2.3 release: the new LLVM compiler driver, llvmc2 (slides, video). It was the project I mentored and did the overall design. A compiler driver is sort of a 'dispatcher' tool, which knows how to execute different tools like the compiler itself, the assembler, and linker in order to produce a final binary from the sources provider. It's a small but really important tool because it needs to know about all different options of each tool involved in the compilation process, manage temporary files, arguments passing, etc. Our aim was to make such compiler driver universal, flexible and easily reconfigurable.
I found the meeting really well organized, so I would like to thank everyone involved and I'm definitely sure next meetings will be as good as this one!
About Anton: Anton is a Ph.D. student in applied statistics at Saint Petersburg State University, Russia. Anton is one of the principal code owners for LLVM as well as LLVM's project adminstrator for this year's GSoC.
Tim Sander writes:
The LLVM Developer Conference at the beginning of August was my first trip to the famous Silicon Valley. Without sponsorship from Google, it would have not been possible. It was a very interesting experience.
The most interesting event was of course the LLVM Developer Meeting itself. I really liked the balance between talks and the possibility to do some conversation in the breaks. Of course my talk about the hardware/software codesign in LLVM is also available for download (slides, video). Currently I am busy working on a bigger example for the codesign backend and some hopefully much improved HW-backend stuff. So probably one fine day these changes might find their way into the LLVM repository.
Besides all this technical stuff I also really enjoyed the trip with all the LLVM illuminates to San Francisco where we did a little bit of sightseeing and had some more time for more or less technical discussions. I also took took the opportunity to take a look at the Bay Area and did a little bit of hiking through the beautiful landscape. Beside all this LLVM related stuff I really found it really interesting to visit the USA for the first time in more than 10 years. It's always very interesting to see the "American way of life."
About Tim: Tim is a Ph.D. student at ICS, Technische Universität Darmstadt.
Many thanks to Anton, Nicolas, Nuno, and Tim for sharing their experiences, and for the generous support of Google's Open Source Programs Office in making this event possible!
Testing the CPython Core
Thursday, September 4, 2008
By Leslie Hawthorn, Open Source Team
When Guido van Rossum mentioned that one of our Google Highly Open Participation Contest™ (GHOP) students, Benjamin Peterson, was interested in doing a more extensive project on Python testing, we were happy to help. Benjamin Peterson recently finished his work on testing the CPython core and was kind enough to send us this report.
Benjamin writes:
And in other news Python and GHOP, Zachary Voase, another Python GHOP student, recently attended Campus Party in Valencia, Spain, and spoke about his experiences in the contest. You can check out the video of Zack, along with some of our Google Summer of Code™ students, on YouTube. (Campus Party site and video are both in Spanish.)
When Guido van Rossum mentioned that one of our Google Highly Open Participation Contest™ (GHOP) students, Benjamin Peterson, was interested in doing a more extensive project on Python testing, we were happy to help. Benjamin Peterson recently finished his work on testing the CPython core and was kind enough to send us this report.
Benjamin writes:
With sponsorship from Google, I've just spent the summer working on the tests for the CPython core.
I spent a lot of time working on the test driver, regrtest.py, in order to make implementing features easier in the future. One of the ugliest parts of regrtest.py is that it kept a list of all the tests that could be expected to be skipped on a platform, so I moved this logic to the tests themselves. In the end, I wasn't able to fully remove Python core related logic from it, but the removal of expected skip lists simplified the program greatly.
I made command line use of regrtest easier in several ways. When specifying tests, one no longer needs to give the test_ prefix. I also implemented test selectors which allow a range of tests to be specified. For these new features, I added unittests. Another major feature is that regrtest automatically writes a list of the tests run, with failures and skips noted, after each testing session. The session can be rerun exactly the same way on another computer by reading the file with the -f option.
Finely grained test skipping is another important feature. New decorators in test_support allow whole test cases and individual methods to be skipped. Decorators range from the generic skip_if, which skips when the condition passed to it is true, to skip_on_platform, which skips if any of the listed platforms are in sys.platform. I hope that some form of these decorators can be included in unittest eventually.
I visited each of the 361 tests to upgrade it to use my new features. I was able to consolidate some tests with each other. A few tests that escaped earlier sprints were converted to unittest.
I really had a fantastic experience this summer. I learned an incredible amount from nice, patient people and look forward to next year! Many thanks to Georg Brandl, my awesome mentor, Guido van Rossum and the rest of the Python core development team, and Google's Open Source Programs Office for making this possible.
And in other news Python and GHOP, Zachary Voase, another Python GHOP student, recently attended Campus Party in Valencia, Spain, and spoke about his experiences in the contest. You can check out the video of Zack, along with some of our Google Summer of Code™ students, on YouTube. (Campus Party site and video are both in Spanish.)
open-vcdiff: An Encoder and Decoder for RFC 3284
Wednesday, September 3, 2008
By Lincoln Smith, Software Engineering Team
Anyone interested in building a faster Web?
We are delighted to announce the initial release of open-vcdiff, an encoder and decoder for the draft-standard format described in RFC 3284: "The VCDIFF Generic Differencing and Compression Data Format."
The VCDIFF format can be used as part of an implementation of RFC 3229, "Delta encoding in HTTP", or to produce compact delta representations for revision control systems. Expressing each successive version of a file as a delta from the previous version can save disk space and network bandwidth.
open-vcdiff is an important building block for implementing Shared-Dictionary Compression over HTTP (also known as SDCH or "Sandwich".) The SDCH protocol was initially proposed at the Velocity 2008 Web Performance and Operations Conference last June, and is soon to be released for review. Its goal is to compress HTTP responses and so provide a faster browsing experience for low-bandwidth users. If this prospect interests you, SDCH has its own discussion group on Google Groups. VCDIFF was chosen as the compression format for SDCH because it is a published standard.
The open-vcdiff team welcomes discussion, comments, and code contributions. Please join the open-vcdiff discussion group to expand and enrich the community of open-vcdiff developers.
Anyone interested in building a faster Web?
We are delighted to announce the initial release of open-vcdiff, an encoder and decoder for the draft-standard format described in RFC 3284: "The VCDIFF Generic Differencing and Compression Data Format."
The VCDIFF format can be used as part of an implementation of RFC 3229, "Delta encoding in HTTP", or to produce compact delta representations for revision control systems. Expressing each successive version of a file as a delta from the previous version can save disk space and network bandwidth.
open-vcdiff is an important building block for implementing Shared-Dictionary Compression over HTTP (also known as SDCH or "Sandwich".) The SDCH protocol was initially proposed at the Velocity 2008 Web Performance and Operations Conference last June, and is soon to be released for review. Its goal is to compress HTTP responses and so provide a faster browsing experience for low-bandwidth users. If this prospect interests you, SDCH has its own discussion group on Google Groups. VCDIFF was chosen as the compression format for SDCH because it is a published standard.
The open-vcdiff team welcomes discussion, comments, and code contributions. Please join the open-vcdiff discussion group to expand and enrich the community of open-vcdiff developers.
Google Chrome: Our Fresh Take on the Browser
Tuesday, September 2, 2008
By Chris DiBona, Open Source Team
You may have already seen yesterday's post mentioning Google Chrome, an Open Source browser now available in more than 40 languages. Today we're pleased to say that the source code awaits your perusal, feedback and contributions. You can download Chrome and check out the documentation at http://www.google.com/chrome, or take a look at the code base at the Chromium home page. For more details directly from the developer team, check out the Chromium Blog.
We look forward to working with all of our colleagues in the Open Source community and users worldwide to make browsing a simpler, faster and more fun experience. We always love to hear your thoughts, so let us know what you think in the comments section.
You may have already seen yesterday's post mentioning Google Chrome, an Open Source browser now available in more than 40 languages. Today we're pleased to say that the source code awaits your perusal, feedback and contributions. You can download Chrome and check out the documentation at http://www.google.com/chrome, or take a look at the code base at the Chromium home page. For more details directly from the developer team, check out the Chromium Blog.
We look forward to working with all of our colleagues in the Open Source community and users worldwide to make browsing a simpler, faster and more fun experience. We always love to hear your thoughts, so let us know what you think in the comments section.