Open source software is used pervasively, with some estimating that over 90% of codebases contain open source software. Many adopt open source software because it’s free and thus cheaper than writing a proprietary alternative. But as the community saying goes: “open source is free as in puppies, not free as in beer.” Once you or your organization has adopted an open source project, there is still effort (such as engineering time, costs, etc.) associated with ensuring that your version of an open source project is healthy and up to date.
While efforts by the research community have investigated the ongoing costs required to maintain open source software, at Google we needed to better understand the resources required to use thousands of open source software packages.
To better understand this cost, we designed a study to evaluate the effort (in software engineering hours) associated with maintaining open source software packages within Google. We focused on the following questions:
RQ1: How long does it take for an engineer (not a maintainer of the upstream project) to update an open source software package at Google?
RQ2: Can complexity metrics predict the relative effort/amount of time to update a package at Google?
Customization incurs additional cost
The majority of open source package updates (at Google) take four hours or less. However, the long tail could be days to months to fully land a package update. We needed to understand what makes the difference between hours, days, and months.
Our results suggested metrics that could be better predictors of update complexity than others. For example, local patches and customization, the number of package dependencies, and the number of upstream contributors, were better indicators of update complexity than the age of the package, or how many teams were using that package internally.
Study limitations and future work
As this study was completed at Google, we acknowledge there are bias and limitations of our findings, including the following:
- Business priority-based sampling: While rigorous, generalizable research studies thoroughly interrogate samples to minimize bias and maximize analytic potential, we designed this study to focus on existing data and minimize new data collection which could impact engineering roadmaps.
- We did not dictate tooling: We did not specify what tools (including genAI-enabled features) could or could not be used in the update process, and as such did not investigate the impact of any specific tool.
- Bias towards Go: Our collected data had significant bias towards Go. Given the lack of variety in our sample, we could not investigate the effect of language ecosystems, but other internal efforts indicated that language may significantly impact maintenance efforts. We encourage this line of investigation for future studies.
- Only active projects: This effort did not include significant data collection on stale or archived packages.
Curious about our approach and findings? Read the full research report: “How much does it cost to maintain a copy of an open source software package (at Google)?”
