OpenCensus: A Stats Collection and Distributed Tracing Framework

Today we’re pleased to announce the release of OpenCensus, a vendor-neutral open source library for metric collection and tracing. OpenCensus is built to add minimal overhead and be deployed fleet wide, especially for microservice-based architectures.

The Need for Instrumentation & Observability 

As a startup, often the focus is to get an initial version of the product out the door, rapidly prototype and iterate with customers. Most startups start out with monolithic applications as a simple model-view-controller (MVC) web application. As the customer base, code, and number of engineers increase, they migrate from monolithic architecture to a microservices architecture. A microservices architecture has its advantages, but often makes debugging more challenging as traditional debugging and monitoring tools don’t always work in these environments or are designed for monolithic use cases. When operating multiple microservices with strict service level objectives (SLOs), you need insights into the root cause of reliability and performance problems.

Not having proper instrumentation and observability can result in lost engineering hours, violated SLOs and frustrated customers. Instead, diagnostic data should be collected from across the stack. This data can be used for incident management to identify and debug potential bottlenecks or for system tuning and performance improvement.

OpenCensus

At Google scale, an instrumentation layer with minimal overhead is a requirement. As Google grew, we realized the importance of having a highly efficient tracing and stats instrumentation library that could be deployed fleet wide.

OpenCensus is the open source version of Google’s Census library, written based on years of optimization experience. It aims to make the collection and submission of app metrics and traces easier for developers. It is a vendor neutral, single distribution of libraries that automatically collects traces and metrics from your app, displays them locally, and sends them to analysis tools. OpenCensus currently supports Prometheus, SignalFx, Stackdriver and Zipkin.

Developers can use this powerful, out-of-the box library to instrument microservices and send data to any supported backend. For an Application Performance Management (APM) vendor, OpenCensus provides free instrumentation coverage with minimal work, and affords customers a simple setup experience.

Below are Stackdriver Trace and Monitor screenshots showing traces generated from a demo app, which calls Google’s Cloud Bigtable API and uses OpenCensus.

We’d love to hear your feedback on OpenCensus. Try using it in your app, tell us about your success story, and help by contributing to our existing language-specific libraries, or by creating one for an not-yet-supported language. You can also help us integrate OpenCensus with new APM tools!

We hope you find this as useful as we have. Visit opencensus.io for more information.

By Pritam Shah, Census team

XRay: a function call tracing system

At Google we spend a lot of time debugging and tuning the performance of our production systems. Some standard practices when doing this involves using profilers, debuggers, and analysis of logs and execution traces. Doing this at scale, in production, is difficult. One of the ways for getting high fidelity data from production systems is to build applications with instrumentation, and then reconstruct the instrumentation data into a form humans can consume (summary statistics, reports, etc.). Instrumentation comes at a cost though, sometimes too high to make it feasible to deploy in production.

Getting this balance right is hard. This is why we've developed XRay, a function call tracing system that has very little overhead when not enabled, but can be dynamically turned on and only impose moderate costs. XRay works as a combination of compiler-inserted instrumentation points which functionally do nothing (called "nop sleds") and a library that can be enabled and disabled at runtime which replaces the nop sleds with the appropriate instrumentation instructions.

We've been using XRay to debug internal systems, from core infrastructure services like Bigtable to ad serving systems. XRay's detailed function tracing has enabled several teams in Google to debug issues that would be really hard to solve without XRay.

We think XRay is an important piece of technology, not only at Google, but for developers around the world. It's because of this that we're working on making XRay opensource. To kick-start that process, we're releasing a white paper describing the technical details of XRay. In the following weeks, we will be engaging the LLVM community, where we are committed to making XRay available for wide use and distribution.

We hope that by open-sourcing XRay we can contribute to the advancement of debugging real-world applications. We're looking forward to working with the LLVM community and other projects to make the data XRay generates useful for debugging a wide variety of applications.

By Dean Michael Berris, Google Engineering