Jsonnet: a more elegant language for composing JSON

A few months ago, we quietly released Jsonnet: a simple yet rich configuration language (i.e., a programming language for specifying data). Many systems can be configured with JSON, but writing it by hand is troublesome. Jsonnet is packed with useful data-specification features that expand into JSON for other systems to act upon. Below is a trivial example of such expansion:

// Jsonnet Example {    person1: {        name: "Alice",        welcome: "Hello " + self.name + "!",    },    person2: self.person1 { name: "Bob" }, }

➡

{   "person1": {      "name": "Alice",      "welcome": "Hello Alice!"   },   "person2": {      "name": "Bob",      "welcome": "Hello Bob!"   } }

Jsonnet doesn’t just generate JSON: Jsonnet is also an extension of JSON. By adding new constructs between the gaps of existing JSON syntax, Jsonnet adds useful features without breaking backwards compatibility. Any valid JSON is also a valid Jsonnet program that simply emits that JSON unchanged, and existing systems that consume JSON (or its cousin YAML) can be easily modified to accept data in the full Jsonnet language. As such, Jsonnet is an example of a templating language, but one specifically designed for JSON data and less error-prone than other techniques.

“Jsonnet” is a portmanteau of JSON and sonnet. We chose that name to convey that data expressed in Jsonnet is easier to write and maintain because it is more elegant and concise, like a poem. This is not just due to syntactic niceties like comments and permissive quotes/commas, but because Jsonnet has all the modern multi-paradigm programming language conveniences needed to manage complexity. One key benefit is the ability to use Jsonnet's mixin and import features to write modular configuration template libraries, allowing the creation of domain-specific configuration languages for particular applications.

Most configuration languages are created ad-hoc for the needs of a given application, accruing features over time and becoming unwieldy. From day one, Jsonnet was designed as a coherent programming language, benefitting from both academic techniques and our experience implementing production languages. Unlike most configuration languages, Jsonnet has a full operational semantics, ensuring matching behavior from third party implementations as well as mathematical analysis. It is a very small and carefully chosen extension to JSON that can express both object-oriented and declarative styles. More importantly, unlike regular programming languages, Jsonnet is hermetic:  Its evaluation is independent of any implicit environmental factors, ensuring that high level configuration will resolve to the same thing every time.

Jsonnet is open source. It’s currently available as a library with C and Python bindings, and also as a command line utility. A real-world example configuration can be found on the website, where 217 lines (9.7kB) of Jsonnet expand into 740 lines (25kB) of configuration for other tools. Learn more about Jsonnet by reading the tutorial and experimenting with our javascript demo!

by Dave Cunningham, New York Technical Infrastructure team

Gumbo: A C library for parsing HTML

We're pleased to announce the open source release of the Gumbo HTML parser, a C implementation of the HTML5 parsing algorithm.

One of the big accomplishments of the HTML5 standard was to standardize the HTML parsing algorithm, so that all browsers see the same HTML document in the same way. So far, most implementations of this algorithm have either been tied to specific browsers or rendering engines, or they've been written in specific scripting languages. This makes it hard to write quick one-off tools to manipulate and cleanup HTML if you don't happen to be working in a language that already has an HTML5-compatible parsing library.

Gumbo seeks to provide a simple library that can serve as a basic building block for linters, refactoring tools, templating languages, page analysis, and other small programs that need to manipulate HTML. It's written in pure C for ease of interfacing with other languages, and has no outside dependencies. Gumbo was built from the start to support source locations and correlating nodes in the parse tree with positions in the original text.

For more information including download, installation, and usage instructions, please visit the Gumbo project page.

By Jonathan Tang, Search Features team

distcc's pump mode: A New Design for Distributed C/C++ Compilation

By Nils Klarlund, Research Scientist - Build Tools Team

For a while now, Google has been using distcc, a distributed C/C++ compilation system, to speed up building software made of millions of lines of code. With distcc, we can build code an order of magnitude faster than we could if everyone had to compile on their own workstation. But even with distcc, compiles could take a long time: compiling the Google Webserver might take 20 minutes. We started looking at distcc to see if we could make it even faster.

We're proud to report that we've succeeded: we've developed an algorithm we call "pump mode", which can be added to distcc to speed it up by a factor of 3. Pump mode works by pushing even more processing onto the servers. Based on an incremental static analysis of the source code, pump mode is able to quickly identify the sets of files needed for the preprocessing phase of compiling C/C++ programs and send them to the compilation servers for preprocessing. This achieves a dramatic decrease in the CPU load of the workstation and of course much better build speed. We have tested pump mode on some open source software and seen improvements in build speed between 50% (the Linux kernel) and 200% (Samba). With simple changes to the project Makefiles, most projects we have looked at would be even faster!

The pump mode extension has been Google's main C/C++ build system for over a year now.

Distcc's pump mode was developed by a small team at Google that included myself, Manos Renieris, Fergus Henderson, and Craig Silverstein. The pump mode extension complements the recently released open source gold linker, which addresses the other basic bottleneck for fast building of C/C++ software.

Distcc's pump mode is included in release 3.0 of distcc. This is the first release since 2004 when Martin Pool, the original author of the code base, released version 2.18.3. Distcc 3.0 contains many other contributions from a variety of contributors, including Avahi Zeroconf support by Lennart Poettering, "lsdistcc" by Dan Kegel, and bug fixes and portability improvements by Nadim Khemir, Maks Verver, Niklaus Giger, Sascha Demetrio, Alex Besogonov, Ben Skeggs, Lisa Seelye, Lei Zhang, Michael Moss, Dongmin Zhang, and others. Disctcc is now maintained by Fergus Henderson.