Improving Developer Experience for Writing Structured Data

Though we’re still waiting on the full materialization of the promise of the Semantic Web, search engines—including Google—are heavy consumers of structured data on the web through Schema.org. In 2015, pages with Schema.org markup accounted for 31.3% of the web. Among SEO communities, interest in Schema.org and structured data has been on the rise in recent years.

Yet, as the use of structured data continues to grow, the developer experience in authoring pieces of structured data remains spotty. I ran into this as I was trying to write my own snippets of JSON-LD. It turns out, the state-of-the-art way of writing JSON-LD is to: read the Schema.org reference; try writing a JSON literal on your own; when you think you’re done, paste the JSON into a validator (like Google’s structured data testing tool); see what’s wrong, fix; and repeat, as needed.

If it’s your first time writing JSON-LD, you might spend a few minutes figuring out how to represent an enum or boolean, looking for examples as needed.

Enter schema-dts

My experience left me with a feeling that things could be improved; writing JSON-LD should be no harder than any JSON that is constrained by a certain schema. This led me to create schema-dts (npm, github) a TypeScript-based library (and an optional codegen tool) with type definitions of the latest Schema.org JSON-LD spec.

The thinking was this: Just as IDEs (and, later, language server protocols for lightweight code editors) supercharge our developer experience with as-you-type error highlighting and code completions, we can supercharge the experience of writing those JSON-LD literals.

With IDEs and language server protocols, the write-test-debug loop was made much tighter. Developers get immediate feedback on the basic correctness of the code they write, rather than having to save sporadically and feed their code to a compiler for that feedback. With schema-dts, we try to take validators like the structured data testing tool out of the critical path of write-test-debug. Instead, you can use a library to type-check your JSON, reporting errors as you type, and offering completions for `@type`s, property names, and their values.


Thanks to TypeScript’s structural typing and discriminated unions, the general shape of Schema.org’s JSON-LD can be well-represented in TypeScript typings. I have previously described the type theory behind creating a TypeScript structure that expresses the Schema.org class structure, enumerations, `DataType`s, and properties.

Schema-dts includes two related pieces: the ‘default’ schema-dts NPM package which includes the latest Schema.org definitions, and the schema-dts-gen CLI which allows you to create your own typing definitions from Schema.org-like .nt N-Triple files. The CLI also has flags to control whether deprecated classes, properties, and enums should be included, what `@context` should be assumed by objects you write, etc.

Goals and Non-Goals

The goal of schema-dts isn’t to make type definitions that accept all legal Schema.org JSON literals. Rather, it is to make sure we provide typings that always (or almost always) result in legal Schema.org JSON-LD literals that search engines would accept. In the process, we’d like to make sure it’s as general as possible, without sacrificing type checking and useful completions.

For instance, RDF’s perspective is that structured data is property-centric, and the Schema.org reference of the domains and ranges of properties is only a suggestion for what values are inferred as. RDF actually permits values of any type to be assigned to a property. Instead, schema-dts will actually constrain you by the Schema.org values.

***

If you’re passionate about structured data, try schema-dts and join the conversation on GitHub!

By: Eyas Sharaiha, Geo Engineering & Open Source scheme-dts Project

Jsonnet: a more elegant language for composing JSON

A few months ago, we quietly released Jsonnet: a simple yet rich configuration language (i.e., a programming language for specifying data). Many systems can be configured with JSON, but writing it by hand is troublesome. Jsonnet is packed with useful data-specification features that expand into JSON for other systems to act upon. Below is a trivial example of such expansion:

// Jsonnet Example {    person1: {        name: "Alice",        welcome: "Hello " + self.name + "!",    },    person2: self.person1 { name: "Bob" }, }

➡

{   "person1": {      "name": "Alice",      "welcome": "Hello Alice!"   },   "person2": {      "name": "Bob",      "welcome": "Hello Bob!"   } }

Jsonnet doesn’t just generate JSON: Jsonnet is also an extension of JSON. By adding new constructs between the gaps of existing JSON syntax, Jsonnet adds useful features without breaking backwards compatibility. Any valid JSON is also a valid Jsonnet program that simply emits that JSON unchanged, and existing systems that consume JSON (or its cousin YAML) can be easily modified to accept data in the full Jsonnet language. As such, Jsonnet is an example of a templating language, but one specifically designed for JSON data and less error-prone than other techniques.

“Jsonnet” is a portmanteau of JSON and sonnet. We chose that name to convey that data expressed in Jsonnet is easier to write and maintain because it is more elegant and concise, like a poem. This is not just due to syntactic niceties like comments and permissive quotes/commas, but because Jsonnet has all the modern multi-paradigm programming language conveniences needed to manage complexity. One key benefit is the ability to use Jsonnet's mixin and import features to write modular configuration template libraries, allowing the creation of domain-specific configuration languages for particular applications.

Most configuration languages are created ad-hoc for the needs of a given application, accruing features over time and becoming unwieldy. From day one, Jsonnet was designed as a coherent programming language, benefitting from both academic techniques and our experience implementing production languages. Unlike most configuration languages, Jsonnet has a full operational semantics, ensuring matching behavior from third party implementations as well as mathematical analysis. It is a very small and carefully chosen extension to JSON that can express both object-oriented and declarative styles. More importantly, unlike regular programming languages, Jsonnet is hermetic:  Its evaluation is independent of any implicit environmental factors, ensuring that high level configuration will resolve to the same thing every time.

Jsonnet is open source. It’s currently available as a library with C and Python bindings, and also as a command line utility. A real-world example configuration can be found on the website, where 217 lines (9.7kB) of Jsonnet expand into 740 lines (25kB) of configuration for other tools. Learn more about Jsonnet by reading the tutorial and experimenting with our javascript demo!

by Dave Cunningham, New York Technical Infrastructure team

FlatBuffers 1.1: a memory-efficient serialization library

After months in development, the FlatBuffers 1.1 update is here. Originally released in June 2014, it’s a highly efficient open source cross-platform serialization library that allows you to read data without parsing/unpacking or allocating additional memory. It supports schema evolution (forwards/backwards compatibility) and optional JSON conversion. We primarily created it for games written in C++ where performance is critical, but it’s also useful more broadly. This update brings:

• an extensive overhaul to the Java API
• out-of-the-box support for C# and Go
• an optional verifier to make FlatBuffers practical in untrusted scenarios
• .proto parsing for easier migration from Protocol Buffers
• optional manual assignment of field IDs
• dictionary functionality through binary search on a key field
• bug fixes and other improvements thanks to 200+ commits from 28 contributors -- thank you!

Download the latest release from our github page and join our discussion list for more details.

By Wouter van Oortmerssen, Fun Propulsion Labs at Google*

*Fun Propulsion Labs is a team within Google that's dedicated to advancing gaming on Android and other platforms.

FlatBuffers: a memory efficient serialization library

Today, we are releasing FlatBuffers, a C++ serialization library that allows you to read data without unpacking or allocating additional memory, as an open source project.

FlatBuffers stores serialized data in buffers in a cross-platform way, supporting format evolution that is fully forwards and backwards compatible through a schema. These buffers can be stored in files or sent across the network as-is, and accessed in-place without parsing overhead.

The FlatBuffers schema compiler and runtime is written in platform independent C++ with no library dependencies outside the STL, which makes it possible to use on any platform that has a C++ compiler. We have provided methods to build the FlatBuffers library, example applications, and unit tests for Android, Linux, OSX and Windows.

The schema compiler can generate code to read and write FlatBuffers binary files for C++ and Java. It can additionally parse JSON-formatted data into type-safe binaries.

Game developers can use this library to store game data with less overhead than alternative solutions (e.g. Protocol Buffers or JSON).  We’re excited about the possibilities, and want to hear from you about how we can make this even better!

Download the latest release from our github page and join our discussion list!

By Wouter van Oortmerssen, Fun Propulsion Labs at Google*

*Fun Propulsion Labs is a team within Google that's dedicated to advancing gaming on Android and other platforms.