opensource.google.com

Menu

Posts from April 2020

Cloud Spanner Emulator

Thursday, April 30, 2020

After Cloud Spanner’s launch in 2017, there has been huge customer adoption across several different industries and verticals. With this growth, we have built a large community of application developers using Cloud Spanner. To make the service even more accessible and open to the broader developer community, we are introducing an offline emulator for the Cloud Spanner service. The Cloud Spanner emulator is intended to reduce application development cost and improve developer productivity for the customers.

The Cloud Spanner emulator provides application developers with the full set of APIs, including the breadth of SQL and DDL features that could be run locally for prototyping, development and testing. This open source emulator will provide application developers with the transparency and agility to customize the tool for their application use.

This blog introduces the Cloud Spanner emulator and will guide you through installation and use of the emulator with the existing Cloud Spanner CLI and client libraries.

What is Cloud Spanner Emulator?

The emulator provides a local, in-memory, high-fidelity emulator of the Cloud Spanner service. You can use the emulator to prototype, develop and hermetically test your application locally and in your integration test environments.

Because the emulator stores data in-memory, it will not persist data across runs. The emulator is intended to help you use Cloud Spanner for local development and testing (not for production deployments); However, once your application is working with the emulator, you can proceed to end-to-end testing of your application by simply changing the Cloud Spanner endpoint configuration.

Supported Features

The Cloud Spanner emulator exposes the complete set of Cloud Spanner APIs including instances, databases, SQL, DML, DDL, sessions, and transaction semantics. Support for querying schema metadata for a database is available via Information Schema. Both gRPC and REST APIs are supported and can be used with the existing client libraries, OSS JDBC driver as well as the Cloud SDK. The emulator is supported natively on Linux, and requires Docker on MacOS and Windows platforms. To ease the development and testing of an application, IDEs like IntelliJ and Eclipse can be configured to directly communicate with the Cloud Spanner emulator endpoint.

The emulator is not built for production scale and performance, and therefore should not be used for load testing or production traffic. Application developers can use the emulator for iterative development, and to implement and run unit and integration tests.

A detailed list of features and limitations is provided on Cloud Spanner emulator README. The emulator is currently (as of April 2020) in beta release and will be continuously enhanced for feature and API parity with Cloud Spanner service.

Using the Cloud Spanner Emulator

This section describes using the existing Cloud Spanner CLI and client libraries to interact with the emulator.

Before You Start

Starting the emulator locally

The emulator can be started using Docker or using the Cloud SDK CLI on Linux, MacOS and Windows. In either case, MacOS and Windows would require an installation of docker.

Docker

$ docker pull gcr.io/cloud-spanner-emulator/emulator
$ docker run -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator

Note: The first port is the gRPC port and the second port is the REST port.

Cloud SDK CLI

$ gcloud components update beta
$ gcloud beta emulators spanner start
Other alternatives to start the emulator, including pre-built linux binaries, are listed here.

Setup Cloud Spanner Project & Instance

Configure Cloud Spanner endpoint, project and disable authentication:
$ gcloud config configurations create emulator
$ gcloud config set auth/disable_credentials true
$ gcloud config set project test-project
$ gcloud config set api_endpoint_overrides/spanner http://localhost:9020/
Note:
To switch back to the default config:
`$ gcloud config configurations activate default`
To switch back to the emulator config:
`$ gcloud config configurations activate emulator`

Verify gCloud is working with the Cloud Spanner Emulator.
$ gcloud spanner instance-configs list

NAME               DISPLAY_NAME
emulator-config    Emulator Instance Config
Create a Cloud Spanner Instance
$ gcloud spanner instances create test-instance --config=emulator-config --description="Test Instance" --nodes=1

Using Cloud Spanner Client Libraries

With the beta lunch, the latest versions of Java, Go and C++ Cloud Spanner client libraries are supported to interact with the emulator. Use the Getting Started guides to try the emulator.

Prerequisite: Setup Cloud Spanner Project and Instance from step above.

This is an example of running the Java client library with the emulator:
# Configure emulator endpoint
$ export SPANNER_EMULATOR_HOST="localhost:9010"

# Cloning java sample of client library.
$ git clone https://github.com/GoogleCloudPlatform/java-docs-samples && cd java-docs-samples/spanner/cloud-client

$ mvn package

# Create database
$ java -jar target/spanner-google-cloud-samples-jar-with-dependencies.jar \
    createdatabase test-instance example-db


# Write
$ java -jar target/spanner-google-cloud-samples-jar-with-dependencies.jar \
    write test-instance example-db


# Query
$ java -jar target/spanner-google-cloud-samples-jar-with-dependencies.jar \
    query test-instance example-db
Follow the rest of the sample for Java client library using the Getting Started Guide.

Using the Cloud SDK CLI

Prerequisite: Setup Cloud Spanner Project and Instance from step above.

Configure emulator endpoint

$ gcloud config configurations activate emulator
Create a database

$ gcloud spanner databases create test-database --instance test-instance --ddl "CREATE TABLE TestTable (Key INT64, Value STRING(MAX)) PRIMARY KEY (Key)"
Write into database
$ gcloud spanner rows insert --table=TestTable --database=test-database --instance=test-instance --data=Key=1,Value=TestValue1
Read from database
$ gcloud spanner databases execute-sql test-database --instance test-instance --sql "select * from TestTable"

Using the open source command-line tool spanner-cli

Prerequisite: Setup Cloud Spanner Project, Instance and Database from step above.

Follow examples for an interactive prompt to Cloud Spanner database with spanner-cli.
# Configure emulator endpoint
$ export SPANNER_EMULATOR_HOST="localhost:9010"

$ go get github.com/cloudspannerecosystem/spanner-cli
$ go run github.com/cloudspannerecosystem/spanner-cli -p test-project -i test-instance -d test-database

spanner> INSERT INTO TestTable (Key, Value) VALUES (2, "TestValue2"), (3, "TestValue3");
Query OK, 2 rows affected

spanner> SELECT * FROM TestTable ORDER BY Key ASC;

+-----+----------------+
| Key | Value |
+-----+----------------+
| 2 | TestValue2 |
| 3 | TestValue3 |
+-----+----------------+
2 rows in set

spanner> exit;

Conclusion

Cloud Spanner emulator reduces application development cost and improves developer productivity for Cloud Spanner customers. We plan to continue building and supporting customer requested features and you can follow Cloud Spanner emulator on GitHub for more updates.

By Sneha Shah, Google Open Source

Announcing the 2020 first quarter Google Open Source Peer Bonus winners

Friday, April 17, 2020

We are very pleased to announce the latest Google Open Source Peer Bonus winners and their projects.

The Google Open Source Peer Bonus rewards external open source contributors nominated by Googlers for their exceptional contributions to open source. Historically, the program was primarily focused on rewarding developers. Over the years the program has evolved—rewarding not just software engineers but all types of contributors—including technical writers, user experience and graphic designers, community managers and marketers, mentors and educators, ops and security experts. 

In support of diversity, equity and inclusion initiatives worldwide, we had decided to devote this cycle to amazing women in open source, especially since it coincided with celebrating International Women’s Day on March 8. We are very excited and pleased to share the following statistics with you.

We have 56 winners this cycle representing 17 countries all over the world: Australia, Belgium, Canada, Estonia, France, Germany, India, Italy, Japan, Republic of Korea, Netherlands, Russia, Sweden, Switzerland, Ukraine, United Kingdom, and the United States.

Even though the cycle was open to ALL contributors, the number of female nominees went up from 8% to 25% in comparison to the previous cycle. That’s an amazing number celebrating amazing women!

Also, we are very pleased to see the number of docs contributors increase from 7% to 15%. Documentation is the #1 factor for project adoption, so this shift is very important and encouraging. To strengthen this trend and emphasize the importance of documentation in open source, the next cycle will be devoted (but not limited!) to docs contributors.

Below is the list of current winners who gave us permission to thank them publicly:
WinnerProject
Matt Mower
AMP HTML
Sergey Zakharov
Android Open Source Project
Pawel Kozlowski
Angular
Jakob Homan
Apache Airflow, Apache Kafka, Apache Hadoop
Chad Dombrova
Apache Beam
Myrle Krantz
Apache Software Foundation - Diversity and Inclusion committee + board
Katia Rojas
Apache Software Foundation Outreachy Program
Greg Hesp
assistant-relay
Beka Westberg
Blockly
Siebrand Mazeland
Blockly Games
Dave Mielke
BRLTTY
Vijay Hiremath
Chromium; platform/ec
Daniel Stenberg
curl / libcurl
Simon Binder
Dart build system
Aloďs Deniel
device_preview
Fatima Sarah Khalid
Drupal
Gregory Popovitch
Filament
Amr Yousef
Flutter
Remi Rousselet
Flutter
Pooja Bhaumik
Flutter
Elijah Newren
Git
Roger Peppe
Go
Oleksandr Porunov
JanusGraph
Tim Bannister
Kubernetes
June Yi
Kubernetes
Karen Bradshaw
Kubernetes
James Le Cuirot
leptonica
Stefan Weil
leptonica
Egor Pugin
leptonica
Bert Frees
LibLouis
Christian Egli
LibLouis
Richard Hughes
Linux Vendor Firmware Service (LVFS)
James (purpleidea)
mgmt
Mike Ryan
NgRx
Stefano Bonicatti
osquery
Alyssa Rosenzweig
panfrost
Carol Willing
Project Jupyter
Mariatta Wijaya
Python programming language
Alexander Neumann
restic
Nicholas Jamieson
rxjs (core member), rxjs-tslint-rules, rxjs-etc, ts-action
Kate Temkin
Several, mostly educational (see in Reasons)
Alyssa Ross
SpectrumOS / Nix
Rosalind Benoit
Spinnaker
Brian Le
Spinnaker
Vincent Demeester
Tekton
Chmouel Boudjnah
Tekton
Andrea Frittoli
Tekton
Simon Kaegi
Tekton
Cameron Shorter
The Good Docs Project
Ando Saabas
TreeInterpreter
Daz Wilkin
Trillian, Prometheus Exporter for GCP, KeyTransparency , OpenCensus
Gerrit Birkeland
typedoc
Wilson Snyder
Verilator
Thomas Oster
VisiCut
Koen Kanters
zigbee2mqtt
Jia Li
Zone.js
Congratulations to our winners! We look forward to your continued support and contributions to open source!

By Maria Tabak, Google Open Source

A milestone to celebrate: 10 years of GCI!

Tuesday, April 14, 2020

 
This year we celebrated the best of program milestones—10 years of bringing together 13-17 year old students from around the world into open source software development with our Google Code-in (GCI) contest. The contest wrapped up in January with our largest numbers ever; 3,566 students from 76 countries completed an impressive 20,840 tasks during the 7-week contest!

Students spent their time working online with mentors from 29 open source organizations that provided help to answer questions and guide students throughout the contest. The students wrote code, edited and created documentation, designed UI elements and logos, and conducted research. Additionally, they developed videos to teach others about open source software and found (and fixed!) hundreds of bugs.

Overview

  • 2,605 students completed three or more tasks (earning a Google Code-in 2019 t-shirt)
  • 18.5% of students were girls
  • 79.8% of students were first time participants in GCI (same percentage as in 2018- weird!)
  • We saw very large increases in the number of students from Japan, Mongolia, Sri Lanka, and Taiwan.

Student Age


Participating Schools

School NameNumber of Student ParticipantsCountry
Dunman High School138Singapore
Liceul Teoretic ''Aurel Vlaicu''47Romania
Indus E.M High School46India
Sacred Heart Convent Senior Secondary School34India
Ananda College29Sri Lanka

Students from 1,900 schools (yes, exactly 1,900!) competed in this year’s contest; plus, 273 students were homeschooled. Many students learn about GCI from their friends or teachers and continue to spread the word to their classmates. This year the top five schools that had the most students with completed tasks were:

Countries

The chart below displays the top 10 countries with students who completed at least 1 task.

We are thrilled that Google Code-in was so popular this year!

Thank you again to the people who make this program possible: the 895 mentors—from 59 countries—that guided students through the program and welcomed them into their open source communities.

By Stephanie Taylor, Google Open Source

Free Universal Sound Separation

Thursday, April 9, 2020

We are happy to announce the release of FUSS: the Free Universal Sound Separation dataset.

Audio recordings often contain a mixture of different sound sources; Universal sound separation is the ability to separate such a mixture into its component sounds, regardless of the types of sound present. Previously, sound separation work has focused on separating mixtures of a small number of sound types, such as "speech" versus "nonspeech", or different instances of the same type of sound, such as speaker #1 versus speaker #2. Often in such work, the number of sounds in a mixture is also assumed to be known a priori. The FUSS dataset shifts focus to the more general problem of separating a variable number of arbitrary sounds from one another.

One major hurdle to training models in this domain is that even if you have high-quality recordings of sound mixtures, you can't easily annotate these recordings with ground truth. High-quality simulation is one approach to overcome this limitation. To achieve good results, you need a diverse set of sounds, a realistic room simulator, and code to mix these elements together for realistic, multi-source, multi-class audio with ground truth. With FUSS, we are releasing all three of these.

FUSS relies on Creative Commons licensed audio clips from freesound.org. We filtered these by license type, then using a pre-release of FSD50k [1], further filtered out sounds that aren't separable by humans when mixed together. We were left with about 23 hours of audio, consisting of 12,377 sounds useful for mixing (7,237 train, 2,883 validation, 2,257 eval). Using these clips, we created 20,000 training mixtures, 1,000 validation mixtures, and 1,000 eval mixtures.

We developed our own room simulator implemented in tensorflow, which generates the impulse response of a box shaped room with frequency-dependent reflective properties given a sound source location and a mic location. As part of the dataset release, we provide pre-calculated room impulse responses used for each audio sample along with mixing code, so the research community can simulate novel audio without running the computationally expensive room simulator. Future work may include releasing the code for our room simulator and extending the simulator capabilities to address more extensive acoustic properties of rooms, materials with different reflective properties, novel room shapes, etc.

Finally, we have released a masking-based separation model, based on an improved time-domain convolutional network (TDCN++), described in our recent publications [2, 3]. On the eval set, this model achieves 12.5 dB of scale-invariant signal-to-noise ratio improvement (SI-SNRi) on mixtures with two to four sources, while reconstructing single-source mixtures with 37.6 dB absolute SI-SNR.

Source audio, reverb impulse responses, reverberated mixtures and sources created by the mixing code, and a baseline model checkpoint are available for download. Code for reverberating and mixing the audio data and for training the released model is available on our github page.

The dataset will also be used in the DCASE challenge, as a component of the Sound Event Detection and Separation task. The released model will serve as a baseline for this competition, and a benchmark to demonstrate progress against in future experiments.

Our hope is this dataset will lower the barrier to new research, and particularly will allow for fast iteration and application of novel techniques from other machine learning domains to the sound separation challenge.

By John Hershey, Scott Wisdom, and Hakan Erdogan, Google Research

References:
[1] Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. "Freesound Datasets: A Platform for the Creation of Open Audio Datasets." International Society for Music Information Retrieval Conference (ISMIR), pp. 486–493. Suzhou, China, 2017.
[2] Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R. Hershey. "Universal Sound Separation." IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175-179. New Paltz, NY, USA, 2019.
[3] Efthymios Tzinis, Scott Wisdom, John R. Hershey, Aren Jansen, and Daniel P. W. Ellis. "Improving Universal Sound Separation Using Sound Classification." IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), 2020.

Code Search for Google open source projects

Wednesday, April 1, 2020

We are pleased to launch Code Search for Google open source projects. Code Search is one of Google’s most popular internal tools, and now we have a version (same binary, different flags) targeted to open source communities.

Googlers use Code Search every day to help understand the codebase: they search for half-remembered functions and usages; jump through the codebase to figure out what calls the function they are viewing; and try to identify when and why a particular line of code changed.

The Code Search tool gives a rich code browsing experience. For example, the blame button shows which user last changed each line and you can display history on the same page as the file contents. In addition, it supports a powerful search language and, for some repositories, cross-references.

Suggest-as-you-type in any search box annotates suggestions with the type of code object, the repository and the path, helping users find what they want faster.


The search language supports regular expressions and a number of helpful search atoms. For a user looking for a function foo in a Go file, instead of sifting through thousands of results containing foo, the user can search for lang:go function:foo to limit search results to Go files where foo is a function and not a struct or a word in a comment.

One example is finding a file using only part of the name. The query file:KytheURI.java goes directly to the file, since there is only one such file.

See the quick reference for more information.

In addition to text search, some of the open source repositories have cross-references powered by Kythe. Kythe is a Google open source project that includes tools to help understand code. Project owners instrument a build of their repository to output compilation information for Kythe. Kythe tools convert this data to a graph. This graph connects definitions to declarations and code references to the abstract objects they represent (described by a graph schema). Google then runs an internal pipeline that combines these graphs for the different languages, prunes unnecessary pieces, and optimizes it for serving cross-references. The whole process runs several times per day to keep the data fresh.

Open source communities use a broader set of build systems than Google. In order to support cross-references, Kythe added drop-in support for Bazel, CMake, Maven, and Go. Projects using other build systems can use Kythe-provided wrappers for clang and javac to instrument their builds; these are used by Chromium and Android AOSP to provide compilation information for Kythe.

Because Kythe is based on the build, Kythe cross-references include links to files generated as part of the build process, such as Java files generated for AutoValues (example here) or protos. For repositories where cross-references are enabled, clicking on a symbol will take you to a definition of that symbol.

Clicking on the definition of a symbol will open a cross-reference panel, showing all the places where that symbol is referenced. For example, clicking on toVName below, we can see the places that reference this method. One of the callers is parseVName, and clicking on that shows the callers of that method.



At this time, we only provide search on the repositories listed below, but we plan to add more over time:
We are also investing in making the application keyboard-navigable and usable with a screen reader.

We hope you find this tool useful!

By Kris Hildrum, Code Search Team
.