Sawzall (programming language)
Encyclopedia
Sawzall is a procedural domain-specific programming language, used by Google
to process large numbers of individual log records. Sawzall was first described in 2003, and the szl runtime was open-sourced in August 2010. However, since the MapReduce
table aggregators have not been released, the open-sourced runtime is not useful for large-scale data analysis off-the-shelf.
) that are partitioned over many disks within GFS
. In order to perform calculations involving the logs, engineers can write MapReduce
programs in C++ or Java. MapReduce programs need to be compiled and may be more verbose than necessary, so writing a program to analyze the logs can be time-consuming. To make it easier to write quick scripts, Rob Pike et al. developed the Sawzall language. A Sawzall script runs within the Map phase of a MapReduce and "emits" values to tables. Then the Reduce phase (which the script writer does not have to be concerned about) aggregates the tables from multiple runs into a single set of tables.
Currently, only the language runtime (which runs a Sawzall script once over a single input) has been open-sourced, and the supporting program built on MapReduce has not been released.
Sawzall's design favors efficiency and engine simplicity over power:
and the sum of the squares of the values.
count: table sum of int;
total: table sum of float;
sum_of_squares: table sum of float;
x: float = input;
emit count <- 1;
emit total <- x;
emit sum_of_squares <- x * x;
Google
Google Inc. is an American multinational public corporation invested in Internet search, cloud computing, and advertising technologies. Google hosts and develops a number of Internet-based services and products, and generates profit primarily from advertising through its AdWords program...
to process large numbers of individual log records. Sawzall was first described in 2003, and the szl runtime was open-sourced in August 2010. However, since the MapReduce
MapReduce
MapReduce is a software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers. Parts of the framework are patented in some countries....
table aggregators have not been released, the open-sourced runtime is not useful for large-scale data analysis off-the-shelf.
Motivation
Google's server logs are stored as large collections of records (protocol buffersProtocol Buffers
Protocol Buffers are a serialization format with an interface description language developed by Google. The original Google implementation for C++, Java and Python is available under a free software, open source license....
) that are partitioned over many disks within GFS
Google File System
Google File System is a proprietary distributed file system developed by Google Inc. for its own use. It is designed to provide efficient, reliable access to data using large clusters of commodity hardware...
. In order to perform calculations involving the logs, engineers can write MapReduce
MapReduce
MapReduce is a software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers. Parts of the framework are patented in some countries....
programs in C++ or Java. MapReduce programs need to be compiled and may be more verbose than necessary, so writing a program to analyze the logs can be time-consuming. To make it easier to write quick scripts, Rob Pike et al. developed the Sawzall language. A Sawzall script runs within the Map phase of a MapReduce and "emits" values to tables. Then the Reduce phase (which the script writer does not have to be concerned about) aggregates the tables from multiple runs into a single set of tables.
Currently, only the language runtime (which runs a Sawzall script once over a single input) has been open-sourced, and the supporting program built on MapReduce has not been released.
Features
Some interesting features include:- A Sawzall script has a single input (a log record) and can output only by emitting to tables. The script can have no other side-effects.
- A script can define any number of output tables. Table types include:
-
collectionsaves every value emitted -
sumsaves the sum of every emitted value -
maximum(n)saves only the highest n values on a given weight.
-
- In addition, there are several statistical table types that give inexact results. The higher the parameter n, the more accurate the estimates are.
-
sample(n)gives a random sample of n values from all the emitted values -
quantile(n)calculates a cumulative probability distribution of the given numbers. -
top(n)gives n values that are probably the most frequent of the emitted values. -
unique(n)estimates the number of unique values emitted.
-
Sawzall's design favors efficiency and engine simplicity over power:
- Sawzall is statically typed, and the engine compiles the script to x86 before running it.
- Sawzall supports the compound data types lists, maps, and structs. However, there are no references or pointers. All assignments and function arguments create copies. This means that recursive data structures and cycles are impossible.
- Like C, functions can modify global variables and local variables but are not closures.
Sawzall code
This complete Sawzall program will read the input and produce three results: the number of records, the sum of the values,and the sum of the squares of the values.
count: table sum of int;
total: table sum of float;
sum_of_squares: table sum of float;
x: float = input;
emit count <- 1;
emit total <- x;
emit sum_of_squares <- x * x;