BigBash It!


Converts your SQL scripts into an autonomous Bash One-liner that can be executed on almost any *nix device to make quick analyses or crunch GB of logfiles. They perform surprisingly well - thanks to the highly optimized unix tools such as sort and (m)awk - and are sometimes even faster than DB queries for larger data sets. You can find more information about BigBash, for instance details on the supported SQL, on the Github page.

SQL Input

--Create your tables using the standard create table command CREATE TABLE movies (id INT UNIQUE, title TEXT, genres TEXT); --Map every table to a file or even to an output pipe MAP movies TO 'movies.dat.gz' DELIMITER '::' TYPE 'GZ' REMOVEHEADER; --The select query that will be compiled to a bash one-liner. --You can use joins, groups but no subselects SELECT title FROM movies ORDER BY title LIMIT 10;

Important: Choose an output delimiter that is not contained in the columns of the files, otherwise this could lead to subtle (or not so subtle) errors in the output.

If disabled, aggregations (e.g. SUM over groups) will be done in memory. This could lead to out-of-memory problems if number of groups is very large.
If enabled, aggregations use an additional sorting step which makes the query slower.


Disclaimer: In no event we take any responsibility for any damages arising out of the use of the generated queries (including but not limited to loss of data and inaccuracies of the results sustained by you or 3rd parties)
Disclaimer 2: This is kind of a hack project and for sure full of bugs. Please open an issue here when you encountering any.