Mutable Ideas

Notes and ideas about Java, Scala, Big Data, NoSQL, Quality and Software Deploy

CRUSH - Custom Reporting Utilities for SHell

It is a collection of programs for processing delimited-text data through the command line or using shell scripts.

My friend @__joselo introduced me to a toolset called CRUSH * Custom Reporting Utilities for SHell.

I started playing along and its really fun, I didn’t explore it deeply but pivoting some data was really easy.

INSTALL

I got a WARNING during the install, because my OSX was missing PCRE. Quickly solved with:

$ brew install pcre

Then just plain compile ./configure; make; make install and we’re ready!

PIVOTING DATA

Consider a dataset as below:

1
2
3
4
5
date,username,count
2012-06-01,bob,856
2012-06-01,peter,100
2012-06-02,peter,573
2012-06-02,john,1609

Pivoting by date:

1
2
3
4
5
6
$ pivot -d, -F"username" -P"date" -A"count" sample.csv

username,2012-06-01: count,2012-06-02: count
bob,856,0
john,0,1609
peter,100,573

Pivoting by username:

1
2
3
4
5
$ pivot -d, -P"username" -F"date" -A"count" sample.csv

date,bob: count,john: count,peter: count
2012-06-01,856,0,100
2012-06-02,0,1609,573

That’s all for now! The utility dbstream seems interesting also.