Spark is good at parallel data processing management. We wrote a neat program to handle the TBs data we get everyday. Apache Spark