Spark: Big Data Analytics Made Fast and Easy

As big data becomes a concern for more and more organizations, there is a need for both faster tools to process it and easier-to-use APIs. Spark is a Hadoop-compatible cluster computing engine that addresses these needs through (1) in-memory computing primitives that let it run 100x faster than Hadoop and (2) concise, high-level APIs in Scala, Java and Python. Increasingly, Spark is also being used to power a stack of higher-level computing tools, including Shark, a port of the Hive SQL engine, Spark Streaming, a stream processing engine, and GraphX, a graph computing package that can describe algorithms like PageRank in a few lines of code. We'll show how Spark's Scala API allows these types of computations to be seamlessly composed. Spark is open source in the Apache Incubator, and has a growing community with over 60 developers and 17 companies contributing.