The vision of Alpine Data Labs is to make data science so straightforward that it becomes a tool for business users as well as data scientists. To this goal, we developed an intuitive visual UI which allows users to interact with Hadoop data and perform advanced analytics. However, architecting a highly scalable and effective platform presents some specific challenges including: supporting multiple Hadoop distributions, supporting pig/sql/R/hive/mapreduce/Spark, and showing visual progress for all the analysis. We have leveraged Scala to address each of these issue by building an agent architecture which uses Akka to scale out to different Hadoop distributions and designed an R-Akka Server that allows Alpine to scale out R sessions. We use Spray + Akka to expose our Alpine restful APIs and have implemented Machine Learning algorithms in Spark using Scala. We have also enhanced the Spark Yarn module via Akka messaging as communication channel.
In this talk, we will specifically focus on the Alpine Spark Integration:
Submitting a Spark job from a servlet engine
Enhancing the Spark client in Yarn cluster mode to enable the Yarn app Listener and the stop Yarn application
Yarn resource capacity callback
Messaging Channel for logging, progress, error handling via Akka
Re-directing print stream and Spark Job Progress listener to Alpine UI.
Job progress live streaming to Alpine UI via websocket