Real Time Machine Learning Visualization...

Training models on massive datasets, even on Spark, can be a lengthy process, during which the data scientist has no visibility into how the model is shaping up. The only way to monitor progress is to view the status of the Spark jobs, which provides no information about convergence or other statistics of interest. In this talk, we will discuss how to visualize and monitor the training of machine learning models in real-time with Spark. With this capability, you can monitor machine learning training from one iteration to the next, observe how the model converges during each iteration, visualize the characteristics of the model in real time, and decide if you wish to continue to train the model. In this talk you will learn: How machine learning algorithms are monitored by adding callbacks to K-Means and other algorithms. The Spark task communication infrastructure that has been built, using Akka to deliver messages from the Spark driver to the job submitter. How HTML5 SSE helps to generate real-time progress visualizations