Twitter Heron on YARN/REEF

Twitter Heron is the next generation streaming system that has been in production for more than 2 years. Heron is used for diverse use cases including but not limited to ETL, BI, machine learning and media processing. In this talk, we will talk about how Microsoft is improving the Heron real-time streaming engine. First, we will discuss how one can deploy Heron with the widely used YARN scheduler to seamlessly integrate with the popular Hadoop solutions stack. Second, we will present the ongoing efforts aimed at making Heron use hardware resources more efficiently by optimizing performance and resource utilization of Heron topologies. Finally, we will round off the talk with a peek at the exciting upcoming future work.