Streamlio: Towards a Unified End-to-end...

Handling real-time data has become a critical capability for data-driven organizations. However, today’s reality is a disconnected patchwork of incomplete technologies that make it a struggle to deliver real-time solutions because of frustrating complexity, inefficiency, and incompleteness. In this talk, we address these challenges with an unified solution for real-time data. An end-to-end real time system needs •  Messaging : receive and distribute streaming data with support for publish-subscribe and queuing scenarios with built-in durability, scalability, and performance using the Apache Pulsar (incubating) messaging solution. •  Processing : process data transformations and analytics with the Heron real-time processing engine, built for performance and scalability. •  Storage : leverage Apache BookKeeper streaming log storage to ensure durability, resiliency, and performance for streaming data. In our talk, we will provide an overview of the underlying three systems and how they are used the core for a unified end-to-end real time solution.