End-2-End Monitoring and Troubleshooting...

Real-time streaming pipelines are comprised of a combination of application, data frameworks and the underlying infrastructure, which has increasingly become containerized. The application code and the underlying data frameworks are closely intertwined with each other resulting in a blurred line between the application and data processing tier. The highly complex, distributed and interconnected nature of these services make monitoring and troubleshooting these pipelines very challenging. In this talk, we will: • Examine the the different components used to build a typical real-time streaming pipeline • Evaluate the importance of modeling the “pipeline” as a first-class object that should be monitored • Discuss the challenges of monitoring and troubleshooting a real-time streaming pipeline • Review capturing the overall metrics for the pipeline that map to specific metrics from each component like throughput, latency, backpressure and error rate. • Provide a set of best practices for organizing information to begin troubleshooting your data processing frameworks when things go wrong • Present a simple way to build a "Pipeline View" that captures the health of each component in the pipeline, as well as the dependencies between the components and gives an indication of any issues in the pipeline at a quick glance • Demonstrate how to visually correlate pipeline metrics and pipeline health to underlying infrastructure issues, so that problems can be quickly analyzed and resolved