data.bythebay.io: Alan Ngai - End-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
thanks Mike okay hello everyone um so yeah I'm co-founder uh CTO OBS Clarity so um OBS clity we've uh built a uh monitoring platform for um streaming applications and today we'll be talking about um you know how to debug and troubleshoot you know realtime pipelines specifically okay um so first things first let's talk about data first uh what what data first application means you might have heard this term kind of floating around um to us a data first application is basically any application architecture whose primary purpose is to extract a value out of data right um so what do I mean by that so you know these are examples of um uh business domains where data is at the center of problem solving for for these domains right you can't you can't do fraud detection without looking at data you can't do personalization without looking at data so the question is what is an application look like that actually tries to solve problems in this domain and turns out there's a lot of moving pieces so let's let's break it down piece uh piece by piece so at the center of it is the data Pipeline and at the start of the data pipeline is your data source so this could be you know your web uh you know web crawlers you could be um uh clickstream through API Gateway or whatnot but somewhere there's a data source that's coming in that data source is typically fed into a message broker such as CF or Kinesis or you know sqs and the like that is then fed into a stream processor such as um you know spark or storm or flank or whatnot and then you know this is where your your core business processes uh sit right here's where you actually combine data from multiple sources do interesting stuff on it and you know ship out the output into a data sync um and these are typically nosql databases you know such as Cassandra or elastic search and then finally um that data is being served U either as a web application or reports um at the end and this is what we call a data Pipeline and it's actually interesting to note that um you can have multiple applications running on the same pipeline infrastructure so for example um you can have um you know a clickstream report you can have a price Optimizer you can have fraud detection running on the same infrastructure often you know using overlapping input data sources right um these are typically run on top of elastic infrastructure so Amazon VMware you know Docker whatnot um you know the the infrastructure is ephemeral you can add and remove host all the time and on top of this you have your surrounding uh you know um microservices and uh and business logic such as you know account management or other web applications and this entire ecosystem is what we call data first application so as you can see it's it's fairly complex and monitoring um you know this this uh system is actually non-trivial um Ops Clarity we actually run on uh data pipelines this is kind of our lifeblood so this is a screenshot from our application what you're seeing on the top left is our automated topology Discovery my cursor okay yeah automated topology Discovery so as you add and remove hosts as you start and stop processes we automatically detect these system uh these services and you know we update the topology for you in real time on the right side you see an example of the anomaly detection so we pull in metrics from your system from your services from your application and we automatically detect you know U you Baseline the metrics and we automatically detect anomalies on top of that and on the bottom is a health roll up so we combine the information we get from the metrics and from you know system and Port checks and into a host Health we combine we roll that host Health into cluster Health we roll that cluster Health into application health and all of this is happening in real time right as a monitoring company obviously we can't be late we can't you know we can't afford to have bad data so um let's talk about the characteristics of data pipelines um so first and foremost um these things are typically heterogeneous um here's a small sampling of the technologies that goes into building such pipelines just a quick show of hands how many of you guys work in companies that use one of these Technologies so it turns out most of you and this is just a small subset right I might even including you know Zer mq Flume Flink data torrent Heron solar etc etc etc and you know monitoring this actually means you need to understand all of these systems at a detailed level because each of these systems are very complex um for good reason right these are running your core business processes therefore they need to be performant they need to be fault tolerant they need to be scalable and to satisfy all these requirements means that each component is actually very complex so taking storm as an example uh a typical storm cluster consists of a master and a set of worker hosts each worker host consists of a supervisor process each supervisor process manages multiple topologies each topology consists of multiple executive processes each executor runs one or more you know Bol end spout right and all of these depend on zookeeper and every single one of these components ship out metrics so you know it's almost mindboggling how much you have to understand in order to monitor one component well in your data pipeline not to mention the other stuff CFA is just as complicated elastic search is just as complicated right next uh highly interdependent by its very nature data pipelines are very very interdependent so if you see a problem on the endide for example um your web application is showing a report with bad data or maybe the data is out of date it's not fresh right so the question is where in the pipeline could the problem have occurred anybody want to guess anywhere anywhere exactly perfect so the problem could be anywhere right and the troubleshooting and debugging process typically involves you know going from stage to Stage to Stage looking at metrics looking at logs SSH intoing it and figuring out what's going on and going back and forth trying to you know correlate problems um and you can imagine how hard that is to do right you put all of this together and what it means is that it's just you know really really hard to monitor real-time applications right um fortunately we've learned some lessons along the way and I'm here to share with you you know kind of the lessons we've learned to help us you know debug systems better um first and foremost put all of your data in one place uh don't use the outof boox solutions that come with these system so capka has its own consoles spark has its own elastic search has its own umari and every everybody else has its own and you can imagine how hard it is to actually do the the you know stage to Stage correlation when you're looking at all of these consoles that all work differently showing different data and you know it's just it's just too too difficult to manage um it's worth the E uh effort um that you invest to pull all of these metrics and put it in one place at up Clarity we actually make it easy because we do this automatically but even if you don't use Ops Clarity whatever system you use put it in place right and this will be uh important for the next stages you want to organize your concerns horizontally so there's a common set of concerns no matter what business you're running that you care about as long as you have a data Pipeline and these are the concerns so you know throughput stuff per unit of time these could be clicks per second these could be docs per second this could be Network bytes per second whatever business you care about there's there's some stuff per unit of time that you care about right in order to tell whether your data pipeline is healthy or not same goes for latency how long it takes to process stuff this could be processing time or CU weight time error rates um how much stuff you have piling up how much data you're losing how much data you're duplicating regardless of whether you're at Tech or search or you know social whatever as long as you have a pipeline you care about these things right and these are actually important for every stage in your pipeline all of these concerns matter for capka for spark for elastic search you know you care about all of these right why is that important now if you organize your data according to these concerns then you know debugging becomes a lot easier so here's an example of all the through metrics for the pipeline I have above uh organized in one place so I don't know if you guys can actually see the graphs but can you kind of detect where the problem happens has occurred in this pipeline guesses going once going twice it's actually the Play application and the reason why it's play and it's obvious is play is because everything Downstream from play you can see that you know there's this uh anomaly happening around this time and the pattern Downstream is the same everything up sorry oops going back yeah everything Upstream from play is normal right and with play you see one of the dock receives being normal and one of the doc process being abnormal so you know you can kind of guess that this is where you should start looking now imagine how hard this would be to see um if you had your CPU metrics your disc metrics your all all the other metrics that you get with your typical tools um kind of munched around together and imagine how hard this would be to see if each of these graphs are on different tools right so now it becomes more evident why we should put everything one place and how why you should organize your metrics according to these primary concerns okay next um organizing your concerns vertically so again going back to uh storm um you know all of all of these guys ship out metrics when something goes wrong it's difficult to figure out where to start looking but if you organize your concerns according to a hierarchy we found it's helpful to basically you know help you navigate so for example you get woken up at 2: a.m. at night um something goes wrong first thing you should check is is my data healthy my throughput and my my latency is okay if it's okay it's probably not as urgent if my throughput and my latency is you know going you know wonky I should probably look into it right away um in Storm's case it depends on Cala and zookeeper so if Cala and zookeeper are not healthy I should probably look into those before I look into storm right um otherwise you know um you know garbage in garbage out assuming my dependencies are okay is Storm itself healthy right is the is the master okay are there adequate resources in the cluster um if the the storm uh surface itself is um not healthy then you know all my applications running on it are potentially bad next my application kpis are they okay U next is my application well distributed within storm so storm has this habit of kind of migrating all of your processes into one machine over time and you have to kind of balance over time um so so you know you need to check on that to see if anything's okay then you go into an individual node are all the jobs running on a particular node okay um if so um if all the jobs are bad it's probably something with that node if if just one job is bad this may be something in your application code and finally your system metrics okay all right great I kind of Bree through that really quickly but I kind of wanted to rush through to to show you an example of these Concepts okay escape and here here you go okay so what I'm showing you here is a our application and what you're seeing here is our topology um it's your you know typical data first application topology so let me kind of walk through the pieces on the right side you have your um your uh web serving app your rails app running with mcash talking to Cassandra and elastic search on the left side you have your data injest pipeline so you have our API Gateway uh feeding data into an injust server uh feeding data into CFA then into spar then into elastic search which is being served by rails right so all of this is autois discovered um and basically we're just showing you the topology okay great um you see a bunch of Reds and yellows and kind of walk through a a a troubleshooting use case in this topology so first thing I'm going to do is I'm going to um is this Zoom let me make it a little bit bigger okay hopefully that's easier to see okay so uh first thing I'm going to do is oh sorry on the right side you have your kind of event log you know all the anomalies all the events happening in the system and on the bottom is a histogram of the events so you can kind of tell uh what time something abnormal happen so first thing I'm going to do is I'm going to zoom in and narrow down the time um to you know just when something bad occurred then I'm going to go in and select all of the services that are abnormal and here you can see the health of these Services over time overlaid on top of each other relative each other so you can kind of see how it plays out let me filter out the uh topology uh hide all Direction and this is just basically the stuff that's going wrong and let's start our debugging process right so again you get wome 2 a.m. something's bad with your report what's going on I click on this uh rails app I scoped down the event log to um you know just what happened on this service and here I'm I'm going to expand the graph hopefully you can see the uh impression count went down so this is an nomin detection on the impressions metric on that machine and that went down so you know what's going on either something could happen on here or something happened upstream and so probably something happened Upstream otherwise you wouldn't see the red and orange Upstream so let's debug that further uh let's look at the elb um and you can see for elb the request count went down let's look at elastic search uh for elastic search you can see the the documents index all went down so that suggest something Upstream happened right um let's go up further go to spark you can see your message wres and your message is red both went down around the same time go back to Kafka messages in and messages out all went down so again Upstream right so very quickly I was able to go from one end to the other and finally I'm going to my my injust server and for this one let's actually just jump to the metrix page and look at the metric so here you see basically the metrics for this uh for this injust server my docs receive is normal my Q size started growing around this time uh my docs process start dropping around that time and my error rates start going up around that time so this is actually where the root CA of the problem is and what we actually did was we deployed a new inest server that randomly drops documents which which basically caus uh this cascading effect at the end right so if you don't have your data organized imagine how hard it is to see all of these those events just in one shot you get wome at 2 a.m. in the morning I to figure things out okay another example uh I talked about uh organizing your um metrics by concern so here what I've done is I collected all the metric all my throughput metrics for my pipeline in one place so here you can basically see the same story as well right everything is normal up until here which and after that everything is abnormal right and you know and I since I've labeled these correctly you can see that it's my Pi in Ser in server that started becoming abnormal so again again imagine how hard this would be to look at if you had your system metrics and all your other metrics kind of mingled together yeah okay all right I am almost out of time um let me go back to my deck okay all right so just a summary so what what do we talk about today uh data first applications are becoming a thing um I assume you guys all know this otherwise you wouldn't be here um monitor data first application is hard I hope I convince you of that um it's very challenging if you don't invest the time to to do it correctly um most people fly uh fly blind today um get your metrics all in one place invest the time whatever it takes um and finally organize your data I think that's the key Point here you know make sure that you organize your data by concerns organize your data by you know components so you can you can actually make sense of it okay and I think that's it for me um happy to answer any questions yes question first question uh I guess you lacation uh there's multiple ways of collecting data oh sorry yeah so the question is um how are you collecting data is it through the logs right um there's multiple ways of collecting data from these uh Services some of them uh you extract data out the logs other other uh systems you use jmx to talk to jmx to pull the metrics others they expose the rest API and you can just pull out of that API so part of the challenge of monitoring these systems actually is to figure out how to actually monitor them um there are systems like collectd and and statsd that have plugins and you know depending on the plugin they may or may not suit your purposes but you know typically you use these plugins and you have to customize them in some form to pull the data so my question two question how do you manage version change uhhuh yeah Engineers that's that's that's what it cost yeah you have to you have to keep up to date another question and that's right that's right um so I mean it goes back to data pipeline right your every component of your data pipeline has to be scalable has to be performant you have to partition your data you know it's it's just stuff that you know you know yahooo eBay Google Facebook they all deal with and and when we deal with the Data Solutions the reason why it's so complicated is that every single one of these components have to be scalable and perform it none of them can be you know a single point of failure so yeah just just hardcore engineering work yeah two questions um one uh what are some Pok Source Alternatives toity to um What In your experience youve got a lot of data um which projects which open source projects are misconfigured or are problematic more frequently than other okay good questions so two questions are what are some open source alternatives to op Clarity and second is uh you know what are issues that you run into with open source Technologies as you try to monitor these systems or misconfiguration now which projects do you see misconfigured most often oh okay good okay so which okay so for the Technologies which ones are hard or difficult to monitor because of misconfiguration okay so okay so the first question is there are no open source Alternatives ofy otherwise we wouldn't exist um so the solution really is just you know you have to what we've done is basically build our own data injust pipeline right to collect data out of multiple sources and and put them all in one place and you know allow you to visualize them in one place and it's actually really hard to do um we started out using nagio and using ganglia and using Gra F but if any of you guys used it it's a lot of manual work a lot of Maintenance and there a lot of you know just a topology Discovery alone keeping up to date with topology is really hard to do so unfortunately there is none um there are computers out there that maybe makes it easier for you to um you know add tags to your resources and they use those tags to group stuff um but as far as open source I don't know of any as far as misconfiguration um uh it depends on how new it is so Docker is really a pain in the ass to monitor if you use Docker at all um some of the more mature stuff such as um uh Kafka is actually you know well documented elastic search is well documented um you know some of the newer stuff like I don't know data torrent um some of the newer you know it's it's really all the newness and and the critical mass of the use of the of the user base um I know that's not a specific answer you're looking for but yeah but that's that's been our experience since you Docker what way did you find Docker I cisor um I can spend a whole session on that but uh yeah I mean it's a combination of I mean there's multiple modes that Docker can run and it can interact with you know there's a Docker Standalone and there's mesos and there's yarn so it's it's really difficult to basically figure out all the combinations um often what it comes down to is running your own process inside the docker Docker container to you know pull data out uh our agent sits outside of Darker Docker um and we have to deal with situations like the port mapping from inside the container and outside container being different and that's that's only if you run it with that specific mode there are modes where it's not different and there are modes where you can't even see the ports so it's it's just really grunt work just spending time and and every one of our customers are different we have Docker customers and every single one of them are different huh one last question so most of the diagnos diagnosing the issue yes any effort to um for us uh our primary focus is the diagnosis because we find that's where we spend most of the time um remediation um there are tools out there to help you you know go deeper into each of these uh each of these Technologies like you know Cafe inspection tools or spark inspection tools and you know they allow you to walk through your code but our emphasis is just on collecting data and monitoring it okay thank you all for