scale.bythebay.io: Edgar Orendain, Build a Modern, End-to-End, Big Data Scala Reference Application
can anybody give me the name of an Apache project that is not a storm spark knife I cough let's say let's say the name of an Apache project that has to do with data storage sorry yeah would you like a green Hadoop all right I'm just gonna oh very nice okay great so hi thanks everyone for joining me my name is avocado 10-9 I studied computer science at UC Berkeley and I've been with horn works since 2016 working on Scala and Java applications with patchy knife I storm spark Kafka knife I've Bari I think I repeated one of those my the thing I'm working on now is we architected and implementing one of our offerings it's a horn work sandbox so a desktop environment for big data applications so this morning I spoke to a few people and it seemed like I needed to make a few changes to the presentation it seemed like people wanted to spend more time learning about the different stages of a big data pipeline rather than going over code for any one specific service so so I've made some changes to this slide and I'll try to I'll try to go over them I'll try to go through everything without too many ups and pauses because he is still sort of fairly recent changes removed a lot of live coding but everything that I removed is still in the repo it'll be available to everybody everything here is documented most of these most of it will go over also have tutorials that go alongside of them and you know yeah I'll make it all available after the talk so so why this talk skull is great everybody loves Scala we all love Scala but when it comes to big data one of the main entry points big data for skull developers is streaming or machine learning and this these entry points coupled with the fact that there's a lot of value in streaming and machine learning it's difficult for us to explore other parts of the big data pipeline and sometimes when we're given the task of you know building a pipeline or working with something that's not streaming you know maybe data ingestion or data routing we tend to stick to the things we know so maybe you know like a streaming solution we're really another tool would be would be a better fit so we end up building a data pipeline in ways that are maybe not the best I'll also cover where it is very important to take scholar centric approaches in stages of the pipeline where typically Scala isn't around so how we can improve pipeline the pipeline with Scala so just to sort of set the stage and understand why these the the pipeline will include some of the services that we go over so gone are the days where most of our processing is done on data in a stationary data you know with the not so recent emergence of the Internet of Things there's a lot of value in in in in real time data and data from the edge and not so much we are working with with the data that's been you know it's coming from data likes the a lot of the commodity commodity devices that that we carry with us have sensors they can communicate and compute they can also compute very efficiently so there's a lot of value in that and the the convergence of these has created an unprecedented opportunity for for distributed data solutions so if for for if we're talking about for zettabytes today then with exponential growth it's 44 zettabytes by 2020 so we're gonna follow a single use case throughout this this reference application and we're gonna we're gonna build we're gonna build the pipeline with this one specific use case in mind so imagine consider a vehicle dispatch company that you know the delivers cargo from one place to another and the trucks have sensors installed on them that feedback to some environment we want the cargo to get to its its destination safely but quickly and so and without accidents so we we perform analytical processing in order to get some actionable insights you know what the best routes are what drivers are being safe things you know things of that nature we also want to predict accidents before they happen so we we want predictive analytics on top of prescriptive analytics and there are a lot of endpoints because you have sensors on every truck and maybe you know you're different different hubs and you know web services that you correlate with sensor data there there's more than there's a lot of data sources and you we have to be able to intelligently route data from these data sources to be the what we'll see are a number of services in our back-end so yes yeah yeah give you guys a chance but yes everything should be based on data and if we're gonna get some sort of action from it and better be reliable so a quick overview of the architecture that will be will be going over and designing here so because we're short on time we're phasing out the play app with scholar J s also spark machine learning again it's all on the repo and to replace code live code examples for each one of these streaming or computing frameworks we're gonna quickly we're just gonna replace them with a streaming analytics manager application so the what we have so just to sort of define the different stages of the pipeline we have we have our edge the internet of anything the edge involves in this use case it's you know embedded system sensors but really can be anything syndicated you know third-party content data likes Twitter feeds you know what have you it's all it's all the edge the next stage is gonna be our data flow management stage or data or sort of our data routing stage that's represented by the Apache knife I logo we're going to use Kafka as our message broker to reliably get data from you know one service to another we have our streaming / processing / computing stage which is that box over to the right finally we're going to persist data to some store in this case we'll use druid but really want to persist data somewhere and finally after analytics are performed right through with with the druid ov up we're gonna try and visualize the data get some actionable insights using super set and schema registry I'll explain in a bit here so so why Apache knife I so first a petronella is a data flow management system if you can see if you if you if you were to do a search for you know how to ingest data how to you know we route data from say some syndicated content provider to your processing framework you know you might you might get you might get a you know tutorial about how to do sentiment analysis on Facebook feeds or word count on Twitter feeds and so you'd probably end up using you know what you're comfortable with maybe spark or storm or flink to bring you the data process and then store somewhere but the problem with always sticking to you know what we as scholar developers tend to know as like streaming frameworks is they are not very great or as great at routing data and defining data flow as things that that already exists so routing is not trivial we let's see actually that's let me just cuz we're fighting the clock I'll go over a lot it's in the actual demo so yeah so let's let's open up a knife fight and build a flow let's see if I get this oh let me share my screen here perfect yeah okay so this is Apache knife I this is going to be where we where we define build our flow so this first node here so each one of these boxes is called a processor it is it you know processor in the sense that you know most sort of streaming or ingestion frameworks I call them so this is a this is a processor and underneath the hood what we have is a data generation a simulator written in Scala and it's just wrapped around excuse me it's it's wrapped by a very light API to get this working in knife I so this this is our edge this is our simulated edge where data is generated and so using knife I we take different processors and perform either transformations or we check on certain conditions and we remove this data to you know some some sync yeah okay great question so yeah there's there can definitely be some confusion we have you know we have streaming we have streaming frameworks we have data flow managers and then we also have ingestion frameworks and those those are those are separate so this is knife I is really powerful when it comes to helping define data flow so bringing in ingesting data from some source moving it around performing some logic figuring out where needs to go and then dropping it into a sink not so much actually performing computation and it is sort of visual programming in that it uses a drag-and-drop paradigm and but I mean it but it's not like this is translated to a single application that runs somewhere it's it you know it runs as different sets of different notes so this is our so yes so this processor here is our simulated data it is our you can consider it our edge we we bring in this processor to to route data we have two streams we excuse me we have two yet but we have a single stream with two types of data being generated from this processor and so router on a trip you ran on attribute will will route truck data data sensor data from the truck over to the left and traffic data over to the right we this is we have if we have time we'll come back to this but here we connect to a web service online to enrich the truck data so we get we get the geolocation of the truck we community with some API and we you know enrich the sensor data with geolocation information these two will come back to and and finally we have knife I take the data and publish it to two different Kafka topics so pretty straightforward I mean it looks pretty it is you know it's pretty simple it doesn't seem like we're doing much but but you know what we're let me get this what we're doing here is we're routing data we're we're defining our data flow and without having to without having to use a design and deploy application like spark or storm where we have to write code compile it then deploy it somewhere you know rather it's you dragon a processor you you know dragon arrow to another box and knife I takes care of where things need to go yeah yes excellent excellent question so it is distributed and they're they're not there have been some there's been a movement to get high availability esque into knife i so i can't speak much to that but but but yes yes it is distributed yes no right so this is this is not sort of a design and deploy application where you know you hit compile and it generates some some application yeah so simple answer's no but I can expand on that after if you'd like we can you take it offline but just just to sort of move along here so Oh sorry can you repeat the question right yes great questions so yeah so I can touch on that now instead of instead of a bit here so knife eye is great because it is it's it's data agnostic you can throw in whatever data you'd like and you you know you tell it exactly how to route it using you know your defined logic structured data unstructured data especially now where there's a lot of heterogeneity and incomplete data you you can you you can define where that where the data goes based on whether looks incomplete to knife I if it's if it's structured a certain way if it meets some schema which we'll talk about in a minute okay so you can the all that it can be defined or specified rather so this is what this is what our two streams of data look like and so this will come into play and a bit here but the the first one represents the data that we pull in from the truck sensor so you know time driver ID name latitude longitude and whether the sensor caught a speeding or not the second one is is congest is congestion information from some some we a web service online or some other sensor maybe outside of the truck so two streams of data so as I mentioned with a knife I you can you can use it to route flows and specify you know where how data should move from from the edge to you know some number of end points it's great because you can use it for you you it lends itself to agile development so you can use it to you use it to debug if there's an issue knife I if if some data packet goes to you know an endpoint that you don't like you can you can replay the logic and tell the knife I to instead you know try a different flow and you know and it does this and you can you know do all this without having to recompile there's a lot of data provenance built-in to knife I so you can see and track where all the data flows from from source to sink so that is great there's a little bit of overhead so whenever you need to do some serious processing you can either you can either route the data to some sink you know some some spark job that takes it over or instead you can create custom processors to run you know run some processing or computation in the middle of a flow and to do that you can wrap your application using a knife the knife IEP I to create a processor so this is just so this is just a stub we won't go over this but this is oh that is it's not what I see so yes so we use this stub and it's available again on the repo in order to build this to custom processors that we have running so if we if you remember the architecture the overview architecture we had something called a schema registry up top that seemed to have arrows coming from other services into it so just to just to sort of gauge how much time we need to spend on schema how many people here regularly use a serialization library so protobuf Avro okay like like half alright so then so we're so then real quick if you're if you're trying to send data from from one service to another you want to serialize the data right you can't just send like a you know JVM object you serialize the data you know either as like a JSON object or like maybe it's just a string of bytes or something and if and you know the the reason you do that is for a number of reasons one is these different services need to know how to communicate with each other without you know without having to explicitly be told hey you're a service that is responsible for reading customer data and then you're a service for reading you know data is from some sort of truck and instead you have one application that can handle both or you know handle the different types of data so any engineer any engineer here knows that that scope creeks and data drift especially in an IOT the IOT world is very natural data drift meaning you know your your data over time as maybe the edge evolves that the data looks different coming into your system and so with with your scope creaking and your data changing you you you need to have a strong schema if your schema isn't strong if it's brittle and then your application is brittle and if your application is brittle then things start to fail and you're in trouble and so so we use schema we serialize our data packets with some you know serialization framework and this is this is fantastic but if you have an immunity if you come in from like a scholar background where you know maybe you're managing you know twenty five to a hundred different micro services that all that all you know expect serialized data or rather just you're just just just data you know to flow in and out your processor excuse me your your services evolve at different rates and so if you make a change to one part of your pipeline or one part of your service then you might have to change your schema you know at every other and every other service you know so that they can still communicate with each other so this could get really cumbersome and when you develop a big data pipeline and there's a lot of different types of data flowing into your system it's it's it may be very wise to use what is called the schema registry so schema registry is a place is a centralized repository for storing for storing your schemas and your serializers and deserialize errs and so we won't see an example but but what you can you so imagine the schema registry as a web service that different services different components communicate with after receiving some serialized data they they ask the schema registry what schema belongs to you know the the data that's flowing in and you know and then the service processes as it should if if something if if a service upstream evolves and it and it outputs data in a slightly different format the the downstream service will still communicate with schema registry and schema registry will tell it hey you know this is the same schema that existed before only now it's changed a bit and you know there's no sort of there's no there's not necessarily a change they need to make to the downstream components to to account for this so there's a there's multiple open-source implementations of schema registry there's there's one by by confluent and then and one by by horn works so we won't go over over registering schema but here is a stub on exactly how you do it so it's it's a it's pretty straight for it's pretty straightforward again more of more information in the in the repo right so sorry so I should probably be repeating the questions so your question was how does a service like schema registry compare to to protobuf or any other civilization library where it's sort of taken as granted that your schemas can evolve so for one schema registries except not a kit can can deal with schemas in different formats and so a schema registry by confluent for example it's only it only works for a bro and at the moment I think the strongest bridge is two Kafka but the one by horn works it you can it deals with adversity realization or any other serialization you can also define your own serializers or deserialize errs and it works with any number of services as long as those services can communicate to excuse me it can communicate with the with the registry as opposed to having to and not necessarily having to be to include specific libraries like you know like having to know about protobuf ahead of time yeah it's a search question we could all I can I'm happy to expand on that a little after but so so let's see so - so recap our architecture looks like at this point is where we've ingested data from the edge we've used a data flow management system like knife I to to route the data intelligently and reliably to some number of endpoints in this case we're using Kafka as our message broker and and we're using schema registry to automatically match impedances between what the data looks like when it comes in from the edge and what the data is expected to look like once it hits our sink so yes sir so yes schema sort of helps us matches impedances automatically or sort of automagically see so so now we we have data and we have this data routed to where needs to be and we should process this data maybe right maybe use some sort of streaming framework so so now it you know now it's really easy we just choose the best streaming framework right it's obviously sparked or if you want right or if you want if you want data governance and security which you know definitely you want in the enterprise when you want to use storm but but then you know you have the problem of well it's you know you can't it's not really all too sexy when it comes to Scala so maybe you should you know there's there's a lot of issue so it's actually you know it's not of course it's not as simple as as we make it out to be or you know as we'd want it to be and you know a lot of times it comes down to whether you should use a streaming stream processing framework or use a stream processing framework like it you know sometimes they do exactly the same thing for your use case sometimes you absolutely want to use a specific one and so this is sort of outside the scope of the talk but Vladimir's giving to talk tomorrow at 1:00 1:00 p.m. so definitely check that out if you want engage different use cases for different frameworks so we're just going to quickly build out a topology using streaming analytics manager just in place of demoing the code for for spark streaming and for storm and flink let's see oh yeah let me share my screen real quick so streaming analytics manager so I've got one I've got a flow belt out here so it follows a it's also a drag-and-drop paradigm and it has value from the the perspective of a of someone who's trying to either build something out quickly you know being agile or if you're not a developer then you can you can build out a topology way to without necessarily writing any code so just to sort of go over to the logic and not necessarily to go over stream analytics manager itself where we're ingesting from the two Kafka topics that we that that knife I dump data into we're going to perform a join on those two the the data and we're gonna check to see if the driver performs some sort of violation like maybe they were speeding or they you know they got out of their own Lane and if and if you know they did perform some sort of violation we're going to find the average speed and and and rat it to a druid store well what color druid in a second in addition we're all we also want to said we want to send the data before it's either the speed is aggregated or rather some excuse me is a verage to a different Jude store and in addition maybe we want to persist the data for you know this case predictive analytics where you know some time after you know not in real time we want to go back and analyze the data or just to you know just to persist threa for a number of different reasons so let's see so this is what apology looks like do I have true it up yes I do so underneath the hood just while this thing is running underneath the hood streaming analytics manager it's really just a visual you know sort of follows a drag-and-drop paradigm to build a an application but underneath the hood and it it's built on top of a unifying unifying model to to deploy this application on a number of different streaming engines so in this case it's going to use storm but it's also it also has bridges for for for storm and you see any spark and see that's gonna take a second to load so oh okay so so streaming analytics excuse me I saw our application as we saw routed to a number of sinks so to druid stores in this case and HDFS or HBase or HDFS and so the next part of the pipeline is data persistence whether that be for just for archival purposes or you want to run some more analytics you know post real times you know post real time range so you know for that there is a number of solutions hive Cassandra HBase druid or just a few of them and and again it really depends on on what your this what the state of your data is and the purpose of your data whether it's just you know to dump into cold storage for later if it's for always-on applications where you need to query the data often or you know anything in our case if we want to perform sub second query in sub second queries where we need information quickly as soon as we can so that we make sure our drivers are safe and so for it for that we we're gonna use druid yeah we saw druid a second ago so druid is a high-performance column-oriented distributed data store there was a talk on on druid right before this one a it's here we go so okay great so so the next so finally the last part of this of the pipeline after we've persisted our data is to get some sort of insight from it right we want to visualize the data we want to see exactly what kind of you know analytics were performed and generally you know see what's going on and not just you know do some computation and then have the data set there so we use some data exploration and and bi tool in this case we're gonna use super set but alternatives are red ash meta based tableau and so let's uh let's just jump in here real quick so the the the the job that we ran a minute ago the one that that that processed the data from the two Kafka sources and dumped into druids stores we have saved to these we have excuse me being fed to these two data sources these two these two cubes and so again these are druid data stores superset is they would connect to a number of different databases druid being one of them there's deep integration with druid so so looking at looking at the a dashboard that I built earlier today this is this is what we're able to visualize in real time using the data that we find in druid which again is being routed excuse me it's being sent to in real time by you know by by knife I which of course is ingesting it from from our edge so so we'll see this update in real time there yes so there are a number of you know visualizations really you know really really anything any sort of visualization that you're able to do standalone you could probably plug it into super set it's originally from from Airbnb and it's now it's now an apache incubated project let's let's actually create a slice so just to see how easy it is to to create a new visualization using the data again that is that is being processed and fed to is made available to super set in real time so the let's say you take violation events we want to group by let's see what it's performing metrics on counter let's let's group by the name of the driver then the type of violation and finally the route that the violation occurred on and that should be it whoops and let's let's get a better visualization let's do a sunburst yes and so so so here's here's a visualization on the the analytics that were gathering from are generated are simulated truck sensors in real time so in this case 22 percent of evolve violations are you know we're done by Rafael and you know of the and of those you know eight percent were because of speeding and you know of those you know thirty percent we're done on the certain route and again all this is updated in real time like and yes so so all this is is is done in real time and again we have sub C sub second queries using druid or our data store so yes let's see yeah thanks for wait did that not you guys saw that right yeah I shared the screen right okay okay good yeah yes great question so the question was when I said real-time what exactly did I mean we can perform sub-second queries using druids druid store but how long you know does it take for all that data for the data to actually get routed and processed and so you can you can define all that you know of course in different parts of the the pipeline with knife I you can you can throttle data when it comes to when it comes to doing ole opera or online analytical processing and analytics with druid you can specify a window a window period you know so that you perform analytics on do some number of some number of data packets over some time giving some you know some some restrictions or whatnot so in this case I think we have I think I think I have it set to like three to five minutes and it could be it could be faster only this is running on my local environment so I think I'm giving it like eight or nine gigs so all of all of this we have running on the laptop with like nine gigs of ram so I had to tone everything down but of course on a real cluster you can sort of yeah sure right so in right so how that fits in with the use case is we're predicting and you can find this on the repo we're predicting driver events before they before they occur given some like something given weather or routes we can predict when a driver is gonna violate how to you know have some sort of violation before happens Oh so that's our that's the that's that's our a streaming streaming framework yes yeah there's a version that uses a spark storm and then we also have one that uses you know Sam and so what that what happens there is the the data is is you know it's it's it's excuse me it's it's transformed it's transformed to look in a specific way so that our you know the the ML lips or have it has an easy time performing a model of it do which which bit right right so oh I think really cut again it comes down to to a number of different things yeah I'd hate to say one way or another but really you can use any one of them I've we have code examples for using storm and spark and they work just as well honestly [Music] using Excel declarative language to the cleaner our flow and then not by transit second blows and yeah yeah the second half was absolutely right and so there's there's I have to have to get pushed what I have now so that'll that'll so it'll look the repo would look a bit different and also it's actually it's in a different branch it's on the master branch so yeah but it's mostly the same so that's my fault I should have pointed that out but but yes your assessment was right so a lot of the XML you see it's just an exported it's just an exported flow for a knife I so it's just there to make it easy to sort of to automate your cluster set up process so it's not actually used in this it's it's there so that you don't have to build out the template yourself or the flow yourself I I will have I will have it named Scala or excuse me scale by the bay after this sure so I think we're have time but really yeah just had a couple other things like you know what things we withheld sort of what you know what what issues we have that use I di didn't point out but otherwise otherwise definitely check it out the links on the bottom it forwards you to the github repo if you're interested in the environment that we used to run all this again it's a local environment using anywhere from like eight to ten gigs of ram with an entire with an entire it's a single node cluster that runs on your machine it is a Hadoop cluster if you're interested in that yeah check that out and otherwise that's it thanks a lot [Applause]