sfspark.org: Denis Kulgavin, Taming the Chaos of Stream Processing

sfspark.org: Denis Kulgavin, Taming the Chaos of Stream Processing

Recording: sfspark.org: Denis Kulgavin, Taming the Chaos of Stream Processing

[Music] Santana's cool guy I'm the founder and CEO of a company called mint data where essentially we help you turn raw data into insights and we do it with real-time stream processing and so today I wanted to talk about something that we've kind of noticed it was really the impetus behind mint data which is that as an industry we want to stop repeating ourselves today we do a lot of things over and over again and there's kind of this great dried don't repeat yourself principle and I think combined with the Scala we can really be much more efficient at the way that we go about essentially building real time applications and enabling real-time stream processing so this light kind of talks about that very notion right we think that there's a great way to dramatically reduce the time that it takes to go from really almost any raw data set to insights for example as he saw in the last presentation with text-based data and doing LSA analysis and we feel that in a lot of cases where today we build and maintain software there is a great opportunity to just define and run things instead for a certain category of problems and so that's what I kind of wanted to talk about today just a sample use case of you know a sample case were essentially something that would be built manually and by hand with quite a bit of effort can really be automated and make us more efficient for actually solving the problems that matter the most so in this particular case let's say that we have an appliance on a network this appliance happens to serve responses to DNS requests from a bunch of clients and so the clients are making DNS requests through getting DNS responses now imagine that we're the vendor of this DNS appliance and we have a lot of these appliances all over the world on lots of different networks and so as the appliances are running they're logging essentially all of the traffic the requests the responses to some sort of a storage could be a flat file an appliance could be some sort of a fancier storage system that's on the network but essentially we're creating huge streams of logs and so one day someone comes to us and says well you know this raw data is great but the first thing we want to do is we really want to enrich this data set so you know we may want to take the IP addresses and enrich them with geographical information we may want to check the DNS requests against a blacklist to see if people are visiting sites they're not supposed to be we may want to do some parsing for top level and second-level domains so that we can see essentially you know which requests are or which sites are more popular and then you know let's say that at some point a business user comes to us and says hey I really like to see reports on this in real time I'd like to understand from an operational perspective kind of how the appliances are doing and I'd like to understand a little bit about user behavior and so we today you know we have these kinds of conversations all the time right we have huge streams of data and we want to build something on top of it so that we can make decisions based on that data and the problem is that today we spent a lot of time between the raw data and I want to just infer insights from it and we think that there are a lot of ways that that can be improved so for example let's say we want to build a very simple real-time application we want to build something that has four pieces on the screen or four visualizations in this case we want to see the requests per second that are happening on the appliance we want to see the historical request rate so perhaps a chart of some sort we want to see a sampling of the data to see you know what's this appliance doing in real time and we may want to get a geographic view to see essentially a a summary of which countries are the most popular in terms of where the requests are going or perhaps even where the requests are coming from if we have a global network of these appliances deployed and so the question becomes how long should this take I mean it's pretty simple right we have you know a bunch of appliances they have streams of data and we just want four simple views I mean this shouldn't be that complicated right but today what we end up doing is when to building quite a bit from scratch right so you know here on this diagram we have for example real time data source so in this case study this is the DNS records but realistically this could be a Dirk West's on a complicated ad network this could be a set of logs this could be events from IOT devices in the field this could be some events on a security company this could be really any data that's moving and so we have this data the first thing we do is we hook it up to a queue and tear of some sort so we've got and RabbitMQ is running around and then the next thing we do is we now have to write some custom business logic to pull things off that queue and then we have to manage connections all the way up to a browser to make sure that all those you know fun things like hey somebody close to a browser tab I'm no longer going to maintain that Stream we have to pretty much build all that stuff from scratch just to get those four views to happen and so if we look at the details of the technology choices we have to make each time is that we say well this is going to be a real time application it's going to be rich it's going to live in the browser so we have to pick an MVC framework so we pick an MVC framework then we end up writing by hand a bunch of views models and controllers to drive these views then we have to pick the networking stack from the browser then we have to write our services layer then we have to potentially add caching if it's a complex enough real time application then we're going to essentially pick our app server tier right we're going to pick for example if we're fans of Scala we may pick the play framework we make it pick servlets JSP really any technology and on top of that again we're going to write more custom logic just kind of wire up the data and then we're gonna wire it up to some queues and a caching layer so something like hazel caps or gem fire for example if the app was complex enough then we're going to add some relational storage potentially talk to some big data systems for file systems columnar storage for document document oriented storage and so we have all these choices and we end up using a lot of these systems and pretty complex scenarios but all we wanted was just a simple real-time app and so the question becomes why do we do this well as developers we want control flexibility we want the ability to choose the best of brief systems because we want to build robust and scalable applications that at the end of the day will be easy to maintain and we'll have low technical debt so that when someone comes to us and says hey the requirements have changed we want to build something different we can say well we have the best tools in the trade we can actually easily make these changes but business users on the other hand all they care about is the lowest development time and cost right so what they really care about is you know how quickly can get this built I don't really care about sort of all the stuff that's half inside and so if meant data we feel that there is a way to satisfy the needs of both people at the same time and so we think that it's possible to do this with Scala and with quite a bit of infrastructure that automates a lot of the things that are unnecessary but without removing the flexibility that we're used to as a developer so this is just a quick slide about our background so we come from a consulting background where essentially we have built a number of low latency high throughput systems and in our experience we saw ourselves building the same things over and over and over again and meant data was borne out of that frustration as a desire to automate the things that don't matter while still leaving us with the flexibility for the things that do so if we have the original architecture with the queuing tier the web tier of the connection tier and browsers and it have real-time data source this can actually be improved and automated for the things that are interesting so in a world with string processing platforms whether it's mint data or something else we feel that the future will lead us to these two components here on the left so we'll have stream processing runtimes how many people here are familiar with apache storm okay so quite a few so by streaming processing runtimes I think that they're here to stay I think the general paradigm of having essentially what we call components or what storm calls spouts and bolts I think it's a it's a very powerful way to distribute logic to run it on a cluster to run it with data very very low latency as measured in milliseconds to do you know millions of events on a JVM this stuff is is here to stay but what's interesting about it is that once you're doing all of that processing how do you actually get it to the end-user how do you actually get people to benefit from that and so we think that in the future there's going to be quite a bit of infrastructure something we're calling stream processing platforms that will take care of all the glue in the same way that we don't write operating systems today we don't write relational databases today we think there's going to be quite a bit of infrastructure in that place and so in particular the way that we think about this is that there be very powerful tooling for defining the application logic in other words for defining what storm would call a topology when on our runtime that we built from scratch we would call it design we think that this should be very straightforward to define in this case you know we've built some components in Scala that will show in the demo and this should be done either visually or programmatically depending on whether you write a unit tests or whether you have a business analyst creating this and once you define this logic you've essentially created the stream processing design then it should be just as easy to define the views for visualizing this data in the browser and it should be very straightforward to actually get inputs from a user to actually drive the application so we think that dashboards are broken dashboards kind of encourage people to have a one-way conversation with their data they visualize the data perhaps they can drill down on it a little bit but there's a very big difference between a dashboard and a true full-fledged real-time application and today as soon as we say we want to build our real-time application we build it from scratch and we think that's broken we think it could be much more efficient and we'll show an example of how that works and so the next thing is we want to be able to flexibly spin up clusters we want to no longer have a bunch of DevOps and scotch-tape to spin up these clusters for different environments and control access for users and so on all of this should be automated in part of the infrastructure so that we can get on with the business of the things that actually do matter to us which is inferring insights from our data and so at the end of the day all we want to do is deliver an application to end-users so in the case of mid data this is a slide that talks about the architecture of how we envision a future like this happening and in fact what we've built out so at the very core of this is the mint data runtime the maintainer runtime is backwards compatible with apache storm but it was built from the ground up to be much more performant both both from a latency and throughput perspective and is essentially conceptually a very similar thing to Apache storm you have components you have stream processing you have a clustered runtime that's able to process data at very high rates and this can run both on-premise and in the cloud then you have something that we call the mint data workspace so this is the place where you can centrally manage and environments manage permissions managed which designs are being deployed the history of designs data provenance and so on they have the visual designer so this is the place we can actually visually define what you want to do as well as defining the visualizations that you want to hook up to the run time so that you can empower business users to actually very quickly make decisions and then the last thing that we have are the mid data real-time application so we'll show an example of this in a second but this is essentially what a business user will consume and to them it will look like something that took months or perhaps many months to build but in reality unmanned data can take just a number of hours or days so with that I wanted to kind of briefly jump to a demo and we can actually see this working in practice so here we're in the workspace application we've opened up a design and we're going to add a component set so we're adding a component set under the covers this is just a maven artifact so these components happen to be built in Scala they could build be built in Java we could even invoke something like Python under the covers and so now that we've added a component set we can actually very quickly start to drag and drop these blocks and each of these blocks will take a look in a second of the code that's behind all of this but each of these blocks is mapped one to one to a Scala class or it could be a Java class and so each of these blocks has inputs as outputs has properties and also configurational aspects I essentially fields that can be injected using dependency injection so for example we did a talk at a meet-up a couple weeks ago where we literally integrate an apache spark onto our platform in one day because it was just a matter of injecting a spark context into one of our blocks and then having a data frame operator be a block on this platform where you can define the types of data frame operations you want to perform in this particular case we actually we rewrote all this these components for this meetup to prove that this can work all in Scala so here we have a message generator it's generating some dummy DNS data then we're enriching the data with the geoip database then we're doing some domain parsing in the domain enrichment block then we're dealing let's see what else we do and then we're I'm doing some blacklist checking we're adding a utility component set in this case the utility component set this is actually a pretty interesting thing so if you have a high-performance stream processing runtime that's processing millions of events per second there's no way that you can get all that data to the browser to visualize nor when you want to and so on our platform is just you know you can have a couple of utility components that slow down the data they sample if they take a top band analysis and once they've slowed it down then they'll actually take that down to the browser so in this case what we've done is we've switched to the visual View for building real-time apps and here we're very much dragging and dropping visual controls and all these controls are just pure html5 so if we wanted to we could build a number of other controls in any favorite client-side framework and we'd have more opportunities to both visualize data but also to gather inputs from the user and put it back into the stream processing runtime so in this particular case we're going to set up the gauge control configure a couple of settings bind it to the data model and then in a couple of minutes we'll see how all of this comes together where basically we've created a real-time application in in this case in seven minutes instead of something that could take days so in this particular case where we're setting up a table control setting up the columns we're binding it to the data model on our platform for example all the messages that are going back and forth are strongly typed and so you have a very clear notion of what's happening between your stream processing components and if you do need something that's weakly typed we also have the notion of message properties so this is a weakly typed piece of key value data that you can pass along with each message if you wanted to but essentially is used typically for things like session IDs user IDs and so on so in this case we're setting up the chart component we're binding it to the data model we're setting up the axes we're essentially doing the data binding portion of this so that we can hook it up to the what's going to be happening in the runtime on a cluster and actually get all this to work end to end so the last thing is we'll configure for example the map component and we have typically two modes replace or so depending on what we want to do with incoming data so here we're going to pick for example the the map of the world we're going to map it to the the data model again and essentially once we hit apply in this case we want to replace so we want to see kind of just a real-time view of which countries are being hit the most and so now what we've done is for all the visual controls that we've dropped in the real-time view we actually have these dotted components or components with dashes that represent the visual things that we just drew and so now that we have those visualizations it's a matter of saying okay here's where they fit into the logic and once we hit run all of this will just kind of automatically through the workspace infrastructure be streamed down to the browser and you know we'll be able to share the resulting application as easily as a Google Doc so in this case we're going to configure a couple more things and basically in a few seconds we'll get to the point where we can actually run this and see everything happening live so while this videos running any any questions or thoughts at this point yeah that's a great question so right now sure so the question was is this open source yeah so right now this is a commercial product form in data and over time we have plans to open source large portions of it so for example the lot of the core platform components I think in the future you will see open source yeah so the question is is there any state in the system that's actually a very important architectural decision so in the case of the meditator runtime we feel that if you want to have stateful processing you can but it shouldn't be at the level of the engine so for example there are engines out there that will behind the scenes try to snapshot state they'll try to replay state we feel that that that shouldn't be in the engine it should be at the level of platform components so in our components yes there are a number of staple components there number of mill sequel engines that we integrate with there are a number of things like hazel caste you know for example if you wanted to have a shared cache on the cluster that you could run but we feel very strongly that this should not be in the engine itself because for example you know the world we come from is financial services data where you have easily over a million messages per second coming into a cluster and you can't lose a single message and essentially both throughput and latency matters and so in some cases you can't afford to save that state and until you've actually done the processing and so so in the case of our engine everything is done there notice that there are no assumptions that are made for you at the at the engine level mm-hmm yeah that's exactly right so you would add the best-of-breed component that you felt was right for the job so if you wanted to do top-end you could do a windowed based analysis based on some short time interval and that you would typically do in an in-memory fashion if you had a business requirement to do this on a longer interval then you would use something like a saundra it would it really be based on the the data that you have at hand okay so let's see other questions before I dive into the code okay so let's take a quick peek at what it takes to write one of these components so in our case we felt very strongly about the fact that we want to change as little as possible about the way that code is written today so for example in the case of a printer component it's just a matter of dropping a single annotation and for all the inputs that are going into the blocks we drop an input annotation and becomes a method for all of the output components for example for components that have outputs it's a matter of dropping an output annotation so for example if we want to take a sentence that's coming in split on spaces and emit a bunch of messages for each word that's coming out it would literally be as simple as dropping an output annotation an input for the sentence messages that are coming in and simply splitting it and there's also dependency injection happening under the covers so for example if I have a random sentence component I have the ability to have any config annotation so this allows me to do dependency injection for any piece of functionality into a platform component and so for example this is how we took a pachi spark and integrated it within one day on the platform we literally took a spark context wrapped it in this config mechanism and we had an in-memory small version of spark running on each mint data JDM literally within a day taking advantage of the data frame operators and so any scholar of java technology is literally at your fingertips within one annotation and one wrapper class to be able to inject so other other questions on kind of the things that we've shown yeah that's that's a great question so it's a couple of things we used to build a lot of systems on top of Apache storm and so one of the things is that we've anecdotally seen that the stacktrace if you look at storm or the call stack rather at any given moment in time is substantially larger than on our runtime so there's simply more things going on in storm that's one of the things we've noticed the other thing is that storm under the covers there they're using threading in a slightly different way so they essentially between two components and they have a transfer queue so they will have basically a context switch between threads when one component is emitting you'll have a context switch to this transfer queue the thread that's processing it and then you'll have from that thread another context switch to the thread of the downstream component from it that obviously is very expensive and it's also necessary in the case of storm the best that we can gather from it is that they were concerned about feedback loops but there are more elegant ways of dealing with that scenario so on our platform you could obviously have you know you can draw the arrow as a feedback loop that's not a problem but we're not entirely sure as to why they have that additional cue for transferring messages between components we've seen that's another case for latency but yeah I mean if you reach out to us we have you know two zip files you can download one with storm one with our runtime and basically can see all the sources run the two demos and see exactly you know in your laptop or on your infrastructure you can actually see us prove out the fact that it's faster than storm other questions correct if you wanted to you could so if you build components using these annotations and run them on the mint data platform if you really wanted to we could give you an adapter that would allow you to run it on a storm cluster we have reasons why we wouldn't recommend it but it's there kind of as a psychological safety guarantee that if you really wanted to you could run it on a storm cluster yes if you if you have existing code that runs on a storm cluster it's very straightforward to run it on mint data all you would do is you would add these annotations to your spouts and bolts and they would magically become things that you can drag drop connect visually wire to the charting and visualization components and you'd be off and running so it's not automated but by dropping a few annotations you could get there very quickly ah that's a good question that's probably for our engineers I know that for for all three cases they've managed a way to implement it I don't know the specifics but if you follow up with me by email I can get you an exact explanation so the question was how do we implement exactly once guarantees correct so this yes so semantically it would be similar to what Trident provides in the case of struggle yes so the question was around kind of infrastructure so our ability to interface with external systems so that's actually one of the the shining lights of data we ship with quite a number of components out of the box and you can pretty much interact with most of the big data systems today without having to rewrite that same glue code that you end up writing over and over for different applications so our ability to work with sort of any no sequel engine columnar storage queueing system relational database all those things are incredibly straightforward and fast to do because he just drop existing blocks and you're up and running very quickly other questions okay great thank oh so for forward ah is so do we support fault tolerance yes so when it's running in a clustered mode we have the notion of workload sharing so for example we have the same notion as parallelism hence yes storm does so for example when you're running this in a cluster if a node dies the existing nodes will take on the workload of that existing component if a new component joins the cluster it will be the existing nodes the existing JVMs in the cluster will share if they will essentially shed some of the workload some of the component instances that are running and that new node will pick up those instances activate them and they will become active on the cluster the way you would typically handle that is you would essentially there are you can implement a callback listener so if you're reading from a queue you typically wouldn't acknowledge to that queue until you know that you've fully completed the stream processing so if you come in a single interface on our platform we use actually very similar mathematics to storm and for the core logic under the covers so that when it reaches zero we know that the message has not spawned any more child messages and that all the processing for that particular source message has completed and so then you can acknowledge it to the queue so that's typically how you would handle this as opposed to trying to replay or store magically replicate the data we did that on purpose because if we build those kind of guarantees into the engine they would kind of come to bite you in terms of performance for the cases where everything is running correctly so we find that the best way is to let to not acknowledge to the queue that you typically reading from until it's fully processed and if something died in between then the queue will either you'll read from the key or the key will replay it too great questions so in theory yes this quran basis with a very straightforward adapter so the same thing for maysa for yarn it's a very lightweight sort of JVM based runtime so anything that allows Java code to run even for embedded systems this would be a good candidate Cory okay thank you [Applause] [Music]