data.bythebay.io: Vladimir Bacvanski, Apache Flink: A Very Quick Guide
Recording: data.bythebay.io: Vladimir Bacvanski, Apache Flink: A Very Quick Guide
great thank you so I'll just share briefly some of the things that what we are doing in our company we do quite a bit of custom development and one of the recent developments was dealing with Internet of Things systems particularly in healthcare and then the next set of clients would like to do that in automotive space so I would like to share with you some of the typical things that we have when we work with systems like that so I'll scribble a couple of things on my tablet here so one of our applications in healthcare is involving having patients try that ok you hear me all right so we have a patient's so we have a patient that usually don't feel too well and instead of having them go to to see the doctor we actually put various sensors on them and then we monitor various parameters like temperature and pressure and now we will be getting also some new sensor that can capture the heart rate and similar and then what is happening with these sensors we are using low-energy Bluetooth and we are sending this to their mobile device and then this mobile device is going to send this through the internet into our systems in a cloud and this is now where the interesting part actually begins and there we have first systems that are dealing with ingestion of data and then when we induce the data we are going to do some processing and also we store this data into data store this is usually some data store that will support the time series data it can be Apache Cassandra which we use the number of projects or openbsd B which is based on age base now when we do this there are two interesting flows that are happening from here one is the batch analytics and that one is interesting because we have these patients who are involved in political trials and then we aggregate the data and then after a fairly long period of time we evaluate the effectiveness of these different drugs that these people are getting and this is then producing interesting reports that can be analyzed but there is usually no great rush in order to accomplish those one of the other areas is that we would like to be able to notify nurses and physicians when something interesting is happening with our patients and this is where we have a very different need this is where we would like to have nurses who can go and attend to the patient when something is happening and the interesting thing here is that this area is fairly a real-time oriented so here we have a real-time and here in this other branch oh we have the conventional batch processing now when we started working with our clients this older systems were typically based on plane Hadoop and later we have been looking at the spark to improve our efficiency but the the problem in this space is that sometimes you would like to react much faster then the the spark is allowing us and we were looking to introduce some true streaming systems in this picture we have been looking into various solutions out there and solution that we have found in flink captured our attention because it enables us to combine most both the batch processing and the real time in the same framework and also many of this analytics that we are doing are going to be using very similar code so using the Kappa architecture so having a combined combination of lambda architecture with the very similar code for both batch and streaming is something that was very attractive to us now our systems in the current version are not really designed to respond to something like a heart failure where you need to intervene with in a really short amount of time but we have clients that are trying to do something really similar but they are operating an automotive space so you have cars that are instrumented with various sensors and they are sending these values there and also one of the interesting things in automotive space is that the data that we are dealing with are not coming only from the cars but they are also coming from environment so there is another aspect of this system and this is the data integration in in real time so we would like to be able to achieve the integration of not only other cars but also with other systems and that was the motivation for looking into applications that are better support the streaming and batch processing so when we look into development of this application so first we have seen traditional approaches that went from warehousing then we have seen batch and then we have seen the growing popularity of streaming systems and the streaming systems were coming with different API with different runtime characteristics and we have seen that there is really a space for something in you so one of the things that is interesting for us is the the set of capability that we are getting with streaming so one of the things that was really interesting in the beginning is that we could combine data from different sources and merge them and have this fusion of data and we have seen something like that in real time systems with sensor data fusion and similar the second thing that was really interesting for us was the development of low latency applications in my company one of the things that we were doing at the real-time system that using oil industry and responding often within milliseconds is very very important thing and now with the internet of things we've seen that pretty much everything can be instrumented and can be sending us signals and having a really low latency below the second is something that you would like to see and we see that in many cases the streaming capabilities are more powerful that then things that we can achieve just with the plane badge and you see looking into what is happening in the world of data systems we could see that there is a quite a bit of interest in in development of such systems and we have been following one project that is coming from European our research the project stratosphere what is interesting for me is that this project is not starting exploration of big data space from scratch like what we have seen with Hadoop but having the background from technical university of berlin this is the team that was famous a long time ago for their work on parallel database systems and optimization of relational database systems so one of the things that was really exciting for me is that the knowledge that we have accumulated in the development of conventional system is not forgotten it does not need to be reinvented but we can actually build on top of that and that this project started from certain premises how to build a new data engine and one I think that was visible is to start with streaming and then a support for optimization so when you are having bad jobs we would like to be able to optimize them based on the data sources based on the nature of our development are we running on a local machine are we running on a small cluster large cluster what is the distribution of data all of these things we would like to have considered for optimization the third aspect is that when you look into machine learning algorithms you would like to be iterate iterating and this is something that traditional a big data systems were not doing very well and finally there is a need in graph processing to maintain a mutable state and various algorithm will be updating this state so what was the desire in this project is to have an engine that would natively support these use cases and not start from batch and then let's see how we can tweak the batch to support this other cases so when we say the native support we are not thinking about running conventional MapReduce jobs a number of times but we would like to have iterations as the normal part of the development as a really low level construct that is built into the system the other thing that is not ideal is if you take your stream of data and start shredding it into pieces and they say well these are the small batches of data and if I process them fast enough it would look like a real streaming system and there are differences between discretized streams and through streaming systems so when you are working on true real-time systems you would like to be able to process events as they come without being forced to batch them the project eventually resulted in what is today known as Apache flank and it has a couple of interesting areas so probably the most interesting part in this space is going to be the core of the system which is the distributed streaming data flow engine the interesting thing here is that the streaming is the native mode of operation of the system and you can view the batch as a stream that will terminate at some point one of the other things is that the system is very flexible when it comes to deployment for the development we prefer to run it in local mode and it goes from clusters and cloud applications one of the things that this picture does not show that the system is quite agnostic when it comes to data sources and sinks so you can connect to a variety of systems from distributed file systems too many of the databases out there on top of this runtime we have the two api's so in the past everybody was starting from back processing a data set API and which also includes on top of it some machine learning libraries graph libraries you think that we have seen in in their systems but what is really interesting is this data stream API through stream processing very elegant API so if you look into the API you will see that it looks very very similar to scholar collections to which you apply different operators and there is exciting work on adding the table interface on top of that and also what is particularly interesting for me is complex event processing we are going to talk a little bit about this today now in the big leather community there is one interesting project that is hampering right now this is a Apache bean which is based on google data flow and apache flink is the most complete a runner for this api today so this is quite interesting apache beam will provide the uniform interface across a variety of big data systems so when we look into a typical application architecture for system light with flink typically we see that we have a variety of data sources then we have a lock system that is ingesting the data we do the analysis with flink and then we can store the result into into various assistance including variety of database systems conventionally many of the applications would get the results from fling put it in a database and then we would query these data in these databases one of the things that is coming later is ability to actually query the flink itself so you don't need to put the data necessarily in the store to achieve that so typical architecture and looking into some of the key characteristics of link in this space is first the orientation of the engine to execute everything as streams pipelined execution we are going to see in moment what is that and also one of the very attractive capabilities is that you have the back pressure from the source all the way to the sink one thing that is of interest is the optimization so in comparison with various other systems flink is trying to do a lot of optimization on its own and is something that we have used to have in relational databases when you have your database there is an optimizer that will calculate the access pass it will consider where at your factors and then choose the optimal optimal path of execution for you data flows that can be iterative stay that you can mutate and save and one of the things that the flink introduced very early was the manage memory so the objects are stored outside of the JVM heap there is a separate part of the system that is enabling you to fill up the memory of your servers and you are not impacted by the limitation of the JVM memory management the garbage collection and you know when you know exactly what is just your access pattern of data what is the expected life cycle then the maynard memory can introduce significant optimization about in speed and ability to store more things so one thing that is very interesting is when you start from a blink program you are going to have a pre-flight part of the execution that is running on client that is going to do basically the compilation and optimization what you get as a result of that is the data flow graph and you can think of it that this graph is the logical graph that can be then optimized and adapted to your actual of platform so the system can choose the right path of execution and then eventually you will deploy that on your cluster that consists of master and the workers so this ability to optimize things is for me quite interesting because we don't need to spend too much time fine-tuning and optimizing the system on our own so we said that the streaming architecture is the heart of the system so how do you build a streaming architecture so you have a couple of things that you need to accomplish first you have the basics and this is a pipelining moving and processing of events through the system you can also replay the streams we needed on top of that you would like to be able to manage the say also you would like to be able to have a backup and restore support of fault tolerance and then you would like to have a nice api's so the api's that we have been playing are interestingly enough coming in several flavors you have a scholar API which is my preference but there is also Java API because not the whole world will move to scala really soon and also eventually we have support for large deployments with a veil of availability and scaling and scale of one of the next thing that is coming is automatic dynamic configuration of system so you can dynamically expand when you have the lead so pipelining is the internal mechanism that is used for processing of events so this is happening in the distributed fashion you have the events that are moving through the system and conceptually events are moving individually in practice you typically have groups of this elements that are moved together in order to improve the throughput and this is something that we have seen in parallel databases some time ago we have the state for each of the operator that is doing the processing and this transformations are going to match two steps like mapreduce and so on and one of the really interesting capabilities is working with streams of data that fit into certain windows and we are going to see in a moment where they are now looking into the state of these operators there is a new feature where you can take out this state and move it to the external system and the current version of link is using the rocks DB rocks DB is embedded database developed at a Facebook very high performance and a very good choice for for preserving the state now event based systems are having the notion of a windows and the typical form of windows draw a little picture here so when we look into the windows we have the first the simplest type of window let's say you have a data stream that is coming like this and then you have a various events the tumbling windows are the windows where you have no overlap so you have one window like this and that the next window is like this and so on then you have the windows that are sliding windows so have your event stream these are going to be the windows where we have a certain overlap so you have one window that looks like this then the second window is going to be like this the third window is going to be like this and then there is going to be some amount of processing that will be happening over event that you have already seen but one of the really exciting form so this is what other systems don't have is the notion of events that are based on time when these events actually happen and when you look into these events so if you peek inside you are going to see that event like this are going to have some payload which represent your data but also there will be the information about the time and what is really fascinating about this notion here is that this is the time where things happen in the application this is not the processing time that you see right now on the server and this is really exciting because for many applications just imagine distributed applications or information that you're getting from mobile devices you can get disconnected for some period of time and there is a notion of the time where the event actually happened this is what you would like to preserve and this is you would like to reason about so this notion of time is extremely important for the time-based windows which are unique feature of of a flank so talking about time we have mentioned processing time ingestion time and particularly interesting for us it is the event event time you can write your extractor that can get this information from the events that are happening now when you're dealing with such advanced mechanism you would like to have ability to fine-tune and to define the grouping of it events because certain events belong together and they can be result of a session the user may have now if you look into other streaming systems how do you identify the things belong together now usually you would say well I'm going to have the window and then every five seconds I'm going to treat this events as they as if they belong together but this is not the case if you have sessions for example if you have a session of the user who is interacting with the website there is some notion where you know okay we are starting the session you have some period of browsing and maybe later we are going to finalize the purchase when we are done with that this is the end of the session and it can take a different time to actually complete so the watermarking is the mechanism that can be used to find you on the processing talking about iterations you have a conventional type of iterations and that is you will use the whole input that is available at this time and data is basically reprocessed and this is typically the first step that you would do but then you quickly desire to optimize these things and and in spark for example we have to write our function that with the operating in a such incremental fashion in flink we have the Delta iterate runs only on data that has changed and that is very convenient because it gives us a significant speed up because we typically process just a fraction of this events fault tolerance for the systems they have been various semantics among the assistants one that was promoted for a long time as acceptable is at least once it turns out that this is not good enough and we will fur to have exactly one semantics for the events and we would like that this exactly once happens from end to end and this is something that flings supports out of the box there is a variation of tenderloin port algorithm that is allowing to to achieve this so exactly once it's very convenient feature for applications as we are working on the fault tolerance checkpoints is a mechanism that enables us to save the state so if something breaks during the processing we can rewind to the logical time of the least last checkpoint and then continue from there one of the new things that we see in flank is a notion of save points and these are the checkpoints that have been taken by the user so they are available externally they will be stored in in a data store and they don't expire and you can even explicitly trigger them and then and use them so you can use your command flink save point and then the ID for your save point and you could even a resume the execution of your system by saying clink run then you put the pass to your safe point and then at the application jar file that will run the system and you can imagine this is a great feature for production because it enabled us to better handle the updates updates of both of our application and updates of link migration from from one implementation to the other and we could even run various what if simulations you start from one a safe point and then you try to run some different execution so somebody recently commented that this behaves almost like a github for the state of the streaming system you can build a graph of these transitions so very excited about this new feature so when we look into the whole set of benefits so we see couple of things the processing that is happening in the streaming engine does not block and that results in a much higher throughput and lower latencies processing than in some other systems I have some references for you later where I point to a couple of benchmarks that can be done at Yahoo at couple of other organizations and there is one that is particularly interesting venture from Korea which is showing how flink and execute things in in parallel and it and this is thanks to the non-blocking nature of execution then we have the business logic is separate from the recovery so we have that the checkpointing interval is the configuration parameter when you do this check pointing in spark you know you have to really write code in order to achieve that and it is a non-trivial code so we get it fairly automatically and one thing that you particularly like is support for fairly rich window semantics so particularly the notion of session windows is a very practical the notion of event time is it's fairly unique in this space and one of the things that is quite interesting you know google data flow has a very rich set of features for processing streams and flink is currently the most complete implementation of this algorithm and there is a great alignment between the beam project and what a flink is offering when it comes to window exim antics a couple of other things through streaming latency so we can process things that are happening within milliseconds exactly one semantics and a low overhead for recovery are very attractive points so looking into code the code that we see here is a scholar code Scala is my preferred choice when it comes to implementing big data systems you know starting from a scouting that was invented a Twitter which was the dsl 44 Scala that would compile Scala operations and collections to the operations on piped architecture from cascading we have also seen similar things in spark and if you know the spark API you will immediately recognize similarity because the inspiration is the same the scholar collections are so powerful and scala provides for very convenient development of domain-specific languages so it is a natural choice I like those api's because they are less verbose than Java API s but flink also has a Java API is for everything so it may be easier to introduce in more traditional organizations so here we have an example arm that is defining the data that we will be processing there is a class word so when you introduce something like this it is going to define set of objects that consist of the world and the frequency we have first example of the data set API which is the batch API and in the APR like this you would read some input and then you would do your fairly normal operation that you would do in a scholar program so you're going to take the lines which is the data set of string we are going to execute flat map flatmap is going to do two things the first thing it is going to do it is going to split the line using the space and then it is going to do the map from each of the world to an instance of the world class where we have the word as a string and then we have number one you can imagine that the input is coming in lines so if you would do just plain map you would have the collection of collections that contain word flat part will execute flatten operator on this and we end up with collection of words that is going through the system then we do the grouping by word and then we sum this and get the frequency the streaming API is very similar you can notice that there is a little difference here instead of reading that text file we are going to get the input from some socket the code that is actually doing the processing look the same you see the lines are the same and there is one line that is different and when you are working with stream you need to think of what is the window and this is this line here we are defining here what is the window and what is the sliding parameter and this is something that is a very attractive they pretty much the same code that can be used both for batch processing and for the streaming systems a very attractive thing one thing that is make it a little bit bigger so here's the variation of this program it is still a bad program that is going to take the inputs from the stream one of the nice things here is that when you are developing with flying you can very easily work in your development environment there is a local mode for execution you don't even need to install flink on your machine it is enough to define dependencies for your maven file and you will be able to run this and you don't need to run any special runner you can just go to the code that contains main and thus a run and I have now my program running so flink is now running and in this version I'm going to open my monitor terminal and i will zoom in in a second so you can see what I'm doing so I started the net cat command so I'm going to stream something on port 9999 and i'm just going to enter so I just make sure that my program is running here alright and now i hit enter and you can see here at the bottom we have the result so what I want to say here is not to impress you with the API but I want to want you to see how convenient is to develop with fling so you're not wasting much time on trying to get these things to run all right so veteran streaming is Flinx interpretation of batch execution which is mapping the bad processing to streaming now I would like to talk to you just a little bit about the complex event processing which is a new feature in fling and I'm particularly excited about this thing because when I was writing my PhD I was implementing complex event processing in C++ and i was using antler and defining my own language and now i have this thing in flink and even works on a big scale so when you're defining complex event processing systems you have a certain set of operation that you want to perform so you want to be able to define the beginning estates so you will start matching events the events can come in different order this events need to satisfy different conditions and need to be a certain type so there is the next event so you can say after this event I'm going to see the next event there will be no other events in between I can also choose to say followed by that means this event will come at some point not necessarily immediately I can filter the event by the where condition i can define within define some time interval so i can say if this happens within this time arranged i'm going to respond and also i can define what is the type of the events so there is a sub type parameter and here is a little example so this is the implementation that is simulating things that could happen in intensive care unit we are monitoring the patients so we have the warning pattern which is too high heart rate events with the high blood pressure within 10 seconds this is so this is the pattern that we want to match and currently a CP for playing supports only Java but you can see it is it is fairly intuitive you can define there is the alarm pattern and then you would say we are beginning with the measurement and we are checking that the type of the event is the heart measurement where the event event get risk is greater than equal than one there is a little formula that would calculate that based on the blood pressure and heart rate followed by so there will be another event we call it middle it is a blood pressure measurement where the risk is greater than or equal to followed by so that third event is coming at some point and it is also hard measurement where the risk is greater than three within time seconds so this is very powerful very very natural way of addressing the rules that we have in a problem domain in our system everybody wants to compare a flink with spark and storm so i could not disappoint you so i have a couple of slides here that i have borrowed from death artisans the interesting thing about flick it is a true streaming architecture and i would the particular likely point here that we have a low latency as opposed to spark which is using micro batching that operates as a medium late as the system so you have the league there you can go and see a little bit further there are some other comparisons and when I'm thinking about comparing different systems recently Apache beam emerged as a framework that provides a very nice model for classification and comparison of features of different systems so the beam is coming with a very comprehensive model for semantics of different big data and particularly streaming applications and it is organized along dimensions so what results are recalculating we're in the event time when in the processing time and how do these different requirements relate so when you follow the link you will be to see the comparisons of them be model itself Clowe the dataflow implementation from google there is the fling and very sparkly the different capabilities of these systems you can see if link is satisfying more of these options than than many of the solar system so quite interesting so follow the link it is also good to learn what are other powerful capabilities that you expect to see in streaming couple of benchmarks I'm not showing this benchmark to start some war about you know how the tests are not done properly which is really when you show the benchmarks I just want you to see that it is something that is worth investigating something that is worth looking for and the advantage is when it comes to performance can be quite significant so here is one from strata San Jose quite interesting measurement 50 million events per second when you when you get rid of kafka that was slowing down the input and here is some other a benchmark that is comparing the latency if you can see if Lincoln storm are operating with the low latency well spark is not performing so well so a couple of things that are coming sequel for streaming this is also quite interesting feature very productive feature Apache calcite is the sequel layer that allows independence from sequel and different stores then I'm extreme ink as i mentioned ability to query the state of the system so you don't need to offload the stuff in the database a couple of other things which means the support and over the wire encryption for the systems so quite active community a couple of indications about the growth and I just picked up this last night thousand three hundred stars and 800 Forks so there is a growing community and at the end I collected two pages of good resources so you have an easy start naturally first the link resources if you are interested in the performance comparisons quite recent reports if you are new to streaming there are two really good articles on Riley side streaming 101 and 102 also that will include various newer features that we see in Apache beam also nice explanation of water marks and a beam itself provides a nice framework for for comparison and finally clean bottle G has a nice set of links on various big data systems and a particular emphasis on flank so that's all thank you very much if you have questions just hang around I'll be here thank you