reactive.community: Apache Flink: A Quick Guide to the Next Generation Stream Processing Platform
very good so let me ask you a couple of question first how many of you are interested in building reactive systems all right how many of you are interested in building streaming systems all right pretty much everybody well if you allow me let me just tell you a couple of words about what I'm doing I'm a founder of company called size Pike we do a lot of custom development for our clients many of our systems are scalable web applications but increasingly Internet of Things applications where we are processing streams of data from sensors and besides custom development we also do consulting so we work with CEOs of number own companies and advise them on the technologies and we also do some custom training which are courses which are adjusted to subject areas of our clients and we teach new technologies in that context just as a side effect developing with Scala is Scala for mortals that moves a normal Java engineer into a productive Scala engineer in fairly short time and then we typically continue with scholar reactive systems concurrency akka spray or akka HTTP and also big data with spark and Scala we are planning the new course on on Big Data and flink and I want to tell you one of interesting announcement and that is about the platform that we have been developing since 2010 this is the agent based system for nodejs so it is similar to actors there are some differences it is a full suite of tools with domain-specific languages and tooling IDs you also have the runtime insight into the execution of the system as cooperating state machines and we have been applying this in areas Internet things in healthcare trading systems a number of applications where we have the concurrent activities of multiple parties in complex scenarios so it is open source now just about months ago became open source and we are currently improving on its documentation it's probably the most needed part so if you're curious about it check it out alright so I want to share with you types of applications that we have been developing lately so I'm going to share with you an application that we have been building for our clients which is a healthcare a company this is the system for monitoring patients so we have patients who obviously don't feel so well and then we attach various sensors to their bodies and we are collecting various parameters so we are collecting first things like a blood pressure temperature pulse oxygen level in blood and so on so the patients are at their homes and the sensors are sending data using the low energy bluetooth to the mobile device so this is usually a tablet that they have around or it can be a mobile phone so these things are beaming the data to the mobile device mobile device is then going to take this data and it will transmit it through the internet into a system that we have at the backend and this is the bigger part that we have in the backend this is kind of the interesting part so we have data coming in into this and the backend system consists of several parts so first part is a part that is doing the ingestion of data so we have ingest functionality and typically with this ingestion functionality we are going to store this data into some form so typically we have a database that we are going to use to store this data and these systems are typically Cassandra which was our common choice for many applications one of the interesting features of Cassandra it is really good for time series data for streaming data and we can query that in near-real-time but also we have options to store this data in Kafka which can be convenient to replay the data and also we can store the data in HDFS open tiers DB open time serious database is the system that resides on top of HBase so if you already have a Hadoop cluster lying around it is convenient to set it up and it will give you very nice visualizations so this is the ingest part ingest part is then going to forward the processing to two different areas so one area here is the real-time part so this is the real-time of monitoring and this real-time monitor is going to interact with the user interface so here we have nurse and the nurse will be monitoring and nurse will be getting some alerts if the values are getting somewhat strange so it is not the system that is really saving lives because if you start to claim something like that you need to go through a very difficult certification so we are just kind of providing help for monitoring so the second part is interesting the second part is dealing with batch processing of this data so this is the bad processor and that one is interesting because that enables us to take this data and analyze it over the longer period of so we can analyze the results of these values and how they behaved in the last three months we can plot them we can see what are the trends and this is used for a reporting that is done for clinical trials and for the users and for the operators of this things this has a huge advantages because the patients can stay at home and we also get much better values traditionally the patients would go and visit the doctor maybe two or three times a week and then they would get one measurement point about their parameters now what is happening in with some of these conditions these people suffer from is you may have increased blood pressure that comes in a spike and it may last for a couple of hours and then it goes down and if you're not at the doctor at that time you will never know about it so we are able to collect this data and and the interesting thing that we have here is this real-time monitoring and batch both of them provide some calculations that are about identifying some values that are unusual and traditionally you would have to implement these two functionalities even in different languages possibly different technologies and you see significant duplication of effort so what we would like to have is very similar if not identical code that we will be able to use for real-time and for batch analytics so they'll give you an idea of why we prefer to have a technical solution that will give us support for both its batch and a real-time in very similar way some other applications that we are working on are applications in automotive industry where you have a car that is instrumented with various sensors and also knows about the driver and the car is connected to its environment so depending where are you with the car you are in certain context and you can engage in different interactions with various third parties including merchants additional services and similar but the overall architecture is similar to the family of systems for Internet of Things so naturally the technologies that we would like to use for this are something that supports a batch processing and streaming and it is also something that is nicely fitting into the Big Data space so some years ago you would think of something like lambda architecture which will enable you to have HDFS and Hadoop processing for batch and then you would use something else for example of Apache storm for processing the real-time data stream but the problem is that very different way of programming for these two lots of duplication you have two very different technologies then we have seen the emergence of Apache spark in the spark model is somewhat unifying the streaming and the batch processing and the way how a spark deal with this is that they are actually simulating streaming by taking the stream of data packaging it in the little batches and then executing those in so called micro batching approach if we go now a couple of years back and look what is happening in Europe European research in data space you had a really solid foundation a research in database technologies data ms optimization parallel databases and that the european research community has seen that in the u.s. hadoop is taking ground there are commercial systems and the europeans did not have a sound answer to that so that resulted in the european research project called stratosphere which is combining the work from technical university of berlin which is very strong database group focusing on optimization of databases some research from sweden they got together and that they said we need to build a new engine for a big data they were looking at the alternatives and they have figure out that they have couple of demands for the next architecture for big data process and and they have organized their thoughts along four dimensions first of all they have thought that they are going to organize their processing around streaming so they are not going to start from batch and then try to adjust it to streaming but they would try to do the other way around so they will start with streaming and basically you have various interesting topologies for streaming and you are processing your data as stream no matter if you are dealing with real-time stream or batch because think about it what is a batch batch is just a stream that terminates at some point that is a batch so they have proven that you can build batch processing on top of streaming now you remember I mentioned to you that these teams have part a bit of background in optimization of data bases so one thing that immediately comes to your attention if you're coming from database background and you're looking at what's happening in Big Data nobody's optimizing anything everybody is going just for the brute force approach if you need better performance you just get more machines to your cluster and where are all these fine things that we have been doing in optimizing relational databases so all of that is suddenly forgotten and if you look for example early hive for a Hadoop basically no optimization at all so this team is figuring out that what they want to do is they want to bring back the optimization and instead of having a relatively expensive mechanism for optimization are done by the developers by hand they would like to have systems that can automatically convert the execution path into more optimized versions so we have the optimization as one of the interesting requirements so optimize the floor if you look into Hadoop and if you look how do we process things in the Hadoop you are going to notice that you are missing one thing let's say if you go to your big data process and go to pig or hive and you look into your program and you would like to execute an iterative algorithm you can write a function in Pig or hive but then to run it again you would typically need to write a bit of a shell code that is going to restart your processing now with different parameters over the files so you don't have support for good iterations and if you would like to have a good iterations you would like to preserve what you have the state of the iteration that you want to keep around you don't want to abandon it so iterative processing and that area is particularly driven by applications in machine learning many algorithms there require you to work with the state that is going to be updated as you are executing your system so you need to have support for iterations and the built-in support for regulations and quite related to that was one of the other areas which is emerged for processing of graphs when you have a graph processing you are going through the graph and you're processing the elements and as you are doing that there is a notion of a state that you would like to preserve so the notion of state is one of the significant points in in the new architecture so streaming optimization iteration and having explicit state by the way you don't need to take pictures of this I'm going to save all of these drawings make a little PDF booklet and then we can say we can share that at the end of the class all right so these are the four things so with the team I was looking into what is that they don't want to do so one thing that they definitely did not want to do is to write applications that would look like this you read the data then you are going to do some processing then you save this data to the disk you do then the next thing in processing and so on the problem here that in this type of processing is that you're losing your state and you need to reread the data from the disk this is your typical Hadoop processing and the other thing that they didn't want to do is when it comes to processing the stream of data if you have a stream that looks like this one thing you don't want to do is start reading this stream into little pieces then you take each of these elements and then you process it and then you are going to produce a new little element that is just showing this this little batch this is micro batching this is what SPARC is doing and for many application that might be adequate but it is not fine for the true real-time streaming applications so that is the overview of the background what is the rationale for the new streaming architecture the project stratosphere which was the European community research project produced number of wonderful technical reports and the core of the new system that became known as blink blink in German means speedy or quick and it resulted in the architecture that has a couple of layers the foundation of the flink architecture is the part that is dealing with deployment and we have the local mode which is very convenient for developers then you have the cluster and this cluster mode can be standalone cluster or you can deploy it on top of existing Hadoop so it runs on top of yarn and you can even execute existing Hadoop loads on it and finally you have the cloud deployment so you can deploy it on Amazon or Google on top of that we have the runtime which is probably the most exciting part and the runtime here is a distributed graph processing engine so it is processing streams that can build various graphs and that runs on multiple machines there are a couple of interesting interesting technical innovations so this is the runtime that is supporting streaming and on top of that you have a to later api's and you can already guess what are these api's one api is the traditional api that is dealing with so-called the datasets and what do you think is this a battery streaming API that's a badge this is a set of data we see it all at once and on top of that we have a number of libraries we have the library for machine learning we have the library for processing graphs called jelly and we have the table like API the next to it we have the more exciting streaming API and this is called the data streams and here we have a couple of new API is a one that is particularly interesting is CEP complex event processing which is which is just receiving its new skull api and there is also a new api that is allowing us to process streams as stables so this is the overall architecture of the flink flink system and you can notice that this is relatively similar to what you have in SPARC only the differences here in the engine the engine is streaming engine as opposed to batch engine that is doing micro berchik to support streams the above libraries are fairly similar in their api's and one thing that you will notice when you start working with flink its api is very very similar to the api of spark and how does spark api look like well it looks very similar to the scala api for processing collections and you will be dealing with the abstraction that you have a data set or a collection of elements and you'll be transforming this data set using number of functions the functions will be transforming these things you have an illusion that you are working with the set of elements in reality you will enjoy distributed processing across multiple machines and results will be combined so it is very easy to migrate a scala developer into one of these systems and that they can be productive very quickly one of the interesting things that we have seen with our clients is we introduced systems like this to the data team and data team enjoys sequel they may do a little bit of Python they're terrified of Java they never heard of Scala and not typical programmers and then we show them the Scala API how does it look like you can zoom some data from HDFS and then you map it into something and you do the grouping and aggregate some values at the end and we show them to three lines and what is the effect that's like wow we have to learn that and then they start poking around the spark they start learning Scala and to my surprise the most take in Scala was actually among the data teams you know Scala emerges as the killer language for writing modern big data applications quite interesting so the api's are going to be quite similar so let me show you example of a small Flint program that is doing stream processing so I will show you first the structure of the project and I'm going to zoom in a little bit so you can see what is going on so here we have a little Scala project how many of you have been using maven at some point in your life so about 75% and you loved it right so but that's beside the point what is interesting is that here we see a project which is an SBT project SBT is called build tool it uses the same organizational files like maven so the interesting point for us is that there is a source folder has a main Scala some package and there is a place for our source code in the source code let me show you where is that in the source code we are going to open a stream of data so a stream of data on particular port and then I'm going to read in the data from this so you can see here I'm getting socket text stream so I will open a socket and I'll be sending data to this port and then you will have the typical hello world of big data systems which is the word count and we are going to do here reading the text lines of text we do flight map because in the text we have collection of lines if we split this into words we will have a collection of collection of words we want only a collection of words hence the flight map take the word to lowercase split it using the word characters and filter those get rid of the empty stuff then we map the word to the tuple that has the word and the number one key by zero so we are doing key by the first field and then we are going to do the sum by the second field which are the 1 1 1 1 the occurrences that we have here and then we are going to print the counts so we will print the word and how many times did we see it and at this point here nothing is happening so you just set up the pipeline you can imagine for streaming the flick optimizer may do some rearranging of the flow and then when you say execute only at this point the system starts to run and this is something that you will find in many of the modern big data system so you pre configure everything and then you get it to run so let's move here to the terminal so I have here SBT already ready so that should now build my project ok so we are ready to run and I also have over here there we go so I have netcat on port 9000 start it and then I'm going to run my program so I will say a run the program with local host and port 9000 ok so a bunch of infamous ages so now the the flink program is waiting there for the input and I can start providing the input in this stream so I can say hello world enter and now when I switch over here you see here at the bottom wall occurred once and hello occurred once when I add a new line over here for example by world you can imagine what will happen and now as this update I have a world I've seen the world two times and I've seen by one time so it is a streaming program that is maintaining the state so this state is global state that will exist during the execution of the program I will show you later usually we want to limit this to something that we call the window so we can say within the last hour these are the words that I have seen or within the last second but you get the idea and I just want to stress again the elegance of writing the code like this which is very very similar to processing ordinary Scala collections so when you look into the applications like this so we have seen here we are streaming the data from the terminal the modern applications with flink typically look like this so you have a various sources of data coming here and they are they are producing some values there we go so we have the data sources they are producing some values and the question is what do you do with these values so one of the things that you could do is you could just send them into your flink system and basically what you do is flink is you do analysis and then if link is going to produce some results and that will then go into some data stores and a flink has a number of adapters for or a variety of no sequel and also a relational databases so you have a number of options where do you put these results now and this is how flink application started but over time there was one innovation that became almost standard in most companies so imagine if you have a source of data and you have over here the system for analysis what happens if I have to shut down my analysis system for a moment I may lose this data so we need to put something in between that will buffer in this data and potentially allow us to replay the data so we can analyze what happened what system comes to your mind Kafka there you go so I'm going to move this thing a little bit over here to the side to allow to place Kafka over here Kafka is like a pipe that we can stuff with data and this is the part that is doing our buffering a couple of other things and that will conveniently send the data to the linked system so this is the modern architecture for our systems now one of the things that you may also do in this setup the data that are coming out of the flink you can pass back into Kafka and possibly some other system can consume it from there so this is almost the archetype for the modern big data and streaming application so this is the big picture of flink and it's code and how do we architect the applications now let's dive into a couple of technical details about flink so I'm going to switch a little bit to my PowerPoint slides and I want to point out a couple of things so to summarize the characteristics to have for a flank first native streams I run everything as streams and that also includes support for back pressure you also have the automatic battery job optimization and fault tolerance you have ability to execute cyclic iterative data flows you have a mutable state that you can keep in memory or in disk and one of the innovations is manage memory you know when you get a big server with lots of RAM if you're running job application you are still limited with the amount of RAM that you allocate for your JVM how many gigabytes do you typically allocate to your JVMs in production but what what are the numbers that you have encountered under 32 in some company they would say 8 gigabytes don't go beyond that but you have a machine with 128 gigabytes of RAM what do you do with the rest so it turns out that both flink inspired by the way they benefit from externally managed memory so you have the memory area that is outside of the JVM and you are also able to get a good performance because you you know exactly when are you using certain data and when certain data is not needed anymore so instead of relying on garbage collection which is a very generic mechanism that tried to handle all kinds of cases you may develop your own specific memory allocation the allocation policies based on what you are doing so that enables the systems like flink and what was the name of the engine in spark that is doing the same thing the tungsten engine tungsten is the custom memory management so this is something that you also see in various in memory databases get the storage out of the JVM program compilation is mechanism that is used for optimization so we are going to have our fling program it is going to be optimized and if you remember relational databases there is this execution pass or excess pass that the database is built first and then we optimize it and then we execute it so if link is doing the same thing but it is doing different optimization depending on what kind of data store are we dealing with so for stages that we have in the maturity of stream processing so first of all pipelining that we can take piece of data and move it through the processing elements ability to do the replay of some parts of the stream is a basic functionality then the next thing is maintaining the state to have the state for the operators which are processing elements an ability to also backup and restore this state above that we are trying to provide convenient API for the developers and har level API is something that we see in a fling as ap eyes that are close to the collections that we have in a programming language and also some integration with batch will fit into that and then above that support for large deployments working with large clusters scaling out a particularly interesting mechanism so the pipelining that we have in a flank is the mechanism that is processing streams that are spread on multiple machines and you have a certain amount of communication between machines in the cluster and you also have fault tolerance so certain elements might be processed multiple times if if needed and the particular interesting is the operator State so this state is defined by the user it resides in the memory and we are able to perform transformations of that state windowed streams are the important element they allow us to view our processing so imagine you have the data that is coming in to the system data looks like this so it is maybe you have something on the time axis and these are my events so these are the events and you can imagine these events are located on this axis so you have a couple of mechanisms how to process these so one approach that you can have is to process these events in Windows that are not overlapping so it looks like this we call such windows oops sorry bandit we call such windows tumbling windows so you can say every five seconds I am going to look what are the events and I will report something on them the other kind of event is event that is going to have certain overlap so look at this I am still processing three elements but I am now shifting by one element you see so I may either reprocess or I may be smart and calculate just the Delta one of the tricks that is often used to improve the performance is to look at the aggregate value remove the first element added the new last element and then you just recalculate a little Delta instead of recalculating everything in your dataset so this type of window as you can imagine would be called the sliding window and one type of a window which is new in in flink is by the way tumbling and sliding windows they in spark as well but something new that spark cannot do is having time based windows and if you think about these windows these are the windows that actually matter for our application they are identified by time that is relevant for our application so that means imagine this you have let's say an e-commerce system and this window here is based on the user session and this is really tricky you know you cannot really replicate that with the tumbling or the sliding window you are interested in all the events that your customer had with you since they came to the site then they were browsing and searching eventually they do checkout and then their session concludes you are not interested in saying what is happening between 1:00 a.m. and the 11:15 a.m. you are interested what's happening when this user came to your site and then until he or she left so this is the new thing that we really like in the in the flink that we have the power of API to express these things so time-based window based on the real world time and that leads us to the different characteristics of time this is also quite unique to flank so when you think about time in the stream processing systems you have three notions of time one is the processing time and that is the time when our system processes the data so you imagine this event occurred patient had a heart attack we take this piece of data and we store it somewhere in a file we read it three hours later bring it into our system at that time our system is establishing the processing time in this helping the patient in any way No so processing time often does not matter but it is very easy to obtain you just look at the time stamp when this event appears in your system and it's very easy to implement you can also say well how about ingestion time you have the data entered your system and this is when you identify the time and then you're doing processing based on that I imagine this imagine you have a customer who is interacting with your system and they have performed various sections on their mobile phone while they were disconnected so they have performed for example transfer of money at 11:00 a.m. they were disconnected they go back to the network at 2:00 p.m. and the injection time will be 2 p.m. does it really matter we need to provide the analytics for 11:00 a.m. when this transaction occurred so you see so the ingestion time typically does not matter either in most scenarios but what it really matters even if this phone was disconnected when it gets back into the system we can do the analytics knowing that this is event happened at 11:00 a.m
so this is the event time that occurred in the problem domain that matters to us and we can get this from sensors from various devices it is only that other streaming systems typically don't have a support for processing like that and that one is a bit challenging because the events as you can imagine can arrive out of order I have the event that was actually created on my phone I was disconnected and depending on the network the events can come in different order and the processing engine needs to be able to deal with that so there is a notion in flink you can establish a watermark and you can say I have seen all the events that appeared before 11:00 a.m. now what can happen hour later you may get an event hey I happened at 10:58 you can decide what we're going to do are you going to maybe restart the processing and reevaluate or it will say sorry too late I'm done with my analysis perhaps you can do something later you know is the event sourcing architecture you will be able to reprocess this and update your value but having this capability in the streaming system is extremely useful to build applications that really make sense to the users couple of finer points about iterations you have iterations where we process the whole input of data that we see in a particular window and we have interesting improved feature in flink which is the Delta iteration so you're going to process only those data that have been changed and the total amount of computation that you need to perform is significantly less for tolerance systems like storm have the semantics which is at least once the problem there is that you may see an event couple of times can we deal with that well we can if we develop our operations to be as important so if we see the event multiple times we will be able to handle it saying okay I have already done this computation I don't need to do it but the problem is that the burden is put on developers they need to develop application to handle that what we would like to have is a functionality that is exactly once so we know that we don't get the event multiple times but only once that requires more sophisticated system and the flink is using what is known as the chain D alembert a synchronization algorithm which allows for exactly once semantics there are a couple of other algorithms but this one seems to be the the winner and one thing that is important when you're thinking about such fault tolerance is that it is not enough for your processing engine to support exactly once it also depends on the elements that you have for ingestion and for the final storage of results but if your sources and sinks support exactly once for example you have a Kafka and flink and you are putting stuff in HDFS then you can have the exactly ones semantics so you need to also watch your sources and syncs checkpoints imagine you have your application that is running and you are preparing to stop it you would like to restart it at some later point so you have the notion of a checkpoint so you will store the data that is being processed in the application and the state of the system save it into some location and then when you restart you can start by reading in this data and then you can restart from the last point save point is very similar to that only it is not used only for fault tolerance but it is something that the user can perform so there is the command line function save point to imagine running your system execute save point you have saved the state into a file that means that when you have either the same system or upgraded system you can start it you can identify what is the path to the file that represents the state and you can continue the processing from that point this is a very practical thing for operations because you can easily upgrade and migrate from one system to the other so to summarize here the benefits of our processing is that processing does not block so we can have checkpoints for any interval business logic and recovery are completely separate so the parameters are defined in configuration and one thing that is really a winner for us is the ability to have session windows that match the things that are happening in the application domain and ability to reason in event time and not in processing or ingestion time and as we mentioned here streaming latency so you are operating with latency of milliseconds which is important for some of our clients which are in financial space when you have a systems like this you can monitor transactions that are happening and you can do the fraud analysis and you can say well this transaction that is happening right now there is a great chance that it is a fragile intersection so you can intercept and interrupt such transaction before or eight comments if you have system that is slow and a system that is for example doing micro batching like spark you may be too late you may be able to say oops a fraudulent transaction just happened two seconds ago and this is why you have the need for a real-time very low latency in order to deal with fraud analytics in real time so if you are in payment systems this is a very interesting feature now I mentioned to you when I was talking about our architecture for batch and streaming we would like to have the same or very similar code and this is something that we have in flink so imagine here a code that is very similar to the code that I was running to you word frequencies we have a class that is having two fields two properties one is the ward and the second is the frequency flink data set API for batch looks like this so we will say we have lines which is the data set of strings and we read text file and then we will do flat map based on the lines that displayed using spaces and then we map each word to the instance of a word passing the string and number one we group by word and then we sum the results creating the frequency and then we will print that so this is badge the same processing now happening on a stream of data that we are passing into the system the input is different instead of lines which is data set I have a data stream I read the data for example from the socket then a my processing code look at this this flat map with the line is identical exactly the same code and now here is one change I want to define within which time period I want to do this analysis and I do that with the window so I have a window that is 5 seconds long and I'm going to do the computation every second so what kind of window is that it's a sliding window so I am every second and I'm calculating five seconds of text that I have seen and the rest is the same so you see so I just defined the window if I don't define window what happens I will have the state for the whole run over all over the program so you see nearly identical code you just define the window and you have the same programming model and you know if you're a little bit careful about writing this thing you could actually write this as a set of reusable functions that you can then compose for both branch and in stream processing so one point that I want you to remember for a fling batch on streaming batch program is just the case of a finite stream you have the view which is the global instead of going through the windows and it is it is fairly easy to simulate batch using streaming but I want to talk to you about very latest addition to flink and this is the brand new API for complex event processing and I'm a bit excited about this topic because my PhD was implementing complex event processing engines his c++ awhile ago so now we have this functionality in in flank and the flink defines the number of operators so when you are doing complex event processing you have events that are happening and you want to express the relationship of these events in regard to each other so you can imagine a state machine where you have some start state you say this is begin this is where I begin to do the match then what is the event that comes next and then you can say this event is followed by some other event where you may have some condition for filtering and then you can say these events happen within three seconds from each other and I can also indicate what is the kind of the event and here is one example that is expressed in Java API that we have in flank and this is the warning pattern so imagine you are monitoring a patient and we have a too high heart rate events with a high blood pressure within 10 seconds to see so there is this certain pattern that exists here and you can see here we have a pattern so this is the alarm pattern we are going to do our measurement and we will say that this is our first event which is the heart measurement where event get risk is greater than equal than 1 so you will do the evaluation of the risk for the value of the parameter followed by middle value which is the blood pressure measurement which has the risk greater than or equal to which is followed by again heart measurement where we are looking for the risk that is greater than 3 and all of that is happening within 10 seconds so this is really nice because it translates two rules that our subject matter experts have when they are identifying certain conditions so you can treat this almost as a domain-specific language for expressing the rules about certain behaviors when you are processing events in manufacturing and various industry many rules are identified like this if you see this followed by that followed by something else then you need to act in some way and currently the API is only in Java the latest version of of flink that will be released is getting Scala API which is much nicer and shorter so so that is very exciting development and a little bit of comparison you know when you have a number of systems in the same space everybody loves benchmarks and comparisons so here are three popular systems this is storm a spark and flink and the important thing for us does assistant support through streaming or mini or micro batches so flink is a true streaming system together with storm and the spark is the mini batch and don't don't take me in the wrong way this mini batch model is still perfectly fine for many application areas but not for all both spark and fling enjoy very high level API based on Scala collections so you will feel very comfortable the fault tolerance in spark is achieved with the resilient distributed data set and fling is using the check pointing mechanism so this is the algorithm used for streaming but it provides state tolerance for tolerance a state is internal so there is a notion of the state that we can update exactly once is what we support for the semantics in flink windowing is quite flexible it's the most flex nearly the most flexible windowing that we have seen low latency and we are going to see some benchmarks looking into the next things that are happening out there one of the very exciting development is Apache beam so what is the patchy beam who has heard about a batch beam a couple of you about 25 percent so Apache beam is Apache project that is taking the Google data flow API and it is expressing it in in our open source fashion Apache beam is probably the most mature API for dealing with streaming and it distinguishes four different dimensions of this engines what are the results that we are calculating where in the event time we are calculating those when in processing time and how do we relate the refinements of these calculations so Apache beam currently is only the API and Apache beam does not have its own native implementation instead a beam has a runners and the runners are existing streaming engines that can be plugged in and this is very cool because you can imagine one day you'll be able to do your development in Apache beam and then change your processing engine from spark to flink or something else and I will not go into too much detail I will just point out here the beam model this is the API the first column in each of them Google Cloud dataflow which you can enjoy on Google clusters in the cloud we have a patch of link as one runner and Apache spark as one runner and I would say it is kind of natural that flink being a native stream engine has more checkpoints than a spark you will see here in certain areas so so that makes a link the most capable open-source runner for the Apache Beaman currently Apache beam is only providing Java API but Spotify has created the Scala API that is being donated to the Apache Bhima project so that is quite interesting and the API looks it looks very nice and elegant so a couple of performance measurements so here we have the performance straw storm with Kafka Fling Chris Kafka so you can see quite a bit a difference and then the fling team realized it is actually if Kafka that is slowing them down so they said let's see how many of this events can be processed without Kafka and it was 50 million events per second so it's a really great performance plus one important thing is that fling jobs are correct under failures so even when we have failure we have exactly ones processing semantics and a storm may see multiple events and I put here the reference you can read more about this benchmark it it's interesting to read how it is set up and execute it also we have here the latency so you can see here the values at the bottom this is a flink and storm very low latency spark latency increases but a spark is really engaging in the trade-off between the latency and throughput in some benchmark you will see this part will have higher throughput than flink but the flink latency is much lower a couple of new features that are coming in flink sequel for streaming so this is what many data professionals desire to have sequel for everything there is apache calcite which is the apache engine for a sequel it is used in various areas so the flink is reusing their results dynamic scaling so you can adapt the resources to the volume ability to query the state so you don't need to store the data on external database you can query the flink system directly for the data that you have inside support for measles the cluster management and better support for security community for flink it is growing look at this two times two point eight three point four times the number of Forks between February 15th and December 15th so the project is catching up more and more we have seen most of the users are still in Europe and this is not strange considering the the origin of the project and the connections that they have there but we have already seen some users in the u.s. apparently in the financial area and I have collected a set of flink resources for you the flink site the some blogs that are very useful data artisan's is the startup in berlin that is actually developing flink it's a couple of guys after they finish their PhD on stratosphere project figure out they want to start the company and I have also put here links to detail the reports about how benchmarks were performed and interesting articles to read about challenges in setting up benchmarks if you are interested in learning more about streaming there are two articles that are excellent some of the best articles that go into finer points of streaming algorithms you'll find them on or Riley this is from the people from Google dataflow team probably the best writing on on streaming and also a couple of other article that you would enjoy reading so that is all what we had in plan for this talk any questions go ahead we have like let's have hundreds or thousands of these where these are kilometers and and I'm going to look for events where they might have a heart attack wouldn't work to figure that out I need to go back load in their entire history I think so so is it as easy as just saying like not only past ten seconds to the past year and load it all in like go back yes so it is an excellent question so if you are having the stream of data that is coming in and in order to properly identify the reaction you need to consult some data that could be very old that is that is a challenging thing that is usually you cannot do it in real time but there are a couple of things that you can do one of the common designs for situations like this is to do what is known as the event enrichment and you would have the engine that will be receiving the events so we have our events that are coming in but for dealing with really old things and if you need to act relatively quickly you have a data store that you can query and this data store contains some values that are often some aggregates so in theory you would be able to store everything all of the old events but in in practice we often ignore a number of things that we know are kind of uninteresting and we put the important values in this store so then when we are processing our our events we are going to have the join and this join will be performed with these two and then the outcome will go out of this now this this Falls and this aggregate could be something that is supported by storm the question is how much space are you going to you know you will typically run out of space fairly quickly this is why we have seen systems like Apache Cassandra and Apache HBase being used for stores like this I have some really interesting performance data for HBase eBay has published about year and a half ago that they have a base system with 15 petabytes of data and they are running two to three million queries per second so that will give you some good performance and ability to store a lot of data so you can imagine as you are passing the events if you have a system with this capability you would be able to do the random access to such data and then identify if you have a certain more complex event of interest excellent question he was beginning he been she's anything like why was this extra information have to take so much space can you just have a thin pipeline just accumulate persistence make noise just a few more pieces of paper for each event it would it all depends on what is your application area and how much data you want to attach that will be easy to do in fact what you can do is you could create here the aggregate with the patient records so you know who is the patient and you have basically a boolean had the issues in the past and you just append this and that can then change the flow of processing that we grab the older values and we do more detailed analysis stores that would do this would typically be fairly simple stores some variation or key value store who could be good we've seen in smaller scope in e-commerce application Redis has been used for that we have used in similar situations Apache Cassandra because we also had so called the long road design for certain areas and that is whenever you have some event you just create a new column and appended in internally flink is built with Scala Anaka there are some part that are plain Java and also there are some parts in the external memory management that are C++ so this is really interesting if you look also into a spark you see this is akka with mathy Scala and a little bit of akka and it makes it natural you know if you'd be building a system on your own the mechanism that we have is supervision and fault tolerance and streaming of data and communication in a concurrent fashion akka is probably one of the finest framework that you could use some other alternative designs maybe build on reactive extensions so for example Java rx or any of the rx languages I know that they're working on it but I have not been following the the infrastructure integration I would say it must be coming you know because these days everybody is going after Mises so I think it's just question of time one anybody else all right go ahead rushing about how powerful is the data structure so it's a good question I haven't seen any example of that I would say that the API looks like it is carefully designed so my educated guess that it would be monadic api that would make sense because requires to be drinking yet but we provide with their own separate what our matured extreme such that each each element generates again in heart interesting I think we'll need to look into that very good well thank you very much