scale.bythebay.io: Vladimir Bacvanski, How to Pick Your Next Streaming Architecture
Recording: scale.bythebay.io: Vladimir Bacvanski, How to Pick Your Next Streaming Architecture
you quick show of hands how many of you are already using some streaming system alright how many of you are planning to use one alright so I'll tell you just couple of words about what am i doing I'm principal architect with strategic architecture at PayPal that means that I'm looking over a number of projects focusing particularly on developer experience in the tool chain that we have to build software and that includes highly scalable systems big data systems variety of very exciting projects in my previous life I was running my company as a consultant traveling around the world working with clients worldwide and I was focusing on better ways to develop software let's resolve their architecture highly scalable systems lately it was a lot of Scala and functional programming and big data systems and one of the things that I have seen in my consulting engagements in the past is that there are particular talents when it comes to selecting streaming systems first of all there is a great allure of streaming particularly as organizations are trying to process more and more data in real time or near real time as opposed to batch and in large increments and often there is a misunderstanding what these streaming systems do so one of the ways to illustrate that to newcomers is to think about streaming approach inverting the way how we think about processing data so in conventional systems we have a whole bunch of data we're sending the queries and we are getting the results streaming architectures are inverting this picture and we are making the queries static we are streaming the data the data is going through this queries get transformed and this is where we get the results and what is interesting in the organization like this the traditional organization of inner architecture of the system would not work because the demands are very different and that resulted in the creation of a number of frameworks for stream processing so if you go and look around what is available there is a whole run range of them the question is which one would be the best system for your application so need to think about different ways of organisations or a decision choices that have been made in creating the streaming systems from the point of view of developers you may focus on developer experience so how do we write our systems that will process streams you can think of a fairly lightweight approach for example like with akka streams you have a library you bring it in in your code and and that is what you use you have now some approaches that are bringing streaming into the messaging systems like what we see with Kafka streaming and then you have the group that originally started this area and these are the Big Data frameworks starting from storm of spark flink and and so on on the other hand this systems have also different solutions to their non-functional requirements when it comes to throughput at performance latency and so on and you see basically two approaches a one approach which is the true streaming where you're processing one record at a time which you see in systems like flink abeam and so on and you have another approach where we are faking the streaming by doing processing in small batches and this is what SPARC is doing with their micro batching approach and there are differences sometimes one approach is better for low latency sometimes the other approach may be better for higher throughput and you need to understand that the behavior of the system in order to match your requirement when you think about what is happening inside of this streaming systems the good place to start is with one of the oldest systems in this space which is Apache storm it's an architecture that is designed purely to processing of streams and it has a spouts which are sources of data in has bolts which are the units that are doing the processing and the data is three as tuples between these nodes the interesting thing is that in order to promote scalability all of these things are replicated and they are actually running in distributed fashion on the cluster the overall organization there is such that you have the master daemon process which is receiving the code distributing the code to workers you have a zookeeper as the system that manages the cluster this is the source of true information in the cluster you have a supervisors which are demons that listen to work and then you have eventually the workers that are running on task utilizing the JVM threading so this is a solution that is giving us a very good latency its system that is used in real time originally the API was the relatively low level so the developer experience in working with such a system is that you need to do more work in order to get your results but there are some improvements there is a library called trident that is adding higher level of abstraction it will remind you to processing of streams like what you have in spark or flank and storm 2-0 is coming soon with number of improvements and you see a system like this is completely dedicated to processing of streams sometimes you need a little bit more flexibility and we have seen that in the system like Apache spark Apache spark was originally designed as a system that is doing a batch processing of big data but they're streaming emerged as one of the requirements the approach that that spark is using is to take the stream of events that is coming in and package them into battle batches of small data so each of the this micro batches is going to be processed and will be producing some results the internal data structure is a derivative of what is known as resilient distributed data set in this case we call it this test stream and it represents the RTD abstraction and you know from spark RTD is going to provide you with the data that is typically kept in memory and it is distributed across the nodes in the cluster and then we can do parallel computations on that the beauty of that is that the programming model that the development experience is a very similar to processing batch and in fact if you compare the two ways of writing the code here is the batch word count and you can see here we are creating a spark context this is way to access the spark we are getting the lines from some text file and the here is the little bit of flat map map and reduced by key that will give us our word count so this is for batch processing if you think about streaming notice what are the differences and this is very cool so notice that the main body of code is identical what is different is now we have a streaming context and we are defining the window and we are defining our input which is coming from a stream and that can easily be networking it can easily be Kafka and so on so this is one big thing that was happening in the usability of the api's of streaming systems programming batch and programming streams are converging and we see that in the systems that are coming after one of the newer additions to this is that you have a sequel you know if you have a data in some application area the next thing that will happen somebody will demand sequel and then they will get it so that is quite popular and we see actually this is very attractive feature because in some organizations you have people who are coming from traditional data processing or data science domain and they are often challenged when they have to work with Scala so they really welcome availability of sequel similar developer experience we see in think the consume ability of the API is similar to spark however the inner architecture of a flink is very different it is actually the opposite of what spark is doing spark as we have concluded is batch architecture which is adapted to process streams here we have the other way around flink is a streaming architecture which fakes batching because what is a batch batch is just a stream that terminates at some point and quite exciting things there was a talk this morning very comparable we see a bigger uptake of flink recently among the companies and one thing that really stands out is a very good latency because it's a natively streaming architecture and a couple of interesting things you know when the flink was being designed there were a couple of requirements like Cola latency for streaming topologies which one could not resolve by going with batch first resource utilization ability to create the data processing pipelines and then optimize them something that we had in data processing with relational databases for a long time you create an access plan you see what are the optimal ways of accessing a processing data and that is missing from many of the Big Data technologies today ability to iterate and the maintain state in iterations is one of the real interesting feature that filling has and one of the use cases was the ability to deal with graph processing also with mutable invisible state now what we see is that number of systems have been emerging and one of the really interesting thing that is happening here is the Apache beam is the common API layer for big data both batch and streaming processing systems it is inspired by Google dataflow which has a very comprehensive very elegant API and that the idea came that one can take the dataflow API and provide it as a common layer for number of streaming systems which we then use a so called the runners for the Apache be the beauty for us is that we will have the common API and we will be able to change the execution engine based on various requirements performance throughput latency and so on also when we talk about selecting the framework this beam gives us very useful mechanisms for comparison because it is separating what results are being calculated wherein event time when in processing time and how do we relate refinements of results and I have put here link when you start choosing your streaming architecture I recommend that you go there and visit and compare the features of different streaming systems before being we have to invent this on our own now we have a really nice a matrix of different capabilities and also you can see what is covered by different systems so you can see here for example what is being computed you have here different function like parallel do group by key flight on and so on and here you have a number of prominent streaming systems beam model which is just the API Google Cloud dataflow which is the private implementation that you could use in the Google Cloud Flinx Park apex gear pump traditional MapReduce Jay storm which is the storm compatible system from Alibaba in open source and IBM the commercial product so this is going to give you a nice idea about what is supported in the api's and what is really interesting just about a year ago when I was dealing with this you could see the matrix like this there were many more empty spots or spots that were saying you know we see great development and progress in this field over the last year or two sometimes you find that running this on big data infrastructure is not such a good idea you would like to have something lighter answer to that is coming through Apache Kafka Apache Kafka allows us to do streaming and what is really cool about that solution is that in many organizations we already have Kafka we don't need to add any special clusters in order to do the stream processing because in Kafka stream processing is done as a completely ordinary normal java application you don't need to deploy it on a cluster and all these things very nicely integrating the whole Kafka ecosystem one of the feature that was recently introduced is exactly one's delivery that caused a lot of discussions on the web you need to understand that this exactly one's delivery works really well when you have the closed world assumption that means for Kafka in the beginning Kafka at the end you have the full control over the input and output sometimes you will see that none of this out-of-the-box solutions is going to work well for your use case and then you have akka streams for coming to your rescue this is an example of one application at PayPal this is the crawler application you can see here we have the list of crawl jobs gets in the database the values from there are streaming the Kafka validator for URLs go to the web it download the pages do analysis of the pages we have some database of URL caches so we don't revisit the pages and eventually we are sending the results to Kafka after that it goes to Hadoop spark for various so analytics and also the new URLs are feeding being fed back into Kafka that needs to run a really fast a number of machines and akka streams provide us back pressure mechanism and you will see in the next slide how that actually works with akka streams we have two great things one is we have a DSL that can very nicely represent our pipeline of processing so this is the actual code and you see the actual code in akka streams it really represents the graph that we have sketched here so this is this is the actual code the other thing that you see here is the effect of back pressure here is the processing URL per second you can see when we start it is really constant and this is the queue size this is the back pressure the beauty of that is that when we write our replication we just express this in the pipeline you don't need to manually deal with the back pressure the interesting thing this is running ten times faster than the conventional java application that was implemented with multi-threading so very cool so if you need to roll out your own this is a really good solution at PayPal we have a lightweight library called scoob's which is addition to akka streams and it is dealing with the large scale deployment and clusters we provide support for monitoring logging even correlation across services extra security a number of other things and you can use just the pieces that you need if you're interested click on this link and you'll get a talk from the reactive summit all right so finally how are you going to make the choice rule number one you need to create the architectural scenarios for your applications so that means you need to figure out what are going to be the event sources what kind of events are you going to send you also want to know how big are the events how often are they being sent you will be sending that into your streaming system environment is important because you want to distinguish the behavior of your system where system is fully operational and everything is running fine second when your cluster is degraded when you start losing machines when you get slower Network so all of these factors should be considered in your environment and then you are going to capture what should be the results what are you going to produce where do these things go in databases file SKUs and so on and you want to figure out what is the rate of results that you need to have so you need to capture that and the reason is that we see that a various benchmarks of vendors are going to be almost unusable if you have application cases that are different from from the benchmark application and what is real important is I want you to performance test your scenarios use the realistic loads these days with containers in the cloud it is easier to achieve that but you need to have a real numbers so you cannot just believe vendor material that or this system is fast the next thing that you will do once you have these scenarios and you have measured performance you want to identify criteria that matter to your project some of these criteria are going to be technical some of them are non-technical for example in some organizations we have a mandate we have to use open source commercial solutions are not appropriate you also may want to look into fit to organization like what are the programming languages in which you are using this and one of the easy ways to do that is to use the decision matrix where you will list the criteria that you have you have weights I prefer to go with just with a simple high medium and low and then for every option you will evaluate and it is essential that you have the performance and you see how this performance actually matches and don't forget to consider performance for different scenarios in your cluster so normal operation degraded you will see that some of the systems that give you a really good latency a really good performance when the machine fails the system may spend quite a bit of time doing some rebalancing in the cluster during that time your system may not be operational question is is it acceptable so your scenarios need to consider such failures so here we are based on the criteria that you have you will choose the winner hopefully without regrets and to summarize different systems have very different runtime characteristics different developer experience you need to architect you need to have architecture aligned with your needs so don't forget first create the scenarios you need to start with that performance test your streaming application identify technical and non-technical criteria that matter evaluate and choose the winner questions and by the way we are hiring so you can talk to me after I have uploaded the slides I think they should be on the website shortly excellent question so monitoring of streaming systems or in general any distributed systems including microservices is is growing challenge normally we see that the systems don't have a really good monitoring and that is why in our organization we are developing and we have developed number of custom solutions there are a couple of interesting innovations in that field for example in some organizations you may intentionally insert special kind of messages in order to have a particular traceability and then you can extract that from your monitoring your representation system I have seen also some people would generate dot notation which will then enable them to generate visual graph of how the message passed through the system which is kind of cool so there are a number of things also I have seen some organizations providing data for Zipkin Zipkin will give you the visualization of the flow of data through the cluster together with some time metrics so if you have a couple of examples of veteran stripping using the same code in your experience so I mean from from your presentation it's not always so this is the ideal this is the ideal one of the reasons for that is we often see that there are legacy big data applications that are batch somebody provided the data and we had the old days in my old company we had a client they would dump file to us once a day it's a batch process and then over time they figure out well there is no need to get the results only after 24 hours perhaps we could get data immediately and then we have this code that is doing the business processing and if we are able to repurpose this code to now works in a streaming fashion that is that is a winner and if you consider the applications that we had at that time which was mostly SPARC we were able to take the business logic perhaps tweak it a little bit and get a very similar code for our streaming system the important thing there was to identify the window or processing and to figure out what is the appropriate semantics we have seen also at that time that our business logic got complicated in the early days of spark because we wanted to have the window that correspond to user session there was no such mechanism in the beginning so we had to compensate for that in the business logic and that changed the similarities between the batch and stream excellent observation so could we use a workflow management for stream processing a very attractive idea and in the past we have actually done something similar we have used apache camel but that worked well for situations where our stream of events was really moderate and where we have the Message Queuing as a source of this events normally the orchestration systems that we have seen could not provide the sufficient support for a high volume of data that we see so in the application that we have done in the internet-of-things space the volume of data was was so big that the workload system will not work but what I would love to see is the visualization of the stream processing and actually you can see some really nice effort one of the system that we have mentioned there was the IBM streams which is a commercial system they have a nice Eclipse based IDE where you have a graphical composer for your stream processing the limit it would depend on how you deploy for example camel or some the some of the other systems but normally the workflow systems have been designed to work with conventional messaging and not a very large number of events what is that number in our case the IOT systems were sending us 1.5 million events per second and we knew that the conventional system just could not handle that thank you [Applause]