Scale By The Bay 2018: Gabriel Claramunt, Swimming in the stream...
Recording: Scale By The Bay 2018: Gabriel Claramunt, Swimming in the stream...
you get me out and this dog is just to play around with extremes and create a a very simple flow and that we can use to play around with that analysis really complicated most like a toy example but the interesting thing is so I want to show how easy is with just using a streaming solution we can do a interesting analytics by with real-time data we don't need in we don't really need to use a big cluster just even without relatively simple machine we can extract interesting data and so we don't we don't really need to spend a lot of resources I think that attracted me to streams is again easy way to start thinking about food functional programming because the basically data flow and data transformation by using streams actually we get many things kind of I already saw that sort of solve for us they may thinks the main components of the thing we are going to use sources of data that can read from files that can connect to different type of queues the most Probab I mean in previous projects there's a Kafka data source for I guess streams that you can use to get data the other side of the stream is the sink he said when the data comes out of there of the stream you can write it on again we can write an into a Kafka topic we can write it into this the terminal we can write into a file for a theme for paying around we can write in it the terminal then there is a transformation that happen between the source and the sink and that's where the interesting part happens when we get the data we filter out we reach them we can split into different streams we can accumulate values that's kind of a very quick overview of what happened with the same I mean most of stream streaming solutions in your for Scala I guess we if s to monix streams are a pretty similar high-level concept each one has a different flavors actually my most familiar with akka streams races in different project than its it's very easy to come up with something that it can be really useful with just a couple of transformation steps so the idea today is to just play around to increase this they have these I have everything prepared to to write a simple stream we're going to create a a simple flow this is based on our on a previous project this is inspiring in into a simple flow of we have customers that requires our products so we have an income source of request that basically request is the customer ID the product SKU and the quantity they require so I have in this case I have a random generator we are not reading from a calf gun or anything you just have a basic an iterator that generates random elements so the first thing we can do with you with the stream is it has right it if I did to the console oops Amon now this is the whole stream running let me see if I can increase that this doesn't they'll doesn't say as much just code so let's try to get more information here I have my storage in these cases false ODB we store everything in this fake data right here we have the customer information and the Purdue information and like every database we have a find for customer and product but returns a future so if I want to use here I will want to do map this will this the input is the top of middle values so we do false ODB find the customer information you hear the customer information is actually this customer is the first element of the double but if I run this actually going to get something that doesn't make sense because this is our future I can internally wait for the future but it won't it won't help is you know looking so I guess dreams offer me wait for singles and I give you the parallelism so this automatically will take care of assembling running the futures will run in the stream and get the the right results out from the future that's a better we'll just get the customer afraid information but actually we are losing their the data from the original topples so we're going to put that back on the on the trip so we're going to this is the customer customer and we're going to copy the original triplet but replacing the one with the customer is in the second place we have the the product information so this is the same we let's find the product process in place though - this is me put a better name here that's yeah so if I run it looks a little better I have the the customer the product they are asking and the quality but also now we want to do something a little more interesting I have a service that gives me the market price for the product they are asking here I create a small service request this this use uses RIT to be I gotta to be if you can it can be used directly as a stream in Agra swimming so basically stuff flow from request and the original value and returns and the output of that flow is the response with the original value so we can match this is the flow use cache connection pool a a casting automatically takes care of pulling the connections handling the the the responses balancing the the requests having requests waiting because obviously this is a streaming solution if the server the file takes more time or responds it's a back pressure so that the initial collision stop sending data so it slows down so we can continue the flow without breaking everything I'm manually handling the the streaming basic what I do here what I do in the flow is just create my request from the original data and this one - is the response from some of that response into this is the this this one creates the hot flow this is how we get the URL I'm having a function to grade that URL because depends on their where I'm going so create a request here connect to the flow and - the response that's another flow it creates a one flow flow then I'm going to reuse here and the input will be the customer the product and the the amount then the amount of products I want to get and I'm going to output bigdecimal that's the market price of the product basic when I do is I take the the product the input this is Apple I get the product I get the excuse a product that will do they the service call [Music] I made the shoulder this is the service it's a very simple service using again like I should be in this case basically since to market price and in the URL to texture the ski of the product returns a value the price on the market if you see the products we have the product we have the spice some ground meat and ice cubes spices very expensive excuse in in that desert planet are somehow expensive but sometime it is very cheap so I'm going to run that service Dutchess will put up there the world service with the market data we're going to run or flow that now at this point I need to get there the market price Oh Soho something.but mix it up already singing except oh yeah at some point I mix it up this I put the probe that would plug that in the wrong place and that will work who if you see but we got we are kind of printing but the whole response we are not doing anything with the response so this is something with the response so the responses response and also the replay there with the customer the product and the quantity our response in the response is an HTTP response so I can you have a response again use again I got to be to a marshal that into a string and then actually this my problem here that this is a future so again we're actually going to use Napa sink with some verification factor and this will give me the price as a string actually we are going to create an object I have the fulfilled object the basic combines everything the customer the product the quantity just America product quantity and the price actually is a big decimal so here we have the hopeful freeloaders and actually one thing we can do is write in to write this into a file but I want to show first one thing because I'm kind of running all the time right there's something actually run off time for showing this but one thing is they are approached to a gatherings to create graphs so I had the same flow that before but then I can insert a broadcast that splits fan out the stream repeats the same data to different outputs and then I can hook hook up different aggregations what is running in this particular case if you see this one it will generate a new sub stream every in this case is every 30 seconds and it will accumulate in that stream the total and aggregate aggregated by aggregator total by-product that's again running using the my service to my world service to ask for the customer price this is one one of the splits that takes is using group within basically it gives you every n numbers of second on or until you feel the connection it gives you a Scala list that then you can analyze with normal this in this case I'm doing a form to get the total and the average but this is slightly different because this actually can generates a soft stream you are not limited you are not generated the collection in memory just the substring this is both things running the this to total and average is calculated by using de this kind of collection and this one is the soft streams that says okay total position by this day it is this by customer is total required by customer I think I'm out of time so you can have questions or things that he has to check something very simple to use to play around and create even something that it can be real the I did a couple of that analytics pipeline for different customer and it's not really that far from this you can run it in like one big box you don't need a whole cluster using AK probably if you want to scale you can use Kafka you can split with Kafka but then yeah you can do a lot of analytics you know Moloch really simple like a string will take care of the the back pressure there's a couple of ways to handle errors in the stream we starting or you can you can create a flow that count how many times you ask for the service add like exponential back-off to the request there's other things that can be done and it's relatively very accessible thank you [Applause]