Text Naoki 12 15 15

basically reduce the cardinality on it so you can do the summarization aggregations computations on sliding time windows all right so on so at the high level there's two concepts in pipeline DB that you should know about streams and continuous views and so when but the data flows in it goes through a stream and goes into a continuous view and users basically query on the continued spew out but not on the screen and so if you're dealing with sliding time windows for example the continuous view is what actually gets incrementally updated in real-time and so basically continuous views reach from the screen and whatever record that goes from the stream to the continuous views is discarded soon after that record is red Oh is the audio better so this is an example of like how you create continues views in pipeline DB so the first one says create continues view average of forever that select average blah blah blah from 1,000,000,000,000 event stream so this is the stream and this is the the continuous view so as the comment says although the stream itself could contain trillions of events what whatever actually gets persisted is actually just in the continuous view so in this case although there would be trillion events in this dream we pipeline DB would just only store a single rope and in its persistence layer which is just a few boats and you can piss be calculate the average of water work that you're trying to calculate another example on the bottom is basically calculating how many ad impressions whatever you served in the last five minutes so clock time stamp is a built-in function of post grade which gives you the clock time sample whatever it now and you're basically subtracting 5 minutes from it you're going back in history like five minutes right and arrival time stamp is a built in for pipeline DB and it basically every single event that comes in through pipelines v's is given the arrival type time stamp so basically whatever this mr. predicate over here is basically specifying five minute sliding time with you and you're basically looking at account for that from pressure screen does this make sense so when we want to detect unhealthy domains in smog news case one we crawl we basically have a stream defined as above oh it's basically a timestamp of when we actually access the URL the Europe itself the domain and the HTTP status code and that's the stream right and so we create a continuous view called eminently unhealthy domains where we this is the predicate so we're looking at the past 24 hours this is the predicate over here and we're basically looking at status codes and storing the the minute of the domain and the sum of like 200-300 so on so this defines like basically continues view over the past 24 hours worth of crawling behavior and so basically after this it's really simple you basically run a slight safely and so this is basically saying just although we have 24 hours worth of data stored in pipeline TV we're just interested in 30 minutes of data and for each domain we basically sum up all of like 200 300 and 400 goods and their count if um this predicate over here is thing we accessed it a fairly good amount of time it's been 60 a negligible site and so if what we're doing at smart use we um right now we just started rolling this out and so right now we basically run this query like once every five minutes or so and we basically send notifications to slap if there are certain websites are domains that we can't access so over here it says this this article from entreprenuer had about 33 44 xxxx HTTP status codes and so on and so that's pretty much how smart use is using pipeline DB right now but of course it's like you can use it in other East cases including real-time av-test performance monitoring and continuous KPI and that's pretty much it thanks for listening you have any questions side back song it stays in the database marked as bad for 24 hours Wow so basically is there a TTL from the query Ronnie's again on all the sites that you categorize as forward for to see if they're back up again yes so we basically have like a cron job basically running this query every five minutes and so because this is this be running this query is running over sliding window like every five minutes that runs the results will be different or all the sites that give you a poor poor that would run a different time interval to test to see if there's still that yesterday ah ok I see right now we don't have them yeah but we're planning River probably going to do that yes yeah how much data is actually stored in the database their system or is it all just dreaming and if you don't grieve anything with everything alright yeah so that's the beauty of pipeline DD so like um basically if we go back here of this stream up whenever the event goes through the screams of the continuous view that's basically um whatever industry ms busily discarded and so what's persistent is actually the continuous view itself and so you can imagine what stored in the continuous view is actually when you do a slug start from continues view so that's what's going to be stored persisted in pipeline TV so even though not yet but if i define a continuous view as a sliding window can be can always different right right what is actually done persistent what is what was lost is persistent if continues for you is not accumulating but just always a sliding window nothing is persistent rather than this well it's being incrementally updated right yeah but but all these values obviously yeah they're dynamically changing but but it's this I guess it depends on what you define persistence is but it's the continuous views is it's incrementally being updated yes can you give a number estimate about the number of events you guys are we just started exploring this so i think the number events that flows through here probably on the orders of thousands per minute or something but it's so far yes they're very similar my food blog my fav proper subset for another do you do recognize that that occasion or are they all distinct and separate there yeah they're all distinct and separate and basically pipeline give you twice to handle it for you transparently so they are sad ok well the extraction is a bit separate ya know that's a good question ah can you question the question was that if you have like several continues views that one might be the proper subset maybe yeah it's a relatively new product so i don't know if it does those optimizations but probably like down here it hasn't done so already i'm guessing that it's going to do it sometimes yes could you be a little more specific about how you set up a stream I mean what ah yeah what kinds of data is expecting you said to however you pushing stuff so yeah so what kind of data that you can store it is basically it's anything that you can store in postscript so like Jason's post great class equal and um how we basically stream the data it's basically there's recently a bunch of log shipping apt services and that services but software like log stash or lump d those kind of stuff so in our case we basically use flunky to stream application logs into pipeline TV yes to all wondering what's the advantage for 3ds architecture book on using storm ah I think so what the um what the developers of pipeline DB says is that you could in theory like roll out your own sliding time window calculations and does kind of things on top of star but it tends to be unwieldy and like if you just use pipeline DP you can just get it up of the box and you can just run a simple sequel parade get stammers oh yes question about your crawl comments pipeline evf I imagine that you get a lot of duplicates with slightly different URLs you don't really see how many of those you see in how you this kind of fashion or said some processing so there's some so we won't try to as you mentioned we try to remove duplicate articles if they show up if it's difficult news events it's not a video store you episode that comes out of the later step we're actually analyzing the articles themselves we actually have some slides from previous talk or now we talk a little bit about how do we classify articles and how do we remove duplicates and stuff but it's an interesting problem right and then use a lot of people will take an article lady journalist and they'll actually plagiarize another journalist work and like copy and paste quotes and stuff it is essentially the same article so we have to our system has to understand that those articles are actually very very similar and we purposely try to find articles are very different let's say when is a conservative take in a liberal take on the same news item and they're written in completely different opinions that's what we were trying to show side by side as opposed to two of the same news items it's a side of the question from the same site you can get to exactly the same page but I varies URLs out you know different givens things like that you have any strategy for dealing with that or you just worth everything sort of yeah we just collect everything and sweaty thanks thanks