BDSBTB 2015: Chris Fregly, Spark After Dark: Generating high-quality Dating Recommendations
sorry I'm yes I I've been talking a lot the last couple days this is actually a superset of yesterday's training I'm going to show some more things here but also there's a presentation that I do there's about 50 slides it's the end of the day so I'm not going to get too crazy with those but we'll just hit some of the more fun ones yeah so there were a few people here at right the training yesterday is that yeah what I'm getting so I'm actually going to cover I yeah I've done more with the Zeppelin notebooks and then I had with the spark notebooks not until Andy petrella gotten down did I actually really start to pay attention to that and thing so yeah he's been helping me with it let me see if I can zoom in all right clear out some stuff here so just to start if you guys want to check this out if you just like Google flux capacitor that's the key name for this spark and docker yeah you'll be able to find all this stuff basically we sort of worked this for the by the base stuff but and then we write like simplified by the base so we can keep it controlled and focus on just Cassandra spark notebook Kafka and spark so flux capacitor actually has a whole ton of here let me show you this is a fun fun little diagram so yeah this is the general if you kind of follow in the magenta there this is what we did yesterday we like had a script that simulated users adding ratings to Kafka and we have spark streaming that's listening and pulling those messages parsing it storing it in Cassandra I don't think we got to any of the ML lib stuff so i'll i'll talk a little bit about that there's little data and lore you have the data scientist guy down at the bottom yeah he's using like Zeppelin or I Python or hive directly sparks equal and then goes up and around so I'm using like elasticsearch to actually store the recommendations I don't really have any good reason for it I could have used anything but they had a nice rest api and i was trying to test out the like the elastic search spark connector and yeah i wanted to write some code so yeah like actually turns out well if so yeah like we'll go through this whole thing with an end-to-end demo so i think probably everyone's here that's going to be here so why don't we start with a live demo if you guys and this can be interactive too so if you guys can navigate to spark after dark calm on your phones or laptop all right like you guys with me here please participate otherwise this is going to be lame and just pick your top three all right like actresses or top three actors just just poke on them so OOP got to get tiffani-amber thiessen in there I love her so Marlon Brando there's there's no rhyme or reason why you would like these people there's no this is a totally politically correct so that's why I'm having guys choose guys girls choose girls it doesn't matter where in San Francisco so all of these are being as you're doing that I'm using the kafka rest api and it's actually submitting directly back to back yeah like through this pipeline so i'm actually going to pull up a notebook where we're going to analyze these likes as they're happening so let me run through you have you guys selected everyone's got their their votes in what we're going to do here is take these likes and then run page rank on it and then we're going to run we're like a LS matrix factorization pretty basic stuff out of spark but and yeah we're actually going to generate real live personalized recommendations for you guys so don't click those three links on the bottom because that's the surprise so there's yeah these three down here so after we're done going through the analysis which would normally be like a batch job or something then we'll actually okay so I think you see how many people okay so yeah we got about 300 likes so far I can run it again and get get the latest yeah if you guys all scenes upland before it's pretty cool you could do sequel you could do hive you could plug in your own you could do flank you obviously spark is the default this is where you would define all the connectors like you just give it the maven coordinates so I'm using the standard connector obviously and then yeah like elasticsearch and spark streaming so here I'm just loading in some of the reference data yeah each of these actors actresses has their own ID and I just scrape some stuff off of like IMDb got some JSON got their images pulled them in put them oh yeah so this is all running within one docker instance by the way so it's actually running out on ec2 this is it right here yeah this entire pipeline is the whole flux capacitor or the by debate training so let me see I should I min so means I meant yeah i'm at the like a sandra command line here so we'll do use pipeline that's the name of the key space from lakes so i'm going to count alright so we got 364 so let me run this one more time it's pretty quick and then we'll take it to the next steps where we actually generate and populate those links at the bottom of the spark after dark page I put a little help button on here it's kind of fun you can send me questions if you want yes I won't answer them but it's hooked up to like zendesk and some chat thing that makes me look like I'm online and written by responsive so if we do a little analysis here we can actually kind of start to see some of the count some of the more popular people Morgan Freeman yet you guys want me to go through the code or y'all pretty good with data frames trying to figure out how like detailed to go because I can either talk about code or I could talk about my presentation or yeah so let's just keep going here I'll circle back on some of this let's go to the next notebook this is generating so there's basically two types of like recommendations right there's personalized non-personalized or yeah two types that we'll talk about here so that the basic ones if I just run this notebook while I'm talking would be just show me who is the most liked from this group right so yeah obviously the totals are skewed overtime that kind of thing but if we get those counts so we've got the count here and then we can run page rank so picture if you're the person on the left and you're you like three or six people or whatever that's your your yeah that's the overall light graph so here we can drive right like the most desirable users the right like the LinkedIn top influencers so that ran here that we got the top ones and then we'll actually so yeah this wrote out to elasticsearch and then we're gonna do the last one here which is personalized so now this is the matrix factorization so let's run this so yeah quick note on matrix factorization you basically take a snapshot of what your relic user to item like matrix here you know right now we're just using like so it's zeros and ones and then you want to factor out basically create two matrices that when multiplied together we'll match as closely as possible to the values that are there right now and then that lets you fill in the blank ones right so when like user number five logs in with just like scan across and find the most probable person that like top 10 recommendations or whatever okay so let's go back to here and we should be able to click so this is top users so this is just the overall count so I showed this graph earlier that was Morgan Freeman was at the top that was at that bar graph if we do top influencers this will be this is page rank and personalizes or one's straight out ALS did you guys all get results for those that actually did this right do you feel good about your recommendations are they accurate I have five dudes the other two my recommendations that and not quite sure how that happened okay yeah so again so we did like that whole thing right now we have ganglia hooked up in here we've got logstash so the problem with these notebooks is that you don't really get a lot of feedback when relic errors happen so the cool thing is we can I've got all this wired up to log stash and then kaban on top so this is refreshing every I think five seconds and showing us any errors it's got Zeppelin in here as well that's part of my filter so you see some of the WebSocket stuff opening and closing but yeah this is this is actually how I was debugging right like during development because you don't see sometimes it just especially with streaming like you don't really see any of the streaming errors so the streaming errors of here hooked in if you've got like log stash pushing the spark plugs into Kabana then you're good okay then one last dev ops ii thing these are kind of fun you can see I've got ganglia set up here so it's pushing system metrics from our so yeah this is the actual spark cluster itself this isn't like the oh yeah yeah this is the overall system right here but then we could see see file system usage thread pools for the spark jobs we can see some of the the file i/o block manager yeah this is important block manager stuff keep an eye on these keep an eye on your GC keep an eye on the heap yeah so there's a lot of good metrics it's really kind of an ugly interface but okay so I think that covers basically what we did yesterday so that'll get you caught up and now I'm going to oh yeah I was going to show one more so this is spark notebook this was actually from the training yesterday this is what we used and if I'm lucky I might be able to get this working yeah if like auntie's here he can help me there's this cool thing that we did where we actually were able to from a line chart here we could actually watch the streaming ratings come in as they were coming in we could see a line like a moving average every i think it was 10 seconds so it would actually populate this I wasn't having much luck with it before so but let me try it so it's going to start this is the spark you I here so you can see Zeppelin actually is a long-running job so just like sparks streaming when you start sparked streaming which I just killed down here yeah that's a long-running job this is the when i connect to Zeppelin or through Zeppelin it's creating this long-running job running behind and every cell that you execute actually creates a mini job or a full job for spark and then any notebook that we connect so I just connected this ratings thing this shows up as a job as well this is tacky on this is the UI for that didn't get a chance to get that fully wired up but yeah tachyons kind of like the distributed cluster right like a ramdisk sort of thing else were running for five hours here that's when I started the docker image earlier today and then this oh so yeah before I forget about this let's try it here we're setting up our sequel context I've got this concept of a rating I found this as part of spark after dark when I first created it I was looking for some kind of dating data set and i found one data set that was used for someone's PhD thesis and ya got I just asked hey can I use it i'm not making money off this he said yeah that's fine so it's basically it's 17 million ratings and it's for think about a hundred thousand people it's super easy if you search like dating data set collaborative filtering yeah you'll find it but it's basically from user to user and then their rating so for the actual demo that we just did that was likes that was just basically 0 & 1 but this is 1 through 10 so that was the actual dating or yeah the data set that we use yesterday for the training and then yeah we had scripts that would send it in so I'm going to try to do that let me get the this was Andy that actually wrote these Andy petrella the creator of spark notebook so here we're actually going to start a spark streaming job in side the notebook which is completely crazy but we actually saw this quite a bit at data bricks people we're like really wanted to do this which we couldn't understand because why would you want to do it from a notebook if you shut down the notebook but in like reality if you do shut down the notebook like you're like local notebook it's still running in the background the problem is you don't really have a lot of monitoring on it so what we saw customers do is actually schedule a new job because you can schedule jobs with these notebooks so for example this is data bricks cloud here you can actually specify a notebook here so it would be the streaming notebook that would take the data and then train the model or whatever it would do and so they actually scheduled this there's a little scheduler here and they just did it every minute so each minute they would check if the job was up if the streaming job is up if not they would just start it up again so yeah they didn't lose data because they were using Kafka but they would be possibly one minute out of date alright so I think we got that and now we're going to start the job the spark streaming job and then in theory i'm going to start the script inside the docker image it's going to populate this chart right here let's keep our fingers crossed for 12 alright let me tail this log make sure oh yeah we already have data all right so Wow so yeah this is just a yeah this is an actual data set which I was looking for because I wanted some some sort of you know statistically normal type of data not just randomly generated so yeah this is the like moving average every 10 seconds or a sliding window every 10 seconds so you can see we're sort of converging on six which I think for the overall dataset I think that was about the mean for all of that so what's interesting is that right yeah so this is andy's notebook he forked it from I Scala notebook and then basically tore it apart and built in all these widgets I think there's d3 underneath the covers or whatever those fancy libraries on top of d3 but yeah like data bricks cloud doesn't even really have this they have a lot of cool visualizations and but yeah not much streaming so I was blown away because basically what what I had been doing and this was the case with Zeppelin also is you just basically hit like refresh on the cell you just keep like keep rerunning the job to show let me get back to that first scrolling within a Zeppelin is really weird like these cells like grab hold of the focus but yeah I would have to rerun this every time I watch a change and it was kind of silly but so yeah he's using WebSockets beneath the covers on that one all rights so more on the dating stuff let me pull up the slides i think i'm just going to blaze through these because it's already five thirty but yeah spark after dark there's me at the Playboy Mansion netflix with my mom that's day two bricks at the booth and then yeah I have to put the plug in for IBM because they're actually letting me be here today and not work or I guess this is my work but yeah we're hiring we're only hiring nice people that was the one thing I asked when I joined I said are they're all nice people he said no now my boss said no we have 4,000 people they're not all nice but like moving forward we can find nice people so yeah I'm like the culture person I guess when it comes to interviews yeah just started this the events spark spark meet up I don't think data bricks likes it but because they haven't signed up or anything yet none of those guys have but yeah I'm actually going to work with some of the engineers I know those guys really well so they're going to come over the next one is September third we already had two really pretty good turnout some of the demos didn't work out you know start the normal stuff but this next one basically the whole point here is deep dive into the code right so yeah like you can read about it it's not going to be the high level you know here we are with verizon and all that stuff this is going to be this is the code these were the design choices made yeah i was there for a lot of these and I mean yeah they're all like public JIRA's to write so they don't always post the private like design docs but they do post enough to understand what's going on and then of course the code is the final answer so yeah Weiss park after dark so this series called playboy after dark was kind of cool I wasn't around in the 60s but when i worked there i stumbled upon these in like the archive room and popped the tape in and saw like like Jerry Garcia Marvin Gaye got very progressive things for the time so I was into that yeah the whole point here is to generate the high quality recommendations which we just did and also demonstrates Park streaming graphics so the more popular library some of the integrations cassano elasticsearch popular dating sites my favorites farmers only I'm not sure if you guys heard of that it's a city folk just don't get it they've had commercials for like two years and I finally looked it up and it's a real site I think I'm on about six of these all right sue spark you guys seen this sparking production yeah this was kind of interesting so right like data bricks just did this poll recently and they were asking right like how do you want to deploy spark do you want us data bricks to manage it for you within amazon and that's the middle one there that's a 30 or that second one says thirty percent and then seventy percent people said no yeah we want to do it ourselves so that's what kind of prompted me to start using docker plus I just want to kind of play around the two it's been pretty successful we had we probably had about 250 people do the training over the last month and there's yeah there's always the random problems if you give them like the USB because we were sure about giving them USBs and so you don't have to connect to the internet some guys like USB ports just didn't work on his mac we had no idea what was going that was the worst yeah we DC to as a backup so the full talk I'd be talking about all this but I think we're just going to hit some highlights here so I showed you this yeah so these are the themes these will also as part of the spark meetup will be like digging in deeper to these so yeah sign up for it come on out you know yeah bring your co-workers offer to speak if you want if you want to help out like diving through code to help repair that'd be great too yeah this kind of funny back in the 80s the first time I really start thinking about parallel ism so yeah this is Peter Brady he planned a date for I planned two dates on a Friday night I wasn't sure what to do so I guess fortunately that week there was a foreign exchange student or something that like moved in that day that looked just like them so yeah like how come mean it was that so Arthur is on the right and he basically yeah he like multiplied himself so he was able to accomplish this task and I think there was some weird thing at the end where the girl that Arthur went on a date with actually liked Peter and you know it's all I'll wacky ending to these 1980s 70 shows so yeah non-parallel you'd have to add these all up oh and that's not good versus Joe login pretty basic spark is parallel and so were the Olsen twins apparently all right so Daytona grey sort this was kind of the highlight of sparks right like recent so the big thing here that they were trying to prove is that spark isn't just in memory so probably the most important thing here is the smallest thing on here which says it's on disk only they like disabled all the caching so all the caching that everyone's been talking about the guy from like sigmoid and all those people talking about filling up RAM so they turned all that off and this is purely on disk and the point was to compare it to MapReduce so the old record what the whole point of this gasps of grey is Jim Gray right yeah famous computer scientists former computer yeah computer scientists and then you are given under terabytes of data you have to sort it as fast possible and you can use whatever you want so the old record was a dupe at Yahoo they were little fuzzy on their hardware they it was bare metal though but yeah they had 20 100 nodes and they had a throughput those bottom two are what you should kind of pay attention to especially that second to last one the sort rate for the overall cluster is 1.4 2 terabytes spark comes along does it three times the throughput with only 200 nodes and a third of the time and this is all on disk so this is more this sort of speaks to the execution optimizations and you know pipelining and those synchronization barriers that Michaelson was talking about yes how do they do it they this was kind of the first round of performance optimizations this was about when I joined so I was able to hear a lot of the like design discussions around here oh yeah I also saw the bill because this was amazon going the wrong way yeah those two hundred nodes were I think they were I ate some things but yeah there were pretty beefy boxes they were 200 so actually that's what the IBM spark tech center is going to be doing I think in like the next couple weeks actually we've just provisioned a bunch of softlayer that was part of the agreement the you know partnership with us in data bricks to do a lot of performance testing around it and basically do these kind of tests but a daily basis on a per like commit basis right that's make sure individual commit leveled not at the end when everyone's piling and commits okay oh yeah that third box is yeah since they already have the notes they decide to try one petabyte like thousand terabytes and see if it's if it scales so it's the exact same throughput the 4.27 so the details actually I think this is the next meetup that I'm having on September third there's three key JIRA's that cover these optimizations specifically for this Daytona sort thingy there was yeah so they paid attention of course to the like shuffle phase which is obviously like the most intensive the like biggest bottleneck here with these like big data processing frameworks so there was a lot of low-level things big one being they switch to Nettie so yeah not a lot of people at like data bricks like Java they're all Scala heads but yeah they actually buckled down and sucked it up and implemented yeah the low-level medi so like a lot of pipe up for reuse smooth out the GC profile yeah and then of course epoll to keep everything in system space kernel space and not have to bounce up to user and rightly create buffers when you're going from the network card to disk and that kind of thing so this is the next realm project tungsten this the first parts of this were in 14 and then I think the final parts are going to are hitting now we got 1.5 which i think is in review the big thing here is this bites two bites map which is like an enhanced hashmap so the key is I like better like CPU like locality rights or not or I like cash localities you're not ping pong and flip you know clearing out the CPU cache and then reloading it so if you picture there's kind of like a handwritten this is actually from a data bricks like white board here but if you have if you're trying to do large scans or large aggregations if you point to your key values like that versus pairing it up like that and then picture you know going down you're constantly clear because that first one points to yeah it's basically right like you can do tricks where your 8-bit aligning and that kind of stuff right so you could do that better with the cache aware so picture all these these Keys right next to each other right and if you're skipping over it you know key value you have to jump over the value jump by the value if you're just looking for keys then you can just scan the keys without your having to like dereference that first pointer and then skip over the value which could be yeah not eight bit of lines ok and then yeah there's also more yeah there's going to be a lot more like jvm bytecode generation going on there's a couple so that's the cryo I think someone mentioned it earlier today so that's the recommended it's it's a serialization like technique and I can't remember who actually built at Navy google or something but cry oh that's the recommended so this is a serialization framework when you're going either across network if you want to serialize in memory so now with yes more by cogeneration like techniques going on beneath the covers we can actually like bypass cryo and yeah do it even faster because we actually know the domain right so we know these objects versus cryo which is trying to do its best with like a general technique let's see catalyst optimizer this will be another topic for the meetup the big takeaway it here is use data frames not our dd's if you're writing to our dd's your old school which is kind of cool i guess if you like by code and assembly and that kind of thing you could definitely impress your girlfriend or boyfriend the key here too is if you're writing Python if you're writing 2 r.d DS in Python Thursday there's tons and tons of extra overhead because you're really pickling things back and forth between python and the JVM so if you write to the data frames it's going to actually generate it's going to generate by code for you from Python I think columnar stuff you guys priority no parque is mentioned a couple times big thing here is storing columns next to each other right you can do a lot of compression your run length encoding all these other things from your from your recent interviews also to column filters partition pruning yeah right like row filters predicate push downs so these are all part of the data sources API which like a lot of these so the parquet implementation there's a CSV there's avro there's a recent orc yeah it just came out I think with spark 14 and these right to a common sparks equal data sources API which basically lets you do push downs and all that kind of stuff right so you you right to this API and then you can partake in this catalyst optimizations yeah these things here so this is the actual so yeah catalyst is a replacement for the hive optimizer the hive query optimizer yeah it the way that it was written it doesn't take advantage of the rat like the spark like job execution and task optimizations and that kind of stuff so they start to strangle that that code base out and yeah this is the new optimizer so any new data source like I'm working on DynamoDB right now that would certain pick up a lot of steam and I just have to adapt if they you know give me some predicate I have to push that down into you DynamoDB so I'm not returning more data right like the Cassandra connector does it the elastic search connector does it so dynamodb you will to spark streaming this is a classic spark streaming deployment you have if you picture a normal spark cluster with four nodes or sorry five nodes so there's master and then for workers you would take one out and dedicate it to the receiver that was if you guys were here a couple talks ago someone asked about Kafka and why they switched off sparks streaming and Kafka and replaced it with something someone asks a question about the direct Kafka streaming so there's this new Kafka RDD where it's not this it's not this dedicated single node receiver anymore so it actually treats the kafka stream just like it would treat a disc where it has offsets within the stream right so every block or each partition that gets created in your spark close knows the start and the end and it can rebuild from that so there's no more right ahead log specifically for this Kafka implementation which was a huge like bottleneck if you're writing out to HDFS every time that for each batch of data that comes in every 500 milliseconds second if you're writing to HDFS that's going to kill you yeah just for the right ahead log so basically this assumes like success right so yeah if there's failure then it has to go back and rebuild it's not always writing like assuming everything is going to fail so we just did that I think I'll skip the approximations it it's worth noting yeah there's a sly brary called Twitter algebra that's pretty cool there's full examples within smart streaming yeah these these like approximation techniques are very useful in the streaming scenario where data is coming in fast and you don't want to use a traditional hash map for example to maintain counts so you just want to estimate it if it's good enough yeah so there's examples in the spark streaming codebase that uses like Twitter this yeah like this countenance catch algorithm and of course hyper log log Redis just picked this up i think he or two ago as a first class citizen you can with like 1.5 k yeah like you can represent 10 to the 9th elements that's that's pretty incredible and at like about 2% here so this is actually in the spark code base it's on the rdd and it's count aprox distinct by key so it's all been built in for you yeah something interesting which I didn't realize until I first ran this spark pie example is actually in the spark code base so they have this like example that is where I like estimating pie and I couldn't figure out why they chose this as a valid example like what's the like benefit here but basically if you if you picture this technique and it's a form of a Monte Carlo simulation which was used back for the for the atomic bomb from the old Manhattan Project what you do is you you know basically picture throwing darts just kind of picking if you take a square so like you obviously know the diameter and if you can sort of backtrack there so if you throw throw throw throw throw and then take the number of red dots divided by the total dots that's approximately and then x 4 that's approximately pie so if you yes we can actually run this and you can have it do multiple iterations and I think at ten iterations it was getting like three point 10 and then if I did a hundred it was 3.1415 you know for something like that so you can see as you do more iterations and throw more dots you actually get closer so it's a nice paralyzation problem okay talked about non-personalized personalized so basically I have ten forms of like recommendations so these are the two non-personalized talk about similarity yeah so whenever you're trying to make generations you have to compare everything to everything right that's the classic all para similarity problem so there's like techniques that you can use to reduce the amount of shuffle because the the naive shuffle is just huge you know picture m is 500,000 n is 500,000 they actually had this problem at data bricks and yeah multiple times at data breaks people came to us so you have to minimize the shuffle right so either you reduce the rows or you can produce the columns so yeah dim sum yeah which has a pretty funny name this came out of Twitter and a guy from Stamford and I'm sure a few others but yeah I know like those two guys so they what they were trying to do is figure out add similarity to figure out which adds to show you on the Twitter app and when they put this particular algorithm into place they actually had forty percent efficiency gain over just straight cosine similarity which is where you're you know using cosine to figure out basically like measuring the angle between two points so that's a way to reduce em how much time and then let's see we got yeah this is like another cool one so yeah this is the best book ever written by the way yeah this mining massive datasets so there's a form of this actually on like github it's also part of spark packages which is that kind of second-tier if you're not actually committing into the code base you can like submit your stuff to the spark packages and it's kind of a 1.5 classes and not not a full spark commit but yeah the whole goal the whole point here is to pre-process your data and riley bucket it into smaller number of buckets that and that pre process step like uses some sort of similarity to figure out which ones would be some more and then bucket those together and then what you can do there is then compared in parallel right so if you have 50 buckets you would spin up 50 tasks that would do the comparison that's called locality sensitive hashing sparse matrices pretty basic oh yeah this was cool when I was at Netflix I actually didn't realize this right about when I got there they were well actually not about five years before I got there they were spending quite a bit of time moving away from explicit ratings which you know five percent of the user base actually did they move more towards this sort of user behavior which is like common call the implicit feedback so searches clicks hovers we're on the search results you actually clicked like how far down as you scroll how long did you watch the movie so we did collaborative filtering here's some tech stuff that nitro guys probably can school me on this but yeah so this is actually from my profile back to the dating thing trying to figure out similar profiles so again you have to compare my profile to all the other like tinder users or OkCupid users whatever farmers only wait what's up oh yeah actually I did go on a date with a girl ahead to know she was crazy she was she was nuts um okay but yeah she had clean teeth and then let's see yeah yeah this guy I'm interesting too so people that I've that I've liked yeah that's another signal so take their profile and then write like run another comparison against everyone else maybe surface on people that way here I just threw a just a common pipeline thing this is right off the spark ml page but yeah like a common pipeline you would get the raw profile you tokenize it split up into n grams right like whatever you have to do build feature vectors run your models and then generate predictions all right like it what you would really clear out any stop words any you know simple words things that aren't high value okay one minute this was another one so I actually got this email about I know maybe six months ago now I keep saying it was last week but it's actually like six months now and it this got flagged and really grizzin to the top and I couldn't figure out why and I realized that my profile actually she she actually commented on things like yeah I have sonicare yeah she mentioned toothbrush I'm from yeah I wear my Cubs out a lot she mentioned Midwest so yeah California like west coast so it's sort of like an indication that this is a high value email so yeah check it out early versus the thousands of other emails that I get a day which is not true yeah and then so there was this really good blog post where the sky reverse-engineered tend to herb basically wrote like yeah so he wrote an app to simulate his actions on tinder and you know figured out the protocol or whatever he did right and he actually has some code to do this it's not spark code it's not ml lib code I haven't gotten around to porting it but where people that you've liked if you you know sort of average out their faces right that's a term right like I ghen face and I think typically what i've read is about 150 hundred and sixty pictures that's good enough and then any future people that come along right like this app would actually run it against this average and see if it was within some tolerance and it yes if so it would flag me and say hey this is someone that physically you might like based on the you know 27 points or whatever it is for facial recognition yeah hi welcome to more yeah so the same guy wrote this conversation bot where he sort of softballs out generic right like tried opening lines and he's not controlling this he's just spraying out to everyone and if they if their responses are you know positive based on sentiment he'll you know try to respond again automatically if it makes it down to the third one then it'll notify on this is someone that is actually sort of interested in you so yeah I think and so I spent a lot of time thinking about what i would say for these tried opening lines like hey what you doing this weekend right but it'll be like Sunday night or something that's put a little intelligence and then this is number nine here so this is yeah maintaining the spark this is after you've found your special someone how do you stick it out yeah there's another cool demo I've seen actually I think this was GraphLab guys that built this they have an open source that yet I think because they're using IMDb so they're probably little nervous about it but if you're a couple it's a Friday night put your kids to bed whatever you're trying to figure out what to watch on prey like Netflix you put in your personal preference at that moment they put in theirs and it'll generate this pathway so it's a it's really combination graph problem plus we're like items items similarity so you know based on plot based on actor is that kind of thing so yeah typically it's about five or six in between but I didn't have room for that or the energy to get yeah it's actually do the research but somehow water world ended up is the top one final recommendation get off the computer and actually meet people and then we're hiring and check out that's supposed to be me but I found out later that I think it's a girl so because of the hair I picked it and she's wearing pink too I guess so yeah I think Holden and maybe suggested that to me yeah my old coworker yeah so that's it yeah you guys have energy for questions yeah YouTube yeah oh so the question is how do I make sure logging is happening on the workers themselves that that are handling it yeah each worker has to have the that log stash agent that's installed and then configured with whatever directories that spark is writing to yeah but like did you say yarn also too yeah cuz yarn has kind of has its own thing going on there with logging right yeah yeah yeah yes if you're in yarn make sure that using dash Jess supervised right so that that thing will restart and then yeah whatever because yeah once the driver dies your spark streaming is opposed which is why you should always use like a reliable source like Kafka I Kinesis that can actually store data the one problem with Kinesis is it's 24 hours as the max that you get not unless your Netflix and you can right kind of sway them to do more but yep right thanks guys so uh it's