Real-time, Streaming Advanced Analytics,...

data.bythebay.io: Chris Fregly, Real-time, Streaming Advanced Analytics....

Recording: data.bythebay.io: Chris Fregly, Real-time, Streaming Advanced Analytics....

okay so today's talk is going to be totally demo based I've been doing PowerPoint talks for the last probably here and I'm I just can't look at PowerPoint anymore I like upgraded recently to the new PowerPoint for Riley Mac OS and thought that that would renew my interest in PowerPoint and it actually the gradient is sloping downward there to a global minimum so definitely not I just get tired of looking at the same slides the same colors so I've got enough demos now I've got this project called flux capacitor also same name as the company so let's just get at it here let me introduce myself real quick I've cheated a little bit and put some slides inside of the notebook so because it's just easier for me to convey this is me Chris fregley there's my yeah that's my emoji someone bought me stickers for my so if you guys want one I have about 492 left or 98 left yeah because I have two of them no one else wants them yeah currently research scientists we're in like stealth mode I just like saying that I don't really know I'm gonna tell you guys what I'm working on so it's just fun but yeah we're gonna bring AI back to the future that's what yeah that's our goal here organizer yeah so there's this meetup the advanced spark and tensorflow meetup probably 70 to 80% of the meetups are actually here at galvanize right the guys have been very nice to me I worked with IBM before who would sponsor all the food and everything so I'm on my own now I'm gonna be doing my own sponsoring if anyone else wants to offer up pizza or beer or whatever please do we actually had Martin O'Dare ski a few months ago I think back in January if you guys remember we wear like hired his son at the spark tech center Jacob yeah who's a super nice kid and yeah his dad was coming into town to visit him and we somehow had him come work the evening shift at the meetup so that was pretty big right like one thing yes I plan to speak after him and that's the worst idea ever to try to speak after Martin oh dear ski because right like half the room left cuz right like he spoke and it was awesome and then the other half went and talked to him after the talk and it was just like me and like four people so ya know for the future don't ever do that working on a book let's see pancake stack i yeah i realize i'm kind of a marketing yeah i don't want to say the word but yeah yeah marketing person here i guess because i just i've got about thirty eight fifty domain names pancake stack this is kind of a mockery of the smack stack but right like which yes i was part of last year this conference actually last year is when we first trained people on the smack stack so it was me and about four other guys one guy from confluence people from data stacks for Cassandra we had a guy from typesafe or akka me for spark and what's the other one we had a guy from mesosphere as well that all presented as part of training so this year I had to up the ante a little bit I pulled in pancake because I kind of went with stack and it just got out of control from there so I actually got in trouble recently for this because I didn't have right like the the first part the Apache word right like before all of them and yeah so this picture got like tweeted out and some dudes from the ASF contacted me directly and we're like yeah like gets put a and rightly put the full word Apache and then I was thinking about it and that yes I reminded them that it's just gonna be like stop you know like it's gonna be all A's and so that's not fun so yeah they let me keep this this particular slide and keep the acronym but it's yeah pancake stack calm if you get and it but there's really nothing there workshop also pretty much always here at galvanize next one is actually not at galvanize June 4th and then back at galvanize here in San Francisco on June 6 where we do a full day and cover all of these so todays thing is basically going to be this that whole eight hour workshop but in 4050 minutes so formerly IBM spark Tech Center data bricks and Netflix so yeah a lot of X their data sets that we're gonna use there's this labeled phases in the wild have you guys heard of this so they've actually gone through and pre-processed there's I think it's 1300 or 1600 like celebrities and you know it's Tom Cruise and the Brad Pitt's and I like to Angelina's and they've they've done all the sort of face alignment stuff and we're gonna build ICANN faces basically run PCA on these faces and make right like real-time recommendations if you pass in Tom Cruise's face right like who so say that you like Tom Cruise right like you're on air like some right like what's the dings like tinder something like that if you're just swiping swiping if you swipe on Tom Cruise who else looks like Tom Cruise so yeah we'll do that one which is kind of fun there's some surprising results there of course movie lens yeah coming from Netflix I have to get some movie stuff in there and then just kind of my own handmade data sets as well so what's the other thing I used to call this talks Park after dark and it creeped some people out I think yeah I was in London speaking and yeah and I made some some like reference to this dating site and got some like negative feedback so I sort of de-emphasized the whole dating site but that's right like that was initially what like flex pastor was was gonna be a mock dating site and to generate recommendations and things like that so actually the spark after dark stuff was kind of inspired by this blog post called tinderbox rather have you guys heard about this it was a guy up in Vancouver it's hilarious so this guy basically wanted to automate his dating his swiping so yeah he does just that right like with the eigenfaces we're basically after I think it's a hundred and fifty swipes right the system learns the sort of average or like I can face the top you know top ten features or whatever and then can start to auto a swipe after that yeah so the guy reverse-engineer tenders API and was pulling in pictures and rarely doing auto swiping yeah here's is like dashboard if you guys get a chance to read it just Google tinderbox it's hilarious the other thing too yes here's the eigen face thing we're actually to recreate that he has this chat bot thing as well that uses sentiment analysis so he basically automates and just sends out just like sort of generic things like hey I was your weekend or right like you look like trouble or something that's one of my buddy's favorite ones apparently that one works a lot so I should start like incorporating that but you can just do the just normal Stanford Corinne LP which is what he's using or like you can use RN ends right like LS TMS and basically figure out how their responses right because it's not like they're really yeah people don't really read these profiles right they just look at the picture and if it's a normal write like hello or like how was your weekend they're either gonna respond or they're not gonna respond that's his theory anyway so if they make it past a couple chatbots then it goes and will actually notify the user and it shows up on his dashboard and then he'll he'll actually start to personalize some of the conversation there yeah so he'll actually read their profile something else too that will touch on a little bit this is kind of the holy grail here is streaming matrix factorization and surprisingly so when people talk about spark or they talk about tensor flow they're typically talking about the training side of things right there's this whole segment and this is what my company's focusing on which is the serving side of things so think of prediction IO think of yeah there's a bunch of them right so it would be nice to actually build the model and then inside of a notebook for example say deploy and then have a cluster and write like have it Netflix scalable actually I'm jumping ahead a little bit but so there's two things that like we're going to start working on which is that side of things deploying and then also streaming mate so these sort of online algorithms right like not a lot of these algorithms can be done online training where you're training every we're like batch interval right like every like 500 milliseconds I mean that's kind of ridiculous but rarely maybe training every 10 seconds right like just sort of varying the weights a little bit and then redeploying the model things like that so and then things like pinning models against each other - right to see which ones are like more effective a/b testing right the multi-armed bandit things like that so basically trying to incorporate streaming and then deployment and then that whole feedback loop right so these are tough problems there's a lot of ops stuff that I don't really want to do but at Netflix we write it's kind of no ops kind of thing so yes I got a lot of experience doing that so this is one of the components that we're using heavily from Netflix which is hystrix so this is where yeah really does anyone know about this so basically if there's right sort of an edge service so at Netflix there were edge services that we had that face the customer that rat-like talked to the devices or Riley devices talk to the edge service that service then talked to like tons and tons of other services that were like behind the firewall so that's all these services for example so you know it would be like the video metadata so that's things like the description the director right like where it was set you know things like that there's also and it's not on here but so there'd be some sort of personalization service or some sort of recommendation service right that would actually serve up the top 10 shows or movies right to that users but it was our service that was sort of aggregating all this so say that that right like dependent service was down or started to write like slow down a little bit or I was getting overloaded around like some way to sort of rather open up that circuit and then let the traffic drain and then let the service probably come back online so that's kind of what you're seeing here this big bubble here that's like the overall volume relative to these other ones you can see and like throughout the day certain services will you know open closed things like that and then there's all these yeah so there's tons of like timeouts and right so after maybe like two minutes of this or really five minutes of this oh and so when you're in that state you're now returning fall backs which could just be the top 10 movies overall so it's non-personalized but from the users perspective it's not the most optimal experience but it's still like happening they can still continue to use the service right so Ops loves it because it shows us as rarely being available a hundred percent but right like we're just in a degraded State and so once that service can recover right then things get back or they can at some point that dependent service can then Auto scale up or right like their group gets involved and then can like make their like service larger-scale out a little bit so this is a lot of the operational stuff that we're working on so let's show some of it here actually let's do this this is gonna be a live live demo with you guys involved so I need your help if you guys can go to how do I do the there we go if you can go to demo advanced spark or demo advanced marcom if you guys can go to this URL yeah you can do it from your phone you can do it from your laptop wherever select the top five technologies maybe that you currently use in production what we're gonna do here is generate live personalized recommendations for probably this particular group so this is just typical collaborative filtering where I like alternating least squares the sort of intuition here is that we'll pick eat will Riley each pick five technologies and then I'll run the algorithms and then it'll actually specific to right like you guys pick some other technologies that maybe you should be looking at based on what this group currently uses okay so let's see I think I already picked some of mine here docker yeah I don't actually use these I'm just clicking randomly at this point but and if you feel like poking around yeah actually don't poke around just yet because it might screw things up oh that that actually explains yes I put it up here for convenience because like during the workshop we if you get lost you can always go back to this page it's sort of the home page but each time that you come back to this page it's gonna regenerate this unique ID just because I'm like the worst JavaScript developer ever so if that if like this ID changes it's no longer personalized to you right you basically have to click on the same page that you have that ID okay so I assume people have clicked let's hop on over here right does anyone use Epling yeah yeah anyone else are like ipython or anything yeah cool okay yes you guys get the workbook so just so now this is going to power this first link here the first link is going to be non-personalized this is just kind of summary statistics top five just yeah just based on count this next link which yeah don't don't click it just yet will be the personalized recommendations so here we ran I just ran here's we're just pulling in reference data here just so that I can display the images here so I should probably show you guys the overall flow yes I know I'm kind of bouncing around here I'm I'm a bit of a spaz when it comes to the live live demo format I say flexible yeah you might say spaz so we're the users in the upper left clicking and click click click this is going through knife i which is doing geo enrichment so it's look at the IP I don't have to use nine five for this it's kind of silly that I do because it rarely knife is a pretty powerful thing that does way more than just Geo but I was trying to find a use for it in the demo and that that was a good use case goes it then goes to Kafka so knife I if you guys haven't seen looks like it's kind of this dataflow thing this is now a Hortonworks project came out of the NSA Hortonworks snapped him up i think six months or eight eight months after they separated from the NSA broke out if you will so let me see here if you click in so this is the actual request handler so my javascript talks to this thing which is just a jetty server just a like super simple version right now you can kind of see some of the data flowing through here looks like 179 came in goes we're checking we're gonna do the geo right here and then we're gonna put into Kafka and there's right like a bunch of failure queues and things like that that it goes to but let me kind of show you real quick what the Geo one would look like oops oh that's a that's an actual process group so here's the enriched Geo these are called processors basically just nodes in this flow this is the database it's the classic like Mac spine one pulling out the HTTP remote host and sticking in so that's that's pulling it out this is actually adding it here to the request and then if it's not known it will stick in unknown and then goes to success and kind of continues on so once it puts it into Kafka it's sort of out of the picture one cool thing about knife I just since were in here is this thing called data provenance which I didn't I hadn't put this term together I didn't realize it that this was a big thing but this is a way to actually track all messages throughout the entire system so you can actually like click on here you can see this so this drop is where it drops it into the Kafka topic if we take a look at I think there's a way to it's I'm downloading it's not what I wanted to do yeah so there's a way to actually go in and look at the data before and after so these transformations like we could see the request before it was enriched and then after so before it was just a host and then like afterward actually has like San Francisco or whatever in it at this one yeah yeah anyway so let's get back to here and then we have spark streaming we could use flink yeah I've been seeing a lot more flanked just in the last couple months at some point hopefully Kafka streams that way I can get a riddle out of this crap here and just use Kafka for everything we're storing it into Cassandra so that's why this next step so yeah this is the overall flow so those are the ratings going there sparks training putting it into Cassandra we're now becoming data scientist person down here we opened up Zeppelin we started running models right so obviously some of this would be done batch sort of offline and you know some some kind jobscheduler like airflow once we get the actual streaming right like online matrix factorization stuff working then it's going to be super incremental and then rattly could be every five seconds where we'll be training but we're not quite there yet that's yeah so that'll be done here pretty soon but we're then going to deploy so to deploy we can deploy to Redis which is pretty much what like Netflix does they have a memcache is huge memcache cluster fronted by this rarely go open-source project they have it's called evey cash I I can't run with the e V stands for anymore but yeah basically a huge memcache so I'm storing it into elasticsearch just because it has a nice REST API and because I'm not a good JavaScript developer I just want super easy rest access to my data with Redis there's something called web disk that's right that's actually pretty cool so I'm probably gonna switch over to read us here pretty soon so yeah we'll deploy it and then when you guys click two and three that's where this data is coming from is from elasticsearch essentially so let's go back here we should be able to click to okay so these are the top five for this crowd which kind of makes sense HDFS up there Kafka Scala since this is basically a Scylla conference and then Amazon so let's do the next we're back to being data scientist person trying out their models here we're gonna do ALS you guys familiar with riley matrix factorization collaborative filtering type stuff got this probably like yeah just varying degrees here so I'll just kind of like level set basically take a snapshot of that matrix so this is each time someone like selected a particular software it's gonna put a one in there yes this diagram is a little bit old this was back in my right like spark after dark tinder stuff where you could swipe left for note four zero and swipe right for one so right now we're only doing ones so this would be if your user number three and or two you lie software there and there so the second and fourth slot which would be right like index one and three or whatever so if you take a snapshot of this which is what we're gonna do right now let me get it started and you factor it out into two smaller matrices right and it's smaller because we're choosing K which is called rank right it's similar to like k-means in the really sort of intuition where we're guessing that there's there's K which is really going to be smaller so maybe ten or twenty right like sort of hidden features that lies within this data set right this matrix and like we're not quite sure but let's write like factor it out so that when this is all done in factored that matrix times that matrix gets as close to this current snapshot as possible right yeah so once we actually get those two smaller ranked matrices now we can do some cool stuff and so one of the things that people do is is change the value of K right and then keep running it and seeing what they're finding so let's do it here I think I picked K ten or twenty so there's the metadata again yes I'm like redoing a lot of stuff because sometimes I run these notebooks out of order and I want to make sure that like I don't screw things up but that's the one right like sorry downside with notebook development if you start kind of poking around and doing cell five first and then you go up to sell to you know things get all screwy especially in a collaborative really multi-tenant sort of environment so here we're pulling the ratings out of Cassandra again rank is ten passing some other hyper parameters here giving it to what's called a LS alternating least squares form of matrix factorization that's just where you hold one side fixed and then you alternate or you hold one side fixed and solve for that side then you hold this side fixed and solve for that side sort of alternate back and forth maximum twenty times if or if things don't change if things stop changing below a particular threshold then stop right there's two stopped criterias here so some data frame stuff here magic so here's the final so now this would be the - oh so this is kind of yeah so here's this transform method now this is on the actual ALS model that was built on the right that current Cassandra data so this is the huge matrix now of the recommendations sorted by grouped by user sorted by confidence so these are the top 1 2 3 4 5 so we should be able to oh and then I save it into elasticsearch later on let me get to that in a second here but here we're actually saving it to elasticsearch right there so if we go back here we should be able to click this and there's my recommendations now if you notice I didn't do a very good job filtering out ones I I selected so it's actually recommending ones that I've already selected I keep trying to it's it's like super easy to fix I just keep forgetting to do it I keep focusing on the hard stuff and then but okay so back to here I just want to point a couple things out so here I'm dumping out the entire right like matrix right the or so now this is a different right like table this is so this is user ID this is item ID and then confidence so this is what I'll be using to serve up recommendations but there's one other there's like two other cool things that like come out of here which is I now have a way to represent my items in this new smaller space right this new smaller rent so each item can now be represented by this new relic matrix or each vector each item can now be represented by a vector that's part of the sort of item factor that's been like factored out so now this is useful because now I can actually real a cluster based on this so right like this whole goal of like feature engineering and stuff is to take right these like right like items and and like users and then break them down in some way into some right like numeric format some sort of feature format right that I can pass into like other algorithms so what we're gonna do is actually so like this would be item 1 this would be item 2 this would be item 3 yeah so these would typically be like point 2 and point 8 and stuff I yeah I should probably change this because this is one of the like major diagrams that I have here but right so again so when these are multiplied together they would end up like this but I now have a way to represent this user is this particular vector then that that second user is that particular vector so I can actually find similar users and like cluster users based on this new data that came out of a collaboration of users and items so this is some sort of hidden you know really meaning to this so what's cool is that right like for example Netflix can take these and find these sort of weird categories of items or these weird clusters so not just by typical genre right by horror and comedy and things like that but actually right like my favorite I was just in Canada last week and it's like Canadian it's like gory Canadian films targeted at people between the ages of 25 and 30 right so they can get that specific with their and that's because and it's not really it it has to be human interpreted right because like it's sort of the same thing with like k-means or right like LD a topic analysis on like corpuses of text where you sort of give it some like arbitrary number of topics and then have it return to you the top ten terms per topic and it takes the human to sort of say oh yeah this topic is about right like biology this one's about physics with so I thought I had a slide in here later but it shows some of the goofy ones that came out of Netflix but really very very very specific one so right like these are powerful because now you have right these clusters that that didn't normally exist okay so let me show you what I do with that here I actually so this model when you build it lets you build this entire matrix which is what I saved and or probably this entire table which I saved into elasticsearch to power the third link but then also you can actually get the user factors so that's the smaller rank matrix going up and down this way and then I can get the item vectors so this is for user ID this is the vector it's just you know it's just a bunch of numbers but that's cool I can pipe that into a clustering algorithm here's I call model get me the item vectors so this is item number one so each technology it's one index so you know seven I think is spark so now I've represented spark yes I can pipe this in and now I can find similar things rightly so yeah so this gives me the ability to do item two items similarity right so I can yeah I'll do that here in a sec I should probably point out to that so now this is proof here for this user ID twelve six six three and for the item id seven so if I multiply those two vectors do the dot product of that vector and then that vector should give me the exact so let's see what's the value here point nine nine two zero zero and so that so that should be the exact same confidence that this big table gave me for item id let's see where's the big table so user ID twelve six six three and at my de where'd it go twelve six six three item id seven is point nine nine two so this is right so this was generated by the actual ALS model and then this is if I actually do the matrix multiplication myself just to you know really taking each of those vectors and then multiplying them together so that would be this thing I'm just doing vectors dents calling blast product the exact same value so let me jump ahead right here to the user user similarity so this is gonna find users in this room that are similar to each other it's actually not that interesting because we don't really know who the users are and it's kind of yeah I don't really want to ask it'll take forever to resolve but user ID twelve six six three is similar to these people here's the similarity what I'm doing here is I'm taking the vector from the user factor so I guess it would actually be that vector because it's in this that matrix over here and multiplying it by all of the other or no and then I'm just finding let's see how's this work yeah I'm just taking that vector and then doing cosine similarity to all the other right like users represented by that same vector and then show me which users are the most similar and we could do the same so this one's a little more interesting let's do the same with item two item similarity it's a big scary warning for myself because I always forget to run these in order sometimes I bounce around in my specific so here we're gonna find items based on this collaboration similar to spark which is item number seven I just happen to know that so based on what we've all chosen zookeeper Wow yarn Cassandra so in a recommendations you know standpoint I would write like maybe recommend because you selected spark I would show these five and then I would you know kind of keep keep adding them on based on what's in your shopping cart or whatnot here we're gonna use k-means on the items so item to item yeah so here's so we're gonna run k-means again on that just that top item factor matrix and figure out what are some of the hidden so yeah here's some examples here's the gory canadian revenge movies that's what it was cerebral military movies based on real life right so again these were humans that were going in and taking that data that they found and right they played with k a bunch of times i can't remember what okay this one specifically but yeah kind of interesting yeah raunchy mad-scientist comedy well like you'll see this I'm like Apple music I finally switched over to Apple music I bought car play for my car because my stereo was like super old and they put Spotify on like page two so I finally I was like alright I'll just get right so they write like Spotify was on the main thing but then yeah they really snap on music and then so you have to like page over to the second page of car play and yeah that's yeah that's accident-prone right like versus all the other stuff I'm I'm doing like selecting music and like texting and everything yeah that's the one thing actually so here we're gonna cluster so let's run this based on this collaboration again yeah how much time do I have it's - okay you oh I I think I just changed these imports okay anyway this would be well that's no fun yeah so this would basically cluster and show right like which and right so this is cool clustering because this isn't based on any metadata right like this isn't based on the genre and things like that this is based on this collaboration I just created my own vectors class the other day and I what's the import doesn't even know the import org spark ml org spark orca pachi spark and now when our vectors if I get that right I'm buying myself a beer good this is like all I've looked at for the last like two years my life so the fact that I even doubted I would get that is kind of silly so yeah totally right yeah that's the only time I allow myself so here we're just clustering based on this collaboration so let's see cluster centers have been determined kind of boring sort of a kind of an indicator of how good your clusters are here within set sum of squared errors basically how how tight are the clusters that I found right if they're all over the place it's going to be a larger number if it's smaller that's better so here we cluster in pretty much everything into zero Oh 2200 and clustered a few 600 into four and a couple stragglers here in one two and three so might need a little work on my clustering yeah algorithm there but one thing I want to show okay so since we only have a few minutes yeah so just to give you an idea right like the workshop covers all these we actually cover a graph right like a bunch of graph algorithms yeah this one's kind of fun here some our path so speaking of relic recommendations here yeah this is kind of a fun one where let's say that you're sitting down so you finally met someone on spark after dark specifically it's such an awesome dating site you're sitting down Friday night Saturday night trying to find a movie to watch you can't figure it out so yes I had to switch these when I was in London because of the negative feedback I got for being poor yet being stereotypical or whatever so I like message in a bottle that's not a picture of me but I wish it was by non-existing girlfriend likes Mad Max and we're trying to find a movie that sort of is in between so typically and yeah let me run this we'll have and so this is a combination of like Dijkstra but we're using the longest path or the heaviest path so picture we're gonna pre-process in between each vertex which is a movie we're going to calculate similarity between all the other movies right so there's this and for our case we're actually just going to use the right like genre tags right like adventure comedy things like that for a larger scale thing you would use like actors and similar plot yeah so Mad Max actually is really very similar to Waterworld which is kind of funny yes obviously Kevin Costner is in at those two movies as well so once we build that graph up we can start at a specific node and say get me the most similar path are like pathway through so from each node picked the greatest edge weight and just keep going until I get to my final destination and so let's see I didn't actually have those two that I could find very easily so I just took Toy Story which is number one here and I took sudden death which is number nine let's see we're using Jaccard similarity so here Jaccard similarity is good when you have sort of binary things right like sets right so your is this element in the same set right like how much overlap is there or I like Jaccard similarity is was it's it's like the intersection of a set divided by the Union of a set and set here is the genres so you have this huge vector of genres you put a one for all the for each genre that this movie represents and run Jaccard similarity that's that's the similarity function that we're using here let's see and build up the similarity graph just ignore that oh here's my attempt that that's probably a break but I was trying to do d3 graphs within Zeppelin which is like the hardest thing I've ever done in my software career starting with Toy Story finishing with sudden death get me the heaviest path so normally you do lightest or shortest path between those two and these are the movies that come in between I think I actually break them down here I thought I did 2 8 10 and 6 so 2 2 is Jumanji Tom so this is 8 Reilly super small data set I'm just doing 1 to 10 just so 2 8 10 and 6 I think there's 10 Goldeneye and heat I'd probably choose Goldeneye out of all out of those 4 or 5 so it would actually show that I was trying to show it with d3 it's pretty tricky one last one that's gonna blow your guys minds because it it's well because this took me about a year to build so it better blow your minds we're back to the eigenfaces thing that I started with heavily influenced by this book if you haven't read this book and you're serious about machine learning and spark we just hired this guy at the spark tech center he was my last hire before I left I was yeah like we were so stoked he's a spark committer he's a PMC guy he's from South Africa he's like one of the nicest people I know and it's it's a tiny bit dated but if you know where I like data frames you if you know rather the old rdd's it's yeah it's not that big deal I think they're actually rewriting it Nick's not doing it but they have someone else doing it just to update it but yeah the algorithms and everything at yeah so I sat down and like read that book and probably two or three plane rides so okay five minutes so PCA we're basically trying find the top components that describe a particular data set so the data set here is that labeled faces in the wild data set where it's each actor right like actress has a set of picture or facial pictures there each 250 by 250 their color we're gonna actually scale that down to 50 by 50 because I don't have a big enough spark cluster to process 250 by 250 we're gonna simplify it down to greyscale not color just so that each rally item within the matrix is just one value for the grayscale value not three for RGB so it's three times less the data we're gonna convert its extracted convert a grayscale it we're gonna standardize it this is very important all right like whenever you get a data set think about how you can standardize normalize it here we're just doing mean not standard deviation transforming it getting so principal components are not actual features right when you get principal components you can't really translate those into anything concretely right that's the the kind of sad part that's right like the limitation about PCA is that you can't say this particular feature or like feature being you know age or write like something like that fed into this principal component you just have to take it for what it's worth and now it's basically dimension no it's yeah so it's like reducing write like these dimensions so that when you feed it into other machine learning algorithms right like we're not passing in the entire right like 250 by 250 pixels like vector that's been flattened but we're actually just passing in this smaller dimension so each so I've basically broke down each image is now represented by these 10 oh no what happened here okay whoo oh my god okay so yeah let me just back up for one sec here I've taken each image and I've broken it down or no so I've I've passed in all of the images right and out of all I think it's what was it sixteen hundred pictures or something these are the top eight components so these are the so in terms of PCA these are the top eight like components that describe all of these pictures right so this is Angelina Jolie this is Tom Cruise this is everyone here you could sort of see some of the features being picked up right like a lot of eyes stuff maybe this is you know hair style here looks like you got some wrinkles going on there each of these has sort of picked up its own so now these are the the now I've taken the vector and smashed it into I like change the dimensions yes i reshaped it to now show the actual eigen face so that says eigen yes these are eigenvectors represented in terms of like pictures here so it kind of gives you an idea and now we switch over here we're gonna actually kay one minute perfect oh so this is the original pixel vector for every image so this is basically the same thing but I did it now with ml pipelines here we're doing the standard scaler the exact same thing we did before this is just a little bit cleaner right like a little bit more new Sparky this is not in the book this was the stuff so I've taken each image and broken it down into its own PCA vector so now this is the smaller dimension this is me representing these images with with some new format of numbers and now I can do the exact same item to item similarity that I did before where I'm gonna pick someone out of that that list yeah there's our boy Tom Cruise and I'm taking so that's the item ID right like effectively I'm getting the item factor which in this case is the PC a vector converting that into something that spark likes I'm doing the exact same cosine similarity item to item similarity and here's the grand finale the top five people that that looked like Tom Cruise there's another girl yet this guy and yeah that's it [Applause]