data.bythebay.io: Grant Ingersoll, Smarter Search with Spark, Solr and Machine Learning
Recording: data.bythebay.io: Grant Ingersoll, Smarter Search with Spark, Solr and Machine Learning
I just to warn you of grimacing in pain it's not because you guys are asking me difficult questions i had a little biking accident this weekend and having a little hard time breathing but we're going to power through so i wanted to start off by setting the stage if you will around how I think about smarter search and at the end of the day I really like this notion of if you if you remember nothing else from this talk I want you to think about searches something that allows you to build very fast and efficient ranking engines and if you think about with all the data that we have it's it's a it's ever more important to know what's important so I like to think about search as a way to help users figure out what's important in their data and so if we kind of take that from a high level we can then dig in and we can talk about what I want to spend the time here on with you today is kind of a top-down view of starting with some use cases around where search can be effective for you and then talk about this thing of solar and spark together and then I'll leave you with a little bit of code you can go check out if you want a little application you can try out and then some tips and traps how many people here actually work on search or of leverage search before okay so a little bit so hopefully for those of you who haven't worked on search you'll see some things and I'll give you some food for thought about where and how you could use search in typical data science applications and if you want to learn more I've done other talks just if you google my name and data science with spark and solar I think you'll see a lot of interesting things that come along with that so if you're not familiar with search I think one of the things that's really evolved the over the say past five years around search is that it's it's going to come a long ways from the traditional put in some keywords and get back 10 blue links you know you heard Ricardo talk about those 10 blue links and whoever that engineer was that decided 10 was the number but these days it's gotten to the case that a lot of these modern search engines they handle spatial data very well to handle numerical data they do joins grouping they do faceting they do all kinds of really interesting slicing and dicing of the data and plus you can give your users a very natural interface to the data so I think ultimately search is one of these democratizing technologies when it comes to data and we see in our company we see use cases across a large variety of areas and verticals ranging from working on log data for customer insights all the way through to doing similarity models between chemical compounds where scientists are actually saying I want to I want to rank my data based off of the molecules in the compounds that I'm studying so you see a very large set of use cases happy to talk about those in more depth as we go so the premise of this talk is solar and spark if you're not familiar with those just some quick list of features around what solar does and what spark does and then we can talk about the joining of the two and maybe talk about some of the ways we can do machine learning for with those things so again at the end of the day if you haven't looked at search I think when you look at this list you're going to be like wow that's well beyond what I thought a search engine could do perhaps you've heard of something like leucine and maybe you're a little bit familiar with that perhaps you've heard of elastic search solar is part of that same who's seen community it's been around since I think 2006 you know at its core obviously it still does full-text search right keywords 10 blue links all of that kind of stuff but for instance most people these days would be really surprised to know that solar does sequel you can do full-on sequel joints you can do aggregations group by sorting all of the things that you're traditionally used to seeing out of a out of a database or out of something like drill or Impala on those kinds of things other things that made solar kind of well known to begin with faceting of all maybe not used to the name faceting but you inherit you know what this is you go on Amazon you type in TVs for your search that's side nav where it says you know there's 27 Sony's 15 Samsung's all the way down the list other things that are I think pretty interesting we deal with the users typically don't know how to specify what they're into they're interested in so things like spelling autocomplete hit highlighting all of these things designed to help users discover what is in their data and at the end of the day I think that's one of the really powerful things about search is that it not only enables people to find what they're looking for when they know what they're looking for but you can actually use all the data you have in your search engine to help them discover other things as well some of the other kinds of things out of the box that perhaps you're doing using other techniques things like deduplication graph traversals doing grouping and joins there's actually a lot of stats capability language detection all of those things are built in out of the box and oh by the way all this stuff scales very very well I think last reported Bloomberg for instance has upwards of 300 billion documents in their solar index Salesforce is in the same realm and there's lots and lots of other companies that scale this quite nicely so you know you can go from playing around on a single node all the way up to thousands and thousands of nodes on the spark side based off of the questions earlier it sounds like most of you are familiar with this already but essentially this is what you gives you your distributed computability very very fast lost of nice bindings interestingly enough you can run it in a lot of different ways we personally run it in standalone mode but works in a number of other cases as well probably the most interesting thing when you start to think about search is this second to last bullet here around the tools for graph processing machine learning etc these are often the tools that you need for the offline analysis of your content so part of the premise here is let's make spark in solar work together what are some interesting things that we can do with that I'll for starters you know you get the spark shell for free in there and we can start to leverage things like building out our indexes in parallel a lot of people don't realize with Hadoop that it actually started as part of the Lizzie lusine project if anybody is familiar with the nuts web crawler that was part of the lucene project way back when and was designed to go out and do web scale crawling and search so Doug cutting my calf Arella they read the Google Map Reduce paper said hey why don't we go implement this for this nut crawler right and so then you could use MapReduce to go and build your indexes out in parallel well spark is that on steroids so you can build out your indexes at scale very very fast beyond that though it gives you a lot of interesting iterative computations I started them out project mahou it was originally built around doing machine learning with MapReduce and one of the things we very quickly realizes these iterative algorithms on Hadoop are just so slow right now of course they're tractable finally because you can at least do them in parallel but they were still too slow and so when spark came along a lot of the mohel community jumped on board with that and has been working on porting all of our MapReduce stuff over to spark other things that are of course interesting being able to do whole corpus analytics when you look at a lot of the modern web search giant capabilities that offline task of looking at all of the content or at least a good chunk of the content is a really key aspect and making sure you're understanding what your users are doing with it and then last but not least being able to join this with lots of other systems HDFS Cassandra etc so we now have a system where we can bring together our perhaps our authoritative stores very effectively with our search indexes or if your search index is your authoritative store you might be able to sync it up very efficiently with your HDFS system where you're keeping all of your logs so on so forth perhaps something you haven't thought about as much since most of you are familiar with spark for you haven't thought about the solar side and just and by the way I'm just talking about solar standalone with spark there's some very interesting capabilities that you get by integrating these two first off that whole ranking problem that we talked about right the search engine is fundamentally really powerful when it comes to doing ranking and kind of saying here's a fuzzy sort of all of this data bringing in a lot of different attributes as opposed to traditional databases which are much more geared towards sorting on a specific column the second item here though I think is one of the things that is often underestimated when it comes to building out these applications is you can massively simplify your operations by goings just solar and spark together right one distributed system is hard enough to in sync is significantly more complex than each of the ones individually Solar has been around for a long time scales very well very proven out very reliable I like to say get you your weekends back now as data scientist we don't always think about those things but trust me you're DevOps will appreciate you when you they know you are giving them a system that is going to stand up overnight and over the weekends and all of that kind of thing the third item up here and I feel bad I always want to go talk to this side of the room 2 i've got a bias i guess to this side of the room one of the interesting things when you look at like a lot of the spark machine learning stuff is a lot of the built-in ways of handling text are just this fourth line here tokens equals string to lower case dot split on white space anybody who's ever really worked on text knows that's an incredibly naive way of dealing with language one of the things that we did as a company was to take all of the lucene analyzers which have been around for a long long time they deal with all kinds of languages they do tokenization they do stemming they do a lot of the feature selection work that you need when dealing with text those all now can run in your spark ml pipelines and so you can set up all of the stuff configure it and just run it in sparks so you don't have to deal with just simple white space tokenization anymore trust me that's a big deal when it comes to getting quality results out of your text analysis last but not least I think this is really interesting is solar provides you a non dumb distributed resilient storage and what do I mean by non dumb right it it allows you to create very interesting types of queries phrase queries wildcard queries all kinds of things that then can be applied and give a ranking across your data as opposed to those kind of traditional hey I'm just going to go iterate through all of the data that i have and kind of roll up off of some sorting column so you get this random access right so if you think about HDFS for instance it's always block storage that kind of stuff well random accessing the HDFS doesn't exist right so if you overlay a search index on to that stuff now of a sudden you can query into your data in ways that are much more expensive other in other approaches we still do all right at table scans too by the way that sequel stuff that I talked about that supports streaming and table scans and is very very fast and efficient last but not least or these last few here very advanced filtering capabilities again you can filter by all of these advanced queries as well so you know a lot of times in our jobs as data engineers or data scientists it's all about reducing down the size of data that we want to work with so we can see what's going on with it and then expand back out from there search engine gives you a lot of those capabilities I've even seen people who use search queries to build out their training and test models and and stuff like that because you're keeping all of that in your search engine last few things it's schema lists when you want I II when you're getting started with a data set one of my favorite things to do when I get a new data set is just a throw at solar and say hey what is this data look like and I can start querying it I can facet on it and now I get an idea of what my schema looks like and then as I get closer to production I say okay here's my schema and then I get all of the type safety and all of that kind of stuff that you want in production last but not least there's a lot of good alternative views of data in their spatial capabilities are very significant you can do polygon intersections you can do find me all the content along a line you can do advanced polygons where you have like for instance holes in them you know hold on a polygon other other kinds of things like that there's good columnar store and you at the end of the day have a sparse matrix available to you so if you start to think about certain operations that benefit from that you can take advantage of those what does it look like to actually talk between spark and solar first few lines here basically just some set up Solar has a notion of collections this is a logical grouping of your content might be all your logs might be all your tweets might be all your financial data that's up to you to decide and give it a name and then you just connect to it like you would any other source in in spark and here I'm getting a sequel context i'm reading from it i'm telling its solar and then i'm loading it and now i have a data frame and all your data frame operations are available to you underneath the hood the spark solar integration is going to do the right thing in terms of query planning all of that good stuff and then the second item here is just you know essentially a word count basically saying how many values are there in this type in the type field which is a column on your documents right if you want to learn more there's a URL it's just our github and then spark solar it's a patchy license publicly available go nuts with it for us we then feed this into our overall application I'm not going to spend too much time on it other than you know when it comes to fusion this is our commercial product that we we build on top of it we primarily use solar for pretty much everything do in terms of storage and retrieval we keep our machine learning models in solar because we know how to scale it we can treat it as a blob store we know how to do the right thing with it we use it as key value store in a lot of places so on so forth we then you spark to for instance drive a lot of our log analytics so for instance we capture all of our own logs back into a different collection in solar all of our query logs all of our system metrics etc and then spark runs uh we have a lot of out-of-the-box jobs already set up to do things like Bill clickstream models or do you know hourly or daily roll-ups of all of your metrics and aggregations and all that kind of stuff and so spark again is kind of that background crank that is saying i'm going to go figure out what's my best view of this data and so we try to automate that as much as possible zookeeper like I said it that's a part of ships with solar at the end of the day that's something your operations people will appreciate us data scientists don't need to worry about it too much but you know regardless that's the thing that kind of gives you consistency in your data so on so forth some of the other things you know kind of shifting gears into the machine learning side of things at the end I'm going to give you a link where you can actually go try a public repository we take a whole bunch of data news groups etc index them in make them available via solar and spark and then you can actually try out some machine learning on this data set some of the things that we leverage out of the box when it comes to machine learning you know first and foremost one of the things you need to be able to do is very easily capture all of that user feedback that you're getting right I think when I think about why Enterprise Search I enon internet search fails it's often because it fails to account for the way users are interacting with the data and now you know most people's first reaction to that is oh well there's not enough users in the enterprise and while that's true for perhaps small and medium businesses it's not true in the sense that there are a lot of inner actions that happen behind the firewall that companies do not take advantage of right so for instance we work with one large aerospace provider where they actually were taking in all of the emails all of the chats all of those kinds of things and using the data that the employees were creating themselves to then define an expert network so that you could go into their search system and say hey find me the expert on spark and it was based on partially on their own declared interests their resumes but also built off of the conversations that those people were having with each other and doing language analysis to say oh that person is who everybody talks to about leucine he's the leucine expert or she's the spark expert because everybody asks her about the spark application that they're running right and so they're able to then more effectively find talent within their own organization right other places you know for instance being able to we're working with a large bank where they're taking and logging and making searchable all of the rooms that people go in and out of within within the company now how you have to use your little key card to get in and out of specific rooms they're taking and logging all of that into a search engine they use it for fraud detection and bad behavior but they also use it to try to identify what resources to allocate where do they need more space all of those kinds of things so number one you know thinking about that up front how am I going to capture that information to we leverage a lot spark ml and my house how comes with an alternating least-squares implementation that's good for collaborative filtering style capabilities we ship all of those out-of-the-box we ship with a number of clickstream models and multiple multimodal recommenders I see a lot of people building recommendation engines they have they have searched being built over here and they have recommenders being built over here well in my world those are two sides of the same coin right and and from what i've seen these multimodal recommenders are much more effective so they're much more effective than than one or the other by themselves and yet companies routinely they have their data scientists over here doing recommenders and they have their search team over here and those two never talk and yet they're very closely related so we've seen quite successfully a number of times where if you take and build recommenders out on top of your search engine you one get a much more simple operational footprint but you also can take advantage of all of the things that search gives you as well so search helps you overcome the cold start problem for instance helps you bring in things like your ranking factors all of that kind of stuff I wish I could claim that I invented this but if you go Google there's a lot of companies out there that are doing this as well so other things that are perhaps interesting to people built in multi armitt multi-armed bandit experimentation framework being able to capture what people are doing in different experiments thinking about what trying to train your business analyst your your your people who are respond your marketing team being able to train them to think about running experiments as opposed to hard coding rules that can be a really powerful thing giving them a framework that allows them to capture those signals and then feed that back into the results last but not least we have a lot of nice support for time series partitions so if you have time series data that can be really helpful as well getting into tips and traps you know it's it's funny and I probably don't need to tell this audience this is much but you know I think we all are enamored with the machine learning aspects of all this stuff and and I could tell you just in my own engineering team if anybody if I ever say hey who wants to work on the machine learning stuff in open source or in the product etc everybody's hand goes up right but what they often don't realize is there's a lot of just what i would call to put it nicely grunt work that goes into that we call a things like pre-processing feature selection sampling all of that kind of stuff and so I always kind of go back to first order principles and say start simple right you'd be so you'd be a amazed at the effectiveness of some very simple things when it comes to deploying and building out real search applications that just kind of work for the large majority of people and they don't require anything advanced other than you know some some good engineering practices so for instance often adding auto-suggest to your search results you'd be surprised how many companies don't have type ahead auto-suggest on their site that alone is usually a huge win and it's not as hard a problem as saying figuring out how to get some incrementally better NLP algorithm to process all of your queries and so you know again before you want to leap to all this advanced stuff sometimes you just need to start more simply there next up I try to get people to think more holistically about what relevance means again so if you haven't worked on search before this is all about this ranking function and I want to when when we go in and talk with a lot of our customers and we talk to them about relevance almost all of them are focused on content right what am I putting into the engine and then what query am I going to run against that and that's all they think about right and especially as engineers they're like okay it's good enough they don't think about relevance necessarily out of the box well they don't think about it until their boss comes and says hey why does relevant suck right and then all of a sudden they've got a lot of work to do to catch up so I try to when I talk with companies I try to get them to start thinking about relevance from day one and I know it's a little catchy perhaps but when I think about relevance I tend to think about the three C's you know obviously content here are still very important what you put into the engine you know garbage in garbage out still very much applies these engines these days are very good at handling this side of the equation we've seen for instance which a power solar ships with what most academics would say are the best in class ranking algorithms out of the box you can actually change them if you want but the second one collaboration this is really that signal stuff start to think right from day one how do I capture user feedback right whether that's what they're clicking on or tweeting sharing or the fact that you know they're emailing this article to somebody in the back end of the you know outside of your application all of those collaboration events you can keep and track and then leverage to better understand what's going on last but not least context I think Ricardo mentioned this earlier in his talk as well thinking about who people are where are they what are they doing you know is this your CEO who's searching versus you know your devops engineer all of those kinds of things can feed into making for a better search in some ways this is personalization if you will and again you know as you're doing that think about how much you want to leverage any one of these so I kind of think about is I've got you know at least three different levers here that I can then apply to say here's what matters in my application whereas like I said most people I think just think about the content side of it right because that's what they have the most control over right because you know users are messy and who wants to deal with messy users and dealing with time and space and location and all that kind of stuff is also messy so we tend to fall back to what we're good at other common traps just I don't know how we're doing on time here but other common traps that I see across a lot of search applications and these aren't necessarily just machine learning-based but tend to play out for people first and foremost there's a lot of crappy you eyes out there so so spend some time on user experience thinking about how users are interacting with your data it's often the case that we build search interfaces that are modeled after what Google or Yahoo or Bing or whatever does it kind of general consumer but it may in fact be the case that you have a very expert user and so an expert user may want a lot more capabilities these engines do a lot of good things and so if you're not a posing that to them then you're short changing them on the types of information they can interact with the flip side is also true you often see very complex search interfaces being marketed to the general consumer and that's just not going to be a happy thing when it comes to finding your data despite all of this fancy machine learning we have out there right most people with their search applications still have lots and lots of hand-tuned rules so this is where you guys as data scientists can help really solve this problem you can often just go in and very quickly learn all of their existing rules and so then you can say all right let's get rid of all those rules they've already been learned and they're in the system because we're bootstrapping off of your logs and then we can think about how are we going to give you a mechanism for doing rules correctly almost all search applications you still do have a place and a need for specific rules for a specific amount of time doing a very specific task ecommerce I'm running a sale for this week I need to make sure that you know widget a is at the top of my results because I really want to move that item right because for whatever reason you your buyer ended up you know buying 10,000 units when they only meant to buy a thousand units so somebody made an error there yeah so keeping going down a lot of times you know some of these other ones are interesting as well second is the first loser I think we often maybe human bias built in here but we often get trapped in that oh this this result needs to be number one that's usually not the case in most applications as long as it's in the first few you're usually pretty good give or take although there you the web and certain applications that may not be the case it's often the case that we as engineers and data scientists end up in some local minimum right I once was building a cross-language query engine where I was going between English and Japanese and I knew exactly one word in Japanese guess what that was the query that I did every single time I wanted to test the system as an engineer and so it's very easy in those kinds of situations to build in that bias and get trapped in that minimum so you want to be careful of those things the flip side of that is also that as if as you as an engineer if you make sure you're capturing all that information when your boss comes to you and is mad about the performance of some particular query you can say hey boss that's nice I understand that that query is important to you but guess what it only occurs in point one percent of all of our logs would you want do you want me to fix that one or do you want me to fix the ones that actually make our money right so you want to make sure you capture those things out of the box the pet peeve queries that's kind of similar to the the Japan query that I was just talking about Oprah effect this is the story that as the story goes I don't even know if it's it might be urban myth at this point but just as the story goes somebody made some big change of their website and they were running this whole big experiment and traffic went way up on their website and they were all congratulating themselves on how smart they were well it turns out that that day Oprah had said hey go check out this website or this book and that was what was driving all of the traffic so just make sure you know you're you're checking your biases in terms of how smart you are as well and then last but not least i think you know obviously we all make a lot of assumptions in what we do and so you know that's where i see a lot of people end up in traps when it comes to thinking about search and machine learning so net take away if i could only do three things when i'm tuning somebody's Search application first you know I always tell them make sure you're setting up a way to measure your results you'd be surprised at how many very sophisticated companies out there where you would look at them and you say those guys are at the top of what they're doing especially in e-commerce and they have very little built-in measurement capable letís in terms of understanding what's working and what's not a freebie usually in search is just handling phrases better any time a user types more than one keyword that's they probably mean a phrase and so treat it as such even if they don't put the little quotes around it last but not least pretty straightforward and easy to build a simple click stream model that how it usually does pretty well with search as well again you know your mileage may vary here so if all of that sounds interesting to you what we've been doing lately one of the things I've been working on as long as well as a few other people on my team is kind of I like to call it 20 newsgroups on steroids so if you all remember the alt you know news groups from way back when and that was kind of the classic classification problem out there using the alt news groups from way back when we've put together a little data set that takes something I think right now we have a hundred different data sources in it it's primarily from Apache project so you know Hadoop mailing lists websites tweets etc make it really easy to set up and run and capture all of that data put it into your search engine and then you've got spark and solar together and so now you can start to explore this data very quickly and very easily and then we've also just have some examples in Scala that bring together doing things with spark ml like lda and k-means and word to vac and it also shows you how to do that leucine tokenization stuff that i was talking about it's all open source it builds on top of the the fusion platform so if you're interested in that go check it out this is also something we will be putting up in it as a production site it even has instrumented on the client comes with a client application that has all of that signal capture stuff built in so it will capture like time on page it'll capture clicks etc and so it shows you an example of kind of a round trip of a search application so with that I've got some time for questions there's a few links for you so thank you very much grand for the amazing talk thank so is a anti sequel supported on solar I don't think it's fully sequel compliant just yet i know that you know this is relatively newly released it covers most cases i know people have hooked it into things like tableau and all of that and the large majority of things they've tried have worked the cool thing is is you know how typically when you do select star from we're like equals and all you have our you know percent for wildcard and asterisk for a wildcard you can actually put in full on queries there I believe I'm not the expert on it but that's my understanding but so now you get sequel results ranked by the search engine okay thanks and my second question is for the geospatial is a solar completely capable to provide geospatial insights is it a full-on like GIS solution know it pretty much like easily no it's not a full GIS solution it's just really good at spatial search kinds of applications you know things like find me all the content in this spatial area right and notice I'm saying spatial not geospatial right it's not just geospatial you can actually do n dimensional operations right so it doesn't have to just be lat long in fact some people use it to model for instance office hours or when a store is open right you can model those kinds of there's a lot of interesting use cases for it beyond just lat lon search so I we haven't used solar in a couple of years we switched over to elastic for the user facing search but I've been taking a second look recently because it looks like Solar is actually much closer to the Hadoop ecosystem and one thing that we do in our spark jobs is we do really large expensive trigram queries we're doing large-scale text similarity you know big pairwise runs across a large volume of long-form text you throw a thousand quote eight you know quote carries or each quotes ten you know are 20 words out of index it takes forever yep so how do you guys have you guys worked at all with say having the index inside of HDFS so you can have really good locality and have it distributed across your compute nodes so that you don't say denial of service or cluster when you have 50 EMR nodes and 10 search clustering yeah so solar out of the box can be stored in HDFS that being said actually HDFS isn't that efficient for storing your search index well we've done with our spark solar integration is if you set those up it can take advantage of data locality or if you put your spark workers right next to your solar workers they can bill bill be a lot faster than if they have to go out across so tends to be you know if people who are already using the HDFS if they want to store their indexes in HDFS fine but we don't typically recommend it because the way that the index works with the block cash are with the way blocks are in HDFS those tend to you have to do a lot of tuning of what's called the block cash and that can that can take you awhile there so it really depends on the use case so but for yours I would probably do a lot of that I would probably try out to spark solar stuff you know it's still going to be expensive but it has gotten a lot better especially you might check out the sequel stuff because that's a whole different approach to to how we get results out it's a kind of a streaming thing so yep chains so how do you compare them a hotel is was the MLM airless and to spark solar available outside of the fusion product like all the clickstream models can I use it outside of the fusion so the first question was how does my how it compared with ml lib and Bella Bay alyssia yeah you know and I'm not as involved in my how these days but they are there's some overlap you know how it implements some algorithms that ml lib doesn't you know and vice versa the my house stuff actually has some rebel capabilities around helping you you write out all your matrix operations and then I'm behind the scenes it'll go in and optimize those for you so it has some query you know some some planning and optimization capabilities there for the most part you know how it sported most things to to spark what we're doing is we pick and choose which ones we want to leverage so for instance that ALS stuff i don't believe spark has an ALS maybe they do so you know i don't know how they compare you know we haven't done fine grain analysis there your second question the sparks solar integration that's all open source apache license the things like the out-of-the-box clickstream models and all that stuff those are things that we build on in a lot of ways what we try to do with our product is just ask more interesting questions of the data that's stored in the open source thing so a lot of companies don't have data scientists on staff we try to give them a good reference implementation that works for most people in most situations but then you know like this room you would take and go customize it all day long because you know what you're doing other people you know maybe not so much more question great yes thank you for the informative talk just one quick question about solar does solar provide a mechanism for a be testing and also for relevance tracking over time to see how you're doing you know how is so just performing she ran away from you I solar itself does not have that built-in we do have that on top of it we actually capture all of the data back into solar right because it's all just log data and you can track your session codes and all that stuff and then you know we do the the variance and all that outside of solar upstream and what we call our pipelines so no you would have to to build it in you know a lot of people will use you know a be testing tools you know they've already bought anyways from from marketing so yeah all right great thank you for your time and thank you very much friend I have to housekeeping