data.bythebay.io: Max Ho, An approach to internal search query analysis
Recording: data.bythebay.io: Max Ho, An approach to internal search query analysis
hi my name is Max and uh I'm the senior business analyst at course hero um today I'm going to talk to you about U approach that we use to analyze our users internal search queries and how we use that to really drive actionable and very specific insights for our uh for our product teams to continue evolve our product so let me first talk to you a little bit about corero so corero is a crowdsource online learning platform where students can access and share all of the study materials that they might find helpful in their courses um we also offer tutoring tutoring help on the platform so corer is a content-based uh business so our core product is our content and the three main types of contents that we have on our website are course materials these are the documents that students might create either as a study study guides or lecture notes or practice problems anything that they might need uh to help them succeed in their courses uh the second is uh the questions and answer pairs these again are the uh the questions that students might ask to the tutors and the answers that tutors might respond back to to the students and that whole pair again is on the website for other uh users to study and go through and the last type is flash cards so flash cards pretty self self-explanatory users can create them they can also study the the ones that others have created so on course here we have more than 9 million pieces of contents all combined to date and it's that's a lot of content for one user to go through so therefore the internal our internal search bar is basically our users's best friend in finding the specific thing that they need to succeed in their class um as you can see this is a example of our uh our homepage where the the search bar is right on top and we basically have the search bar on virtually every single page page on our on our website so why is it so important to look at our internal search um well for one search is the direct voice of our customer they're literally telling us through the search bar what they need to succeed in their class and as I said before uh search is one of the is basically the biggest way for users to find the content that they need on our website um and that's the most efficient way so uh focusing on this will have a huge impact on on uh our customers um experience on our website so thinking about how this analysis will help our business so we not only need to look at exactly what users are looking for which is what they type into the search bars we also need to look at are our contents catering towards what they are looking for and are the contents quality good enough to find uh to cater to their needs and are we able to give those contents on on rank them on a high high on search result Pages for for users to find them easily so there's a whole array of uh different things we need to look into to do that and therefore we need to there are also a lot of the different uh search engagement metrics in between that we really need to dive into to really drive the actionable insights from this analysis so here's all the metrics that we're looking at so uh for today we'll talk about the metrics from uh demand which is you know obviously what users are looking for and then we'll look at a bunch of all a bunch of the uh the search engagement metrics as well as our supply side as well as the um the user satisfaction towards our towards our our content that we give them I'll go through each one of them uh uh in more detail so the first one is demand demand obviously that just uh how how much is is a potential search query or a keyword being looked up by a user um so from here we're just doing basically the volume of the search the second one is a click-through rate so this is is more on the engagement side so that tells us for every search that a user would perform How likely is it that they will click into one of the search results the third one is the mean reciprocal rank so this slide is pretty full but essentially what it means is it's it's a measure of where users are clicking on the search result Pages a high mrr or means reciprocal rank means that users are likely clicking on the first couple of searches uh or first couple of search results whereas a low Mr Means users are likely going deep into the search results likely on the second or third page uh the fourth one is the conversion rate so we're using conversion rate to measure uh user satisfaction towards uh towards the content we're able to deliver to them so conversion rate from here we're saying for every search that user does How likely is it that they will ultimately convert into a premier subscriber the fifth one is a search coverage rate and this is a a supply type of uh metric from from here we're basically measuring for every search that users would uh enter into our bar uh How likely are we able to give them a certain set of results and over here we're setting 20 and the last one is a document download rate and this again is a measure of uh satisfaction for our users from the content so uh again this is more geared towards a a user who has already subscribed and once they do a search How likely are they to download the document and further study that document so now let's talk a little bit more about uh the actual search text strings themselves so the text strings are very specific um they they can be very specific they can be very broad there's a whole wide range of them and I I'll show you a couple examples later and to really Drive uh actionable things for our business we really need to focus uh our efforts on the very uh the very specific things that we can actually do things on so for example our key business questions here would Define it as we want to find the most popular subjects and Concepts that you uh our users need and as well as the courses that high in demand that uh that they need help in succeeding uh as well as uh the type of material so either that's study guides or lecture notes or uh practice exams or stuff like that so here are a couple of search C examples as I mentioned before the first one as you can see is very very specific it's basically the entire uh question prompt um and then you can see then going down to list then you see users might be searching for a specific subject and a specific type of uh content uh that they're looking for for example Spanish uh vocabulary uh flashcards um users could also be searching for a specific course or at a specific school or they could be just searching for a general concept like uh general theory of relativity so now because we have all these different very specific types of things and we really need to focus down really to the subjects and and the Very actionable things that we need to do uh the way that we kind of parse out our search is basically we extract those important keywords from each of the search uh strings and then we basically group and by all of that um all of the keywords and then quickly you can see that if you just look at the first the onedimensional you can see that chemistry for example pop on on top that's the most s out there we can then go another level deeper and attach pairs of strings that contains Cam and then tell us a little bit more about what about cam so in two dimensional we can see that there's cam study guides and these key these pairs of keywords will show up a lot together and that's something we get a little bit more insight into uh what users looking for then we can again introduce another level then we can say okay well cam study guides from Stanford University that's what users are really looking to to get help in so we can keep doing this more and more in more dimensions and then you can start beginning to get very specific insights but keep in mind that every Dimension you add that exponentially increases like your data volume and then the complexity of analysis so now um one thing we need to look at separately is really the the different types of users uh there are two two main types of users we'll focus on here is the visitors which are the new users and the premier users which are the users that have already uh bought a subscription so we these two users have slight different uh slightly different flows in their way from searching to actual content consumption so we'll look at these two a little bit separately so we'll F focus on visitors first so here's a typical uh flow for for a visitor so a visitor will come on to our our landing page uh they will perform a search on our internal search bar and then if they see something that they like they'll click on the content and if that content is indeed what they need then they'll go ahead create an account convert into a premier subscriber so now we can take the keywords the one dimensional two dimensional thre dimensional keywords that we talked about before and plot it onto a quadrant on Demand on the x-axis and the conversion rate on the y- axis so basically this is telling us uh where our strength are so very clearly we can see that the terms that is being sought after a lot by our students and it's driving a lot of subscribers that's obviously our strength we're doing well there so that's that's awesome but we can also look at sort of the bottom left of the of the slide and look at the opportunity area and see a lot of some some of the terms are highly s after but not really driving a lot of users through our website and that's the area that we can kind of double click and go in and say what's going on there so just kind of giving you an example of what kind of the data looks like uh this is just uh example data so uh the first three sets of Search terms you can see those are our our search our kind of strength quadrant right so we got uh relatively high demand and pretty high conversion as compared to the rest of the the Search terms then we got the second set of the Search terms which are the our opportunity quadrant right the demands are uh just as high if not higher than the one from the search quadrant but the conversion rate is significantly lower than the strength uh quadrant then we can focus on so again if we want to double click into that opportunity quadrant and just dive in a little deeper we can look at the search coverage rate which is the the fourth column there you can see that the first uh search term marketing 101 Cornell that one we have roughly about 86% coverage which is much higher than the other two so then again going back to the quadrant view now we're introducing we're again remember we're focusing on the opportunity quadrant now we're uh splitting that on the uh the search coverage uh Spectrum on the y-axis so again we're looking at contents that are highly s after not driving a lot of conversion and if we don't have a lot of search coverage that means we don't have content for it then the software is we need to invest and get more content there if we look at the top left of the of the chart we can see that um if users are looking for those contents a lot we have a lot of those contents but users are still not converting off of them then that's a whole whole different problem so potentially we can hypothesize that it could be maybe the relevancy of the content or it is it the misalignment of our content quality and the pricing that's not uh uh making the the users to basically pull the trigger and and turn into a premier subscriber so now we can dive even more into that and really peel that part again so now we're looking at uh the next metrics we'll look at to kind of spread out that quadrant is the click through rate um so again highly sought after com uh contents we have a lot of contents but users not really converting so now let's see are they really clicking on their search results or are they not clicking on search results so again looking at the Spectrum on the low clickthrough rate if users are not clicking on the search results then then that's pretty pretty clear to us that the search results are not relevant on the other end of the spectrum if users are searching are clicking on the results but then they're dropping off after seeing the content then that tells us it's really the quality of the content or the pricing of of our uh subscription that's not really matching up for them so that's that's the area that we'll we'll need to focus on to solve that problem we can then also look at uh the mean reciprocal rank to kind of dissect our uh our search algorithm and see how ranking is doing so um regardless of the click array or the different or the different metric we can look at if users are uh clicking on or the Search terms have low meaning reciprocal rank that means users are really going deep into the search results and that means either our uh top search results are not relevant so we probably need to uh uh improve on on on the rankings um and if the main resal ranks are high that means only the top few research results are relevant so as you can see like all of these metrics will tell us a little bit something and that's where we can really Drive uh Point our products to the right direction of what what are the uh problems we need to solve so now let's uh let's look at Premier users uh Premier users uh very similar to how we do it for uh for the new uh visitors so again we'll plot out the um their typical flow a premier user will log in again they'll inter uh perform an internal search um and then if they see something they'll click on a Content page if they ultimately like that content they'll unlock it get access and then download it to study more so very similar flow to uh to what we what we have for the new visitors and then again you can see we can follow the very same approach plot these on the demand and uh document download rate which is the ultimate satisfaction measure for uh our content delivery and again you can see that there's a strength quadrant a lot of users are searching for certain key terms and a lot of users are downloading those contents to study that's great uh on the other side for the opportunity quadrant high demand Low download rate there is something that we need to work on there and then again diving in we'll look at the search coverage do we have contents to answer those if we don't we need to get more content if we do what's going on so very similar approach so the key takeaway is really uh by doing this analysis we're really able to find out one what users really need to help them succeed in their courses and two do we have what we uh do do we have what they need to help them succeed and everything in between is sort of the ways for us to take uh basically improve the user experience and then the uh the different metrics and probes will send out that's really um help us to get there so uh this is an example of how uh the analytics team our analytics team can uh really influence our products uh road maps as well as strategy um yeah that's it question so one of diing together how do you extract from these like you uh so yeah so oh yeah yeah so the question is uh how do we extract so one of the keywords that we extract together is actually a pair of keywords so how do we extract so I think that's a good question so we actually that's actually one of the specific things that we look for cuz if you just have study and you have guide like two separate things that's like you need another dimension to really pair them together so those are the specific things that we look for say Okay study guide these two got to be together but how do you do a system could be next TOS systematic how do you oh yeah so we so we actually so what we do is we parse out every single Tex uh every single word and then for those pairs that we know needs to be together we'll we'll set those uh special conditions and then we'll pull out all the fillers and then from there we just then just group them all together and say what are the top highly saw after what are the you know highest converting ones so the process to together like together aary uh yeah I mean like if we know that study guy like study and guy needs to be together then we'll have like special rules for that but aside from that then we'll just have to uh combine all them and just plot them on the yeah on quadr yans for to theer toy again the meaning of the query um yeah so so I mean I think for that that's that's a little bit further down the line for us um right now what we're really trying to focus on is just uh what what can we deliver and then really where to invest in terms of the actual content uh so we do parse everything out even if it's a long even if it's a big question what pars it out yeah and by definition those uh long like the longtail searches those are very very specific so it's very un it's very unlikely that they will have like a couple of users searching for the same exact thing yeah follow on that you actually do an do you do AO between each every you have um clarify that what what do youan by bull and R bull like both the terms should be part of yeah oh yeah yeah so if it's if it's a two Dimension then it's got to be and so then these two Cur these two key terms has to be has to appear together that time you deter whether these two terms are stud guide do you phrase it do do phrasing in real time to figure out the dimensions uh no so Dimensions we we we have to preset that dimensions and then that's how we able to plot that onto the the quadrant so then the study guy just think of study guy as one term rather than two terms yeah no thank you cool [Applause] thanks