data.bythebay.io: Daniel Doerr, How to Use Big Data to Inspire Consumer Confidence
Recording: data.bythebay.io: Daniel Doerr, How to Use Big Data to Inspire Consumer Confidence
hey guys it says my name on the slide my name is Dan I've been with credit karma for a handful of years on our data science team uh really excited to be here at galvaniz at data by the bay today um I think I have kind of an interesting question I was hoping we could chat through and figure out how we could solve and it'd be interesting to get uh you know your guys' take on how this might work for you uh but before we get started I did want to ask I know this is day five of the conference a lot of people have been here how many of you have been asked what does big data mean or what is Big Data uh you know so far in the last few days any show a hands here no nobody's been asked what is Big Data oh all right all right well I'm going to ask a different question uh so for you guys it'd be interesting to hear maybe some feedback if you've got it uh how does Big Data measure the results that we're driving uh you I mean anyone have any particular metrics that we measure I think uh I've had a few conversations over the last few days as a quiet audience we might have to get people doing jumping jacks or something but uh but uh over the last few days I had some conversations I heard some things like people are measuring well uh how many people do I have interacting with my product and coming back at a later time to you know to reuse my product uh how much revenue am I generating per user how many interactions am I generating per user uh a lot of these metrics they they really boil down to being uh business metrics uh and so I think that's actually kind of interesting uh what would your response be if we kind of changed the question and asked well how do you measure the user impact you're driving uh with uh you know Big Data anyone have any take how you might do that I mean it depends on you SK right so uh let's say we predict customer likely to buy then we measure you know we run test and control and see know which one give uplift in terms of Revenue so we meas directly revenue or the close rate which which you know counts closing faster than the control so have yeah no I think that's exactly right and I think that brings up kind of an interesting question you know like as Big Data like do we all agree that you know someone interacting with our product number of interactions they're they're using or the amount of Revenue that we're driving does that directly translate into the amount of value we're creating for the users of our product you know I I'm not sure that relationship is true I think it's uh my take um people use S called kpi ke performs indicator and that indicator is context depend whether that CH into the performance me yeah no exactly right yeah and so I think in this case here I think that's a question that we as a group have an interesting opportunity to to answer um so to get to what I'd like to chat to over the next like 15 20 minutes on is you know how can we as the Big Data Community really go beyond whichever metric we're using and measure and drive meaningful change in the lives of our users uh so I think this question is really interesting for a handful of reasons um you know first and foremost if you look at the number of interactions people have in their lives and the percentage of them that are influenced by Big Data and the group of us in the room actually a pretty large number I imagine so uh you know I'd like to walk you through a little bit of well how did we go through taking a look at this question you know on our team uh you know and what would it take to actually incorporate one of these metrics in an optimization we're doing and hopefully by the end of this you'll have an idea of a couple steps you could go through in your own businesses with your own teams to try and ask this question and maybe take some of the optimizations to the next level for your users so uh how does this apply to credit cards Karma uh for those of you who aren't familiar with Credit Karma uh we're a uh online personal finance platform and I think our goal is really to to try and enable people to make Financial progress to enable that progress for everyone and when users come to our product we provide them access to their report their scores uh as well as uh Financial tools and content and uh personal finance Marketplace so back to the mission for you know for one more second here making fincial progress possible for everyone like how would I actually go about measuring that I think that's you know it's an interesting question I mean that in and of itself you know it's kind of meant to be an aspirational goal but how could I as a data scientist use the data I have to measure my progress against that that uh guidepost uh well I mean I'd probably have to start by measuring Financial Health and as it turns out Financial Health is actually really hard to measure depending on the industry you're in or the um you know the the type of information that you'd use to measure Financial Health uh there's a wide variety of it a lot of different data points um and what you're really trying to get at could be different so let's see if we can try and find a metric that uh you know matches the type of information that I have at Credit Karma to use and the type of impact that I could potentially have on my users so what would it take for me to decide on a metric that works um well first and foremost we're we're really looking to do some sort of optimization with this metric so ideally it should be quantitative um so taking a look at a handful of quantitative Fields uh if you look at economics for instance one of the metrics they frequently use for measuring Financial Health is consumer confidence which is an interesting metric you know that that metric tends to include at least two components one an element of what's someone's uh outlook for the economy as a whole what's their opinion of you know the economic health but what's this person's personal financial Outlook you know how how much uh you know what can they expect for themselves and so I'm not sure credit karma really has an opportunity to influence you know someone's expectation for the economic health but I think their own personal financial Outlook that there's probably some Synergy there so let's see if we can break that down a bit so if we're trying to break down personal financial Outlook you can generally break it down into two terms uh at least two terms rather uh one what's the strength of someone's Current financial situation and second uh what's their ability to handle unexpected financial needs uh so you know looking at the information we have how could we take these Concepts and try and quantify them one you know the strength of someone's Current financial situation well we have access to someone's Uh current debt situation and what sort of assets do they have are they someone that's likely to be impacted where you know by a financial need very easily or are they someone that's resilient to it in the event of a financial need you know what sort of options does this person have available to them and how well do they know those options and so I think you know these are all things that we could actually quantify with our product uh you know we should be able to quantify somebody's current credit score maybe uh we should be able to quantify what's their awareness to the types of products they have available to them and so I think this framework seems like a pretty good fit but um until now it's still just a framework so how do we actually go through and work this into one of our optimizations uh so I'd like to introduce you guys to John uh John Doe that is and so he's a data scientist like at least a handful of us here in the room and you know John came to Credit Karma today so what do we know about John well as it turns out JN has a pretty decent credit score it's probably a little better than mine uh he doesn't really have any cash on hand because he just recently made a large purchase uh and he needs and he needs some money to pay for a car repair you know he needs a couple thousand dollars because as it appears his car fell down a hole nobody finds that funny nobody all right there we go we got we there that's much better guys um but yeah so he needs a couple thousand dollars and so what's Jon's first you know thought his first thought is well I don't have cash so I need to borrow let me take a look at some credit cards to see if there's one I can use that should give me an opportunity to cover the expense but the flexibility to pay as I earn the money back so where are we going to start so let's start by searching through tiers of different credit products that we could use so we'll SE we'll take a look at How likely the products are to be interesting to John and How likely those those interactions that he might have with them are to be successful and so we'll find that there are some tiers of products that are very interesting have really great features and rewards and terms but are really unlikely to yield a successful interaction like Jon's unlikely to be able to actually have that product and we'll have things at the opposite end of the spectrum so let's narrow in on that second tier here something where John's eligible for and they're still somewhat interesting so now it's a ranking problem how do we rank all the different products that we have in that in that setup well I mean in this case we'll again we'll look at the products that are interesting to JN and the products he's likely to be have a successful interaction for and we can take these and we can sort them and present them to them but I think the question I've been asking is if we leave it here you know and we sort these options for John allow him to select one of these credit cards have we actually used our data to measure the amount of financial benefit that we've you know that we're creating for John I don't think we have I think we've kind of come close but I think this is where Big Data really needs to come to the rescue so so how can we do that well we can start by remembering that JN is just one person and we actually have the opportunity to uh draw from the experiences of a whole bunch of people just like of the different users of your products um and you know in our in the case of our product we've got about 50 million so there's a good number of people that uh you know we can use to draw from you know draw from their experience so how do we start identifying users like John well let's take what we know about JN we know what he's here to do today we know his credit score uh we know his report um and we know a few other things about him and so let's take the other users we have and try and filter them down and understand well what did they do in situations like this once we've identified well what have they done in situations like this you know how do we measure the future Financial strength of these sorts of people which brings us back to that question I was asking so we have that framework we can look at well how strong is someone's Current financial situation and how you know what's their ability to handle an unexpected need so if you look at the data we have we have someone's credit report and scores which we can use for a couple things we can measure well what's their ability to borrow does their credit score keep them does the their credit score make them eligible to take out products to handle certain types of needs um how how debt Laden are they is their current situation such that if a large Financial uh need came along they would have to borrow um what sort of assets do they have on hand and so I think uh someone's interaction with our type of products should give us the opportunity to measure that so we should be able to take a look at the data we have and well let's look at what happens when someone chooses to do you know option a option b or do nothing you know what are those metrics look like over time can we look at 12 months 24 months down the road so so we can take a look at that for a handful of users that look like JN um in this case they're all different colors but uh you know a handful of people like JN and uh you know we should be able to quantify this so we should be able to create some sort of composite metric that quantifies well how much improvement in their ability to borrow have we seen how much improvement do we see in their you know the strength of their current situation their ability to handle um you know business as usual um and the next step I think is probably pretty clear for you guys what would you guys do next if we can quantify uh you know the difference we're actually having what's the next step did I come in products well but before you do that you probably have to build a model [Laughter] right so so let's take all the information we have and let's let's build a model right and I think uh you know there's probably a handful of people in here everyone has their own favorite technique I think you can look look at this problem a few different ways but in essence we're going to take the information we have the type of interactions that could potentially happen and make you know and what we saw 12 and 24 months down the road in terms of Financial Health for these users and see if we can build a model to predict that for John so we should be able to do that to some degree of you know to some degree so now that we have that model let's actually make some predictions for John you know let's take what we know let's push him through the model and we'll see that as it turns out some of these products are likely to create more financial benefit for JN in the long term than other products and I think this brings us back to that optimization problem we were looking at before and so I think this is really where things get actually interesting so now that we have you know these handful of products we can say well we were able to evaluate the options you came here to look at and the options that you might not have otherwise considered um but we still have to figure out in the context of our business how do we take the current business optimization that we're doing and incorporate this measure of user benefit into it uh and so I I suspect that for each of us in the you know each of you in the audience here and for myself at each one of our businesses that conversation will be slightly different um and so I don't know what the right answer is for everyone as how specifically we can incorporate that but what I do know is that we can do a lot of different testing and so I think uh you know one of the tools that we should be able to rely on here is well let's test what happens when we rank things different ways and we use this you know expectation of user benefit in a handful of different uh you know with a handful of different methods uh one of the things that I you know I think is really powerful in this situation is we don't know which of these types of products Jon is you know most likely to choose in the context of some of these have different benefit for him so we should be able to sort them rank them and allow Jon to choose and so I think allowing Jon to choose and allowing our users to interact you know in a way that's benefit beneficial for them does two things for us one it's more likely that our users actually Come Away with an outcome that's helpful for them in this case we were able to share some context as to why we're presenting the options we're presenting and sort them in a way that's most relevant for JN or at least we suspected would be relevant and uh Jon was able to make a choice and able to solve his need um and we know that you know down the road JN will be slightly better off uh you know compared to the other options he could have chosen so the other thing this does for us is it actually provides another data point that we can use uh you know to help optimize further decisions down the road and I think that's really why we're all here it's like you know how do we use all of the data we're collecting every day in terms of the interactions and both what interactions our users are making but what things that our businesses are doing to the users and how can we use those to make better decisions down the road so to kind of bring this back to you guys you know so how can you know what did we see in this process here and so I think we saw that there were a handful of basic steps I think one it was really taking a look at whichever product where you know know any of us could work on and what's the objective what's the mission of that product you know are we looking to you know in this case we were looking to drive Financial benefit for users make Financial progress possible and so the metric we chose was uh you know what is your future your future ability to borrow and the strength of your future situation for some other products it might be slightly different but once we've done that let's take a look at the tools we have available for that you know what what data do we have at hand what metrics can we use um what interactions can we use in our product to help drive towards that metric uh third uh you know now it comes down to a business conversation now that we have some sort of metric you know how can each of us do some experimentation do some exploration and understand you know what is the impact this you know introducing this metric into our optimizations has on the business so that being said you know question for you guys is you know how will you go about using some of the same framework to drive uh benefit for your users uh you know it might be useful to take a couple seconds here and just turn to the person next to you and let them know like what are you going to take away from this like how are you going to try and use this to help your users I think we've got about 30 seconds or so for that nobody nobody oh we got some people chatting about it in the back yeah take a second and turn to the person next to you you guys over here are sitting alone so we're going to have to work on that yeah just I guess we didn't use this cool all right looks like we've got a [Music] question I will facilate from the because what happens you always have lied data from whatever Source dat Source but more data points customers the better you can make Improvement in their lives more profil I like to add more Behavior dat to make you know the or whatever I want and then I have a question do you guys use the other side of the data which is the what I see is all all lending lending lending but you miss out the investment and then when you m this two data you can do much more and so just to repeat the question and so he was asking uh you know we saw a lot of chatting about uh credit information and lending data do we use any investment data I think uh right now Credit Karma doesn't deal much on the side of Investments but we do have uh a product that allows users to connect uh some of their financial accounts and allow and you know allow us to help them monitor those for fraud and things like that um but uh yeah with that being said um you know the investment data would be pretty useful because I think that you could build a much better picture of someone's Financial Health and make it easier to drive some meaningful Improvement there uh but I guess we have some time for questions uh do we have any other any other questions in the room so unfortunately for us we have the the Lu and the pain of of being involved in the financial industry and so what data we use um and how we use it is we have to be really careful and so it it really depends on the type of decision you're making U because we have to make sure that we're protecting users uh from you know you can look at ecoa or any of the other types of uh regulation around Financial in institutions does that mean you use allocation and stuff like that um in most cases is using location for like a financial decision would would not be allowed so um so we couldn't use that but um if we wanted to send you a greeting uh I'm sure we could use you know a location location sensitive greeting but uh actually I asked sometimes healthare there's a correlation between some health conditions in certain locations in the country I thought maybe there's certain correlation with that too SP how yeah if you look in San Francisco we spend way more on housing and I'm I haven't figured it out I'm living here I don't know how you guys do that have you deal with survivorship bias in your data especially the positive and negative ends of right so the question was how do we deal with survivorship bias uh in the data that we have um I think it's actually a really good question and so um if you look back back in our history we really only pulled credit information for people who came back to our product and so it was uh not as easy to measure uh you know the financial impact you were driving for people who didn't come back and so uh you can imagine there's a handful of other things we're doing now to try and get visibility into that picture but uh in general we're trying to make sure that we do have visibility for people who come back and don't come back do you ever get uh user like how do you collect user feedback in case your recommendations your essentially your model engine did something was wrong the user gets frustrated like why is this to me and then you get a lot of that or is your data kind of you don't uh so the question was um you know how do we get uh how do we gather user feedback in the event our recommendations are incorrect or not really what someone's looking to see uh you know I'd like to imagine that they are bulletproof but I know for a fact they aren't uh I we make really strange recommendations for me all the time but uh you know don't insert yourself into the data that's what they always say um no I think uh we have a handful of ways to gather some user feedback I think you know if you look at product reviews uh you look at other sorts of interactions people can have but uh the rest of it I I'm not sure I'm supposed to share so I'm going to dodge that question so we've got about one minute left here so any more uh uh one maybe one last question so do you make your data available at all and online data um no no no uh nope no the question is do we make our data available uh no as far as I know uh we can ask our lawyer but as far as I know that the data is not shared um and it's uh stays hidden but um yeah other than that any other ones great no thank you guys a lot this was uh this was really good uh hopefully you guys seem much more awake than you were at the start of this presentation so it's a good thing thanks [Applause] guys