data.bythebay.io: Yu Jay Huoh - Making Predictions Under Lending Regulations
Recording: data.bythebay.io: Yu Jay Huoh - Making Predictions Under Lending Regulations
thank you uh thank you everyone for coming out um so this is just a brief outline of what I'm going to go over today so first I'm going to talk um about Earnest and what we do um then I'm going to talk talk a little bit about how we model credit decisions at Earnest um I'll touch very briefly on the Equal Credit Opportunity Act um so in the description I mistakenly um listed the Fair Credit Reporting Act which is my fault um not going to talk about that going to talk about this guy um just wanted to clarify um and lastly I'm going to talk about how we use Ensemble methods to sort of um satis safy the requirements from the Equal Credit Opportunity Act Right so um Ernest is an online lender um we were founded in 2013 um we mostly do uh personal loans and student loan refinances um one of the key differentiators about Earnest is we collect data from many sources right so first we get the full credit report from one of the credit reporting agencies so like Equifax or Experian TransUnion those guys um but we also get transactional financial data so when applicant applies we can see all the transaction history going back um and this is from uh companies like indu or Yodi so if you're familiar with those uh companies that's the type of data that we're getting as far as I know we're the only company out there that has access to both of these data sources at the same time for like the same um persons right and so um one thing unique about earn is we try to uh really we really want to emphasize using technology and data to make better lending decisions right so we're collecting all this data and it doesn't make any sense if we're not going to use it to do anything right um so one of the first things we tried to do and you know an example of trying to emphasize technology is we tried to automate our underwriting process through modeling and so what we wanted to do was sort of build model models that would replicate what our Underwriters were deciding for loans right so our original plan of action was so you know very simply um We Gather the data right so this would be both the applicant credit characteristics but also the outcome of the decision so like whether or not we approve the loan um then we sort of curate some features around these characteristics and then we train a yes or no model on the outcomes right and so um none of these steps is particularly hard right like your data might be poorly organized or like not structured well so it might take you longer to gather everything um it might you might spend a long time like curating you know the world's best features right that it might you might spend some time there but the actual modeling is relatively straightforward right so our Underwriters are humans and human brains just you know aren't wired to process that much information at the same time right models on the other hand are are very powerful right if you give them more information they'll they'll do much better right and so our our sort of initial approach here was just you know um engineer a bunch of features and like throw it at your favorite algorithm right so this is not a hard problem right you you come up with what like 200 features this is a01 outcome they solve this problem in like the ' 80s right we're not putting this on kaggle um so this is an examp example of our results from like doing this model right so I built a um I went through that process and we built a random Forest um this is the RLC curve on uh these are out of sample predictions right so uh this curve is really steep right this model is great um area under curve of 9479 right um but it turns out we're we're not quite done right so um it turns out that lending is a very highly regulated industry um and these are a few of the laws that all lenders are subject to whether online or like in person um whenever I see a list like this with like Usery laws on it I always think to myself like ah stupid people in the 30s always ruining things for the rest of us right so this stuff is like you know fun to joke about but it's actually really serious stuff right these things sound ridiculous to us today but that's because these laws exist and have and like institutions have been following them for so long um so the main law that I'm going to talk about is the Equal Credit Opportunity Act um so this law was enacted in 1974 to prevent discrimination in lending um so it explicitly prohibits certain information from being collected at all during the application right so we're not allowed to ask whether or not you're married or if you have children or planning to have children um it also prohibits specific attributes from being considered at all right so we can't use race or gender or age in the um when we make our decisions for whether or not to approve you for your loan and this last part which is sort of the most relevant one um this act requires the the lender or creditor to give specific reasons why an application was rejected right and so this is to provide transparency to make sure you know everyone's following the top two rules right so if we sort of go back to our original model right this is sort of not enough anymore right we also have to be able to given all the people that we decline we have to tell them why right and so stepping back for a little bit right all of these machine learning methods that we're familiar with like random forests which we used here um gradient boosting machines or neural Nets these are all very powerful methods um but I'm sure you guys are really familiar they're also very opaque right and so here we've built this random forest and I've you know tuned the crap out of it it does great um but given this new requirement how do I back out the decline reasons for anyone that the model says the decline and so you might be tempted to um look at the importances right and you can try to like make some guesses on like why someone gets declined right so if we say you know I apply for loan I get declined and we sort of like look at this stuff like uh your credit report isn't good right um I just want to point out that I've sort of um masked the the variable names uh CU I don't want to like reveal any of our secret sauce but I swear this is real like this this is a real thing um right so you can try to guess stuff based on these importances but at that point you might as well be doing this um this is one of my favorite photos that comes up a lot at work um I realize this is a stock photo um so if the author is out there I swear if this thing takes off I'll buy this photo off of you um but until then thank you for the free photo um so what are some other approaches right so a traditional institution would try to dial everything back right so they they limit the variables that you look at or like restrict the types of models that you consider building and um you almost always end up with some silly risk scorecard so um if any of you guys are familiar with uh working in finance I'm sure you've seen something like this right so your risk score is equal to your FICO plus your income plus your assets and then they like ding you a little bit for your debt right and then if your risk score is high they give you your loan and then if it's low then they decline you and then because you know this thing is so simple and the variables are are very easy to understand right it's very easy to back out the the decline reason right so if you get declined and your FICO is too low they tell you your credit history isn't very good right um if you get declined and your debt is too high they tell you you have too high of debt right and if you're income is too low they tell you something like ah your cash flow is low um so we found this very um unsatisfactory right like we weren't gaining anything if we just settle on a model like this and so you know if we if we go back to our original plan and think about how we went about building our models right like the actual decline reason data that stuff exists right the law like we're we're abiding by we're collecting all that data like for every loan that we that our Underwriters were declining they were also selecting these decline reasons um but in that original plan we never actually like Incorporated any of that data into our model building right and so uh we sort of we wanted to come up with a way to incorporate it into the process and the way that we settled on was instead of building this uh you know this large model for uh yes or no outcome right instead we build a bunch of much smaller models right one for each decline reason and the model is sort of predicting whether or not this decline reason can be applied to the loan right and then um once we have these small individual models we can take an ensemble out of all of them to predict the final yes or no right and so what we're doing here is we're sort of imposing a bit a bit of structure not super not a lot but a bit to gain a bit of transparency into the actual like what the model is doing um so this is how it looks like in pictures on the left here we have uh you know all the stuff that we're collecting in the middle these are sort of um individual models right so there's a model for high debt there's another one for like Mis payments right and these um produce like a 01 right and then on the far right we take uh an ensemble out of all of them to sort of make the final decision um I wanted to talk very briefly about Ensemble methods so in general there's sort of two paradigms when it comes to Ensemble methods um the one everyone's most familiar with is uh taking a bunch of weak Learners right and then getting a strong learner out of it so all these things like random forest or adaab boost or boosting machines right they take very weak like decision trees right so um in adaab boost if your tree only has one node it's called like a stump right and the collective of all your trees right that gives you a really strong model right so the other Paradigm is you take a few mediocre Learners and then you can get a strong learner out of them and that's sort of the approach that we've taken back here right so um each of these models in the middle is sort of it's it's an average it's an okay model for whether or not we're going to decline the loan right so not not every loan with low assets gets declined some of them get approved but when we look at all of them in aggregate we can make really good decisions um so these are so I I ended up building random forests for the individual decline reason models it turns out you can build anything um I just had all that stuff set up um so on the left here these are the importances from our forest for predicting low savings right and and then if you look at the top four features these are all um based on their account balances on the right we have um the importances from our model for predicting uh if someone is in collections and everything's based on the credit report um this is the same stuff uh this one on the left it's high debt to income ratio um the top two are related to income the middle four or the next four are all related to the amount of um monthly payments that the borrow is making right so uh the the next four like the first two is the income which is your denominator and then the other four are the uh numerator right your debt um on the right is sort of one of my favorites so this is high debt most of the credit report related stuff is related to account balances the two really cool ones are the transaction ones in the middle and those are related to whether or not the borrower is getting assessed um interest charges every month right so whether or not we we're able to tell if they're actually like paying off their credit card every month or if they're holding a balance um so um to do The Ensemble step um I used the glm to model the final decisions given the outputs from the individual models um I picked the glm because it's simple and also because it's really easy to back out the decline reason reasons right so um what we can do is we can use the effect size right so um we it's just the the slope from our glm multiplied by the score from our model and then we we pick reasons based on the highest ones and then maybe apply a bit of like thresholding and business logic to make sure we're only picking relevant reasons um this is an example of you know doing the effect size math which is just multiplying um the second column is the uh slopes that we estimated in our glm the third column is the output from our individual decline reason models and the effects on the right um and we take any effect that's larger than 02 right so for this person we declined them we would tell them they have low assets and high debt to income um so this is the RC curve for our Ensemble model that's the thing in gray and black um the the red line is the RC curve from our original um so every time I see this I always get very uh I feel really good about myself um but you know we we lost very little signal um but we gain all this transparency like even the area under the curve like barely changed right and and the best part is this same model or this model with ensembles over 96% of the decline loans we were able to correctly identify a decline reason right and we didn't have to restrict any of the features that we can consider we we could have built any model that we wanted I just picked random Forest because we had that set up but we can build any like you can thow your favorite thing um and we didn't lose very much accuracy um so that's all I have today um any [Applause] questions they each so the question was in my data sources do I check if they're correlated with each other like if I'm using the minimum set um uh so before this step like before the stuff you said here you see here um we did a bit of feature selection to prune out the stuff so for the most part after pruning they're reasonably uncorrelated that's question I was curious um these are former formerly rejected loans versus so the the data that we use to train the the models is um what our humans were deciding like what what decisions they made yeah so the question was what I meant by glm on this slide yes uh this is generalized linear model it was a logistic regression yeah uh the question is what is the ratio between positive and negative samples um I cannot tell you explicitly because that would imply like our approval rate but they're they're not it's not super skewed it's decently balanced so the question was um for the intermediary models um whether they output a yes or no or the score and so they they actually output a probability yeah if you were using just logistic progression looking at the coefficients of this uh features what would you have um so do you mean like if instead of using random Force here I use logistic regression for like all of them um I remember having an intern do that um the difference wasn't super big um I think we we could have so I he he looked at a bunch of things he also looked at using like uh um uh gradient boosting machines right and uh the Improvement was marginal um we we settled on random Forest because we had a lot of it built out how many trees so the question is on how many trees that we ended up using um each of these is about uh 500 yeah just nothing nothing special like they were they weren't they weren't like super hard right like humans don't do that much yeah I guess how is a problem and how you deance is a problem you you to solve you know how many you said like yes how how you so the question was on um how unbalanced was the data and how I solved that um the data was pretty balanced for the most part um that that actually went into like how we settled on these reasons so it turns out our Underwriters pick like um they have like 20 reasons to choose from um we settled on the nine most popular ones and those nine were not super skewed or anyway yeah um I think I'm out of time so thank thank [Applause] you