Mining Noisy Transaction Data with Neura...

data.bythebay.io: Frank Taylor, Mining Noisy Transaction Data with Neural Nets

Recording: data.bythebay.io: Frank Taylor, Mining Noisy Transaction Data with Neural Nets

so yeah thank everyone for attending I wanted to get a since we have a small crowd I kind of geared this talk for a high level for data scientist who might be interested in doing this type of work or just to share what I've learned but also to maybe people in finance so do we have any people from finance may be interested in mining transaction data I don't know or is it all like pure data scientists in the room and so anyway the Golda presentation is really to just share a use case I started working at earnest about five months ago tired as a data science is there and one of my first task in darkness was really to start exploring our transaction data to see if we can begin to make sense of it and utilize it the purpose of my talk tonight is to share some of what I've done with this data from a very high level so I'll discuss the algorithms I used in minor detail and then discuss how we can use them in subsequent modeling further down the look down the road so I mentioned neural Nets really because that's what worked the best for this particular use case I when I first embarked on this it wasn't really you know we just had a lot of data it was noisy it was messy was there was no labels and we were just trying to make sense of it so we tried you know clustering using various methods you know using things like levenshtein distance and stuff to make sense of the text data but what ended up working was a rather simple architecture and I'll talk about that in more detail so my goal here is really just have a high-level discussion or a presentation on you know it could be for someone in finance who wants to leverage you know may take away some of the stuff that I've learned and take it to their data science team or for data scientists who maybe are doing similar type text analysis so you know I expect people here to have a little bit of knowledge about neural Nets noted neuron is a loss function things like that what if you don't feel free to ask or just email me or something afterward so my goals today I'll briefly talk about who we are earnest and what we're doing and how we're leveraging data science just briefly how can we take this transaction data what is the data itself and how can we use it to leverage enhance our products and both build new products that are going to you know help the company but also help our clients so so a big part of the talk is really just around taking the transaction data and then encoding it you putting it basically converting it from a format for text into numbers and then further modeling after that so then I'll talk a little bit about next steps that were some of the stuff we're working on currently that's not necessary the results aren't really ready for presentation but that we're working on so who are we at earnest we're building basically a modern bank for the next generation our mission is to democratize access to high-quality financial services and we believe that we can interact with customers to build a synergistic relationship that benefits them and their financial status as well as benefit the company in the first step in this process is really to give college grads this is just like a first iteration is to be go after people who don't have a lot of credit and that's typically young people it could potentially be somebody who's very old but generally speaking it's young people who have a college degree and they don't really have any established credit but we think that there's enough we can learn enough about them with modern data science to not just underwrite loans to them but actually give them competitive logs and we also have you know ideas around or products around you know interacting with them on a very regular basis so it's not just a static credit report you know give them a an interest rate and sign the dotted line and that's it we want to interact you want to offer new products we want to engage with them in a way that it improves their financial status so the transaction data that I was tasked to work with it's about I think currently it's about maybe a hundred million rows it's about a terabyte and a half of data roughly and the problem is is there's no labels around it it's noisy and we want to be able to leverage it we know that people are using these transactions they're buying stuff with it they're going to places and we're getting you know signal of that we know that they're taking out other loans there other financial products they're spending their money in different ways you know there's heterogeneity and how people behave we feel that we can capture a lot of that in this transaction data so the data just so you know is growing very very rapidly especially in the last month we're really seeing it take off so what is this data just briefly when a customer comes to Ernest and they they fill out an application they they are required to let you know when they undertake a loan and they actually sign on and become a client they have to link the required to link their checking account savings account if they have it credit card accounts what have you the vast majority of the data is checking accounts because where most of our client base as of now is younger people most of them have a checking maybe a savings and maybe a credit card some of them actually have investment accounts which is cool but most of them is just checking and savings so the data just it consists of a timestamp the amount of the transaction and text and the text is what is really what i call the noisy data it's very unstructured we don't you know we see text for example you see POS transaction in anaheim california or you might see like transfer to XYZ account you know and there's information often like financial institution I apologize guys not sure what I forgot to turn off my hot corner there okay so where was I so you know we get a lot of information there's many rows of data but the data as you can see can get messy so so we can see that like for example see we can see like for example that you know there's a point of sale purchase at Anaheim California and it has an ID blah blah one two three four five and we can see that a different user came along and for a different amount did something that looks like an ATM withdrawal and a NH ma or MCA probably is Anaheim California but you know it would take a human to read that and obviously it's the same store it has the same ID and a lot of transactions don't even have a store ID they have other information that really doesn't mean much of anything and then we have the exact same transaction from a different user probably has a different financial institution and we see that we see ATM withdrawal in anaheim california which is much clearer to read so that's really the crux of the problem that's the keys to makes how do we make sense of that and that's what I'll be talking about here so some of the things we'd like to do with the data i just want to mention this briefly so we know kind of what we're aiming toward making sense of the text is really just the first step in doing a lot of other things so you know we could look at free cash flow in the transaction data and make underwriting decisions and we could do we could look at history and if we see red flags like derogatory transactions such as payday loans overdraft fees things that might hint at financial distress we could actually have like more sophisticated models which is something we're working on currently like change of state model so state space modeling around you know if somebody had a financial hiccup in the past and maybe their credit report doesn't look ideal can we separate somebody who is just you know in essence a bad money manager somebody who really just as bad with their money we don't want to learn to them versus somebody who maybe just had a hiccup you know one lifetime event where they're generally good with their money but there was a hiccup something happened they got sick sorry I got a lot to that maybe something happened you know that they got sick or there was some hiccup in their life that caused them to have a financial problem maybe it was a divorce a death in the family something serious that caused this to happen but going forward we have no reason to believe that they're not going to be a good customer so those are the people we really want to try to capture and we believe that the transaction data I can give us some ground truth data to to to get closer to that so just as a first pass to see if we could encode the information we decided what we're going to do is basically vectorize the text in some way either using like bag of words or traditional NLP methods word tyvek and build an appropriate feature space with both the timestamp and the amount of the transaction and then feed that and if we can get labeled data which is something we had to do manually Li literally SAT and had a party and a competition and you win a prize if you label the most transactions we had a set of criteria we did some evaluation on the labels themselves to make sure that the labels were actually high quality and they were actually a very excellent quality we were happy about that and it was a lot of fun to get everybody together and actually have a labeling party that was fun and then so once we have labels we ended up with about 22,000 labels we tried out a handful of different I say we really was me doing all this we had a you know I tried a handful of different models and I I basically trained several versions of a feed-forward neural net basically a multi-layer perceptron is what it ended up effectively being gradient boosting and random forest as well as a logistic regression model which I think was a good baseline model so just briefly you know machine learning models need numbers and typically you know if you're if you're parsing like Shakespeare or parsing you know New York Times articles or Wikipedia you have a lot of text and the text is typically in in English or perhaps another language but it's it's it's written in such a way that it's it's in line with that language and it's it's a little bit you know there's a lot of methods that are in place to understand that what we found with this particular data set is that it's just really messy traditional animal PDF it's just you know for example bag of words where we take a vector and depending on the words in that particular row we you know we give it word counts to form a vector this didn't work very well correct they're our customers data nobody else's so when they linked the account it goes back pretty far when they link their account it goes we see historical data yeah I know it seems almost big brother but you know they're agreeing to do it to get to you know because we can offer them alone in a competitive rate where they wouldn't otherwise be able to do it and we also feel that you know some of the products we're building out right now are going to even extend that to more people so we feel like we're going to offer a product that's good for people and it's not just milking them I mean if you go to a poor neighborhood you see check cashing places everywhere and you know other financial products that are out there especially the personal loan products that offer personal loans at just extremely hideous rates and that's and often these people are good to pay the money but it doesn't show that on their credit report so you know there's an advantage to them too to give us their data yeah so one thing we actually have not been able to discern is like so we see across financial institutions like if you're a Wells Fargo customer and I'm a Wells Fargo customer and you know other people are well we see consistencies across wealth Wells Fargo's customers but we also see that the encoding is also dependent on where the transaction takes place and I'm not sure where that trade-off is because this text data that's in our financial transactions it comes from something called a payee name which comes from one source and the text is concatenated with other text that comes from another source so it's a combination of all of the above and it's messy yeah but what we do have is text and you know often we can make sense of it and thank you for the question so where was I here so in order for back of words to really work we need lots of highly informative words like I mention them you know we want something like Wikipedia or you know New York Times articles something where the the text is is clean and what we have here is the opposite of that we have you know very short sentences with often maybe just four or five words at most 10 usually and the words are often abbreviated or encoded you know concatenated with no spaces in between things like that and that ends up making you know a messy a messy arrangement but we do have the benefit that our data specific to banking transaction so like I mentioned a minute ago so we have a finite number of institutions and those institutions tend to have patterns across that institution so it means you might see a weird encoding for example transfer to but you'll see that repeated across customers and across institutions so because it exists and it happens enough and often enough we can make sense of it so it turns out that looking at co-occurrences actually works quite well you know so you can imagine like transfer to and transfer to and then some transaction account or some number they co-occur in a similar manner so you can see that the word to and transfer and to along with ex fer would co-occur and in a latent vector space we would have a closed vector they would be considered similar same thing with payment to versus payment from we could see that to and from would be highly informative in that in that regard and because they would co-occur with you know various forms of the word payment like pymt or other you know PMT whatever form those color currents is basically build vectors for each of those occurrences of the word that are very similar you can imagine you know using bag of words that simply would not be a case you know PMT would be treated differently pymt and the actual proper spelling of the word payment all of those would be treated differently with the word vectorization model it works actually quite well and so that i will speak a little more so a lot of you who are familiar with NLP are probably already familiar with word vectorization zat least you've heard of them if not used them I'm going to talk about both words avec which is actually a class of algorithms several different algorithms and the glove algorithm which is one that I've really come to like it's practically the same thing it just scales a lot better you're doing with pretty large data sets or distributed systems so so the words of act like I mentioned is actually a class of several different algorithms it predicts the word giving given it predicts a word given what words are around it or vice versa it can predict the words around the word given the word if that makes any sense Martin mola famous comedian once said writing about music is like dancing about architecture so we're going to vectorize about words it's a it's an art so this is a predictive algorithm like I said it takes an input and the input will be either a word and it will predict the words around it and that's called the Skip cram method or it will take the the context it will take the words that are around a given word and try to predict the word and both achieve pretty much the same goal of vectorizing the word explain that in just a moment so I pulled this off the internet I don't know if this is the best representation of how we're tyvek works but basically each one of these inputs like so in the Sebo that's continuous bag-of-words model it takes the words that are around a given word it vector eyeses them with a one hot encoding so each one of those words is a one hot encoded vector a huge vector like of the size of the vocabulary and it multiplies those by weight matrix which is the size of the vocabulary and the size of your latent dimension space so each one of those basically gets a row in a matrix that's in the middle layer so you can think of that as an n-dimensional neuron or neural that hidden layer and then the output is an n-dimensional word vector in a very similar process happens on the other side on skip Graham you know in this case it's just a softmax there's a loss function you do you know you basically maximize the conditional probability to learn the word vectors and in the other case it would be a multinomial distribution that you're working on there so so you can learn the posterior distribution of the words based on the softmax of the output later so I'm going to show this really quick for those of you that haven't seen this before just I think this is really cool and i'm using capitals along with their countries to show how word to vet can capture semantic associations and I'll talk briefly about how that kind of what that means for transaction data so you can see that there's you know countries here and capitals there and you could take Spain for example subtract Madrid add Lisbon and you get a vector it's pretty close to Portugal similarly you could take Japan subtract Tokyo and add Ankara and you get something that's pretty close to Turkey it's kind of interesting that Japan and Turkey are both altaica languages I'm not sure that might be a coincidence it might not be it's certainly not a coincidence that China Russia Japan are all linked together because you know they co-occur with words like East maybe Asia you know Italy and Greece Spain are going to co-occur with words like Mediterranean southern Europe things like that so those co-occurrences actually encode these words in this vector space in a way that is both semantically meaningful and well semantically meaningful for now so you could imagine similarly to like you know words like transfer to and payment to get also similar associations and we found that to be the case we found that like even weird encodings of like payday loan sources where a lot of those places you've never heard of it actually was able to capture them in a vector space and we had a cluster of those which was very interesting I want to briefly talk about glove for those of you that are not familiar with it it's very similar Tour de beckons I think it's practically the same thing if they're both implemented in the simplest sense and what it does is it takes basically every word and calculates a PMI value for you know co-occurrences of those words so and then what you do ultimately is you you factor that matrix into a lower dimensional space just like you would for you know non-negative matrix factorization or some other method and what this does though it actually it's because it's just counting words and the way the algorithm is is framed it's really simple to implement this is a spark it's literally like you know chunk of code maybe you know 20 lines of code and your you can do this on huge amounts of data and get really good results because I ran my word tyvek in spark I didn't use any negative some subs amp or negative sampling or subsampling I just did a straight brute-force approach to where davec I got nearly identical results in fact almost exact same results using this as i did with word to that using the skip Graham method of word today so this is just a brief summary of how we vectorized a document first we strip out all the number and character sequences that could represent account numbers anything that's private information we did some heuristics to try to get names out of there it's not a hundred percent accurate but it works pretty well we lower case we tokenize and once we have a tokenized version we can spit this into a word tyvek model so now we can celebrate we're halfway there we have a vectorized our entire corpus now we have all of our texts we it's in a vector space assuming the vector space has been done like we've trained the model properly we can probably use this for something more interesting so what we ended up deciding to do because most of our transaction data is actually retail the vast majority of our young clients are you know they're young most of their stuff is happening in their checking account and their you know their swipe in their card everywhere they go so that's almost fifty percent of the data it's like 40-something percent of the data its retail transaction so the first iteration was let's just separate that out let's build a classifier that says this is a retail transaction or this is something else so we first took a pass we labeled 22 transaction 22,000 transactions and we literally like i said earlier had a competition to do that and I set up a range of models as I mentioned earlier logistic regression for a baseline I feel like it's interprète belitz easy to implement that and then some boosting and ensemble methods I'm sure you all most people here are probably familiar with these and then the multi-layer perceptron which really was just a simple feed-forward one hidden layer or two I tried it with two ultimately it works better with one hidden layer and an output layer a softmax output layer or in this case logistic help earlier so just a little bit about some of the tools i use i did this in apache spark mainly because you know this data is probably small enough i could do it on a single machine but what i did was i built it out to on spark so that it scales easily because the data is just getting bigger by the day so i expect probably in another two or three months time this is something we really truthfully need at least a you know at least a three machine cluster for so ultimately once we have the word vectors we can use the amount of the transaction we can use the financial institution ID we can use timestamp ID for example did the transaction occur in the morning evening or did it occur by hour or by day all of those things you can use what I ultimately use was just the amount for the first iteration the amount and the actual vectors themselves and just so you know because there's multiple words in a row of data I just take the centroid so each word gets a vector and I just take the average of us effectively the arithmetic you know the mid the think of it is the centroid in the vector space I did a five-fold probably overkill but i did a five-fold grid search using cross-validation and then I also tuned parameters both for the models and but also on the words of X model as well and the the key thing to tune on the word tific model is a number of latent dimensions so a couple these are just some further notes that I made corbis has to be large for this really to work well you need a lot of transactions luckily for us that wasn't really an issue we had a lot of transactions I didn't do any subsampling for those of you that are familiar with word to that gets really common to subsample frequent words this is just you know you want to you want to down late words like to and from and you know things like that I found that subsampling reduced performance and I think that's you know words like to and from were highly informative there were a lot of other you know what would be considered stop words and most NLP projects were actually informative words in our project at least reasonably informative so ended up not even doing any subsampling so then we just take the centroids and then further features so like the next iteration and one thing I've done one iteration of this but haven't really tuned it whereas you actually take the time and look at the temporal aspect so because like I said one of my ultimate goal is to build state space modeling around this and talk briefly just about for those of you who are familiar with neural nets and I don't know if you've done anybody and anybody hears done like distributed neural Nets but distributing a neural net causes a lot of headaches you really only want to do it when you need to you know there's several issues related to parallelizing oh so basically just you know there's different approaches to parallelizing you can take you can copy all your nodes each machine and you can basically learn subsets and then average your weights afterward or another option is to actually send partitions of your nodes so send some of the notes to one machine some of the nose to another machine we did the former basically just did a copy of note on each machine and then you know once you've done that you can average your weights afterward there's issues around that you definitely you take a hit and performance when you start distributing the neural net and you know some of that has to do with the loss function you know just the you can imagine the topology of your loss function is going to vary as you start taking subsamples and sometimes merging those and updating those at the right time is is a tricky matter it doesn't always work how you think it will I mean I've seen you know I've done two iterations with the exact same settings and have one come out really nicely and the other one just spit out garbage results so yeah so just really this is the little chart of our flow here so you just tokenize you know we vectorized the words train a classifier evaluate then tune and when we tune we both tune the word vectors and the classifier then classify again and then evaluate right now we're developing labels so this is ultimately you know what I'm presenting here is a binary classifier but ultimately this is actually like an 18 there's 18 classes that were classifying but initial results for an hour just on the binary classifier so logistic regression got us eighty-six percent the boosted trees got a 88 as well as random forest got us the same result the multi-layer perceptron with the single hidden layer got his 94 so that actually surprised me I think with a good feature space I would expect boosting and a single hidden layer neural net to give comparable results so that kind of surprised me but I think it hints that there's still optimization on the feature side on the word divx side that we could do so we can probably improve that a little bit is my guess some of the further encoding this is for some of the other modeling that I've been thinking about and some of the stuff we're building out so one of the things we're working on now is it's not an lstm it's a simple hidden Markov model that takes in this data you know it looks for changes of state and so right now we just have two states one is like building wild horses depleting wealth so we want to you know separate the people who maybe they're good with their money and then they stop getting good with their money and then they go back to being good with their money we want to be able to detect that so that's one of the things we're working on and trying to speed this up so I think we're running out of time just yeah here's some resources that i use i mostly spent this in you know i got a lot of help actually from adam gibson at deep learning and sky mind who's behind deep learning for j and i'm not ask allah programmer I mean a java program where I've been programming in Scala my background is more in C++ and Python so all of that was new to me but I did find that their libraries and easy to use even if you're not a Java guy or a Java person sorry oh yeah spark net has a great utility I didn't use it for this but I did play around with it i found that it was it's got potential I thought it was useful especially for distributed computing were ultimately everything I did here was on a distributed system so yeah that's it that's the talk so anybody has questions that's our victim hello did you betray me yeah but she never planted do you think there's any train back into the that is the same time so to answer that yes you can treat the the word tyvek is so let me go back I have a one of these here let's look at this one so imagine right here let's look at the Skip grand model you have an input which in this case a word outputs ultimately that's you know our first layer is really multiple words it's a sentence so you can have multiple inputs as opposed to one so it's not it's not this it's that with a few more input it's something we have thought about and I've crudely implemented but ultimately it's just a neural net right it's a you so you can imagine that you can learn this those outputs feed into another layer and perhaps into another layer rate so that you could do that with a simple one you know one pass back propagation I you know I'm not really successfully done it yet just played with the idea I've coated it up a little bit but haven't actually implemented that but I do believe that's possible yeah I mean ultimately this this model and also some of the subsequent models just to touch on what you're talking about is I see it as one model with multiple layers that's one thing I really like about the deep learning architecture and the concept of deep learning is that I'll turn that back on in a minute you know one thing I like about deep learning the concept is you can really incorporate many models into one depending on how you frame it if you do it properly I think you can do exactly what you're saying not to mention you can do you know you can imagine each one of those is a you know you could stack those on top of each other and have something like an LST em right where you have it's basically a recurrent net as opposed to just you know one back propagation one input to output the output ultimately feedback into another set of inputs which is just a copy of the first one right so yeah two marks our eyes we closed between multiple team uses a machine or GP as possible so two GPUs I think it's too i have to look at our i don't remember because it's on our it's our spark cluster and its i think the instance that there's only two versions you can get one has i think a tour for the other one has to we have build a smaller one I think it's to a smaller person that the halogens a dispute between two 2d views on the same machine not really because I didn't really even use the GPUs i mean i just i mean i did so i used them i use them only just sort of experimentally but overall like to get the final result I'd ended up not use it up just because there were some complications around that and a I'm sorry what was that well so to that end actually did excellence it captured because of the word co-occurrences it captures really well synonyms it does it very well actually that's one of the probably biggest strengths of what I've seen word to Vic doing is you know people can say two things two different ways and use different words but if they mean the same thing you know if you have enough data to train your model with those co-occurrences which will tell the model that the in in that latent vector space that those those words mean the same thing so it actually works very well for that and that's what I saw what I was most impressed with is like for example when I looked at payday loans you know all the closest words in the vector space everybody so I was just saying that actually it's the opposite that it works very well you know for example when I looked at payday loans and if you look at all the the closest co-occurrences or the closest I'm sorry words in that vector space that they were basically a lot of them were weird and coded words when you looked at them but when you actually looked at where they occurred the transactions were payday loan transactions so it worked very well I mean most of the time it's not perfect but it from a modeling perspective it was very well deep learning for J no it's not distributed per se but it's a it's a simple matter to do this distribution yeah so you can imagine you can pass a model to one node pass a model to another node and then spark makes that really easy and deep learning for J just so you know has some spark examples on their github repo so if you just copy those and use that as sort of a baseline or a template it's really easy to distribute it now just be aware it's easy to code it up and make it run when you start distributing your neuron that it's hard to get to work rate like there's a lot of considerations so don't be surprised if your model just you know takes a big yeah anyway you understand anyone else I was going to answer you though in terms of parallelizing the GPUs my understanding is that you would treat them just each like a node right I think I think it depends how the graphic cards are paired together like a lot of gaming environments you can treat that like one graphics card I think with cuda but i'm not sure in the java deep learning for j environment I'm not that's why I don't we weren't able to really do that had I done this in Fianna or something I might have there might be a solution to that yeah lots of stuff like that we could go down that rabbit hole but yeah after work so we just that along with the amount of the transaction and 10 some temporal information well in this particular case that was just the amount to the transaction so is yeah it was all the vector it was all the weights for the word vector or for the row vector the centroid vector and then the amount of the transaction normalized in some way scaled in on Eliza yeah that's all and there may be simpler ways to do that or better ways but that's you know a good first pass simple just take the centroid here dishes case yes yes exactly so like I did use a I'd playing with a I didn't see any so to answer your question I did do that I didn't see any improvement but you know I didn't it's one of those things when you're at a company sometimes you iterate on something and you don't see an immediate result you just pass it that was the case I suspect just probably more we could do around that but one of the dirty things that I did do is I just took every time I saw one of those sequences of numbers I just did like I had a regex expression and instead of just stripping it out you just replace it with a tow cam yeah and the token could be like you know account number or something right i think i just called it acct or something but I you know you don't really know it's a lot of it most of the time it's is it's exactly right we don't know the point is it's just a token that represents you know whatever that might be yeah and I just let that be anything at one i forget what i called it but it was i put in a placeholder there and this is something i did like four months ago so i don't remember exactly what i called it but I put a dummy token in there I didn't see any improvement in the model in terms of the final output but it makes sense to do that certainly I think you know in many cases those tokens represent an account number in many cases they're a store ID in many cases they're in some cases are a phone number and a few rare cases are like Social Security numbers which don't ask me how but I mean people you know in fact one of the I think this is one out of like you know I found one I didn't look at every single transaction 80 million or whatever I was looking at it I found one that literally said alimony payment to the person's first middle and last name their address their phone number i mean just had everything and I was just like waffle somebody just copy and pasted like a whole bunch of text in there that didn't belong there but that was like one out of a you know however 80 million that I saw yeah yeah I hope it was helpful I