data.bythebay.io: David Talby, Hunting Criminals with Hybrid Analytics
Recording: data.bythebay.io: David Talby, Hunting Criminals with Hybrid Analytics
so thank you hello everyone my name is David talby I work at atigo we a software company based out of bellev Washington and we are a fully integrated Big Data stack meaning that we we build complete Solutions so Solutions include the Big Data infrastructure the data science and application layer and uh what it means that it works particularly well for problems we actually need models that uh that learning production updating production right because we we take responsibility for the actual uh business outcomes of the production system and this is one of the problems that shows this very well so the the problem we're going to talk about today is the the challenge of finding Rog traders in a financial services uh scenario this is a an example of a project we did with the with the top five US Bank and uh the challenge is to find case of internal compliance issues which is a big problem it's a big problem because right now the way we're fighting it is really with regulation over regulation which makes it very hard to tell between so you know real real bad actors and people uh you know who just need access to banking that we we are not giving to them it's a big problem for the banks uh they are they are being hit with internal internal issues which they are having trouble finding about in advance it's a big problem ER financially so the top UK Banks lost about a fourth of their revenue not not profit Revenue in the past five years to fins and obviously it's a in terms of just the Public Image it's it's an issue as well so so there's a real interest from the banks themselves to solve this problem both to make sure they're actually doing it right uh as well as as the financial incentives that goes with it so when when you're trying to look into this kind of problem what we did is H we had we had access which you can do if you're a Trader you you know you give out a lot you give away a lot of access in terms of your your communication your trades so people can see what you're doing in terms of you're trying to do money laundering a bribery political contributions breaking Chinese walls there are many ER ways that where you need to open up your communication to allow for that so we can get a stream of Communications email IMS and such we can get transactions and then the goal is to find out if someone is doing something funny and there are three interesting challenges when you do that the first problem is that you know this is a number of uh true negatives that we have meaning that when we get the stream of of messages of transactions about one in a million is what human experts would consider see there not even a true positive but even a suspect right something that is relevant to look at okay that's one in a million this means that just about any supervised machine learning algorithm you would try will tell you immediately no matter how you how you try to game it that all messages are good and you know you're getting those you know five nights of of precision and accuracy and there's just no way there's just no way to go and beat that number so really you need to take a different approach you need to start ranking some of your suspects and looking into them that's problem number one so it really is a need in a hch problem problem number two is even if somehow you were able to get Beyond this and and and get some reasonable training data of real bad examples the next problem is that you have no training data because there are only several hundreds or several thousands bad examples that happen in one year and it takes a very long time to invest investigate and actually validate one training example right so you know if you've been following cases of of financial fraud in the news it takes usually large teams of forensic accountants and lawyers several months to even agree if there was a crime and if there was a crime really what was the the actual regulation that was violated you know in in a long stream of of you know concocted scheme that that someone brought up meaning that you you cannot you cannot go in kind say okay just give me give me a definition of you know what's FR or give me enough examples and I train to that it needs to be much more of an online version where it's a semi-supervised problem you get some hints and you get more examples over time and you need to train the system while you in production the third problem that you have was you know even if you were able to H somehow get over the previous two problems uh this an ad this is what's called an adversarial learning problem okay an adversar learning problem is a machine learning problem where you know once you give me training data I train on it but once I start making guesses you actually change your behavior so that my my results uh decrease okay H and and specifically in financial services you know if we were all Traders every time someone would get caught right we would all you know go into a room like this and you know and going to compliance training and people come says look you know here's what this person did they did one two three four here's how we C them and you know if you're a you know you're sitting there in the audience you take notes and and obviously you're not going to try the same thing so what you see is that almost every new type of scheme that's been uncovered is new okay which Al makes it hard both for you know both for human and and machine uh machines to to try to understand what's going on so we need a different kind of approach here so just to summarize we we do have a need ltic problem and the approach we'll take is what we call hybrid analytics which really means that you you can't you know no one trick works and and we see how we take different signals from different kinds of approaches to make that work there is no training data okay so we have to take a semi-supervised learning approach okay meaning that we we start uh with the low Baseline we go into production we start getting hints and we learn while we are in production okay and because it's a it's this is really a textbook example of of an adversarial learning problem ER similar you have to have an online feedback system right so so another important thing with those kinds of problems your offline metrics your Classics or your Precision recall on your offline training that almost don't matter right because you know that it's not going to be what you fac in production really what matters is why in production what's your rate of success and can you keep it over time so when we talk about hybrid analytics really the question is okay you know we we're trying to to catch this Ro Trader who's planning something what are we what are we really looking at okay and what we need to look at are the the different kind of signals that we are trying to catch right so so you know when someone start thinking about the fact that you know they've been in back for five years they had everything and it's time to plan the you know x million dollars and get away ER things change right so the first thing is they do begin to think and talk differently right they become disgruntled they think that somebody owes them something right they think that you know people are against them they think that you know the world is extremely unfair against them right and you can see it in in how they talk in who they talk to right in how they behave H when they start doing something you definitely see that they change who they work with so sometimes they start working with people that are completely outside the Departments right to try to to coordinate different kinds of scheme sometimes it's the reverse they actually sent their business with only one or two people but you see those differences you see those changes obviously timing of events is important what you want is you want to be able to have some kind of routine analysis and then you could look for simple things like you know you maybe I'm starting to come later in the morning right you can see that my emails and transaction are starting later in the day or or it may be the case that I'm actually I'm starting to transactions usually it's not usually the Sunday 400 a.m. thing usually what happens is the end of quarter or end of month when there's a large stream of you know payroll or tax or other transactions and you can so hide hide within the crowd better obviously people start actually breaking rules but rules in the financial services uh space are are not easy to Define first of all you do have tens of thousands of of pages of of just regulation regulation which are very hard to encode but on top of it I don't know if you've ever been ever been involved in a you know in a tax tax audit or any any sort of Financial audit there's the regulation and then there's the so the standard that the Auditors expect right which is much more of a fuzzy thing that you need to be aware of and obviously what you're looking for is a combination of all these things and one thing to not I me if you're building those systems is you're not looking for specific signals or specific you know U mid midnight email or or or an email to to a person that I don't usually work with you're looking for a change in Behavior okay and what what you what you're trying to see is you're trying to catch people in Decline before they actually do something that's that's regrettable analytically each of those types of algorithms means that you need to look at the different set of algorithms right so you definitely need strong natural language processing you need to be able to understand emails text in depth and we show you some examples of what we have to deal with you need to understand graph analysis and Link analysis uh one thing I can tell you as an example we we have a we've done similar projects in healthcare in healthare I can tell you if you have one Doctor Who You identified as a as a bad as a bad actor then the doctors who who refer patients to that specific person are seven times more likely to also be B actors okay so you know here's a here's a free feature that was one one of the strongest features we we have found H you need to be able to understand time serious analysis very well to look at routines look at your daily Cycles weekly cycles and deviations from them obviously you need to be able to understand rules and most importantly when you're building your models what you must not do is just take those types of features and combine them into one classifier that fails completely in general in machine learning when you take those completely different types of features and combining into one classifier what's going to happen is that some features are going to completely overtake and dominate over others just because it depends on the classifier so you know some classifiers natural language processing you have you know feature pair diagram you're just going to have too many features other classifi is going to have maybe the time series features will dominate so what you need to do is build separate models and separate classifiers for the different types of signals and then build emble models on top of those and that's how you go and U improve and actually have those features impact what you do any questions so far okay so what we did here is want to have something that you you can play with uh so we we build an example that's based completely on open source tools so it's it's python based the jupyter notebooks are available on T's GitHub and you know they've been there for a while many people have tried and you know hecked away with them and and actually sent us some some very interesting things and the goal there is is just to show a reference architecture about how you go and build those systems so it's all open source and there's a you know AWS Market jump start if you just want a one click thing in terms of how how the notebooks work H there are two levels of data processing in such an architecture the first one is what you do is a stream of emails and transaction that comes in and what you need to do is you know in this case we we use Kafka and then we use spark streaming to go and Pull and process the data as it comes usually you know we're dealing here with several terabytes per day so this actually is a big data example you know most data science example are actually not big data example this one absolutely is um and what you do is is you just do the basic per transaction features right so so if it's an email and I can move over to notebooks here and show some of the examples in the Neo not notebook but if it's if it's an email you can look here uh let's see this is on the transactions this is on the email so what we do is we we we tokenize the content we tokenize the subject we have two types of sentiment analysis here so passive aggressive and dominant submissive uh we do some uh topic modeling right because you want to understand the topics and running topic modeling regularly is very important because there's a lot of jargon in in the text here H there's features you want to run on the transactions as well and so here we process them we look you know whether transactions are you know consider large or not whether consider an usual time of day or time of week transactions and and basically what you do here let's go back to here you're just doing the basic per transaction processing okay and then you can then you go you aggregate them into users there are some examples here H some of the things we look at are things like you know when do you send emails who do you send them to which domains you send them to how how you sentiment changes on the language font the language for the one thing I would say is you really need a strong natural language pipeline here and you need one that's custom to this okay so if you you're dealing with with emails you need to look at things like you know replies forwards you need to look into attachments we have things like especially when people are trying to hide something they will use a foreign language right so you know Latin alphabet by speaking a foreign language they would attach an image and in handwriting WR a note if they use if you use I am or text messages obviously the grammar is not a complete you know nobody uses full grammar when they do that but on top of it there's a lot of jargon a lot of code words that have been used that you need to learn other than that you need to be able to understand negation you need to to be able to understand compound sentences you need to be able to understand who we are talking about right so it's very important you know which person I'm talking about which project I'm talking about what action am I asking someone to do okay so if I'm you know texting my wife because someone told me which they shouldn't have that you know an m&a is is is going on H that you know we need to be able to track that um and and there are so so that's that's some some of those examples you'll see in the code here right and and some of what you know whatever is open source we we added um on the graph analysis space I'm not going to talk a lot about but one thing to note is also when you're working know those kinds of needle in HC problem you need basically different graph analysis problems than than probably what the the usual ones you know we all know in love so this just an example if you use let's say you have you have this kind kind of visualization this is just dimensionality reduction for two dimensions and I'm trying to find those two green clusters which are two two interesting cluster events if you use K means on this right it's just going to to differentiate between the left side and the right side right so K means tell you here's the center of one cluster is the center of another cluster right because there's so much background noise right if you look at the SC look most of it yeah just there just points all over the place okay and this is how most of these algorithms work there are other algorithms okay that would do better okay for example one one one one that we've used here is the bregman Bible clustering that will basically tell you here's one cluster here's a second cluster everything else is noise okay and and one of the things we've had to do is actually go back to research and implementation of other algorithms you should definitely want which is something you definitely want to do the next thing that you need here is you need a second layer of processing H that is not at the transaction layer but is is at the user layer right because you know there's never one email that's going to you know to to read out the whole scheme nobody is that stupid H it's going to be a a long stream of of things that tied together between many people in many events so what we do here there's a a separate spark process that runs either every second every few seconds or every few minutes depending on how much data you have that basically retrains in production and builds a new per user model okay and what we do here we build features that are per user right so so if the message level my NLP features are what's the sentiment of a message at the user level it's what's the aage sentiment of this user right what what is the change in the sentiment of this user from the past day or the past week okay so that's a that's a different level of aggregation similar for graph now I'm looking at you know what are the communication partner of this user H in terms of you know are they changing the number of people they're communicating with their graph size their click size and so on as just as a tip it's more important to look at changes than to look at the absolute numbers so what what I do here is I uh build those user level features per user I apply the feedback right so I do have human agents who are looking at my leads and say yes this is relevant this is not relevant and I rebuild those classifiers okay and you know with enough data you can really do this every few seconds and then with the feedback from the agents right and with retraining both things like you know your topic models but also just reing the graphs you actually have a system that learns here while in production um and what you have here you have the notebooks that show show this training process where we apply the feedback uh we use uh here what we we we actually package U package the python Scapes within U Spar SQL udfs so we can actually do the both the feature engineering and the training in the distributed fashion and you have the notebooks that go and do that um and let's see so just just to summarize what we have here you have the notebooks you have separate notebooks H one for the overall just go one more time one for the overall demo and then there's separate notebooks separate notbook to look at specifically the graph features to look at the time series features so we we do gener generate new Arma models we we we look at different types of outliers I actually we build there are features that show you how you build people's routines and then there there are some email features here mostly focused on topic modeling and sentiment analysis and then the main demo notebook it shows you how you going combine these what you need to do if you're interested is just you know just go on download and play with them H obviously some of the more advanced features we have are not there but what you have here is all open and and I do think you know having done this for several years that this is the the correct reference architecture for thinking through these types of problems where you have you need to have one stream you look at each message or transaction but you do need to have that second layer that is bch and scheduled we go and retrain model that really is per user or in general per actor that that you're looking to go after what this enables you to do is first of all actually actually get some traction and we have a system that has you know human you human investigators who go and look at the rank results and and work them and we be a to get some very sort of like you know best ever results from doing that the other very very cool thing that happens here is you have a system that actually learns okay and and what happens is that you have a system that in production changes its models H due to changing behavior of the actors due to feedback from the investigators uh which is very nice to see because what happens after a few weeks of a few months you actually go back to the system just to learn what's going on right it's usually the reverse because usually today we still r on data scientists to go and update models and do feature engineering here we we actually got to a point where we do the reverse to understand what's happening thank you for Cle questions yes uh so on the on the topic of the rareness of the event you're searching for yep um I actually work on fishing detection it's a similar problem it's about one in 10,000 so not one million but we found that waiting the the examples works very well does it not work when you get to the one in a million um yes so so yes so one one one easy very easy cheat is to come and say yes just take the few examples you have and either weigh weigh them higher or or duplicate them right with some variant in your training data ER we did this at the beginning just to get some results H the problem is that that we had here is that the scheme change too quickly right so so yeah you know with fishing the good thing is yeah once you found the scheme it you know probably many people will try it right and and with fishing or you know lot a lot of time you know with ax evasion those kind of things you have very few sophisticated vors right but then the majority of people just try to do same things right here we we just don't have the luxury of the longtail yeah but that that definitely works yes yes can you talk a more T Ah that's that's an excellent question so the question was about how we test the classifiers and the real challenge here as I was saying is that you you your usual offline metric are are I mean they they're basically meaningless right because to a point you know if I just you know if I just make very small changes to my training data I could you know double or triple you know my my recall that's one one thing the other problem is that you know really once you put this in production right then by definition people are going to react within a few days or a few weeks right so so so really what you're looking for is uh the team of human investigators working on this looking at their feedback okay so at the beginning we're looking to see how many leads we actually following up on right because when you do this the so especially when you start with the NLP things the first thing that comes up is the monthly compliance email right this email goes up to everyone come say here's who we caught last month and here should be careful right because it hits all the keywords right and then one or two people come say this is not relevant right and then the system retrains and it goes away right so but one good thing is is you just have the feedb from the people either you ask them or just you see what they're working on to see are they actually getting good leads right are they actually working the top ones H and the other thing you can you can really only see what's happening in production in terms of you know over a few months are really you know catching more or less H which is also how to measure uh because it's not um you know very often it's like you know if I start stealing money it can be finished with you know a talk with my manager or talk with HR right and off it's you know there's a long time before ACC comes in right and very often that even the bank is unwilling to tell you as there was actually something there because they they' rather keep it lower but a lot of it is is your online feedback right the off what we have learned is the offline metrics are almost useless have you been able yeah so so I think that that's similar to example here yeah when when you want to bootstrap you you definitely want to do that yes yeah questions okay than thank you