scala.bythebay.io: Greg Makowski, Telco Fraud Detection in Kamanja
Recording: scala.bythebay.io: Greg Makowski, Telco Fraud Detection in Kamanja
I you oh thank you very much so I'll tell you about one of the use cases using our open source product come on Joe I'm from lega data and so we've combined use a product that's written in Scala so that's the relationship to the conference point and okay fine so c'mon Joe just to put kamata in context with other open-source products that you've been hearing about today we would be a little bit like flume or other systems we would have Kafka or mq streaming data into us we would be a major compute engine referencing state from HBase or Cassandra or various other places and then streaming back out to Kafka or other things we have security integration with kerberos we have once and only once transaction processing so barclays bank could use us for processing your your paycheck depositing it once and only once no more no less we have a number of financial services large clients top five in the world and top five in the US and that's a significant requirement that they have of us so we use internal clouds at the bank sites or external clouds as needed we have different adapters for either the streaming and also for the the data stores so some clients have asked us to add various other things like influx DB so i'll be talking about one of our consulting projects in a problem-solution result format it's mid project we've got half a dozen people working on this right now or have been for first couple of months and we've got a number of months to go so it's talking about two main pain points fraud and kpi reporting so this is covering many South American countries for this one telco company they want to modernize our data infrastructure they're currently using the Oracle Big Data appliance and it's gotten just completely unusable they've got various kpi's key performance indicators that they're using every day and they need to run their business to make decisions during the day so calls blocked calls drop by tower various things like that and it's taking oracle26 hours to compute some of these things and they need it in two hours or four hours so we were coming in and providing the Big Data consulting to to replace that with an open-source stack again since we have a lot of experience working with banks and other large companies are part of our consulting is to provide the the insurance policy they could download these open source products but to get them working together to get into work together in a reliable system is something they're less confident about hello click ok so one of the problems is robo color fraud so we've got our evil robot in the top right there and what he's doing is he's calling out thousands of people that our customers at this telco and other customers as well then but they call and they hang up so I call you and maybe you're one of the curious people what I missed a call who's that from I don't recognize this number well in the South America they're constantly buying these prepaid phones the identity is constantly changing so are the numbers so like in the u.s. you'll recognize somebody's number but here it's constantly changing soon as your minutes run out so you call back and you say hello you know wondering if it's his girlfriend he missed and then he gets a charge as if it was a 900 number and he has no idea he was calling a 900 number back and so so he's getting billed that um and the angry few angry customers that hit that look at their bill you look at it said oh my god get out of here why am I getting charged for this call so they call the telco the telco wanting to maintain the relationship gives a refund hence they're paying for the fraud or else at risk of losing the customers so this is why it's a pain point for the telcos so they've got tons of these robo calls and so the fraud investigation finds that there's a prefix related so there will be outbound calls but it'll change from day to day so like in the u.s. you know if you have some phone number the 123 would be the the prefix and that would represent a part of the telco network switch and so all the calls will be coming out from one but they can't filter them all well they can't say I'm not going to accept any of those calls because you're my good customers I don't want to exclude you too throughout the good with a bad so that's part of the analytics that's going on is to find out how to distinguish the good from the bad by the by the behavior of the phone calls and and what's going on so now that's on the the fraud problem there's also just a problem of data complexity so why is this bottom row showing up yeah okay so the call data record is from the erickson stack so erickson would be the hardware Network telco provider they're also populating all the database fields and this isn't something that's easily changed it's just part of the vertical application so they have 40 to 50 million people with his call data record structure and a few million people with isp records in that structure so the source systems generate a lot of small binary UNIX files and that's not going to be changed it just has to be dealt with so the Oracle schema is 72 800 700 800 tables our data sources are about 20 different data sources and you multiply that by say ten to fifteen fifty fields per source so just multiplying say 20 x 30 something in the middle that's 600 now multiplying that times say 300 fields 500 fields per field you know we're getting up to 200,000 so we have to scale not just in data volume of 40 to 50 million people but in complexity of all the different fields coming in and so in order to be scalable part of what we were doing in this project is not just be scalable and volume but also in the data complexity so we were developing a DSL that's called jtm and i'll be talking about that in the solution so first of all go through smaller parts of the solution then i'll build up to the larger architecture so the design goal is using open-source ecosystem including hamana provide a big data platform for the bi use cases and be scalable in complexity so that's definitely a strong reoccurring theme so there's a lot of data cleanup that's needed which sounded surprising to me when i first heard about this project but all the phone numbers coming in from all the different countries all the different data sources oh it's very old systems a lot of legacy systems needs a lot of the cleanup and transformation and enrichment and so to get out of manually coding in Scala the data cleanup for 200,000 fields then that's where we are getting to doing cogeneration with the dsl so the dsl is at a higher level that could support a vector of about you know 800 similar fields and do the same thing for all of these you know for this set of fields do the same thing for all of these fields and we had to be robust to changes in the schema for new releases from the ericsson stack you know we can't be hard-coded so be flexible or die so if you're familiar with OLAP reports be dimensional tables and so rather than updating a relational table and then re aggregating it to all that cube 1 2 3 and 4 as a given record is coming in that record would update all the different OLAP cubes directly at the different levels of aggregation real time so it's getting clean up enrichment joining with any third party data it needs or look-up tables and then directly populating the olap so that way you can just feed any front end systems real time to the reporting bi systems so if you have at the end of a minute you know revenue for minute 42 and then update all the KPIs over the dimension so you know you have end of every second and if every minute and if every hour update as you cross those boundaries you know so call and hang up rate at two p.m. and then update all the KPIs for that so there's a lot of logistics but if we can do it all real-time then they get all of their KPIs immediately in real time and they decided to stick with the existing Oracle dashboard tool they had already paid for it it was working and it wasn't a scalability problem and it was something we didn't have to rebuild so we could focus more on our value add so we were fine with that too here's one example integration so this is a dative is integrated with the Google Maps API there's a slider at the top here what the refresh rate is for updating the screen from every half second to 20 seconds we can update all of the values the dots are cell phone towers the radius is not coverage although we could display that it would be so it could be something anything from cell phone block drops or blocks or whatever attributes you have in your OLAP cube that you want to report and then you can just play that for the geography zoom in and out to whatever geography want and you can take a look at all the things so the compute clusters going to lambda architecture so we'll have some compute real-time clusters and some storage and batch clusters have a data lake that can manage that for the analysis after the fact so for the and these would all be AWS notes so the command you would be running in the top in the compute cluster for the real-time analysis so as I said we're continuously collecting the data the source systems aren't going to be changed they're generating a lot of small binary files that we're cleaning up from the jtm is adjacent transformation management so that's the acronym for the dsl so you specify the jtm and then it compiles to scala or Java code so you can have a shorter specification here and then it generates a much longer amount of code in Scala so yeah and that was something we just needed to do to keep our sanity for the problem so this is a new feature for us and this project that we're developing on that but we're going to add it back into our code base in kamagra and so I'm not working on this team I'm managing a couple other projects but I happen to be in the bay area so I could present at the conference I'm not in South America so I said why so what why use this jtm why not code everything in Scala and they were just describing how many person hours it was so much easier just to code in the dsl and then generate the Scala code was a hundred times longer than the specification code so any time you can use cogeneration like that it's a big deal I'm glad you last five dollars check in the mail it's so nice to have an audience paying attention thank you so I'm not an expert on this our CTO has been generating this so I've got two screens of this part of the same section so we have a header imports code and aliases advantages of this as we can embed java code Scala code and also libraries so if there's libraries you want to bring in to do the transformations in Scala or Java then that can get embedded directly into here without any problem or extra effort because it's just a specification for cogeneration so it can carry long anything in here so then for some of the transformations and they're looking at what some of the outputs could be so I won't spend along a lot of time on this and let's see Oh some of our solution architecture so here c'mon Jay is so we're working on the real time engine where Kafka would be feeding us with a data integration we can be working with the MapReduce data lake a spark for computation we sit on top of zookeeper and are using that and I think for this one they're using HBase but there's a variety of other things in the stack that we support so when you download c'mon jet c'mon jörg then it comes with Kafka configuration as part of it and HBase or Cassandra you can choose we have a number of examples that you can download you have things out of the box already integrated and you can just start running for the particular tellico client then coming in we're reading in we have a variety of inputs external sources ways that we're processing it a lot of different metadata that's kind of describing the systems what we're reading in the land architecture this is all the continuous decisioning the commands apart so handling a lot of the transformations continuous aggregations doing all the KPIs and then this is a batch layer you know HDFS to do data lake so we've got various layers in here that we're feeding into and then feeding back out so we can have different oh that views Excel a mobile app or on a desktop view and a website so submit project results so right now here we've got kamancheh one of the things I didn't mention before is where a PM ml consumer if you're familiar with data mining p.m. ml is a predictive modeling markup language it's a type of XML so when you train a data mining model then you can save it as p FML and we're integrating not just with open source systems but with a lot of commercial systems so if a bank is using SAS enterprise miner then you can save your neural net as be FML and we can execute it so its spark can generate p.m. ml but not execute it so we are a PM ml executor so we can execute that as well for to my current projects for adding are so while I can generate p.m. ml and are for maybe a dozen algorithms are has a library of maybe 220 predictive algorithms or as I come across other algorithms that are might not be predictive but something I need for the given project um then I can pull that off the shelf so we have a speed layer and a batch layer and then getting to the output production systems so we're able to get the incremental KPI calculations available mid day so three four hours later as opposed to 26 hours later much cheaper so that's been a big win people are working on the fraud detection we had to get the various data plumbing data cleanup going first and then then we can start working on the fraud detection more so that's the in process product part a little bit more on p FML in the top left corner data mining group talks about p.m. mo so all these rows are the different vendors many are commercial some are open source so and the columns are different data mining algorithms that are supported so the purple would be supported by P FML so for example in our then here it's showing ours generates about nine different algorithms you can see very common algorithms would be decision tree regression clustering and then other things are spread out other rows that are have a lot of cells populated are embedded in them so it's not a surprise that they have so we're integrating with both commercial as well as open source systems so any other questions or anything I can help with yes um so in the they haven't started doing that yet so there's another project I'm working on right now its credit card merchant attrition merchant meaning in London when you swipe a credit card that restaurant or hotel would be the merchant and so now i'm forecasting for that customer who's likely leave so i basically go through a wide variety of a dozen algorithms let them compete to see what's the most predictive and we also support dags so i can have four models at the first stage feeding into a model at the second stage kind of taking a vote or combining the scores as well as with the input fields another problem i'm working on is security for a very large company and they have a lot of computer system generated logs and they want to be able to detect acts and so they nor they understand why particular logs are generated for different computer systems so they can be pulled out and so you can so for that i'm looking at something that's a variation of a priori rules Market Market Basket analysis there's one that's oriented towards sequence analysis and so I can find a sequence of logs that are commonly generated for different reasons so start removing all the logs that are generated for reason one reason to reason three so quickly and go from a bajillion logs to five percent of a bajillion and then perform another stage analysis I bring this up because there's a variety of different algorithms we pull and pulled a bear I've managed a couple of projects with Stanford NLP natural language processing so that is very computational and it fits very well inside comansa being a Java library and so there's a number of like an eight month project for large bank and a four month project for news corp ask me later I can talk about that I've reached my time thank you for your interest [Applause]