The Security Wolf of Wall Street: Fighti...

data.bythebay.io: Thibault Reuille, The Security Wolf of Wall Street

Recording: data.bythebay.io: Thibault Reuille, The Security Wolf of Wall Street

right guys hello um this is a presentation titled the security Wolf of Wall Street fighting crime with high frequency classification natural language processing so we're super excited to be here today thank you so much for inviting us data by the bay this is really great so actually one of the first times we're actually speaking in San Francisco we speak like in a lot of different conferences but not so much in San Francisco so first let me just introduce myself real quick uh my name is tibo and I am the manager of development research uh at op DNS uh you may or may not know me with another project that I've worked on is called open graffiti which is a data visualization platform using openg G and 3D Graphics but uh more recently I've developed the Avalanche project which we're going to talk about in this presentation and the main focus of my my research you know obviously data visualization 3D Graphics graph Theory but especially real-time systems cool and uh thank you for having us and thank you for attending um my name is Jeremiah I'm really excited to be here uh friends with Alexi um I met a while ago at a text text processing Meetup um so yeah I'm a scientist at Open DNS uh now Cisco um and uh basically went to school in San Francisco got my masters in computer science uh as mentioned before worked for Evernote and Uber uh also Mandan which is now far if you guys know about that and I just like to solve interesting problems and uh proud s SPCA puppy owner and uh we got a really nice presentation for you guys so really cool stuff so you guys will be excited thank you Jeremiah so the the the agenda for today um so introduction then we're going to talk a little bit about the Avalanche project and the research pipeline so essentially I'm going to describe um the whole kind of data processing pipeline that we have and a couple of data mining techniques that we're using you know in real time um on our cluster and then we're going to talk about NLP rank which is one of those classifiers who's running on this data processing pipeline which is doing a fishing uh detection okay cool thought let's get started so yeah just a little bit about open DS Labs I like to think where it's kind of data science meets network security and data science is kind of new to the security field so it's kind of still still being developed but uh we feel we're at the Forefront and basically every day we deal with over 100 billion DNS requests per day um and we a little bit more about us we we're now uh have a proxy so we process about 10.1 million requests of HTTP traffic and now including https too um so yeah on a daily uh basis we write detection algorithms do security data analysis uh distributed systems and uh Big Data engineering and this guy's the king of data viz so yeah thanks cool so with that much traffic one of the main challenges that we have is really to go real time on this uh so first allow me to say something I do know what real time means and in this case I'm just going to use like a um kind of a soft definition of real time what I actually mean every time I'm going to use the word real time is near real time um so essentially yeah if you're if you're familiar with real-time system so please forgive me for that it's just easier to say um so the Avalanche project I would like to start this presentation with a quick analogy because really it's very close to what we do and this is the title also of the of the presentation um really the analogy with what they do in stock market and like the algorithmic trading first you have a stock exchange you have usually a qu server and then they run a couple of strategies and that Loops back you know through an execution um kind of an engine to execute the trades and then they obviously measure how the strategies are doing portfolio risk management stuff like that but they also store historical logs you know like the historical uh data of the stock market to be able to run uh their models and to do back testing on those models and really when you think about it it's not that far from what we're doing really for us we see like a huge amount of traffic coming from resolvers again like 100 billion queries per day um and really we have we store those logs you know we aggregate those logs in S3 or like Hadoop a lot of different ways um and we run those models you know we do some back testing which is our machine learning kind of training process um and really the execution that we have is uh to to have an impact directly on the resolver so doing some blocking whitelisting or domain tagging or any sort of thing that is going to affect the traffic in some way um and obviously to do that we run a couple of models couple of classifiers and for us really the analogy with the portfolio and risk management is to have some sort of predicted impact on our users obviously if for some reason one of the classifiers were to block google.com you would think about it twice we have all sorts of techniques to avoid that obviously but um yeah so just um really like I think it's interesting to understand that we we have to deal with super super fast and and heavy data and we have challenges that are very close to what they do on the stock market so this is exactly the reason why I started developing this project called Avalanche uh and it's an open source project so a lot of the plugins that we've developed for that project are actually intellectual property that we have to keep but the core engine is actually open source and you guys can um actually check it out if you're interested so it's a realtime um data processing framework it's completely modular parallel and distributed design and it's uh written using Python and zeromq um and zeromq is widely known in a finance world if you're familiar with it it's a great great Library seriously like kudos to these guys they've done an amazing amazing job um and really for us is the platform a lot of a real-time open DNS models these ones are private as I said so NLP rank which uh we're going to discuss today but also we do some sort of DNS tuning detection a lot of DGA detection as well and fast flux these are all um uh challenges that we have in this the security world and you may or may not be familiar with those but I invite you to check them out so really the the whole Paradigm of the Avalanche project is to be able to design data processing pipelines this way so it's like a graph like a data processing graph so all these nodes that you see here are plugins threads and then you can also assemble um nodes together so threads together um to create to merge them into only one thread and that's what I call a rack so really the idea is to develop this whole processing pipeline that's going to come from one or several uh um Origins and be able to process this data as fast as possible using zeromq and python plugins and then when I zoom in on this pipeline the node level this is what happens so basically a node is a plugin some code I'm going to show you some code after that um and it has an input queue and output queue which are both managed by zeromq sockets so really zerq sockets can be seen as a you know standard sockets smart sockets that's the way I like to put it there's a lot of strategies the way you can actually manage those sockets how they handle disconnections how they queue up messages you know I really invite you to to read the documentation it's it's it's really amazing work um so yeah so all those nodes are connected together and this is how the code looks like here you have two examples of plugins if you were to use the Avalanche uh engine so the first one is like a simple version you just have one method process message and it returns uh the message which means that the message is sent to the rest of the pipeline so the yeah the following noes um and then the other one is one that can run like kind of in Standalone mode so it's in all it runs its own thread so in the infinite Loop so here it reads data from the input like kind of do some processing here like I'm uh actually loading it in the Json message and then you can actually send it to the next nodes as well so these are very very very basic templates that you can use uh to actually start playing with it okay cool so that was the Avalanche project now how do we use that you know inside Open DNS and really this is interesting part because this is uh tool that we actually wrote to be able to process all this data that we see so first I'm going to do like a high level View and I'm going to zoom in progressively so at first we have our DNS resolvers right here and it turns out that we actually store some of the logs in S3 uh so the resolvers basically send lock chunks every 10 minutes or something like that and we have a 10 minutes lock chunks that you have to process coming of all from all the different regions in the world all we have really a worldwide presence so we have resolvers everywhere um and then we run this data we replay those uh lock chunks through Avalanche and then it gets sent to production so meaning blocking you know white listing domain tagging all these things okay now let's zoom in on the Avalanche cluster so only on this Avalanche node there so it's actually composed it's a it's a really simple group grid uh kind of architecture with a Master Slave strategy and right now I have eight Amazon instances these are um C4 2x large for people who are uh interested in AWS configuration um and really the master is acting as a a load balancer so essentially it knows which slave is busy or not and just organizes the work and which slave can actually replay uh which lock chank and one really uh key important part of design is that it's actually a fire and forget uh strategy which means that um if for some reason the master cues up the lock chunks so we have too much work we can handle we basically start dropping dropping uh older uh chunks so for us it's more important to process fresh stuff fresh content to discover new stuff we don't know about to stay as real time as possible rather than make sure we can process everything because we have so much traffic anyway so really for us it's important to find fresh and like really um accurate stuff um and then after that you know after this processing obviously we centralize all the results and it's how they get sent after that to production now if I zoom in to uh the actual slave uh level what we have is basically four processes uh per instance and these are basically Avalanche pipelines so the first Avalanche graph that I've shown you before um and they keep you know processing this data in parallel and again this pipeline is also composed of a lot of different threads and total right now we have uh eight instances four so 8 * 2 32 and composed of the pipeline of a lot of different threads which are described here so this is a simplified version of the this data processing uh pipeline so at first we have obviously the master who sends the work and then uh the streamer who replace all those logs and then the first thing we do all our uach just to decrease the amount of traffic that we see the DNS traffic is extremely redundant so we really really want to you know um decrease the amount of traffic we have to to process um unless you're doing some some time series calculations or you need the actual uh time stamps of the DNS request there's really no need to reprocess the same thing um and then it goes through a domain tagger so this is where we do the pre filtering so uh wh listing blacklisting things we already know basically we take out things we already know we have a couple of um uh we have a matcher that basically takes out a couple of ips uh sinkhole IPS uh especially uh and then we enrich this information uh by the goip the max mine plugin so this helps us to get the ASN information from the IP in our logs and then this thing is split into a couple of classifiers again this is not there's also the fast flux that is missing here but today we're going to talk about the NLP rank one here and then once this process is done we have to go through what we call a minor uh stage right and this minor is really um I'm going to describe it later in the presentation but but the idea is to use other indicators external indicators coming from other apis even ours um to be able to to do kind of graph detection techniques apply graph detection techniques on it then we have some filtering phase just to make sure we take off like a couple of um details of false positives and then it goes directly to production cool now a couple numbers because I think it's uh it's really uh evocative it's it's really nice to see those numbers these are numbers that extracted from a log and this shows you uh the the the message rate that we're actually seeing at the resolver level and I chose um the Amsterdam resolver which is kind of a not the biggest not the lowest kind of average resolver the biggest we have is Miami uh lowest I'm not sure Singapore probably um and I took you know a a lock chunk at noon UTC which is noon uh in Amsterdam I mean around noon which is peak time usually in our logs so query log are so if you're not familiar with the DNS traffic really the way you can think about it is question and answer query log is between you and the resolver and off logs are between the resolver and the authoritative Network that is behind um and really what you're seeing here is the number of user uh requests so here we have 10,000 um requests per seconds at noon and at midnight like something like half of that and I run a couple of benchmarks with Avalanche so really we can process 30,000 messages per second which is one message every 33 microc so really you see the parallel with the with the kind of what they do in the stock market when you have to deal with data so fast um so honestly it's I can't really take credit for that Zer mq is doing all the job um all the work is is you really this this socket um message processing engine that they have is is completely amazing um so really we can run our logs replay our logs three times as faster than Amsterdam me again Amsterdam is not the biggest one it's like an average so I think it's kind of cool to see those numbers to have an idea of the speed of the traffic cool and so the the last part of the pipeline describe this this minor script uh and really I want to talk about this graph oriented data mining technique it's going to be a little abstract but um please bear with me um so as security analyst security researchers we have to deal with you know a lot of different internet entities that we see no logs domains URLs IPS asns uh hashes binary hashes for viruses that are helpful to do uh signatures for antiviruses um emails and reg AES as well uh you know a lot of different ways to approach this data and a lot of the external apis a lot of external vendors have indicators as well such as investigate this is us for open DNS Max M virus total Showdown or even like we can grab the HTTP content as well to build our own indicators uh also like some um some custom models and this is the this is the case of NLP rank today um and really uh one way uh to aggregate all this information in order to to take an accurate decision is to model it in graph format and really this is probably the most beautiful data structure you can use uh in our case um so this is just a simple example so uh you can have a precise idea of what I'm talking about let's say you have a regx and that regx matches a couple of domains in your logs and this domain is registered you know through the the who is data that would be a certain email registrant registrant email sorry um this domain can point to a couple of URS uh those domains could be hosted on the same IP which would belong to a certain ASN and this ASN could be located in a certain country uh this URL could be uh hosting some viruses some malware um you know using certain hashes and signatures so really that's a way to aggregate all of these indicators all these entities and to connect the dots to be able to have a to step back and have a a larger um idea to see the bigger picture basically so now that you uh see that really what we're doing when we're doing a data exploration once we have aggregated all those indicators really we're doing a bread first traversal so um if you're not familiar with it it's a simple graph algorithm think of it as a Facebook um you you start from your own profile and you visit all your friends the friends of your friends the friends of your friends of your friends and you do that as many times as you want um so for us we start from a domain from a URL whatever and we explored all those indicators and we try to take accurate decisions based on the neighborhood of a certain node this is why having a graph representation is really important um however with so many uh you know input and seeds you know to mine from uh might be super long to actually uh operate that this is why we have to opt for like a distributed breath first traversal so really this is what the miner is doing it takes all those outputs from the Avalanche pipeline mining all the external apis HP content uh running some uh some machine learning uh you know algorithms and trying to take make a decision all at once um and this is how we actually do classification so really you can think of it as um I love to call that Lambda mining CU kind of a functional graph exploration process um this is how again we we can apply some rules some threshold like do some topology uh based detection so the The Wider problem is called subgraph isomorph isomorphism detection It's actually an NP complete problem this is why we have to opt for U those um uh smaller kind of Solutions cool now let's move on to the next part and we're going to talk about NLP rank which is our amazing you know ficient detection plugin using machine learning that Jeremiah has been working on for the last what two years one year something like that okay and we have to do a quick transition thanks all right thank you cool those down yeah thank you sorry about all right can you everyone hear me yeah cool um so basically uh this is the text track so let's get into some text mining right so um basically uh using uh NLP to find fishing Pages or fraud detection um so just to give you a little kind of example at the at the at the beginning so this is a fishing uh uh domain so basically you could see from the who is records uh this is a this is a UI for our product investigate uh of Open DNS basically you going to investigate different uh domains from BNS traffic that we see uh you could see it's registered or created February 9th of 20 2016 updated February 9th 2016 okay so and then we saw that that Spike just like around February 9th so and you can see the the top line is what basically I I I see every day to view our results so this is the time it was detected the time samp uh the um here's the domain and basically this is the score you can think about it as a probability that is a fishing page uh and this is the page that basically I trained it on um and I I'll get into explaining it but basically the probability that it's 99.9% a fish okay and boom you can see that that's what the actual fishing page is from our traffic so just to give you a little flavor at the at the beginning just so keep you guys interested so let's get into it so basically the overview of the model is it's a fraud detection system using NLP uh and unsupervised machine learning techniques and traffic features to identify domain squatting and brand brand spoofing in DNS uh basically this is a common technique uh by Fishers and uh AP uh cnc's so and just a little pay homage to uh the kind of origin of our data that I trained it on is a fish tank if you guys know about it and it's it's pretty well known in the security Community basically it's a crowdsourced uh kind of uh BAS system where basically uh the community submits fishes poent potential fishing Pages people uh vote on them basically it takes a little time to like 20 votes to actually categorize it as a fish and then also with the brand and then we we push that out to the whole community and it's consumed by a lot of people uh you know um Google safe browsing for example I mean uh we uh we Brave the new the new browser um just a lot of antivirus so basically uh it's a kind of a staple in the community um so yeah and I'll get into that a little bit later too so just a little uh psychology behind uh kind of fishing so it's kind of falls under the category of human computer interaction right so what kind of links do people typically click on you know how do people get fish what are people and that has to do with people's interests right or and so kind of the topics of Interest would be you know money of course uh news secured your software updates and now social the social networking all these sites coming out so um just a little psychology behind the the attack because it really has to do with uh kind of like human and social engineering right so uh just for this presentation we focused on bigname Brands uh I just want to kind of like emphasize that like the the most fish Brands and you guys I'm sure have seen like PayPal fishes and receive them Apple fishes asking for your Gmail creds whatever well as Fargo so yeah just for this presentation focus on that so this is kind the kind of overview uh kind of like diagram I show for basically I'll keep referring back to it um as I kind of like step through the the pipeline of the model um and so you guys will get a better better picture so basically for the input it could take DNS or HTTP traffic uh like a domain name or full URLs um so the first heuristic I have uh and this is just a filter is uh basically an ASN filter uh and you could think of ASN I have a white list we have a white list as TBO TBO mention uh for uh um Zer mq uh the Avalanche basically through through through our pipeline so that filters out a bunch of traffic so we deal with so much traffic the first step is to remove everything that we don't need to inspect because text mining is computationally intensive right so then uh after the white listing we have a ASN filter and you could think of the ASN as your ZIP code on the internet right so if you were in some kind of like Gmail update or Java update right uh most likely if it was a domain name registered by Google um it would come from Google's ASN or oracle's ASN right so you could think if anything coming out of that zip code that's has like some kind of like Java update um you know you could automatically think it's some kind of like uh weird kind of like you know anomalous right so that's the first step just filter out as as much as we can so then the off logs come in and basically uh basic uh the Avalanche will enrich them too with couple of other things like that the ASN from the goip lookup and they will filter by that right so the second characteristic right uh getting into some text processing now is we're defining a kind of like malicious language among internet fraud right and this kind of has to go with kind of the human CU computer interaction and psychology right you're advertising some kind of update right some kind of Bank update you know to your bank account update or you know software update or or you know some kind of verification uh login right um so uh I used the edit edit distance algorithm here and uh kind of like some cust RX is kind of applying aomata Theory um kind of defining different languages among among the domains and I'll I'll get a little bit more into that so here so what I did first was kind of just some basic data analysis out of the URLs and domains I on fish tank and kind of in general a corpus I buil built of AP um cnc's I extracted words that were in the dictionary and I also stem them because sometimes they might use the root of the word right so basically um yeah mail news soft you could think software server update right online uh free login secure right um and so basically that was that was the first step um and and then basically also just extract the most likely do or the high the top domain or top brand names so Apple iCloud Wells Fargo Bank of America yeah B OFA so um basically uh one of the ideas came up with is looking at uh brand names plus what I call is an advertising action word you know so basically you know yeah Java update um Wells Fargo secur verification online.net right something like that those those are most likely you know like what it would what these fishing domains look like right iCloud service update what whatever so but the basic idea is you have a brand name and you're advertising kind of a you know some kind of action to get the user to click on the link right here's some examples I mean and uh so basically all also what I what I did here's an example of using automata kind of like the um kind of defi defining the language among regexes that would you know for the edit distance sometimes it'll be a oneoff like PayPal with a one at the end instead of the L but we can't set a a a large distance because otherwise we'll flag too many FPS and too much too much computation right so um basically what we do is build kind of like a a a a language among these so that that allows like different um you know variations of these letters so with a basically we're allowing a higher distance but only in certain cases right that's the kind of idea um here's just some examples of fishing examples and you could see from here the advertising action words wellsfargo.com billing. account. update my account. wellfargo.com online accounts so you got you guys you guys see it get the get the idea right so these are kind of anomalous already right you know Wells Fargo online you know update do Bank of america. update. login.in so here's some examples of like AP uh cnc's and you know in the security Community it's kind of like a one of those kind of you know markety terms but uh I just use it as a label data set so uh it stands for advanced persist threat it was a term created by the Air Force so but it's thrown around a lot at conferences like RSA and stuff like that so but but basically we're just using it as a um label data set and you could see these uh domains exhibit that kind of um those kind of features too and these are more targeted attacks but you could think of these as kind of like you know Target at big companies right um you know Lockheed um Sony uh so just a lot yeah so okay so yeah basically I explained this kind of creating a malicious language so that's just on the on the uh domain name when we see it in the traffic if it exhibits these features flag it right and then we'll do I'll get into the content mining right and just a just overview of the edit distance so uh shortest path dynamic programming uh from algorithms class uh Google so this is a three edits right so goog with with the zeros one three edits Dropbox with zeros Bank of America with you know change the O zero Z the O's to zeros e to threes you know I to one so yeah three edits so just a basic overview you guys already know okay third here stick is HTML content mining so this is kind of where it gets interesting so first once we the the domain exhibits those features in the traffic we do a get requests we check for form fields on the page we get counts uh convert to tfidf space which is another text processing algorithm uh we apply latent semantic an uh indexing basically a topic modeling uh we grab the topics of the web page so you could think of it as basically content mining web pages right um and then we compare cosign similarity to a trained uh basically a corpus right that's already converted to LSI Uh u space and we just compare vectors in the Corpus so any input page comes in exhibits those features flag it grab the content convert it to LSI space then we have a corpus already converted to LSI space Compares through each of them just iterates through and finds the coine and that was a score I showed you in the beginning the 99.9% that's the cosine similarity score so you think of it as a document similarity problem uh these fishing Pages often times they'll just copy the exact uh HTML uh from the uh the original page you know wellsfargo.com the login and basically uh use that right so that's that's basically a one of their primary method so here kind of going on against like with the AI idea we're recreating a security analyst basically to do our fishing detection for us you know you could think of what a when a analyst or researcher investigates a website that's in question you look it in a tour browser then you get a summary of the page and basically you you you you decide what the page is about so we're creating that creating a summary of the page or getting topics of the page right I just like this quote unsupervised learning is the future it's all about the features and that's what we're using LSI unsupervised algorithm uh a lot of these days so much data is being generated uh we can't even uh process it even through supervised algorithms you know humans have to come up with those features so we need a way for the machine to generate algorith or features right that's my only Point uh so yeah I'll go through I use Jensen Library highly recommend that text mining check it out and it's being developed on by uh Google sum of code they hired a couple of interns but it's awesome so knowledge Discovery algorithms and it's it's super awesome in Python yeah so yeah uh building the Corpus basically uh pages from uh all different brands uh focus on bring bring brand names and the reason I have to build a corpus is because there's ' 90s PayPal 2,000 PayPal all these kind of like older fishing sites so and that's not updated with the original legitimate site right now right so tfidf um basically this is the kind of overall um just algorithm so basically it's not only if the term is relevant basically if it's it's if it's high frequency but also if the term is rare so we want to kind of merge that together right to find the important words and you could think here I just gave an example on tfidf on Shakespeare's MC Beth so the top 10 is you know McDuff McBeth McDuff you know fear so topics right fear uh murder that's that's what the play is about right so just to for example witches um LSI kind of like nasty linear algebra long story short uh lot of uh kind of Matrix decomposition um so yeah we have uh basically Pro decompose so we have the x is the count Matrix um m is row uh and N is the number of columns and this could be also the tfidf Matrix we pick value K which is the magic number the black art black magic number right and then we decompose X into three matrices M by K Matrix where m is terms K is the number of Concepts and S which is the K byk Matrix kind of like emphasizing the variation right among the topics and V transpose which is what we're interested in at the end where K is the concepts and N is the number of documents right so that that's what we'll be comparing right the ve the vectors so yeah and then get the cosine simility which is basically like the normalized dot product sorry going a little fast I'm a little short in time but I just want to get to the results um so yeah then here we process the documents and then push them to our block list so we are blocking real time now so and then we get kind of analyze the rest of results rebuild retrain right all right let's get to the results so this is kind of what it looks like so it gives back a top however many scores we want but we're taking the top one and we're blocking based of that kind of whether is the closest to so if you have a input query Wells Fargo that's that's the input domain we caught uh and then it Compares and you can see here it got closest to the actual Wells Fargo the legitimate site so I have legitimate sites and fishing sites in my Corpus right here's some uh kold cross validation for just for proof uh and this is trained on the fish tank data um but yeah any questions you can talk me later so here's some dedicated examples so when I say dedicated this this is dedicated domains set up for fishing so PayPal D update um- account dotinfo versus something site that already got that got hacked like WordPress you know you know something mckas you know.com com.br and then they have paypal. D account. update so basically it's it's it's a site set up directly for fishing not a hack site uh serving fishing fishing Pages too so this is dedicated sample so the setup so here's iCloud server

Ru detected or 98% cosign similarity and yeah B basically we detected it uh kind of um here uh 5'8 so you could see right here on the May May 8th we detected it it was created a while ago but still we're detecting it right away and we block it right away right iCloud server. Ru boom Russia Moscow uh web apps paypal.com you know DET detected uh February 11th created February 10th we saw the spike February 11th blocked Real Time Boom Google drive.com 95% you know we we see it basically and what we're doing also we pivot on these domains so look at this email address s mosovich yandex.ru kind of looks suspicious that's the page boom and now there's a bunch of fishing Pages associated with that email address this is what we do what happens when we pivot on the IPS too so here's security apple.inc detected 94% close to iCloud boom it's in iCloud and look at that uh trusted Gmail so same email address um and have much more fishing on there right Apple ID privacy which is right here first one boom it's all fish gmail. TK 96% right looks kind of yeah funky so basically yeah um and we pivoted on the IP and it's all fishing on that IP so we block that IP everything that comes up for there automatically block so this is dedicated kind of fishing examples same thing that Yahoo reply was one of the domains down here boom Google secured Center uh yeah so then pivoting on that email address yep and now these sites are just being secured Google so they haven't even came up with the site yet they just get spin it up getting ready so it's kind of predictive so I don't like to throw that out too but it's it's uh one of those kind of terms the buzzword terms but basically Twitter I just emphasize here is that we're detecting different brands Wells Fargo Bank doo. Ru comp my sample so this is domain so here's the fishing page and here's if you look at rural mos.com boom that's the original page the the the landing page right so these are hacked Pages google.com Stephanie lell boom you get the point uh and here's just some more compromised page I didn't put the landing page but that's what we're detecting it's kind of like as example I just want to show we detecting a lot of Brands obviously we have false positives but basically what what model doesn't but here what we're actually finding is that Mark monitor which is a service that does brand name protection which redirects back to the legitimate page this is a good kind of like test set within live data so we're on the right track and we were able to filter those out based on like the who is records so now we're not really block blocking those even though kind of Mark monitor already does that Ser so it's a whole company based on this kind of idea so yeah you could see Mark monitor right here here's another uh example of a page that we we hit on but pay pay for install. Ru which basically directed from that play play. google. which is no one will miss this page if we block it anyway and it looks kind of suspect anyway CU there's some fishing or like typo squatting on that email address here's some more results a lot of lot of n High 99 so you know High 90% so stacking model so we're combining this model with our Spike detection model so we could see uh uh kind of fishing large scale fishing campaigns happen in real time you can see this is a iCloud Safari page so iCloud page and it's has over 2,000 so basically and analyze SP request some by one of the other models aot one of our a couple of our researchers uh developed so stacking models that's the only point of that other Clues basically some other crap in the page mirrored from google.com so um yeah here's some like just one of the bad uh kind of like carbon act domain so basically Russian AP uh they kind of like exhibit those update Das java.net that was a banking Trojan um yeah so I I I won't go into this but because just cuz for sake of time but basically uh there's just a lot of crap out there uh yeah and so for future work reduce validation time on fish tanks and push out to the community so it could be used to to detect fishes so basically anything that people push to fish tank does not have to be verified anymore it could just be run through this algorithm and push out uh integrating with the Cisco proxy and the email Corpus building testing and tuning the algorithm and detecting target attacks and moving to aat spark in that direction so yeah cool thanks Jeremiah yeah so very quick conclusion because I think we're going a little bit over time but um you know in the security industry like it's a lot of people usually claim that you know they do machine learning but actually when you start scratching the surface it's not really there I really would like to give kudos and shout out to Jeremiah who's been working hard on that and it's actually something that's running live blocking live traffic every day we see like new uh stuff that is you know happening on the internet and really like if you look at all these examples um sometimes the domain has been like set up maybe a couple weeks ago but we see the first request 10 minutes later boom it's blocked and that's for me that's really impressive um if you're interested in in you know learning a little bit more about what we do we have a Blog go to labs. open dns.com and we can see our can daily task and you know we blog about everything we do um special thanks to obviously the analyst team who helped us review a lot of those domains marketing team for those slides obviously um design team Tasha and also obviously the data by the bay uh we wouldn't we wouldn't be here today without them um we don't have much time but let's do maybe a Q&A session short one I mean if we don't have time you guys can always come see us you know we'll be happy to answer your question and hear your feedback how to dat do you have to keep your models or your Corpus um that's a good question that's a really good question yeah yeah okay okay uh so basically um yeah I mean I I go through and I update it like basically pretty often like weekly uh but the thing is we just have to keep on top of certain Trends and uh sometimes they might modify the actual legit the legitimate page like you know whatever the iCloud landing page or Wells Fargo so we like basically periodically add that to the Corpus too but uh I'd say an ideal time is you know monthly but we see like uh I ideally like weekly or two you know bi-weekly uh and I think uh one of the things that I'm working on really is kind of analyzing a way that we could just retrain from the fish tank data automatically and it updates the Corpus you know every every so often I think you know maybe daily or kind of like a you know a couple times a week would be ideal you know just to keep on top how we is it possible for you guys to yeah so if if if if if we're not saying that many false positives in the top traffic actually uh but the thing is um basically uh if it's a page we filter out the mark monitor stuff but if it's a f false positive we have a popularity score to kind of check against too um but basically if no one's really going to miss it and no one complains then then we don't really care but I mean for the for the the top stuff that's automatically blocking there's almost n yeah it's it's pretty pretty pruc one so so people guys yeah basically yeah they'll say that but but usually they're they're kind of not like they're not necessarily fishing but they're kind of suspect Pages anyway yeah yeah and a lot of you know coming from like kind of like some weird provider in Russia you know and kind of you you saw the the false P that you know they have other yeah sorry yeah yeah uh but they basically there's other stuff that's on that IP or like domain or like email address that's already bad too so it's kind of like suspect anyway yeah might how do you decide of function how many for posi you likeus for negative and tune in yeah so what we do is we set a very like a kind of like a two different thresholds we have uh basically one thing that flags and automatically blocks the stuff that we're like super confident about like very little false posit like none and then we have like this kind of other range where we have like review that sends to like our security analysts that they review and sometimes some of them are a little bit more false positive prone but some of them are are still fishing so we're trying to work that too while we get some more features and stuff but basically the for the stuff that's automatically blocked there's almost none I was wondering um how much back for bu you get from looking at the Domain in url content um I mean if we could flag just based on the domain name of course that would be nice right but the thing is uh there's um you know a lot of stuff like the DNS is a jungle uh and there's all these type of stuff that looks bad that might not be or kind of having people register kind of weird stuff anyway uh the content mining is is basically to create its own analyst within itself that we don't so we want to be sure because if we block something that's not really we don't have too much like uh you know confirmation about uh especially kind of like new stuff that comes from Google like that might be on some kind of like weird ASN that doesn't have to do necessarily with the ZIP code we we want to be uh sure we're not doing that you know and uh basically the thing is that we we don't want is customer complaints so we want to do have as in-depth models as possible you you know and it's still faster than humans so you know way faster so I mean literally from all we need like like TBO said all we need to do is see one query flags on that and does do a lookup and then it's blocked from the compare the compare compare to convert it to Vector space and then it's compare against the Corpus and it's automatically blocked so within that time period I mean that's humans can't do that anyway so we're still faster so yeah it's it's not really just just takes more resources more machine kind of power but uh it's pretty cool problem I think and kind of a lot of companies do it kind of this idea of categorizing the internet content mining because this can also be used for we we've also used this algorithm to discover kind of stuff on like mining the dark web so for like drug sites uh child porn pornography um kind of all different you could use TF and topic modeling to discover those pages too which is kind of cool so and this is kind of how they do U website content classification this is a search engine algorithm so yeah cool all right thank you very much thank [Applause] you