scala.bythebay.io: Alex Kozlov, Endpoint Security with Complex Data
Recording: scala.bythebay.io: Alex Kozlov, Endpoint Security with Complex Data
you so i'll be talking about cybersecurity i don't expect you to know anything about cybersecurity and i am working currently it each security that got next slide actually so i came here in 91 just recently it became very popular to tell that I'm an immigrant have been here 24 years i was still in st where I did my ms and PhD I was at glow their phone number of years and what is one response to this conference so I know everything about Big Data and currently they are really i but i'm not sure it will big data this fashion was used to be able to know about it and I guardame software engineer at each security and the idea is to combine big data analytics and security to give some interesting results and products let's see how it goes by the way it is one of the lead groups how many people know what group theories here all right good so a group database the cell transformation with transforms a logic into itself and the render some mathematical concepts so the idea behind apply and group theory to security is that by exploiting certain symmetry properties of an object's you can actually increase the signal to noise ratio that applies to frequency filtering or some other concepts in physics but in security what happens right now the attacks are distributed so you have to basically observe your signal across multiple machines and there are no products which can do it right now you can install antivirus one machine but it doesn't solve the problem of coordinated attack on enterprise let's see actually let me see so one aspect of it security is definitely big data I can talk about big data for a long time one thing that it's interesting big data it's basically big data is everything that traditional up the bmes was not supposed to solve or they were not optimized for so we have very good relational database including Oracle but the problem is when you go to complex data like arrays or structures yes you can do it but it's going to be very slow so that aspect of big data is definitely scalability change in schema schema evolution but it doesn't see what happens if your machine goes down replication you want to store data in multiple places data pipelines it's very related to functional program not going to talk about it later streaming and three to five visits velocity variety volume and additional people came up with veracity and value basically the idea is the same that it's something that Oracle haven't thought about it or haven't really optimized for it so sometimes people compare big data to a train so you can probably do things faster with a sports car but if you were dealing with large amounts of data then the great technology so much better for it all right and finally of course my talk will be about scholar so what's interesting about Scully and I discuss it a little bit of it in my book and we can talk after the talk wats why skull is so appropriate for big data and as I see it there are two properties of scholar is lazy relation intimate ability which make it very appropriate for big data for particular for distributed programming you know that spark framework is written on scallion that is written for that because these two properties make distribute programming much much easier if you know that object is a musical everyone who did distribute know that you can optimize social things and the second thing is in scholar data becoming this primary you apply functions to data the transformations so you are focusing more on the data rather than on the functions are going to again I'll go a little bit later about it as well finally a scholar and spark and incorporate traditional tools the data scientists use including java python and are so cyber security there are different aspects of cybersecurity one is definitely probably forensic analysis when people go to compute and recover deleted files the other is just firewalling what we do actually is mostly retrospective analysis so want to record every packet with disney honest work we did pass on your network and so that you later you can say well intrusion detecting happened and we can tell you what data was stolen right now people don't do that if you hear about like DNC break breakthrough or break down or any other walmart intrusion attacks most people people even don't know what data leaked because they don't have record to it so so as I told you one of the approach which particular companies is being able to record everything so if you know in the Middle Ages people used to build modes around the cities or water features so that intrudes can pass later on you don't see much you don't see much in San Francisco the only mode i'll have is that i know it's about its round larry ellison house I don't know for what reason but it does no longer works the same with security right if previously fire will work the problem is that people now command come up to your enterprise with your own devices and all bets are off you can do very interesting things on the network actually seen people at the conference which are scan in network for vulnerabilities in your laptop so conference are very good tool to steal other people code on data and this is the map of intrusion detector which happened pretty recently mera outbreak in October 21st so a lot of services were done in luzon Twitter so what we do is the usual big data stack for log producing you can see very familiar tools I think it's pretty well established thanks to three across most of all Facebook Google and companies like Twitter which build this wonderful tools now we can apply these tools to the other problems so what's the current status we ingest a lot of logs so first of all its web proxy look right at the outside the company you have a web proxy which translates your request to the outside world and probably blocked some of the requests and run our companies which do web proxies blue code websense below the network's dhcpd HP establishes mapping between your mac and IP address that's very important as well Active Directory our big companies have Microsoft work directly it's basically the standard it keeps maintains the database of people work in the company and provides additional information like how many people logged in into the systems and well there were some security exceptions a specific release doc have been talking about endpoint win again there are companies which provide clients on the endpoints which collect information so what's end point for us end point for us in any external device which comes to the organization for example your cell phone is an endpoint so we want to know everything that happens on the endpoint because it might be security risk again cell phone probably dunes to you we can't really install the client on your cell phone but we can look at the traffic which emanates from your endpoint and there are other services or tools which we collect from potentially from our partners security van information management and data loss prevention or the acronis each other there are about 10 different kind of specific tools to collect information in the logs based on the network traffic so we collect up to one terabyte of data per day and do certain models so I'm not going to go specifically into machine learning models but one of the major models we do is pure analysis so basically there are ganization in the enterprise like marketing engineering all of them have specific behaviors and we compare at behavior of individual with these groups to overall behavior if there are something different from overall behavior then we market as suspicious coordinate activity I don't know how many people heard about command and control model well probably no not me but it's it's it's basically the modern software or the madden malware developers they develop not only the program which still the data is a very devilish cooperate on different nodes so you'll have malware injected one node and then it slowly traverses your network and infects multiple machines and they talk to each other they communicate so they can potentially send very small bits of information from each machine so the total data loss will be very high but it's very hard to detect because each one sends a few bytes of data every day time series analysis it's a pretty standard for anomaly detection and signal chain and what single changing if you have one suspicious of it in front where you have fallen star it's probably well it just an event it's not likely but if you have certain events in sequence that really make you suspicious right particularly if it's a predefined interval so era guess it's enough about that so so what's endpoint behavior so want to point it to this end point so cell phones is one and point the others maybe cash registers collisions can change and retail stores you want to monitor what exactly depends on the sketches that which process were on which ports were open how much traffic emanated through this port in a simple example currently we have software which can produce several hundred thousands of costs and monitor about several thousands of process language host so it puts everything into database and we can run oolitic tools over this database so we can talk a little bit about object area gramin design so traditionally what happens in object any programming you probably know the paradigm the rockettes dogs we can basically aggregate common behavior and say that heads bark dogs now or the other way around doesn't matter but the real problem is that what we do we actually package in our behavior with the data in my case actually don't care what IP address does right I don't know what IP does before I analyze the data so I'll always should behave this way there you can say we can we can call the function say whether it barks and now so whatever well only thing I care that there is a mac addresses shaded IP address and there are multiple potential dns entry associative dynasties and also i want to track the history of these associations i want to see a trigger through time right how you do it it's to the implementation side it's not to the design so what happens in java in most cases you most likely package behavior in the class and then you supply the data file and you don't supply the jar which analyzes this file and everyone who dealt with the sequence files in Hadoop you know that's a pain because if you name it class you no longer can he read the sequence file because you already have a class named recorded so it's a very bad paradigm because basically what you say I'm have to ship the data and now I catch the behavior and this my data mismatches behavior I cannot do any longer anything with this date so again the scholar you can certainly do object-oriented programming but I don't recommend you right and the next slides basically gives the solution to it its color pattern matching so you can familiar with probably most people here familiar with scholar pattern mention but basically what it looks like it looks like if you know about c++ compiler there is a virtual table virtual table basically structure for each call if you have a class then you call the virtual table and it knows exactly what function to call but in C++ and in shower this is encoded in the jar file in your code in skul you make it more explicit you can at the runtime say if this is AP address I do this thing if this is host I could do that thing and in the default case I just returned some arbitrary steen right so ok this is returned unit but ok we can we can make it writ on the stream but anyway in the key in the default case you can do not in what way for example so the other thing which matching helps you is identify interesting behaviors using the skull imagine it's very easy to to analyze your data so usually your data as I tell you want to trace behaviors very common pattern and its advertising user monitoring or security field you have an idea when a user ID or machine ID and then you have a sequence of actions and the timestamps and that's very important for profiling right and what you want to see if at times than one some action one happens and the next time stamp at some action to happens so scholar gives you very good way to analyze these sessions and basically find this interesting behaviors so additional feature is guards I don't know how many people use guards and practice but I find it very useful in addition to finding this sequence affection you can also say these parameters have two different less than x I'm out so you can find all the sequences within certain period of time this is very useful facilitate such translation and other things so current status we are collecting a snapshot of runyon protests on the host and port of thousands of endpoints converted suspicious behaviors and purchases to alerts and analyze all this data in spark notebook or Impala Impala has a great support for nested structures right now using parka falls and again parka Falls is something any virtually based on product above it's very efficient storage for the nested structures so I'm I'm mostly done with my talk have any other questions here is basically a report from the i would say women battleground on from the field this is actual code which how engineers implement and daunting which i personally take pride of its teachings in a functional program so there is a way to write this code in red in in very java like and there is translation to the skull equivalent of that so i won't want to speak to talk little bit mad i want just to say first of all it's important to use the match operator because we have let's see instead of huge in virtual table as in c++ we basically say if this condition this like i said then do this and do that the other thing which i notice what happens if you learn more functional programming your program becomes much wider and much shorter right you write pipeline so it's kind of squeezed from the top to bottom usually right and i can explain it in next slide you'll see then my next place will be a shorter and shorter so the other thing is exceptions right it's nice when you teach something at school and then you come to work and say well it's actually does work in practice it's something different so you'd be surprised if you like try to compute the square of a number or the next prime and in mathematics you cannot say disillusioned in dijon something because there was a network drive right it has to return something right and that the functional approach you always want to return something is that may be exceptional cases where you plug out the glory the power cord and there is special way of dealing this in scholar right you can do let's see yeah catching catch radiation word you can actually return an option in an exception and you can do the same with logs instead of log in the data somewhere on this you can actually return a logic logon object to the back end 22 which encodes everything that happened with this particular record okay I'm running out of time we can discuss it a little bit more after the talk but this is kind of implementation of logger that instead of logging something returns something there's a complex object vision cause the whole history and that's that's not easy usually because if it's very complex object you have to articulate ebrahim you have serialized and digitalized on other machines so you have to be careful about those things I don't know energy solution somebody does let me know finally there is definitely some even even if you don't use skull you can use function and on functional approach to programming so red is non-functional green dysfunctional we can able discuss there is a one line encoding of Fibonacci number generator there is equivalent for generating the prime numbers it will be given as an exercise and my advice to scholar developers particularly the spark is in green festival you can start with crunch and crunches framework which is works on original MapReduce even though it's not spark known as not in limiting and memory it teaches a lot of you a lot of pipeline in concert and functional programming concept from the top I think data / plies think about serialization sterilization can be very expensive use initially use very simple case classes or playing objects and prepare with deal with acceptance login and mountain these things are not easy you need to think them before you input something don't don't do organization programming in Scala its lead you nowhere we better build programming in shower don't launch complex servers within your spark tasks like HTTP server and don't pass complex objects between different tasks I think that's it so if you have any questions you can ask ask them now or I'll be here in the halls after the talk you can always come and talk to me including this code snippet thank you [Applause] you