Scale By The Bay 2020: David Aronchick, Owned By Statistics: How Kubeflow & MLOps Can Help Secure...
[Music] hey everyone um thanks so much for the opportunity to come talk uh this is my first uh presentation at scale by the bay um but uh i went to the last one last year and uh it was just one of my favorite conferences just the the density of stuff so i feel really honored that i even have the opportunity to come chat with you and and thank you very much and thank you for attending uh so other than that um off we go so um i'm going to talk about uh machine learning and ml ops and specifically how people are really running in a pretty vulnerable state in a lot of ways nowadays um and what we can do using things like mlaps to secure and inform your ml workloads um but also you know ideally have a much better you know model over time so um with that you know i i am i work at microsoft um and and microsoft has been you know in ml for a long time uh really you know contributing back to the community in enormous ways um uh you know we very much understand that that machine learning takes place in open source and it really has been transformative the things that that we've taken and built on top of the things that we've given back and people have built on top of from us you can see some of those things here from microsoft research vision and speech and language and so on um and the reason we needed to invest so much is because it touches literally every aspect of our product line uh whether or not it's windows or office bing xbox um hololens you know and of course microsoft for search um we simply could not do what we need to do every day on behalf of customers if we didn't spend a lot of time in ml and and when i say on behalf of customers i mean just in absolutely enormous numbers you can see here uh you know 180 million users a month uh interact with with ai backed features in office 18 billion questions asked of cortana and and 6.5 trillion security events are evaluated every single day on behalf of users whether or not that's locally or in the cloud um you know to try and capture you know new security issues and and things of that sort on behalf of users the the numbers here are simply so large that we could not handle this kind of data without ml uh doing filtering doing intelligent understanding and so on so with that um you know unfortunately we say a lot of this and it sounds great but then um we in the ml industry really don't do a great job um helping people understand how to use it ml for better or worse today still is really hard and the reason is is because you saw all those numbers um that i just gave uh oftentimes those numbers are related to um you know very sophisticated models and and those models are valuable make no mistake uh models you know are the the heart of any machine learning system but it's not just about a model if you want to actually change the way your products work and roll them out to customers you have to build an entire model pipeline and so that might be data ingestion data transformation validation ultimately evaluating the model and then doing it again at scale and in continuous pipeline and then rolling it out um you know you can't just throw these things over the wall you need to integrate it into your serving stack your monitoring stack logging and so on it is only if you look at this in a holistic way that that you're actually able to get the thing out the door so let's say you're a data scientist and you're like well i don't care about that i really actually only care about models um the answer is you do and the reason you do is what you see here um more often than not you a data scientist might create the best of the best you know better than human performance around a model but that's not enough because it's still after in this case 11 months sitting on your laptop not being used in production because you weren't able to integrate it with the the ultimate application with your sra team with the rollout procedures so to solve this much like they did with uh software engineering and coming up with something like git ops uh folks have gone through and implemented ml ops and the idea behind ml ops is very similar to git ops where you think about that inner loop that a data scientist is going through today on their jupiter notebook you know on their laptop or on their private vm or whatever it might be continuing to iterate iterate iterate um until they get the right metrics area under the curve outcomes whatever it might be that they're looking for then they take that and they check that code in and it kicks off an automated developer and ops pipeline and so you that's where that that model comes in and is now tested just like any other library or any other piece of code it's continuously integrated continuously deployed and then continuously delivered to that that end point now that doesn't necessarily mean go live um you may have a a plan for how you go live in a more sophisticated way that's okay but the sooner you can get it into a go live ready state the better you're going to be and the more responsive you're gonna be able to be and it won't be some catastrophic change that requires many many people to all come together in order to roll it out so that's the essence of mls um when you do that you get a lot of benefits uh of course on the left-hand side automation and observability the ability to automatically move that model in uh attach it to standard logging endpoints whether or not it's a flat log whether or not it's a hosted log service whether or not it's more sophisticated logging uh such as model tracking over time and things like that um and that's all in immutable deployment constructs right you're able to say oh this was exactly the sha of this particular model when i checked it in this is what happened and i'm basing it on that rather than you know and and uh no shame involved here but ssh into the production server and swapping a directory uh obviously that's not so good the second thing is you can now do things like static validation again because that is code that's been written and checked into a get a git repository uh you're able to do a suite best practices around that uh static checking linting you know avoiding syntax errors and so on um you're also able to at that point do things like check for bias and explainability and things like that again um if you're waiting to do that when you get to production you're obviously going to miss a bunch of things and then finally around reproducibility so again if your entire pipeline your and your code and your models and your data are all checked in um that's not something that someone can go back in time and fork the entire repo on giving you the ability to audit and the ability to reproduce if you go and spin up a brand new environment you can now download this and attempt to reproduce the results which is going to accelerate overall so at the end of the day you get you know what i like to say is velocity plus security for ml so um you may be asking yourself um uh you know wasn't this supposed to be a talk about security and and my supposition here is that ml ops is the baseline for security um and you're like well all right it's ml how bad could it be i'm going to talk to you about three different types of attacks today and how you can use ml ops to help defend against them so first we'll talk about how your attacker gets your model to lie to you um i'm going to walk through for those that already know ml this is probably going to be pretty redundant but for most folks you know it's interesting to explore a pretty basic pipeline so what it might look like is this right first you're going to ingest a bunch of data uh you're going to engineer and split that data into both training and test data sets and then you'll train on that data and then ultimately validate against that data into the future and then finally once that the system is trained you package it and roll it out for survey this is a very very standard um ml ops pipeline that you see here so we're going to build a circle detector in this case what i'm going to do is i'm going to stick a whole bunch of circle examples into my training data and say go train on it and then i present it at the end um my my public endpoint with a circle in in this square and say is there a circle here uh and the serving model says it's a circle so you know thumbs up this is working great now let's say i come along and present uh the same input except now i present a square and it still says it's a circle you're like well that what what the hell happened there um uh you know it was working so well before these two are obviously different one should be a square and one should be a circle i don't understand what could possibly be going wrong in this case it so happens that i didn't realize that all the bits for a circle happen to be inside the bits for a square and you know foolish model person that i am didn't realize that i have to have counter examples it's got all the same pixels the model's like ah it's good enough that's a circle and so the failure i made here was not presenting enough counter examples for the circle so that this is a circle and this is not a circle in order for me to figure out what's going on um now you're like oh wait a second you know this is a foolish example this never happens in the real world i have of course bad news for you so here for example this um comes from uh a uh paper where they introduced a great framework called lime um and and i should say all the papers that i reference here um are going to be listed at the end i there's absolutely no way i could give you this talk without uh incredible work by researchers all across the industry so in this particular one you can see here they have uh wolves and huskies and you're trying to predict one versus the other uh and in this case i presented a whole bunch of examples and only one wrong that's pretty good right well it turns out that when i look at this uh in the center column is where i was detecting um huskies and on the right and left i was detecting wolves uh and in the center part you can see well i'm kind of pulling out the features that look like a husky but on the right it's really only the stuff around the edges and you're like what's going on there well when you drill into those pixels it turns out that what i really built was a snow detector and so i was able to detect huskies in one case but in this case uh you know snow in the other not great uh and that's very much very similar to that um uh you know circle versus square problem that i was talking about earlier uh you know it of course gets even worse uh here for example i try and detect for ships but in fact when you look at the pixels for that that were the most meaningful in determining the model it actually was detecting for water uh here you can see i was detecting for trains but instead of trains i was actually detecting for tracks and uh in one of my favorite ones uh here you're detecting for horses um but uh you're actually detecting for copyright notifications apparently uh in a lot of stock footage uh well of course there's copyright notification and that became the most interesting pixel uh you know it continues that that obviously is is just uh academic research here you can see tumblr tried to create a not safer work detector um it failed in both directions unfortunately uh both with false positives uh and with false negatives that green square was actually something that was not safe for work um in this case i was able to or not i was the researchers here were able to attack the model by simply adding an owl in there so um you had that not safe clearly not safer work object uh that that was blocked appropriately but then simply by adding an owl to that picture uh you can see it there very funny looking owl uh it was able to pass the picture so again both false positives and false negatives can be an issue here um it gets much worse especially as we start getting into more security domains here you are on the left-hand side layering stickers over the top of a stop sign and they're able to convert it uh from the in the model from being a stop sign to a speed limit sign obviously that's pretty bad for self-driving car uh and on the right-hand side they 3d printed a shell and put it over the top of an actual tortell a turta turtle excuse me and they were able to uh force the object detector to um declare it as a rifle uh some more fun examples are fun and quotes obviously here uh you can see airplane uh you know if you were building a satellite system that you wanted to count planes on a runway here's a model that you might use unfortunately they were able to by simply putting stickers on the backs of these airplanes they were able to hide them right so again no big changes no changes to the model just layering a sticker over the top and they were able to hide them um and here you have um you know it's not just about hiding it can also be about confusing it in really pretty dramatic ways here um the researchers were able to layer over glasses on uh people and convert them from one person to another um again you know pretty bad stuff and and like i said these aren't just research examples um at the end of the day this actually impacts people's lives in pretty dramatic ways so here you can see amazon's face recognition system uh falsely matched with 28 members of congress um uh when when looking for mug shots again you know not great um and the problem isn't even just that right like it's not like this kind of stuff is now layered over all of society and you know you're not even able to opt out even if you want to so are you terrified yet um the answer to this is there are many tools to defend the first is you just you know watching a talk like this and being aware of these problems but you can add many more edge cases and detect for bad data uh you can add different evaluation metrics and and uh basically train different models to kind of be an adversary against yourself i can't stress enough how valuable it is to have your own red team attacking your models constantly either automatically or you know on a regular basis and alerting monitoring but most important is where you take your production data and you layer it back into your model training um so that you're able to say whether or not things are are getting better or worse and and fix them over time um and like i said there are lots of tools to defend here but um uh the most important part is being able to iterate on that loop as quickly as you can and the way to do that is with a pipeline and when i say a pipeline it looks you know very similar to pipelines you probably already use it's something like ci cd github actions or jenkins or whatever you like adding a series of modular components mixing and matching picking whatever makes sense for you your domain your industry and then continuously measuring that models do go stale very quickly and you are the the number one person who should be monitoring that and updating that constantly don't let your adversaries be the one to alert you that your models have gone bad uh so to walk through what a basic pipeline might look like and all this code is available um i i uh you know did a lot of this work in kubeflow and you can go to the kubeflow website and see a lot of those things but there are many many examples out there in this case you take something like jupiter uh where you're going to probably be doing most of your model training in the first place just for exploration you check that into github and then in this case we're going to use kubeflow pipelines uh cubeflow pipelines just think of it as a vessel where you can drop in whatever makes sense to you in this case we're going to use kale kale is a tool that's available for kubeflow that automatically converts a notebook and metadata into uh components that can now be run at scale um in this case uh python so you take that then you continue along with a feature engineering step in this case maybe i'm going to use an external spark cluster to do my feature engineering and so cube flow pipelines will call out to that spark cluster spark will go through and execute on it and then at the end it will hand back to um qfl pipelines to continue when it's done i'll do the same thing with my training and in this case i'm going to call a second cube flow pipeline could be in the same cluster it could be a different cluster it could be in a different cloud um but i'm going to call out to that pipeline kick that pipeline off excuse me and then hand back at the end then i might do a hyper parameter sweep with a again a another kubeflow pipeline where i'm going to scrape against hundreds of different pipelines that i kick off simultaneously get the you know result back pick a winner and then ultimately serve and in this case we're going to use scale to convert that serving artifact into something that's valuable um and scaled underlying all of this is going to be metadata storage metadata could be stored anywhere it could be stored on a nosql sql database could be stored in just s3 it could be store or you know your blob of choice can be stored as flat files it doesn't matter but you do need to record all the results of the inputs and outputs and executions of each of these steps and layered over the top of that you're going to layer infrastructure excuse me layered under that is infrastructure where you're going to pick and choose the infrastructure that's most appropriate to your domain so that's the first example and an example of a model again all the code is available out there this is a fairly tight talk so i didn't have enough time to actually show you the the demo but but it's all out there and you're more than welcome to go and look at it uh the second attack is where an attacker takes your models so in this case what an attacker is trying to do is download a copy of your model now it could be a very good replication of it because they just want to save a bunch of money but more often than not they're going to use that as a way to lever up and attack your model in other ways um uh it like i said you can also do crazy things like extract private information we'll get into some of that extraction attacks later um but you can all you know one of the things like i said is is really being able to construct adversarial examples if if people just attacked the public model then you would see that happening and may be able to take action they were able to download pretty good copies of your model then they may not have to attack your model very hard at all in order to create constructed adversarial examples we're going to talk about two attacks the first is what's called a distillation attack and the second is called a model extraction attack and i'll walk you through both these so the distillation attack uh that's where you have a black box model um over there on the side and you begin to probe against that model to understand exactly what what passes and fails when you present an object to it so in this case i'm going to start presenting objects here i present a heart and i get a failure then i present this pentagon and i get a pass then i begin to pat you know present a lot of examples and in this case you know i'm starting to get kind of a shape for what this model is looking to detect and then i present a lot of examples and i get this final one here and as you can see um you know it's starting to form a shape anyone want to guess what it is uh it's a nina simone detector no it's not it's a triangle detector and um and from this now that i have all these hundreds and hundreds of examples up here i'm able to use those to create a duplicated model because that becomes my training and test validation sets um and so now i can have my people or my adversarial attacks go against my duplicated model and never touch that original mod now the question is is like okay you you said you said you could go off and do this attack how many attacks would it be before you got to for example 99 accuracy in this case it's pretty scary right um they were able to reverse engineer amazon's logistic regression um uh model in just you know under 2 000 queries and they were able to get to the big ml decision tree survey and under you know just about 4 000 queries so at the end of the day you get to about 99 accuracy in about 5 000 queries which is two queries a minute over you know two days it's incredibly hard to detect for something like that um you know and you're like well maybe i'll block a vpn or any or you know something like that but these numbers are so small that if you did block that you're going to block a lot of real traffic and even if they did it doesn't prevent me from spinning up a lot of vpns and attacking it that way so these are really hard to detect the second one is a model extraction attack where you start to probe against things like nlp now nlp stands for um natural language processing uh you know the new generation of nlp started with some of the work from google around bert you may have heard of very very popular and really changed the way a lot of people were going to uh went through and created transformers for looking at language uh this is the underlying basis for many of the cognitive service and and other apis out there uh in this case uh this is azure's and and um you know we base it off a derivation of this and the way it works is uh it's really really cool stuff what you do is you present a whole bunch of information and then you're able to ask a pretty high level question and it gets at an answer and as you can see here there's a lot of like words around this there's different instruments there's different uh you know the the actual answer is split in the sentence and so on uh yet it's pretty good at doing it uh you're seeing better than human performance the way they attack this is kind of similar to the distillation attack you basically present it with a series of words in this case it's random words or you could also do wiki words from the corpus itself and you present those and the model doesn't know that you're bad at coming up with sentences it just tries to respond and in this case as it responds it tells you some of the underlying architecture of the model so in this case it comes back with a response and and i'm able to use the the success or failure of that and those words to map to the original model and really i'm able to do it in a lot cheaper way right here you can see one tenth of the number of original samples i was able to get the 72 percent accuracy or the excuse me authors of the paper were if i use one to one i get to 86 and if i use 10 times as many i get to 89 so you know very very accurate uh very close to the original of 90.6 uh and while the original models may have taken many millions of dollars to train and that is not an exaggeration uh here you can see uh you know i'm able to get one million translation queries for under two thousand bucks so pretty cool stuff um at the way to use ml apps to defend like i said it is really hard uh your best bet is to continually be updating your model to get better uh and you use that with that continuous flow from the final day of final results into the original um but realistically your model will be stolen what's going to be far more important is how does how to host it how to serve it how to have high performance and things like that uh shorthand is spend the majority of your engineering time on the left hand side not on the right hand side so that's the the models now um let's talk about getting at hidden data um you know a malicious user may you you oftentimes will train your models against you know proprietary or private information uh and then you'll put your models out there and you're like well you know it's all are you know opaque uh the reality is it's probably not and you were probably already having this problem today um you just weren't aware of it so in this case for example um here are some hidden data leakage examples uh left-hand side this is a ways recommending where it might look into my history and find um historical meetings that i went to and leaked that out um maybe it looks in my network graph here's uh twitter for example and again these are all mocked up examples but i promise you they're happening in real life where your friends graphs might be evident where it's recommending who i should follow and it's telling me the other people that that should uh that um i should follow uh or maybe it's just community information here you have a an exercise app uh that is recommending places for me to run and in this case it leaks out um where other people have run and in this case this is a private military base and they were able to figure out exactly where people could run inside the military base indicating the architecture of the base now like i said there's nothing so bad that it can't be made worse especially with ml and in this case it's called secret memorization where you're able to uh prompt the ml model with a starting phrase and then it will fill in the rest uh and again you know this is something you're probably already having a problem today but in ml models it's very opaque and the worst part is like i said it's oftentimes coming from my probably private corpus of data uh which is really not great now here it's just recommending the rest of a sentence but what if it gets into more private information for example let's say i'm able to prompt the model with the start of an address and then it fills in the rest of the address or the start of my phone number uh or my relationship information uh or again where it gets really bad is when you start to prompt it with known things please put it on my visa which has this as a starter code for visa and then it presents the rest of the number or uh you know my social security number is and this prefix for a social security number is known basically if if you know where i was born roughly like geographic part of the country um these first two numbers are clear and this last number comes from a very small set usually between uh you know two and six digits uh and so then just by prompting on this the model might spit out the rest now i have your social security information uh there's some cool ways to defend against this um uh you know one of the things that happened is is injecting a canary where you inject into your training data something where where there's a known uh you know figure here and i'm able to detect for that figure automatically really creative stuff but it's not a solution right the model still contained my private information i need to now go back and defend against it and you're starting to see other things around things like differential privacy which will begin to help but but the honest gotten it problem here is that a model that is working perfectly is going to leak data it's supposed to uh because it's supposed to feel like me um okay so summary is you know ml ops gives you a lot of best practices repeatable workflows and so on uh and it really is the basis for security it's not free but it certainly is uh you know a a nice framework for getting going um and it's up to us all of us who are doing ml training and other data solutions at scale to go and work on this on behalf of all the people out there in the world the front line workers and the real estate people and the environmental engineers who don't have phds in statistics it's up to us to provide these tools in a safe secure and repeatable way truths that can't be avoided you will be attacked your pipelines will have issues and the game is all about mitigations of harms and quick recovery and uh that's it oh here's the uh final thing as promised uh here are all the papers feel free to screenshot it send me a note uh there's my twitter there's my email um send it my way i'd love to hear from you and uh with that thank you very much [Music] you