SBTB 2023: Mihai Maruseac, Model transparency for AI/ML security
Recording: SBTB 2023: Mihai Maruseac, Model transparency for AI/ML security
[Music] to paraphrase Taylor Swift it's me hi and I'm the speaker and I will talk about model transparency for model AI ml security so as you know this is a talk about AI you've heard a lot of talks about AI I use AI or ml interchangeably I know in theory they are not the same uh AI is larger than ml but for the purposes of this talk let's just consider them the same and just for contrasting the software that doesn't use AI let's call it traditional software and now if you look again at the V diagram between the two you see that there are some things in common so let's focus on those one of the things in common that we want to focus on is the history both traditional software and AI based software started first as hobby as research people in Academia built computers internet was built in Academia to connect several computers and so on and then it became commercial then became a commodity people can use it the same happened with AI but just at a much faster SP uh pace so first we had people in Academia doing research with trying to understand which images were dogs which images were uh cats and so on then you started having some commercial uses for AI like Google translate like like this and now with the CH GPT and so on you started having AI as a commodity so everyone can start using AI everyone can start building their own gpts and so on actually the history goes a little bit more as the application started being more and more mainstream you also started having more and more hackers and more and more exploits and more and more uh cyber security incidents both in the traditional software and both in the AI and now let's give some examples for exploits in the AI let's just focus on GPT based models so we know for example that you can build an evil model where you hide some malware inside the neural network this is because most neural networks are actually T in complete the framework that implements them have has operations to write to files to send Network packets and so on so you can encode your marware inside the big llm and then at some point if a specific prompt is given to the LM you can trigger that and because most models are Opa nobody can know what's happening and there is also the possibility to build like a GPT for Chan where it's a GPT training for Chan and it's going to still act normally so still give you text for text not do anything malicious but the text that is giving back to you is bad and this is not just theoretical just research we have also seen it in practice so we have seen chat GPT being poisoned we have seen poison GPT that has been actually published on hugging face to spread fake news and this becomes even worse when you start using AI to build more software so we have GitHub copilot that generates code for you what if GitHub copilot were to be hijacked and now it generates code that contains malicious users and then somebody can exploit this or even worse there is for example a demo GPT that builds a prototype of your app in minutes you just describe what the app should contain demo GPT builds it entirely for you you can even deploy it if it works and in most cases it works and this is not just commercial users this also affects high performance Computing so spack the package manager for high performance Computing now uses llms to try to guess which packages should be updated first uh such that the entire update plan won't C ises so now if the llm gets confused the entire High the entire cluster can be be brought down okay so we need to merge security and a together and actually there are several ways in which you can fill the Gap we can have security from AI we can build applications that are secure against whatever malicious Ani can build to us or we can bring security via AI we can build applications that for security that are powered by AI for example there is Socket that uses llm to understand opusc code and tell you if this that opusc code is malicious or not and there is also security for AI where we try to bring security for deploying a models and so on and basically that is what I want to focus on during this talk and the motivating example is this scenario that happened some months ago uh attackers were able to find out several terabytes of data uh used by Microsoft a researchers data as data they could manipulate that and so on but what was wrong what was worse was that in the buckets they could also see ml models and since a lot of the models are pickle files and again they are not easily inspectable somebody could maliciously modify the pickle file and then later when you want to use the model Anything could happen on your system because now it's no longer doing just just not no longer just an ml model okay so let's look at another example this one a little bit more theoretical so let's consider kagle and let's say I'm a bad attacker that I want to explode kagle and I train two models one it's evil model and the other one is the GPT for CH that we have seen previously if I upload them to kagle in the model card I will have to put a description of course I can put some fake descriptions like I could say that evil model actually detects malware and the gp4 Chan is a friendly chatbot and publish them like this and then I hope that users would download them with use the models and then I would exploit the users later of course users if they see the name evil model probably they're not going to download it or they're going to sunbox it but I can always rename the model so instead of evil model I can say it's a nice model or instead of GPT for Chan it's a GPT for friend and now users are going to say oh yeah it's a good model if especially if I do a lot of advertising and a lot of promotional material around it users are going to deploy it and use it so the problem is that on their own the models are not publicly attributable to the user so I can exploit the thousands of users and they they could say oh we detected that this model is malicious it's okay deleted the model I upload a different one nobody can attribute that to me nobody can detect all of the models that I release are bad so to solve this we actually need to look at the entire supply chain for building a model but basically we should we should protect the entire ml supply chain what is the ml supply chain let's think about what we need to do in order to train a model first we do some data set preparation we do some model training and so on now that we have a model we publish it somewhere such that users can then use it in their applications or their experiments so there are three pillars and if we Zoom a little bit more into that let's say now I'm a ml developer and I want to build a model the first thing that I do is take some data set and pick an ml framework and split the data set into the training test and the test set so here I just picked Tor FL but doesn't matter I can pick whatever so I already have some inputs the data set and the ml framework The Next Step that that I do is I pick a model architecture and I code it in the ml framework of choice so like a sequential model in tensor flow with caras doesn't matter it's the model is the data set and the architecture now I need to combine these pieces together and do a training once I train the model I publish it to some model Hub then I wait until somebody downloads it and once somebody downloads it they use it in inference for their products and now I can exploit the model if the model was malicious I can exploit the user this diagram here looks very similar to the diagram for traditional software I just switch the labels so a developer has a source repository where they write the code this code uses some dependencies you put them together you get a build for an artifact you publish this artifact somewhere and wait until somebody downloads it and uses it so if we squint a little we could say maybe we can do make my model or Basel build my model or whatever build system you want to use my model and treat training a model the same as building a normal software and going by this analogy even that attacks that we have for regular software for a traditional software they also apply for models so an attacker can compromise just the model Hub so I trained the model I was okay training the model I didn't do any anything malicious in that I upload it to a model Hub and then the model Hub gets popped and somebody manipulates the model there or I publish the model in some Cloud bucket and then that cloud bucket gets access by external attackers and they modify the model or an attacker can be even more malicious they can try to exploit the training process itself to inject uh bad behavior during the training I can also be wrong myself I can either use a vulnerab dependency that as a vulnerability and then that haer can exploit that or by mistake I can submit bad code and the model that I train is bad so there are multiple possibilities in which Mal work can be ingested in a model and now protecting the supply chain means putting bullets putting bubbles around parts of this to protect them so let's start with the simplest one let's say I trained a model and now I want to be sure that after this point nothing can happen to this model without me knowing or the users of the model would know that something happened to the model since I since I last trained it so in order to do this we are going to use a solution called signing the same as in traditional software you you build your software you sign it and then users would verify the signature before using it and for signing we selected six store which is the same as let's encrypt for software for but for software artifacts so the process that we want to do is let's say I'm the model trainer nice I the model trainer and I train the model and then in my code on I do model Dove instead of just writing the model to this there is the signing flow that happens behind the scenes so this signing flow would get my identity for example my GitHub username or my email and so on there will be like a popup to select which identity I want to go we go to Sixt CA to get a certificate and then we'll use this certificate to sign the model and put the signing event into a Sixt transparency log and return to me back okay the model has been inserted into the transparency log here is the set of here is the pth in the Merkle tree now I can publish this model to the model hub together with the transparency log with the proof of including in the transparency log and then the model Hub itself can verify okay this model has been signed I verify the signature I can put it in the model card Associated to the model I can put a label that says this model has been verified uh you can trust this it's not a malicious model it's not a model that has been just bumped over the fence and nobody cares about it and then later when users want to use the model they can either trust the model Hub the badge there that says verified or they can run the verification process themselves and signing and verifying the models is quite fast so scales quite scales very well so I try I we tested from small models all the way to big llms and we never had more than 10 seconds to wait okay so this solution only helped the compromise of the Hub but what if we want to protect the entire supply chain what if we want to record what's the what was the source that has been used to train the model or what if you want to also record the databases that have been used in the model or the ml framework for this we're going to use another solution for uh the software supply chain called Salsa Salsa basically generates a Provence file associated with the entire build process or train process in this case so what we did was we add edit a GitHub actions where you can trigger the GitHub action you can say I want you to train a model and generate provenant and this is the format for the model so here for example I can pick caras model for tensor flow but actually we support both tensorflow and py torch and other models can be supported in the future okay so at the end of the gab action run we get two things we get the model and we also get this Json file that is the provenance and people can look into the provenance by themselves or they can run a salsa verifier automatic command and verify with that let's look at the provence manually just to see what it contains so if I cut it and pipe it to JQ to see what the Json contains I see that it has some signatures so nobody can manipulate can change the provence after it was generated and it also has a payload which is B 64 encoded so next step to do is to look into the payload itself so if I look into the payload first it contains a field called subject this subject is actually the model that I trained and it also has a check sum this check sum should be exactly the same that used during the signing and verif during the signing flow for Sixt if we look more into the payload we see that we also have a builder so basically the trainer for the model and here we can record like you have used this code you have used this U training framework you have used this databases and so on we can also record them in the materials portion the materials are more if you think of salsa like the recipe for cooking something the materials are the ingredients and the Builder is how you combine them together to get the final product the model and in our case for materials we just record the repository with the commit but this should be enough to see exactly what has been used to train the model because the entire code for downloading the data set and so on is in the repository and all of this can be automated so salsa verifier can look into it and verify the provence or in case there is you have a a model that you have trained for hundreds of CPU hours and uh then you discover okay the framework that I have used has a vulnerability and now you can decide do I want to spend hundreds of CPU hours to train all of my models or I can look into this provenance and identify exactly Which models have been trained with that vulnerable framework and only train those okay so in the future we this is still incipient work so in the future we still have to do on the model signing work we want to increase performance of ashing we want to make it as fast as possible so basically users won't notice any delay at all and we want to integrate it with more ml Frameworks so instead of users having to save the model and then sign it we want the entire flow of model Dove do the signing and so on as I was describing on the previous slide and also the integration with more model hubs and model cards so if you go on K if you go on kagle if you go on weight and biases whatever you go to get a model you would see the verified badge on the salsa side we want to scale it outside of GitHub actions so for example using tecton if you train on gcp or on AWS we also want to support uh tensorflow saved models or very large models which uh split the weights into multiple files because right now salsa verifier only checks the check sum so if you sorry if you have multiple files uh and I flip the order of two files now the model no longer behaves properly but salsa verifier will still see all of the check some that it was expecting so it will say yeah the model hasn't been tempered with so this is something that we have to work in the future and and as I was saying we also need to record the data set Provence we want to know exactly what data has been used in training the model maybe exactly what images have been used and so on and there are other standards in supply chain that you can also bring to the ml word so for example there is this bomb software build of material which just records each dependency which their hases and so on and on the vulnerability side there is a Vex standard there is a CV and so on so you can imagine bringing those two to the ml word but the conclusion is that we can bring the same protections from the traditional software to the ml word and we don't need to reinvent the wheel just because ml is so popular right now and finally on the last slide I'll have four QR codes the first one is to the repository where we have the proof of concept so far the second one is the blog post about this then the third one is a op ssf open source security Foundation uh working group on ML where we can discuss these topics and other similar related ones and finally the last one is from Google Google secur framework which is the framework that encompasses this work plus work on alignment and privacy and so on and that is all thank you very [Music] much