Taming ML’s Wild West: Practical Model Signing with Sigstore on Kaggle| Mihai Maruseac, AIBytheBay25
Hello everyone. I am Mihi Anusak from Google open source security team and I'm going to talk to you about practical model signing. Basically how we can tame the wild west of ML. So let's think about three years ago when JP got launched the future became reality. We started seeing a lot of models, a lot of AI applications. Everything that we saw in science fiction books and movies became a reality. But there was still a little bit of a problem and that is if I can reuse the XKCD cartoon, we have all of the top of the castle where we have all of these fancy tools and so on. But that tiny piece at the bottom the AI supply chain security, AI security
And why is this a problem? In general, people think there is training for a model and that there is an application that uses that. But actually there is also a step in the middle. So you have a big company that trains foundation model to do any language processing on the internet. Then you need to fine-tune the model to work on the specific application that you want it to work on. So for example, fine tune a model for coding, finetune a model for healthcare and so on. Then maybe there is also quantization and other fine-tuning and so on. And then finally the model gets deployed into an application and at each stage there is a separate team of developers that work on this. So you have ML developer team one for foundational training another team for fine-tuning and then software engineers that build the application around the model and in between these teams the model get put into storage into some model hub and then taken from model hub and put back into the next stage
And now we see that there are many places where malicious insiders could compromise a model and could create havoc downstream. And let's actually do a demo for that. I'm going to have a lot of demo in the talk. So I am going to switch to this one where I am going to build a trainer and I'm going to train a PyTorch model. This is a very small model that does digit recognition. So nothing too fancy. and loading PyTorch and it should train. First epoch finished
I have four more epochs taking a little bit of time. The plan here is I'm training this model then I'm uploading it to a model hub that I can also control and then I have a user that downloads the model from the model hub and uses the model. One more iteration and okay, now the model is trained. So the next step is to upload this model to the model hub. And now I'm going to switch to the user and I'm going to download the model and then use it. This is a PyTorch model. So use it. So because this is a digit recognition, I'm passing an image and I'm asking the model to tell me what image is in that what digit is in that image
The model will work. Sorry, this was the happy part. Okay, now let's switch to as an attacker. So now I'm going to put on malicious insider have I'm going to go. Okay. So now I'm going to be on the model hub and I'm going to run a simple script that I stashed in here to hack the model. This is a PyTorch model. So I'm going to hack that one
And this takes a little bit. And now let's switch back to the user. Suppose the user wants to download the model again and wants to use it. And this time because the user doesn't know that the model got hacked actually all of their data now gets sent to a third party, their passwords, they get hacked, their bank credentials and so on. So this is what we want to protect. Let me switch back to the slides. This is also important because if you go on hugging face and you search for the llama model, you see that there are two different providers. One of them is Metal Lama
The other one is Facebook Lama. One of these providers is actually the meta company. It's the real one. The other one is the researcher trying to prove that just looking for the model name doesn't tell you anything about what is in the model. And you can get compromised by just downloading a model that you think is the real one. Sure, in this case, if you look at the other statistics like number of stars, number of users, presence of model cards and so on, you can find out which one is the legitimate model. But all of these are metrics that can be gamified and can be faked. And there is actually research that says that in ML models that have more stars are more likely to be malicious than models that have fewer stars on average
Okay, so coming back three years ago, we saw the future become a reality, but there was no way to verify trust and integrity of the models. And traditional approaches for traditional supply chain integrity don't work at the scale that ML has. And that is because models are not just on single file. In general, they are collection of files and they are not just on single format. And each time a new model format gets released, you would need to go and work with that to find out how to protect the integrity of that. Furthermore, as a trainer, when I release the weights for a model, I want to make the model as as usable as possible. So I would release the weights in multiple formats like PyTorch, safe tensors, tensorflow, keras, and so on. But as the user, I only care about one specific format
If I use pytorch, I only care about the weights in the pytorch format. I don't care about anything else. So we need to support a way of signing more more files than what we verify and also we have the scenarios when we have fine-tuning quantization and so on when only a portion of the model changes and everything else stays the same. Okay. So going back we have some questions that we need to answer. We need to know who trained the model. We need to know if the model has been fine-tuned since that point. So also need some lineage
This model came from this previous model. We also need to know if the model has been tampered with between training and us using it. And then for all of these answers, we need to make sure that we can prove the answer. And we have now a solution for this. If you run pip install model signing, that is a library that will support model signing and you can use that. And I'm going to show you a demo on how that works for the scenario that I just showed before. So let's switch back to the trainer. And this time I'm going to run train pytorch model but at the same time sign it
So basically this is going to run the training code again and at the end of the training process it's going to compute a hash of the model and sign that and for signing here because we also want to protect the identity to prevent to determine the identity of food train the model. The signing process itself is going to use an OIDC flow. So it's going to ask me to go to a browser and get a code and that code is going to be used in the signature will turn will contain my identity. So let me switch to the browser for that. So let's say I'm logging in with GitHub. So it it it will have my GitHub identity and I copy this code and I paste it in here in the signature. And now this model is trained and signed and I can upload it to the model hub. So let's upload the model
And let's upload the model signature. And now from the point of view of the user, they will still download the model. They will download the model signature and then they would run a command to verify the integrity of the model. And in this command we have the identity of the trainer. This is my email that I use to log to GitHub and the identity provider which in this case is GitHub because I use the GitHub my I use my GitHub account to sign the model. So if I run this, the verification succeeded. But let's say somebody lies and says actually this model has been signed by mihi.mosak at GitHub or I could pass another identity. In this case, when you try to run the verification, it will tell you actually the identity doesn't match
And now let's switch back to the trainer. Let's try to the hub. Let's try to hack the model again. If I download the failed model and I try to verify its integrity again, I'll get an error. It will tell me that the hashes are mismatched. So I will see the compromise. Okay. So now let's switch back to the slides
Okay. So right now AI is an intrinsic part of our tech stack and we also have a way to verify the integrity of model and the identity of who trained the model. Uh we launched a model signing 1.0 in April 1.1 last month and we also separated a specification for the signature into a separate repository such that this can be targeted from other languages outside of Python like Rust and so on and now we need to drive adoption of model signing and build upon this and basically the model signing format the specification we sign a collection of arbitrary files so this way we don't need to hunt every new model format and incorporate signatures into the format itself we can just sign everything that is under a specific directory. This will also help us sign data sets in the same way. And also we are private key infrastructure agnostic. We support signing with six store which is like the demo that I showed you. But we also support signing with traditional signing approaches like signing keys, signing certificates and so on. And basically this allows us to build uh provable integrity and trust in the supply chain
If you go to this QR code, you'll get the repo for the MS format. The previous one was for the model signing library. Okay. So I'm going to show you an example on how signing works with six store and ML. So we start with the model trainer the in the middle of the diagram and as the model trainer trains the model into some pipeline the workload identity token connected to that pipeline will get sent to a sixto certificate authority and the six door certificate authority gives you back a certificate that is valid for only 10 minutes. You can use this certificate to sign only once and after that the certificate is invalid and even if it gets leaked and somebody else tries to sign in your name that signature would be invalid. Once you get the certificate you can sign the model and the signing event is stored into a six door transparency log that you can monitor. The important part about the six door transparency log is that you can monitor the log with your identity
And if you know that you released 10 models, but you see 11 entries in the transparency log, that means that somebody trained a model into your name. So your identity got leaked on the internet and got stolen and you know exactly which model to revocate. As part of the store transparency log, you get backlog inclusion proof so that you can also monitor uh verify integrity of models offline. And now you upload everything to the model hub. So you upload the model, the signature, the the shortlive certificate and the inclusion proof for ver verification. Now the model hub itself can verify the signature and displayed a verified badge and users if they trust the model hub and they see the verified badge, they can just use the model. If they don't really trust the model hub, they can download the model, download the signature and verify itself. And I'm going to show you a demo for that
So this is Kaggle, but rather than showing you screenshots for that, let me switch back to the terminal and let me switch back to the trainer. So now I'm going to upload this demo model, but rather than uploading it to my hub, I'm going to upload it to Kaggle. So as you see here, Kaggle is telling me that the model doesn't exist. So it creates that. And you also see that it creates this Kaggle signing JSON. This is Kaggle upload process signing the model for me and uploading it. And if I go to this link here, this goes to Kaggle and you will see here six status pending. Right now Kaggle is verifying the integrity of the model that I uploaded such that I cannot send a model with a fake signature
And after a little bit this six status will change to verified. Uh I'm going to pause a little bit just for time and I'll come back to this one. Okay. So that's the demo on Kaggle. We also have integration with the Nvidia model hub and I'm going to do a demo for that one. So while the Kaggle is verifying my model here, let me switch to the Nvidia model hub. So let's say this is a model from Nvidia. Sorry, this is on dark screen
And I don't know how to change it to white. But basically what's the important part you see here there is this signed badge. And this means that I can go back here as the user and download the model from Nvidia. So I can use this command ngc registry model download version to download the model and this will download the model from the Nvidia hub. And then I use another ngc command to download the signature. And this will download the signature from the Nvidia model hub. And now that I have both the model and the signature, I can verify. And for Nvidia, I have to use the certificate, the public signing certificate that Nvidia uses
So the only thing that I change in my command is verify certificate and I pass a certificate chain. and I pass the Nvidia signing certificate and this validates that the model that I downloaded from Nvidia is exactly the one that got trained. Now let's switch back to the Kaggle one. I think the verification should succeed. Yes. So right now because verification succeeded Kaggle changed the status and six door status now says signed and there is this attestation link that gives you a notebook which you can run still on Kaggle infrastructure and verify the signature later. So there is this middle layer of trust where the files might be tampered but the notebook but you can still trust the Kaggle infrastructure to run any notebook. And if I go back into this into this model into Kaggle the the Kaggle directory into signing JSON, we see what how the signature looks like
It's basically a JSON file and we have the DC envelope that contains the files of the model that get hashed and everything else. But we also see the verification material. That's basically the transparency log inclusion proof that you can monitor. And if I go here into the TLOG entries and I copy this log index, I can go to this website called record and I can search for that log index. And I should I should see the entry that got added to the transparency log. And we see here under validity, we see a certificate that is valid for only 10 minutes. And at the bottom we see that this is signed with my Kaggle username account using the private email the private user ID. So even if I change my Kaggle username to something else that I cannot fool people to say that now I'm meta and I release model in meta name
And what I can do now is actually search for this email. And it should share. It should tell me all the models that I have signed with my that I uploaded to Kaggle. And I have signed. There are 40 of them. If I know that I only released 38, that means that two of the models have been released by somebody that compromised my Kaggle account. Okay. Okay, now let's move back to the slides
Okay, so that was all on model signing. So what we prevented so far was a malicious user compromising the model while the model is in storage. That's what we did with model signing. But we are not done yet. The malicious user could just shift a little bit left. So a state action could compromise a model trainer in some big company to make sure that they compromise a model as the company releases it. And in order to do that, let's actually show you a demo. The last demo for today
Uh I am going to switch to No, I was on the right one. I'm going to switch to this Jupyter notebook where I'm running some setup cells to set up the infrastructure and then I'm going to run this function to train a model. In in this case, I'm going to train a TensorFlow model, but it's the same it's the same thing. So, train a TensorFlow model that does digit recognition. It shouldn't take that long. And after that, the malicious user inside the company is going to run an additional script, an additional step. They're going to actually hack the model and then upload the model to the model hub. And they are going to also delete this to cells in the notebook and tell the users the model into the model hub has been trained from this notebook
You can inspect it. You can inspect the notebook there there is no mention of hacking. You can just trust the model. But actually they and now when the user downloads the model and tries to use it, they get compromised. So what we want to do here is actually modify the notebooks to generate a provenence of the data of the cells that got run and in what order they run. And we have a proof of concept for that where actually this was a notebook that I introspected where I recorded the cell execute every cell that got executed, the time it got executed, the hash and the code of that cell. And now we need to make this tamper proof and in a way that is both human readable and machine readable such that machines themselves can inspect the accuracy of the provenence document. Okay, switching back to the slides
Basically this is what we want to do. We want to build an attestation for training and make sure that that attestation is temper proof and verifiable and this was the demo that I showed you. So basically what we can claim about ML models we have the integrity claims that is the signature basically making sure that the model has this specific hash from the training and users can verify it. We have provenence claims where you can say this model has been trained on this notebook or from this specific data set on this TensorFlow platform version and so on. And then we also have generic ML properties where we can say this model has scored this much on this benchmark data set or this model has this model cards at the release time and nobody can lie about them anymore. And at the foundation of all of this is signing and hashing. So that's the work that we've done so far. But we also want to progress again for the future and make sure that all claims about the models, data sets, metadata and so on, they are temperroof
So once they get once you made the claim, nobody could alter the claim. The claims are tied to an identity. So making sure that you always know exactly who made the claim. And we also want to make sure that all of these claims are generated by the tools used by the ML developers. We could for example train models on GitHub actions and rely on the fact that GitHub action artifacts are immutable and can generate also provenence but no ML developer is going to train models on GitHub actions. So we actually need to go to Jupyter notebooks and other workspaces that ML developers are using and what we want to do in the future is extend signing adoption. So more model hubs should support this like hugging face and so on. Uh integrate with model frameworks imagine saying torch.load load into a pipeline and PyTorch will see that this model is signed but the signature doesn't verify
It will stop loading the model at that point rather than getting you compromised and integrate with model pipelines sign agent card sign MCP servers and so on. We want also want to build on top of model signature toward like salsa for ML a package manager for models. So you could say like ML install some data set or ML install some model and you'll know for sure that that model is not compromised and it comes exactly from who trained it. And then all of this tamper proof metadata can also serve to introspect the lineage of a model. For example, you could detect that this data set uh is out of life is expired in this jurisdiction. You could uh ingest all of this metadata and find out which of the models that you are using for that jurisdictions are now illegal and you need to retrain or prevent people from accessing them. And on a diagram on a diagram form it will look like this. Every time we generate something we sign it
So in this case we sign data sets we sign models we sign model sources. Every time we use something we verify the signature for that. And I also show in this diagram some compromises that get fixed by doing this. And that is all for this. If you want to get involved, there are two working groups in the open source that work on this. One of them is open source security foundation. There is an AML working group. Uh we're meeting next Monday at 10 a.m
Pacific. And under this working group, we have special interest groups that are working on separate areas like model signing. We just met this morning and the next one is in a month. Um we have safe MCP for doing the same thing for agentic development and there is a cyber reasoning systems where we are trying to use AI for security. So using AI to find vulnerabilities using AI to patch code and so on and on the other side there is coalition for secure AI that is more towards standardization and enterprise use. Uh we are meeting every Wednesday so next Wednesday should meet again. I think I got the date wrong there. uh we just published a model signing white paper at that QR code and there is a second part uh work in progress where we are looking at adoption levels and maturity levels for supply chain integrity protections and if you want to get involved either of these forums you can also talk out to me and I'll give you more details on that and just as a moto we want to make sure that today's intelligent creations don't become tomorrow's security nightmares because we are moving fast and forgetting Yes
Sorry. [applause] >> Thank you. Mihi questions. >> Thank you for the wonderful talk. I think that's very applicable to basically every ML project. Um I have a number of machine learning models uh we open source in the past and we want to open source more of them. Um I would like to design those models. Um two questions there is like uh from from the workflow perspective we have our generic model then we quantitize it and then we generate like an onyx version of that
So each of those versions would get their own signature is that correct? Okay. And then in your workflows um do you like um is a human signing that or can you use like ML like ML pipelines? We use TFX pretty extensively. Um is it something what do you use like behind your se behind the scenes on your side because then becomes like a security issue again like like what's the workflow when you say like signing um thank you. >> Yeah so uh each time you generate a new model you'll have to sign it and we are actually working under co to create the lineage chain between the derived model and the parent model. So in the signature for the derived model there will be a field that says this model has been generated from the previous model and you can refer to the models by their hash. Uh on the integration side and how to can sign how you sign them. Uh for these demos I signed them as a human but the Kaggle process when I run the upload to Kaggle that was actually using the model signing as a library and basically what we support is actually integrated model signing into different workflows and we have some work right now with TensorFlow TFX TensorFlow to integrate signing into there but we still need to incorporate like with PyTorch framework with PyTorch lighting and so on such that you can actually do do the signing as the work as the workload identity rather than the human identity. >> Any more questions? >> Thank you again Mihi
It was insightful session. >> Thank you. Thank you.