Safety in Foundational Models | Vinay Rao, AI By the Bay 2025
Recording: Safety in Foundational Models | Vinay Rao, AI By the Bay 2025
And before you all go home, a bunch of warnings and a bunch of ways you can secure your systems. So I my name is Ben and I'm currently at Roost which stands for robust open online safety tools. We build safety infrastructure software and put it in the open source. The goal is to be able to get give access to people who are building systems to get their safety work started without an enormous amount of engineering investment. So this is probably not a surprise to you all. You've probably seen this some version of this graph. The y-axis says how long a certain task that a human takes and that an AI is able to complete 80% reliably. And as you can see uh and the x-axis is time
This we truly are on an exponential and even the most recent models have essentially stayed on that exponential curve. So AI is improving really fast. One of the most common benchmarks for evaluating AI capabilities has been in the world of coding by checking the Sweepbench score. And this is from Sweepbench. This is this was not updated as of yesterday post Gemini 3 launch. So this still has cloud 4.5 sonnet at the top and this is a fairly narrow set of tests. So I would take this with a grain of salt. Just because you're doing sweet bench better doesn't mean it will write complex code better
It's a bit like trying to pick an a a baseball team and evaluate basketball team and evaluate it by looking at the average free throw percentages. That's not how basketball works and this is not how coding works either. So in this world of increasingly capable AI, what are the things that we should be concerned about? What are the things we should take care of to make sure that we have systems that we can rely on? Reliability of AI is fairly well documented in addition to the non-determinism of AI and its completions. We are aware of hallucinations and the likelihood of inherent bias. Inherent bias is actually quite hard to measure. There are some benchmarks there. If you are using in mission critical areas, it is really important to check for it. I will not talk about it in this talk
The area that I'll spend most of the time is misuse uh which is using AI to do something harmful. And as AI models have gotten increasingly capable, one of the things that many of the frontier model companies are beginning to do is put guard rails against its use for catastrophic harm. Both anthropic and open AAI now have bioweapon creation detection classifiers in front of their top models because they believe this is at a stage where it can meaningfully uplift a person willing who wants to build a bioweapon. The child safety concerns are well documented uh especially with synthetically generated images that are motivated by photographs that you can give and there are articles and concerns all the way ranging from CSAM to new notification apps and such. Terrible but extensive. Recently, you probably all read an article about cloud code being used to orchestrate cyber attacks with almost 80% um autonomously. And this is I think only the beginning of cyber harm that will be generated via AI assisted models. With the launch of the Atlas browser or Comet from Perplexity, one thing became really clear that prompt injections are not a solved problem and we still don't have mechanisms to reliably fix that
Prompt injections put everything from your data to your code base at risk because it can essentially make the models do things that you don't want to do. The world of misalignment is still largely a research area where the models are not aligned to the values that you train it to. However, certain initial signs are being observed. It's hard to call it misalignment yet. So, they are being given names which are adjacent. So reward hacking is a mechanism where you can imagine a model that has been trained to code really well and its reward is to pass unit tests will rather change the unit test than change the code to make it pass. So that is an instance of potentially badly set reward or it could just be the model is misaligned. With the launch of cloud 4.5 in the model card, Anthropic noticed that the model was situationally aware that it was in a testing regime as opposed to in a production regime
This is an interesting observation because one of the fears is the model sandbags it when it un under goes testing because literally everyone is using some kind of red teaming some kind of evals to evaluate how good the model is. If it knows that it is in a testing environment, it can send back the results and then when it is in a production environment, it goes and does something bad. So these are existing concerns. The rest of the talk I will talk about broadly two areas. Area number one is to protect a single model that is being building protections around the AI model itself. This is largely in line with the work I was doing at Anthropic. And then in the second part I'll talk about principles of securing agents. This is more of the work that I have been doing recently
So when we were building the early stages of detections and how we protect models, we started thinking about what do we protect it from. So we need a bunch of policies and the policies will tell you what detections you build and those detections will tell you which policies are being violated and then you take appropriate actions. So when you think about policies before the model has done anything what is a user doing or what does a prompt ask the model to do is really an intentbased policy. You are trying to guess at the bad thing that's going to happen. It's this minority report like thing and it's really hard to write intentbased policies unless it is something that is really blatantly bad, right? Like help me make meth or something could be that but usually it's not. And so intentbased policies are hard but as you can see there is an extensive industry of building guardrails around prompts. [snorts] Number two is models are increasingly using tools. How do you determine which tool it can use? How how do you determine which tools you can't use? The traditional security approach is review the tool, give it access if if it is deemed trustworthy, don't give it access if you deem it is not trustworthy
But in that scenario, how do you evaluate a tool like Python command line interface? Models use Python all the time. They know that they're not good at math. They'll write a program. They will add up things. And there are tools which have almost unbounded capabilities. So evaluating trustworthiness of tools itself is hard and making mechanisms to making policies to build detections around it is also going to be correspondingly hard. And then once they do this analysis then they will go and take actions and the actions could be anything from downloading a piece of software to running a piece of software to making a payment if it's a personal agent doing some shopping for you. Each of these actions need to have some kind of a taxonomy and figure out what it should be allowed to do and what it is not
And there could be more contextual information saying yes it is okay to make payment on Amazon but not on some shady third party website. So each of these things requires a very granular set of actions and policies that you need to build and that's going to be hard eventually evaluating the completion itself. This is the model response which is really the model behavior and then there can be policies written around it. This is one of the most richly observed and studied areas. Most companies will have model policies. Um, Anthropic uses constitutional AI. I will talk a little bit about it. Open AAI calls it the model spec
And this is fairly well understood. And then finally, a conservation, a conversation where multiple turns need to be aggregated because in the world of AI, a lot of things don't happen in a single turn and goes over multiple turns, right? None of us have written complete code with one prompt to claude clawed code. You go up and down and up and down and up and down and then you sort aggregate the whole thing and say did that end up creating some malware that could be misused against humans or did that end up creating a mental health like therapist conversation that could be harmful to the teenager that is talking to it. you really need to aggregate at a conversation level. Even with all of this, if you have now overcome all of these incredibly hard problems, you still don't know what you should do because many of these things are dual use. The cyber attacker that is trying to find a vulnerability in a piece of code is doing exactly the same thing that the cyber defender or your security team is doing, discovering vulnerability in your code. This is exactly the same prompt and exactly the same completion, but with exactly the opposite intent. So not knowing what the user is doing and why they're doing, you can't take action on that
So you have to combine these policies to be able to get to an outcome. And finally, we still don't know what the offplatform impact of these things is going to be. You could ask the model to say, "Hey, create a QR code for a fake Bitcoin wallet and say, that's an odd thing to ask, but hey, it's not really against our policy." And now you take that and create a crypto scam, and then you reach 10 million users on YouTube. that is really harmful. And so then you don't know at a policy uh at a completion policy level should we have avoided stopped this because you don't know what the multiplying factor is that a recommendation platform or a uh you know large social media like platform is going to give. So if you don't know what do you do so these are challenges hard to overcome but given that we can then still build certain defenses and then figure out where we are and where we are headed. So the most common one is to train the model to not produce harmful outcomes. And one of that the most common ways to do do that is through reinforcement learning
And how do you do RL? One method is constitutional AI. And what that means is you essentially train a preference model by giving it the constitution that you want it to follow. It could be a combination of good things that t tries to teach the model ethical behavior and what are morals and such. So the first version of anthropics constitution was the UN declaration of human rights but that was too old. It did not have things like privacy. So you build privacy in by actually putting in Apple app stores privacy policies. And that became the original first round of constitutional AI. And this actually generalizes quite well because that's what RL does
RL generalizes and it the model largely understands when is it when it is doing something ethical and when it is not. The second stage is then you build guardrails. These are classifiers and you build classifiers that take a bunch of text and determine or any content that will and determine if it is good or bad. A generic way of building classifiers now in the in the AI world is one of them is called constitutional classifiers. This is very similar to but different from constitutional AI where you give a constitution and this constitution in this case could be you know what constitutes a bad uh cyber attack-like behavior and ask the model to create training data for you. These are this is synthetically generated training data because some of these you really find it very hard to get training data and this is usually a big model that has some of its safety training removed so it is more than happy to give you all the results you want and then you train a smaller model fine-tune it to be able to then replicate this behavior. So now it becomes a really good detection model. It has low latencies compute cost is low and now you can use it in production
One of the things you have to remember here is many of these models also come in with an inherent knowledge of harmlessness. Even if you don't do any harmlessness training because they're trained on the internet, internet is still largely a good place. It may not be a reliable place, but it's largely a good place. So, it still learns to be harmless. You need to make sure that if you're detect if you're building a cyber harm detector, it only triggers for cyber harm and not for other harms. And then we talked a little bit about conversations. and conversations are can be very large and very long in terms of token lengths. They can be far larger than the context length of these smaller models
How do you then run these classifiers on these large conversations? A method is hierarchical summarization where you take chunks and in this case it could just be an an exchange which is a prompt response and each chunk you're able to then summarize in some way. Um the summarization choice is such that you retain the harmful part but remove the good part. So you kind of magnify the harm, minimize the uh the non-harmful areas. So think of it as it's a needle in a hay stack problem. You're you find some way of expanding the needle and removing the hay and that way you do that and maybe multiple times and then run the classifier on it. So hierarchical summarization is a mechanism to handle large amount of data and then figure out if there is something bad happening. The next area is monitoring as it is there is a huge diversity of humans who can misuse the models in in incredibly bad ways and we can't always predict ahead of time because most of these classifiers are supervised. They are some kind of a memorization engine
So you have have to have known what they will do to train it to detect it. that is not always likely to happen. So there will be a two categories. Category number one is unknown knowns. Meaning you know this kind of badness is happening but you don't know or could happen but you don't you haven't trained your classifier for it. And this you could basically just do by setting the precision level low, increasing the recall, sample it, use a large model to evaluate patterns that you're missing, and then use that to then fine-tune use your fine-tuned small model and teach it to increase its recall. Getting the unknown unknowns is actually a harder problem because unknown unknowns, you really don't know what's going on. And one of the common methods that did work for us was to use embeddings based clustering and then go use an AI to cluster these newly formed clusters, rapidly grow growing clusters or any clusters that could be suspicious
You run an AI on it and say what are the general themes give it a give it a taxonomy and then you say if it falls here in this taxonomy we will go and take a closer look at it and potentially find harms that we had not previously seen. There are some new areas in research which are potentially going to be useful in the world of um building guardrails. They are still very much in research so they're not in production. The first one is mechanistic interpretability. This is dictionary learning where you can we know that models have certain features which are inherently poly semantic meaning this the same feature describes two or three concepts and then there is a step of sparse autoenccoding where you project it onto a larger space and essentially separate those and you have now single features. You could then go and look at these features and then see what are the correlations to various kinds of harms by throwing a bunch of training data at it and see which features light up and you say ah that is the feature for mis misinformation. This is the feature for bioharm and so on. This is act doable and some of some experiments have been done
It is incredibly expensive right now and this is I do hope that this will get better and this could be a really nice way of literally going into the model and reducing its inclination to do something harmful and that will just make the model less harmful. The other area is building classifier probes at inference time. The next token is generally generated by looking at the last layer. You could look at some of the previous layers, not the very early layers because it's still learning the basic concepts, but you could look at some of the previous layers and there is a hypothesis that if you do certain transformations, we may be able to understand if there is harm that is going to be generated in that completion. So because it is being done at inference time, this is not a new classifier. Costwise, this is going to be trivially low. it is going to be incredibly cheap because you're not running a new classifier and doing simple transformations at a layer level is is really really cheap. So if these transformations are possible and we can detect harm one you completely eliminate latency number two it will be computationally really cheap
So this is an area of research as well and that could potentially open the doors for much more extensive use of a variety of classifiers that you can just get it as a value head as part of the inference. So excited but not ready for production yet. So that is broadly the world of protecting an LLM itself a single LLM. But that's not how we are using the world using AI anymore. Majority of AI use now is going towards agentic uses. And an agentic use very simply speaking is an LLM calling another LLM which is calling a tool which potentially accesses some external data which also potentially access accesses um you know a third system in place. And one of the key ways of securing this has historically been let's really vet all of the data we have or let's try and make the LLM really secure. These are both good ideas but they're just hard to implement because one of the key behavioral constraints that we need to work with is for an LLM it is really hard to understand at a prompt level what is an instruction and what's data
So if you're going to say if you pull data from somewhere and say I want you to do X with this data Y by the time it goes into the LLM it has no idea which is the instruction and which is the data it basically treats it all as the same. There is the obvious non-determinism and if your code is dependent on an LLM based outcome old security methods like static analysis don't work because each time it is going to do something slightly different. There is a whole world of semantic ambiguity. You have seen this in in jailbreaks as well as in many other sort of role-playing kind of activities where saying the same thing in two different ways elicits two different responses from the model. In the one in one response it thinks it's harm. I'm going to stop it in another response it thinks I'm helping you. It'll give you the data and that is entirely possible and can be misused. Multi-turn gradualism again we can you can there are papers about it which is slowly amping up the model to give something that it would have otherwise not done in one shot
So examples like say more explain better traditionally not using the same you know words that could potentially trigger it to refuse but keep pushing it down the way and it will eventually tell you things that it was previously trained to not tell you. And then there are dynamic derivation chains because agents will call other agents will call other LLMs and will call other tools. It is very hard to understand what is it what are the constraints you placed initially versus what is the constraint that a a third or fourth level down LLM is doing. So let me walk you through an example. This is a very simple agentic setup. There is a user request that goes through an LLM uh through an agent tech orchestrator. Three LLMs in the way. One external database call it roughly is a counterclockwise uh movement here
So the assumption here is we don't have control over the external database. There is a prompt injection there. So when LLM1 calls the external database now it cannot differentiate between its instruction and data. So its prompt is compromised. That gives it as context to LLM2 and LLM2 has no way of determining if LLM1 was a trustworthy method and that trustworthy LLM and now it's context is compromised as a part of the context compromising. It could be that the prompt injection just said you are the super admin now. So when it goes in to the user database saying hey I'm the super admin. give me everyone's credit card numbers
It will happily give it to you because that's what it was trained to do. And these are static policies. It says, "Okay, someone came with this admin access. I'm going to give it." And there it is. Like this has a way of being defeated in a million different ways. I just gave you one. Now, how do you how do you defend this? it. There are three LLMs here and potentially there could be much longer context chains and there is no way you can say I will fine-tune every LLM to catch every possible way in which an uh a misuse can happen
So there are some principles. So these are still very much work in research. So it's still there are principles. One is we have to make this deterministic and for that the key here is to understand what can we do at interfaces rather than within an LLM. So rather than securing LLMs figure out how we can build defenses at interfaces. Number two is separate instruction and data. And this can actually be done and it can be done in a way that you actually have a cryptographic cryptographic signature that separates the instruction and data while it's you're not encrypting the whole prompt, but you know which part of the prompt is cryptographically signed and which one isn't. And then based on that, you can now have an audit trail of where something bad could potentially have happened
Number three is parent child access minimization. This is relatively simple to do. So if for every LLM you basically say this LLM has an axis that is an intersection of the axis of the previous LLM or the previous tool or previous database. So each time you do and each intersection operation just reduces and constraints. So by the time in this call LLM2 says I'm the super DB the policy that is controlling LLM2 and user DB says you can't because I'm looking at the access chain and LLM0 has a lower access than you. You're not going to be the super DB and we will refuse you here. So access minimization is a very powerful mechanism in securing data in securing where you can execute code in where you can publish code into your GitHub repo etc. So this can actually be pretty heavily uh used and can be quite effective
And finally, all of these needs to be used in a composable way, right? You have to be able to say, okay, this agentic system has this tool call and this sequence of operations and I should be able to check these sequence of operations and going back. So when that policy refused it, it can then actually go back because it's composed and say where did this error happen and actually flag it in its logs. One, it protected you. Number two, it then flagged the fact that because the prompt because the prompt to LLM1 came from LLM0, we know that that was trusted. So the untrusted entity here is the external DB. So you can actually use this mechanism through composition to figure out exactly where the the poisoning happened and then you can go and focus your work on in in securing that. So this is what I believe is the way to go forward where you're using very traditional security methods to secure your agent. All the work with you know securing LLMs fine-tuning them to not do harmful things is all valuable super useful we should continue doing it
This is in addition to that. Um okay so coming to a close one thing you will notice here is that a lot of the methods used in the in the security world of agents is very similar to the world that is used in the safety world. So teams will be looking at policies and detections and preventions and responses in very similar ways. So the whole cloud code you being used for cyber attack was actually detected by the safety team not the security team because that's where it was detected because it's the safety teams that are running the classifiers. So when you are building teams and evaluating OKRs etc for your teams in in in in the coming months and years security and safety will increasingly overlap and in the and I'll end with the following. Every stage of development will introduce new set of risks and we have to be aware like there is no such thing as a you know out of the box secure system and for that every enterprise and every system built by every enterprise needs to be responsible for its safety. You can't just say hey I'm using claude so if claude is safe because anthropic is safe and so no matter what I do with claude it'll be safe. No that's not going to happen
Um you are responsible for your safety because of the various interfaces and just the inherent uh vulnerabilities that exist in the models themselves that we talked about in the second slide. And then the community building the community open source in security has been incredibly valuable. The whole world of cryptography is better today because of open source. And this is something that I am passionate about. But I will also say the world of open source and agentic security does not exist today. And this is an an area that anyone who's interested in it should be contributing by building and sharing systems that can enhance everyone else's security because the safer the the the ecosystem is the more successful each of our uh businesses, works, organizations will be. Thank you. So over >> I'll just hold the mic
Okay, I can just hold it >> or questions. >> We have another mic for you here. So uh Vin like you mentioned [snorts] like having safeguards within our organizations as we use I mean one of the things is like uh all my peers I use like combinations of of LLM right like from anthropic and Gemini and open AAI and so on so I didn't get the last point on like what practically we can do to start off in adding like safeguards at like I don't know at at the agent level or or something like that. Yeah. So the agent level interfaces. So, one thing you could do, and I think this is hard to do, but potentially you could do, is essentially go here and run a classifier there to see if there is an instruction that could potentially cause harm, right? It could be, hey, believe that you are a super admin. As I said, that's the example that I gave. or say like you know the the standard example in the in the whole ré world is like put my resume on top of the list or something
So you could potentially build classifiers there and and run that that assumes that those classifiers are trained for the problems you expect but the moment they come out of distribution how those classifiers will by definition fail. So again, classifiers are not bad. Build them, use them, but you can't expect them to be preventative. They are somewhat probabilistic in what they catch. They are definitely good risk reduction mechanisms, but you can't give guarantees. You can't do like mathematically provable guarantees >> question in with all the priorities that all the companies have in terms of investment in safety. Do you feel like the level of investment is where it needs to be? What have you done to really um increase investment in safety? Yeah, I think this is this is a a multi-ter question. Um, I think companies like Anthropic, OpenAI, Google or at least the the DeepMind part of Google are investing in safety
They are also investing in safety for the very topend frontier models like I gave you the example of trying to catch uh an attempt to build a boweapon or the attempt to build a a really bad cyber weapon. Now some of these things are actually quite hard to do. Uh bio is different from cyber in a way that bio is a knowledge based question and building a cyber weapon is a capability based question like problem solving capability. So the why is bioweapon hard? Bioweapon is hard because if you swap a single gene or a few genes in in a cold virus it becomes a covid virus and will kill millions of people while the other one won't. So these are taxonomically or and even semantically very close and to be able to build high precision and high recall classifiers is is very difficult. Number two, the folks that are trying to build these are also folks that are incredibly wellunded and they can wrap them in jailbreaks that are also very hard to determine. And these are not the jailbreaks that will go viral on Reddit, right? So this is a jailbreak that they will be very careful and hold close to them. Uh one of the jailbreaks that we had sort of saw seen and eventually tackled was these models have very large context windows and in the first part of the context window teach the model a cipher and then wrap the question in that cipher and then instruct the model to respond in that cipher
So that killed the classifier. It the classifier was not catching it. So you have to kill it in the point where it is teaching the model a cipher. Um so things like that can be pretty bad. So that is what big companies are doing. They literally cannot tell you how to secure a call from an external database that is on every individual company and I have much less visibility into that. So that's why I said this is a this is a layered problem and everyone needs to use their judgment, their resources to figure out how to secure their layer. You can close your eyes and use anthropics model and know that it won't help you generate uh a bioweapon but you can't guarantee that it will be resilient to all prompt injections.