LLM Avalanche: Charles Martin: AI is hard; LLMs can overfit

LLM Avalanche: Charles Martin: AI is hard; LLMs can overfit

Recording: LLM Avalanche: Charles Martin: AI is hard; LLMs can overfit

thank you thanks for coming some of you some of you may know me from LinkedIn because uh last week we had a bit of a scandal I published uh post by a paper at MIT where someone had trained a custom llm that they thought could Ace all the MIT courses and you know to my surprise it went absolutely viral I think and now we're almost out of 850 000 Impressions something crazy like that and you know it was a extremely controversial the MIT actually had to retract the paper and it's so astounding you know you have MIT Stanford um MIT Stanford Harvard Boston University's all these people trying to train this llm you know basically training gpt4 to and claiming the capacity entire MIT curriculum which you know obviously it's a little fishy but you know give them a benefit of the down it turns out that they may have actually overfit their llm so this brings up a really important point you know llms are hard to train you can overfit they hallucinate they have lots of problems so we need better tools so when I want to talk to you today is about the Weight Watcher tool on a Weight Watcher is a tool for doing data free diagnostics for deep learning it's an open source tool we have over a hundred thousand downloads it's been featured in kdd icml the paper itself this actually is taken from a snapshot in nature so it's been featured in nature and what it does is it allows you to analyze a deep neural network without having access to any data no training data no test data it takes the model takes away take it looks at the way matrices it does spectral analysis and then analyzes the eigenvalues and it uses some some techniques from theoretical physics and computational Neuroscience to tell you something about your model and someone surprising you could do this but it actually does work and it's been a subject of research I've been doing this now for almost 10 years and you know with Chachi PT I decided to take it out of the garage and show you how it works so here's an example this is an example of looking at Bloom so this is the largest blue model and I compare the small one which is 560 million parameters to the Big Bloom and what you see is that there's a layer quality metric this is one of the variety of metrics the tool gives the quantum metric Alpha and the closer Alpha is the two the better the layer is trained and what you can see is that and smaller Alpha is better now if the alphas go into the red region the layer is over fit and if it goes into the yellow regions or once it's under fit what you can see is that both the blue models are actually trained fairly well but as the models get bigger and they're trained on more complicated data sets the alpha start approaching two now this is actually predicted by Theory we have some theoretical papers that show this and this is related to some very deep ideas in computational Neuroscience but as a tool you can see these signatures of emergence and and this is the idea so these signatures of emergence which tell you just how well your model is going to perform and I'm going to show you that this actually you can actually learn quite a lot from this so let me give you an example here's an example of the very famous or a falcon model that was just released compared to llama and you can see that the Falcon layers all are within our quality metrics maybe there are a couple that might be a little underfit but pretty much even the small Falcon model looks pretty good when you look at a llama however there are a large number of layers that have quality metrics in the yellow zone which mean that those layers are under fit What's Happening Here what's happening is that the models have been trained with these chinchilla scaling laws and this healing walls are right but the data sets are not there's something wrong with the Llama data set it has too many duplicates too many problems the Falcon team what makes their model so special for those of you aren't aware of it is that they spent an enormous amount of time curating the data and removing duplicates and you can see this you can see that the quality of the Falcon data set is better directly from this plot now this is something which is actually published in nature that I mean I'm not you know not I'm not hallucinating right this is actually based on our Research into why these models work so this is a good example I'll give you a a deeper dive you can drill down and look at how the information flows in your model so you look at Falcon versus llama and you just sort of sort of look at the green one the green dots and you can see that the Falcon information flows fairly solid it the layers are all within the white zone the Llama model you know it was it was doing well for a while and then about halfway through the layers the layers just stop converging the information doesn't get to the layers and you can see this using our tool and it's very clear what's going on again we have many many examples of this GPT computer vision models this isn't specific to llms but this is what we're doing here's another example you can also use the tool to look directly at the Deltas of your model so we for example we can compare the very popular vicunia model to dromandera and you can see that bakuni again is trained pretty well most of the alphas are in a range between two and six is where you want them the drama Dairy layer it looks like it was trained well but it looks like there's a lot of Alphas that are just too small and that we suspect is a signature of overfitting so you've actually over fit the wires and we have several metrics and techniques you can use to detective layers are overfit in fact we're pretty sure you can even go into your data set and you can find the specific data points that you're over fit on and then remove them so that's one of the things we're working on right now here's something which you may really find I've seen as you know that a lot of these models they lie to you they hallucinate they make stuff up turns out that our quality metric is anti-correlated with the truthfulness that is the better the layer is trained the better your model is trained overall the better is it lying to you just like a person and again this is It's a weak correlation but you definitely see it and and again this is basically I just went to the hogging face open llm leaderboard poured out the truthful Nexus you can see that using this you can evaluate different base models to see how well they're trained and then this is just sort of an example of you can see that if you look at models like Bloom chat or GPT neox these models appear to see the signatures of emergence they're trained much better and yet at the same time they tend to hallucinate why does this work you know I'm giving you a tool it's an open source tool you can use it to evaluate your models to tell do you have enough data in your model are your layers over fit are they under fit we can do many things to the tool the reason it works is from a theory and physics called self-organized criticality and if you look at this if you look at the activity of spiking neurons you take actual real neurons and you culture them and you study their spiking activity they exhibit a property called self-organized criticality and if you fit the correlation structure of the spiking neurons they exhibit parallel Behavior it's actually these tools from computational Neuroscience which Weight Watcher is using to analyze the correlation structure of your models and again this is something this is based on something called the critical brain hypothesis which was proposed by per Bach back when I was a grad student um and it's it's called self-organized criticality and this is fairly well established science if you talk to people who do computational Neuroscience or experimental Neuroscience they're aware of this and what we're doing is using these techniques from computational Neuroscience to analyze the weight matrices of actual neural networks of llms of fine two Del Limbs and we can give you actual quality metrics which tell you how well your model will do how well each layer is performing if a layer is under fit or overfit if there are anomalies and we can even provide you remediation strategies so the goal of this talk I came here today because this is an open source tool you know I'm trying to get it off the ground as a product but right now it's an open source tool we have over a hundred thousand downloads um it's you know it's still an alpha tool it's not even release one yet it's under the Apache License it's again it's been published in nature and a number of other high-profile ice um AI journals we have a Discord channel to support the tool and we're just looking for people early adopters collaborators maybe some investors if anybody's in the audience but we're looking for people who are fine-tuning their own models or training large models from scratch we're going to provide the tool to you and help you show you how to use it to detect these problems to monitor your models in production and to fix problems that you can't find any other way and that's the talk hey [Applause]