Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 1

Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 1

Recording: Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 1

We're going to review some of the basics of large language models. I think we have like mixed um levels of experience here. So, this is if you have never heard of a large language model before or used one, um this is going to be horribly insufficient introduction to what these things are. Um if you've been, you know, using them every day, you're probably going to be uh uh it's going to be a little bit repetitive, but it's important to know like at least a little bit of how these things work. Um then we're going to cover kind of one of the core skills of building language model applications or just prompt engineering. Um kind of one of right now one of the trickiest parts of getting these applications right. And so, we'll cover some like kind of tips and techniques to to make this to make this work well. Um and then in the like as we get towards the afternoon, we'll talk about augmenting language models, which is how you take all the information that you need the language model to have and inject it to the into the model so it can solve the task that you actually need it to solve

Um so, that's one of the biggest subjects and we'll cover it um for the most time. Um then in the afternoon, we'll talk about kind of like how do we systematize this? So, once we have the pieces in place, how do we actually go from, you know, a system that kind of works as demo and make it into something that works well in production. So, we'll talk about uh a philosophy for building language model applications kind of um that's analogous to test-driven development and we'll talk about kind of the some of the operational aspects of building language model applications as well. And then in the afternoon, we will kind of like do a a walk-through of a of a code base like an example code base um of a kind of like a language model application that's actually running um in the Discord channel that uh there's a link to here. So, we have a a Discord for FSDL that has like a you know, most of the folks who have ever taken a class with us. Um there's often like pretty interesting discussions going on there. So, uh feel free to jump in the Discord here. Um it's a good way to stay in touch with folks after and also interact with our our our our uh ask FSDL chatbot as well

And so, we're going to cover today like two different two very different ways of building this chatbot. The first is like the simple easy way that doesn't work very well, which is to use the like brand new features of the assistance API that OpenAI released last week. And then we'll cover a kind of more complicated more in-depth process of building this application, you know, the old school way as of 2 weeks ago, which works quite a bit better today. So you'll see like sort of two different ways of building an application like this. So let's dive in. Um All right. First of all, just a little bit about FSDL. We really try to sit at the intersection between like what you think of as a traditional machine learning course that focuses on fundamentals and techniques and algorithms and things like that as well as covering a lot of more practical aspects of building applications

And so with a focus on like actually shipping things and not just writing papers. And we also try to mix in a good amount of state of the art. So it's one of the reasons why we we typically do these classes like once or twice a year is because we're always updating them with latest materials. And and so hopefully we'll cover some stuff that you haven't seen before. Um just a quick introduction. So I want you to start by imagining a world. Okay. In this world, it's there's disturbingly simple computer programs that can make mimic human cognitive processes

Humans are starting to form emotional attachments to these chatbots. Computers can play chess, they can write mathematical proofs, they can pass difficult exams, and you can actually even get intelligent investment advice from AI. Right. So what is the world we're talking about here? Is it the future? Is it the present? Is this like what LLMs have enabled? Actually, no. This is all from the past. This is a This is This all happened in 1965. Um so ever since kind of the dawn of artificial intelligence, people have been making claims about the capabilities of these systems. And we've seen isolated um, individual examples of human level or past human level capabilities from computer programs

Um, so, you might ask, what's different this time, right? Why why is it actually why do we feel like this is an exciting time to talk about AI and it's not just 1965 all over again? And I think the key difference is that now, rather than having a bunch of handcrafted individual computer programs that can each do one task really well, we're starting to have a single tool that can do a lot of really powerful human level or even past human level things, um, which is this tool called a large a large language model. So, what is a large language model? Um, large language models, you know, as the name suggests, they uh, they model language. Um, so, what does that mean? If you imagine a paragraph and you imagine writing a single word of that paragraph at a time, if you learned how to guess what the next word was going to be, if you learned how to predict what the person was going to write in the paragraph, you'd get pretty smart. Um, and that would be especially true if the person wasn't just writing text, but they're also writing all kind like uh, logic, math, um, chemical formulas. If you could always predict what was going to happen next, you'd have a lot of knowledge about how the world works. And so, turns out that large language models are really good at this. And they can predict a lot of things very well. On top of that, large language models, um, on like unlock kind of what I think of as like a new paradigm for computing, which is language user interfaces, right? So, in contrast to graphic user interfaces from the uh, you know, the the the current or previous area of area of computing

Um, language user interfaces, this is not a new concept. Um, it has its, you know, a Wiki a Pedia page and people have been talking about this since the 1960s. Um, I'm having a little bit of trouble with slide. Here Here go. Um, but what's really powerful about LLM's is that they've unlocked like new ways for us to interact with computers by just typing like asking what we want from them rather than needing to have a human design what is possible to interact with. So language modeling, you know, according to a lot of folks who are deep in the AI world, um, often like folks often consider language modeling AI complete. Um, what that means is like if you were able to model language perfectly like if you were able to predict the next word in the sentence as well as you possibly could then you would have a system that is, um, you know, like you could consider artificial general intelligence. So what does AI complete mean? Concretely it's a pretty slippery term

It's like not something that's very precisely defined. Um, but the core idea is that language modeling is really hard and if you could predict the next word in all kinds of different things like chemistry and math and science and history then you would have to have a very, um, condensed model of the world in order to predict what's going to happen in the future and that condensed model of in the world is like in some sense some form of intelligence. So before we get into like how to actually leverage these tools and what you can build with them want to talk a little bit about what happened last time. So, what happened the last time, you know, people got really excited about AI in the 1960's and after that. And so the the history of AI is kind of a history of like overselling and under delivering. So each of the past two or three times that folks have have started to feel the way that they felt now which is like hey this technology is amazing, the world is going to be so different in 5 years, the the hype has outpaced the actual impact. And that's led to several AI winters where people sort of felt the hangover of of of this over promise and it it led to AI falling out of fashion for long periods of time. And so why was that right? I think one of the core reasons is because, you know, the the the high expectations were coupled with amazing demos, but they were never actually matched by the product experiences that people were able to create with them

Um so, the high hopes like were never actually realized for most people. Most people never interacted with one of an AI system of this previous generation that could do anything useful for them, right? So, people eventually get disillusioned with the over-promising. Um kind of like I guess kind of a little bit like what happened with crypto a few years ago. Um And so, you know, and there's like numerous examples of of people sort of complaining about this over the years. So, I think like the key point here is in order to prevent this from happening again, um it's not just about like how amazing a demo OpenAI can produce or how many papers are getting published or like how what people say this is going to be able to do in 5 years or 10 years. In order for this technology to to do anything, it we we need to actually build products that create value for people. And the good news is this time around, um there's a lot more happening than just research. Um there's all kinds of uh like products that are being developed

Um ChatGPT itself is like one of the fastest growing products of all time. So, there's hope that this time it's actually real. Um and finally, there's tinkering, right? So, people are actually like the the most encouraging part of all of this has been if you spend time on Twitter or like talk or, you know, around the Bay Area going to LLM meetups, there's thousands and thousands, maybe even tens or hundreds of thousands of people that are just tinkering with this technology trying to build useful things with it. Um which feels really different than when I got into the field back in 2015. Um so, that's the good news. There's a lot of people that are trying to build products with this technology, and uh there's and there's a lot of tinkering happening. Uh the bad news is is in AI, this has always been true and it's still just as true as ever, the gap from like having an amazing demo, like a thing that you can create in these days in like 15 minutes that shows the promise of like of what you're trying to build, the gap from that to having an actual working product is huge. Um so self-driving cars are a classic example of this

Uh you might remember back in 2016, 2017, there's a ton of hype around self-driving cars. Um you know, there the demos looked like these things were basically ready to go out into the world. People thought that we were, you know, uh a handful of years away from never having to drive again. Now in 2023, six years later, uh we're just starting to see kind of the earliest phases of this technology actually coming to fruition. So, a long hard six six years, hundreds of millions, billions of dollars spent to go from what looked like it was already working to something that is just barely starting to work. Um and so that gap is actually maybe bigger than you remember because uh while people were, you know, the last time people got really excited about self-driving cars was, you know, five, six, seven years ago, um the first time that people got really excited about self-driving cars was actually in 1988 um when uh researchers demonstrated that you could have a a vehicle drive itself on highways um all the way back then, right? So, um I think like the hope would be that we're in the 2017 era of large language models or potentially even past that. Um and then the fear is actually maybe we're still in the 1980s. Um and uh and so let's let's hope that we're not there

But, you know, again, the I think the thing that would point you to believe that we're further along is that there are um a handful of breakout products powered by this technology. Um so, you know, ChatGPT being probably the most prominent, uh but GitHub Copilot being another like really powerful example for those of you who have tried it. Um and then another kind of like favorite of ours at FSDL is Descript, which is a a video editing tool that's powered by LLMs to make video video editing way easier. Um, so there's not, you know, I I think maybe when we first taught this course uh this version of the course back in April, I was hoping by now there'd be dozens, hundreds of LLM powered applications that people were using every day. Um, there there are not yet, but there are a handful. And so that's really promising. So, what we're going to cover today is like how to build these like great LLM powered products. Um, if you want to try to help avoid the AI winter or build an amazing product that couldn't have existed like maybe even a year ago

Um, it's a lot more difficult than just building the first demo. And so we'll talk about some of the ways that you can you can kind of get from that first demo to a real working product in production. But before we get into any of that, um, we're going to talk a little bit about just kind of some of the basics of large language models. Like what is this technology that we're working with? Um, first of all, you know, one thing you might be asking is like why do I even need to know this? Like isn't an LLM just an API that I can call like any other API that I've called in the past? And I would say like 80% of the time you can you can more or less treat it that way. Um, most of the time when you're working with LLMs it's a pretty good mental model is it's just a black box that can uh that you know, you'll develop intuition for as you interact with, but it can, you know, take in some text as an input and generate some text in response. But the other 20% of the time when that intuition starts to fail you, it can be really valuable to know at least at a high level what's happening under the hood for these systems. Um, and so what does that give you? It can give you like sometimes as you build with LLMs you're going to start to see you're going to start to hit their limits. Um, even though the limits are really high, you'll hit them very quickly

And so if you can if you understand the way the models work, you'll start to build more intuition for when they might fail and why they fail when they fail. It can also help you find bugs. They're still, you know, the the technology is getting closer and closer to something that you can just plug and play out of the box, but there's still kind of like subtle bugs that you'll encounter as you use these things. Um like tokenization. And uh and some of the kind of like core intuitions about machine learning um are still really valuable to know. So, LLMs you can you can you like most of the time when we interact with them, it doesn't feel like what it felt like to do machine learning 5 or 10 years ago. Um but a lot of the but under the hood LLMs are still machine learning. And so some of the like kind of core principles of machine learning like uh overfitting and underfitting, which we'll talk about in a second, still apply um just in more subtle ways

So, let's first talk about machine learning at a high level. Um in traditional programming, what happens is we a developer um builds a program and that program takes an input and it produces output mostly in a deterministic way. In machine learning, the role of the programmer is different. So, rather than producing the rules of the program itself, instead the role of the programmer is to produce training data. Um training data gives the uh the the algorithm examples of the behavior that it's supposed to have. And then once the model has been trained, then it's able to look at new novel inputs um that follow similar pattern to the ones it was trained on and look at those and produce outputs. So, if you've like spent any time in machine learning, you'll be familiar with like many different types of machine learning that people use. Um kind of at the at the the highest level, there's basically three kinds

So, there's unsupervised learning. In unsupervised learning, what you're trying to do is you're trying to learn the general structure of uh some data. And the reason why you'd learn the general structure of some data is because you want to be able to produce similar data to what the model was trained on. Um and so that's kind of like that that's the closest to what language models are doing today. Second type of machine learning and probably the most widely used one in practice is supervised learning. In supervised learning, what you're doing is rather than learning the structure of the data, you're learning a relationship between two different types of data, Um between an input data like an image and then a label like you know, a a description of what's in the image or a translation or a transcription of some audio. And then finally there's reinforcement learning. Reinforcement learning is probably the most different type of machine learning from the others

And in reinforcement learning you're rather than kind of operating on some data that you're given, you're instead learning to interact in an environment in the world. And so reinforcement learning has been applied to like mostly these days video games. But it's also relevant in the language model world which we'll talk about in a little bit later. So these days you know, that's historically like if you look at machine learning you'll see those three types. These days there's mostly just two that are used in practice. There's supervised learning where you are you know, taking some input and you're mapping it to some deterministic output. And then there's self-supervised learning where you're doing something similar. You're taking an input and you're mapping it to an output

But in this case that output is what happens next in in in the sentence or in in the game that the the AI agent is playing. So how does this work? There's a lot of different algorithms that you can use for machine learning you know, dating back decades and decades. You probably heard of a lot of these. Logistic regression is a simple type of machine learning algorithm. Support vector machines were really popular type for a long time. Decision trees still widely used in practice for a lot of applications. But one is becoming dominant and that's neural networks which is also called deep learning. So what are neural networks? Neural networks are a machine learning algorithm that's inspired by the human brain

And the way that the brain works, you know, if any of you are are neuroscientists or biologists I'm sure I'll I'll I'll completely butcher this but at the highest level the brain is composed of billions of neurons. What the neurons do is they receive electrical inputs on one side and then they transform that into electrical outputs which are passed to other neurons in the brain and ultimately by composing many many neurons you're able to produce very complicated types of behaviors. And so you can also think of the brain itself as having like high-level inputs and outputs that are transformed through the process of like billions of neurons interacting with each other. So for example, you know the high-level input is you might see a cat kind of running across the the floor in front of you and the output that that might produce is you might say there's a cat. And so there's a really complicated process happening in between those two steps but the the inputs and outputs are very simple. So how do we formalize this in machine learning? In machine learning we also have a concept that's called neurons but the neuron is like a highly simplified mathematical model for what's actually happening in the brain. So rather than transforming electrical impulses instead the the neuron in a neural network basically does some like really simple linear algebra to transform the the inputs into an output. And so the inputs in a neural network are um are basically like a vector of numbers

So the this like the simple input that you that you might want to put into the to the model needs to be transformed into a vector of numbers. And then at each step in the neural network it's you do a matrix multiplication where you take that vector of numbers and you transform it into another vector of numbers um and as you do that more and more times with some other functions applied in between you are able to produce really really complicated transformations. How complicated are these transformations? Well, it turns out that neural networks like with some assumptions the main one being you need to have one other thing called a non-linearity happening in addition to just the matrix multiplication, you're able to actually model any function. So neural networks can model any function that you would want to model. So this is a very general class of algorithms. And so you know, the mental model that you should have when you think about what an LLM is doing is it's taking a vectorized representation of the sentence. So it's a transformation of the sentence into just like a long list of numbers. And then it's doing a bunch of linear algebra

It's like multiplying that that vector by a matrix and then it's multiplying the resulting vector by another matrix and by doing that many many many times it's able to transform that that input in arbitrary ways. So that's kind of like the high high level intuition for what a neural network is. So the I think the the relevant question is like, okay, if a neural network can represent any function, if it can do anything, then how do I get it to do the thing that I want it to do? How do I get it to have the behavior that is important for me? And the process that we have to do that is training. So in training you take some data. That data is usually consists of a set of inputs and a set of labels. In computer vision those inputs might be images and the labels might be you know a description of what's in the image, like a cat. In language models the input is the sentence and then the output is the next letter or the next word or the next token in the sentence. Then we take we we take the this like potentially massive set of data that we have and in language modeling the data is in fact massive

And we sample just a tiny little batch of that data. So we might have billions and billions and billions of sentences that we're training the model on, but at any given point in the training we're going to sample just 10 or 50 or 100 of them. Um we use the current version of the model, even if that model's totally untrained, and we make a prediction. Um so we say, "Okay, what do I currently think the next uh the next output is?" Once we have that prediction, we're able to use that to compute a loss function. So the loss function is a mathematical way of formalizing like how close was our guess. Um so we predicted that this is, you know, an image of a dog. Um in fact, it's an image of a cat. Um that's that's uh that's an error that we can that we can then use to improve the model

Um once we have that error, once we have that loss, we're we um use this a process called backpropagation to adjust to um take that loss and use that to adjust each of the matrices in the neural network. Um so that loss uh the difference in what the target value was, like what is the actual answer here, and the answer that the model currently predict is predicting is um backpropagated through each of the matrices in the neural network um by basically using really simple calculus. Um and that produces an adjustment to each of those matrices. Um so what's happening is the model over time gets incrementally better and better at guessing what that output is um because we're because we're essentially doing like some really simple calculus to um determine how the matrices should be updated to make the model's guess closer to the true output. And you do this over and over and over again, um and the average loss, like the average difference between the predictions and the outputs, goes down over time um until eventually you have something that looks a lot like um the output that you wanted to have. So like sometimes it's uh sometimes I reflect a little bit on like how wild it is that um just doing this really really simple process of you know having random predictions and then doing a little bit of calculus to try to make the predictions closer to the true outputs produces something that feels as intelligent as language models. So pretty pretty amazing phenomenon. So another part of how neural networks are trained that's I think relevant to understanding language models is in uh in machine learning in general um you don't train on all the data that you have available

You instead split your data into a subset that's used for training and then two other subsets that are used for validating the trained model in different ways. um The reason this is important is because models are really powerful and language models have tons and tons of weights or parameters in them. um Really big matrices you can think of it as. And so the effect that that has is that models can are very good at memorizing the data that they're that they're fed when they're trained. And so really common phenomenon in machine learning is you train your model it looks like it's perfect and then you try to make it generalize. You try to make it work on any other data other than the ones that were in your training set and it performs so much worse than what it was initially than what you thought it was going to perform based on training. And the reason is because the model is memorizing the specific examples in the training set rather than learning how to generalize to other similar examples. So to avoid this we have data that we you know we know the answer but we are not actually training the model on and we use that for for validating whether the model is doing what it's supposed to be doing on novel data points as opposed to the data points that we fed it in for training

So this is the reason why I think this is like one of the conceptual parts of machine learning that's important to understand for LLMs is because this same phenomenon applies to experimentation with your prompts. Um so if you if you've spent time doing prompt engineering um then you probably had this example where you you know you have a few types of inputs in mind like a few documents that you want to summarize and you work on your prompts and you improve that prompts and then uh it it's doing better on those three documents that you tried it on and then you tried it on like a fourth document and you realize it's actually doing much worse. Um so this is a similar phenomenon where um you know anything that you like anything that you use to tune the model anything that you use to try to like improve the outputs of the model um it's like the um the result that you measure on those things you should make sure that it works on other things as well cuz ultimately the goal of these systems is to generalize not just to work on a handful of examples. Um so you know one one thing uh one challenge of training models is that it's really slow. And this is particularly true with like modern deep learning models where often times these are trained on millions hundreds of millions of or even billions of inputs. Um and so pre-training the model is like kind of the core training process where you start from nothing a model that doesn't understand how to do your task even a little bit and you end up with a model that solves that task pretty well on this massive amount of data. Um and so pre-trained models are things like GPT-3 GPT-4 um or any of the open source models that you that you might have interacted with. But one of the nice things about deep learning is that once you have a model that's been trained on this massive amounts of data you are able to um just keep going with the training process um on a much smaller amounts of data um which is maybe the data that is relevant to your task um rather than the one that was relevant to the task of the people who trained the original model

And since you're training on much less data this process is much faster and you can uh get pretty good results quickly. And so you know, these days for most of you, you may never actually train a model from scratch at all. Um you may actually almost always start with a model that someone else trained, and then just adapt that model through the exact same process um to work for the task that you're trying to solve. And uh one of the great things about, you know, working in deep learning in 2023 is that there's tons of these pre-trained models available. Um so, you can go on Hugging Face, and you can find thousands and thousands of models that someone else has trained that you can um that may not work out of the box perfectly for your task, but you can take those models as a starting point, and you can use a lot less data to make them work for the task that you're trying to solve. Um so, Hugging Face is kind of the best place to go look for a pre-trained models if you're trying to, you know, find a good starting point for your task. Yeah, 180,000 of them. It's crazy

And this is actually I think this number's out of date. It's probably over well over 200,000 now. Um one of the other great things about working in deep learning in 2023 is that uh before around 2020, pretty much any task that you could think of had its own like bespoke neural network architecture. And if you went to a conference like NeurIPS, a lot of the um which is kind of the main um conference for deep learning these days, there uh like a lot of the focus of the research community was just making incremental changes to model architecture, which means like changing the um the way the way is that the like the matrix multiplications are arranged um in order to incrementally improve performance on specific tasks. And so, computer vision, natural language processing, speech, translation, these all had their own bespoke model architectures that people put a lot of effort into creating. And so, if you're trying to um get up to speed in any new part of the field, there's a lot to learn. But now, um pretty much all of the field is converging on a single neural network architecture called the transformer. And so this is really powerful because it means that the a lot of the the tooling, the infrastructure, and the knowledge is starting to consolidate

And so you don't have to learn as much in order to start building a good model for your task. So this all started with a paper called attention is all you need. Um and this is kind of uh like very groundbreaking at the time. Um and a lot of people when this paper came out kind of saw that this was likely to become a sort of a dominating force in the field, but it's it's difficult to underestimate like how much impact it's had in 6 years. So I'm not going to review like the details of this architecture, um but what I will talk a little bit about is like why does this why does the trans like why is the transformer become dominant? Um so the transformer is as becoming dominant because it's very general-purpose, right? So it can represent any function. Um you uh you can easily optimize it to solve whatever task you're trying to solve, um but both of those things are true for deep learning in general. Um the amazing thing about the transformer is that it's highly efficient to train, um which means it's super parallelizable, uh which is which makes it like very very well suited to training on large numbers of GPUs, um which is, you know, like allows people to put to devote like tons and tons of computing power to training these models, um and you know, spend in some cases like tens or hundreds of millions of dollars to train a single model, um which was with previous architectures was just never really possible because the it didn't scale well enough to for that to to make any sort of sense at all. Um unfortunately, we still mostly don't really understand why transformers work though

Um so there's a lot of work that's that people are doing in the research community around um what's like often called interpretability. Um Yeah, here we go. Um and so a a lot of this work is coming out of a group of researchers at Anthropic. And if you want to like try to build a deeper intuition about why transformers work or what's going on under the hood, I'd recommend checking out some of their papers. But, um unfortunately, like I think this is at best a highly highly incomplete picture of what's actually happening inside a transformer. And um I think it's fair to say that like most researchers feel like the uh we don't actually know what's happening inside of these of these models, which um can create a lot of problems, but is is uh is the reality of the technology and it's probably not going to get any easier. Um okay, so we I intentionally didn't talk about the transformer architecture just for the sake of time. Um but, you know, you might be asking like is it actually worth spending the time to understand the way this architecture works or maybe even to code it up yourself

Um so, I would say it's like really not necessary. You know, you can if you have a high-level intuition of machine learning, it's going to be helpful for you. Um and beyond that, you can most of the time just treat it as a black box. But, um it's also not very hard to do and it's pretty fun. Um so, I would recommend doing this. And it will like deepen your intuition a little bit for what's going on and um potentially help you avoid that like that one random bug that could have sucked up two days of your life um if you hadn't understood this stuff. And so, uh Andrej Karpathy has a amazing sort of set of YouTube videos um re-implementing GPT-2, an earlier version of, you know, GPT-3 and GPT-4. And the implementation is like less than 400 lines of code from scratch, um including like all of kind of like the what's happening under the hood

Um so, you know, this is like a a pretty low-effort kind of weekend project or, you know, if you have a slow day at work or something like that, it's a it's a fun thing to do. All right, I will pause there. Um see if there's any questions on kind of like fundamentals of machine learning. I know we went through this really quickly, but um hopefully this gives you a little bit of intuition if you haven't seen this material before. Yeah. Is there a sample validation set? Yeah, so the question is examples of validation versus testing. Um, so if you recall we um, rather than uh, we rather than training on all of our data, we instead split the data into three subsets. Um, and the reason why we split it into subsets at all is because if we didn't have any data that what the model didn't train on, it would be impossible to know whether the model was going to generalize or whether just memorize that training data

But one question you might have is like, why three subsets? Like why do we actually need three subsets? Why not just have two subsets? One for training and then one for validating. The reason why we have three subsets is because um, often times when you're training a model, you don't just like press train and then you're done. Like, you often need to try a lot of different things um, in order to see whether the model is like in order to find the right uh, model architecture or the right set of parameters or hyper parameters for your algorithm um, or the right initial data set or like all these other different factors that go into training a model. Um, and so every time you train that model, um, you need to like to in order to tell whether it works, you evaluate it on this held out data that it wasn't trained on. And so it turns out that if you do that enough times, um, and enough like isn't even that many. It can even be like a dozen or uh, or a hundred times, then um, even though you weren't actually training on that held out data, by selecting the hyper parameters or the model architecture that works best on that held out data, you are um, sort of unintentionally overfitting to it. Right? So like, you you your human intuition about which model parameters are working best is selecting the one um, that works the best on the whole the held out data that you're evaluating on a lot. Um, and so the reason why we have a third held out data set called the test set is to sort of put a check on that um, on that process of overfitting

So even if you are like trying all the different things to make the model work as well as possible on the on the validation set, that's sort of first held out data set, um you still need to check whether it's uh like whether it's working on a third held out data set to avoid the the problem of um I just guessed too many times and I eventually got the answer. Yeah. Uh you you mentioned that uh Hugging Face has 180K uh different could be more, of course. Uh So, if I have to use that in production, all right? Is is it the right place? Uh should I be concerned? Concerned about using Hugging Face models in production? Right. Um I would say I would be concerned about using any model that someone else built in production um because you you don't really know like the the key thing is even if you're using a pre-trained model, um you still need to validate that it's going to work well for your task. Um and so we'll talk about not in a ton of detail for Hugging Face models, um but we'll talk about like kind of some high-level ways that you can uh you can try to test out whether the model that you're grabbing off the shelf works for your task or not. Maybe I'll take one more and then want to keep moving here. Uh hi

Uh my question is mostly about uh usability or like large language models where we at the at the moment. Um I work for uh Walmart search and when I know when our team looked at this um at this point, it seems like it's not feasible with current pricing model. Like for example, you know, we uh probably partner with Azure and uh the pricing scheme is basically it's too expensive if if you hook it up to every user query. Uh where do you see like um the what's going to happen in this sense? Is it going to become cheaper or where's the way to go? Yeah, so um the question's around like cost feasibility. And so for some applications um like modern large language models uh are just way too expensive for using them to make sense. Um cost is coming down really rapidly. I think even last week OpenAI um introduced like a I forget the number, but it was like a pretty significant cost decrease for GPT-4. Uh does anyone know what it was? 3x? Yeah, um 3x cost decrease

And that's like I think their second big cost decrease this year. So I think like the the um the cost of running these these models is coming down like something that is much faster than Moore's Law. So um still good to know about what the cutting edge systems can do because what's cutting edge today is going to be in 2 years is going to be like pretty cheap to run. All right. Um So let's talk a little bit about large language model. Um so large language model is kind of a fuzzy term and uh like a lot of researchers kind of hate the term. But when I say large language model, usually what I mean is like a transformer-based model that has at least a billion parameters. So like all of the matrices in the that neural network together um there's more than a billion of them

And it's trained uh like the task that it's trained on is to take a sentence um take a paragraph of text and predict the next token, the next word or the next character that's coming in that sequence. Um so I want to talk a little bit about how modern large language models are trained um with a focus on the data that they're trained on because the data that large language models are trained on is really important to understand in order to build intuitions for what these systems will do well and what they won't do well. Um so let's go all the way back in time to you know prehistoric era GPT-2 back in 2019. Um so GPT-2 um you know was a 1.5 billion parameter model um and sorry I'm having trouble with the slides here. Does that change if I do that? Okay. Yeah, I guess it's just not showing up here correctly. Um which was really really large for the time um and now would be considered you know pretty tiny by large language model standards. So how what data was GPT-2 trained on? Um the authors of the GPT-2 paper found that um if you just like basically take all of the data from the internet um which you can find in this data set called Common Crawl which is just a like an open-source data set that um has a web crawler that basically pulls data from all websites

Um the this data has massive data quality issues. So if you imagine just like random data from the internet um some of it is really useful and high quality like thoughtful blogs, news articles, Wikipedia articles, but most of it is just pretty much garbage. It's like kind of random like HTML tags, uh advertising, stuff that you wouldn't really actually help a model understand the world better. Um and so they they formed a subset of the Common Crawl data set um called WebText and really the way that this is formed is they they looked at um Reddit data. I think when people think about this they think of like kind of all the um all the comments and all of like the nasty stuff that happens on Reddit, but in reality this was looking at the outbound links from Reddit um and in particular the ones that people on Reddit had uh flagged as being higher quality by you know up voting it on the Reddit system. And so this is the data that it was trained on. Um and so they were left with like around 8 million documents or around like 40 gigabytes or so of text. Um so this is GPT-2

GPT-3 was, I think, like a one of the first sort of big breakthroughs in language models that um started to make these things feel more general purpose. Like um I I remember interacting with GPT-2 and it felt like the model was doing a reasonable job of guessing. It would like felt impressive for a model, but it wasn't actually doing anything useful. Um GPT-3 for me, at least, was the first time when um it seemed like language models were actually going to be really useful very soon. Um And so GPT-3 wasn't really an an advancement in any way in the underlying technology. Like it wasn't an algorithm's advancement. It wasn't a um an advancement in the way that things were trained. It was just like GPT-2, but 100 times larger

Um So, rather than 1.5 billion parameters, it had around 175 billion parameters. So, a lot of engineering went into scaling it up like that, but not a lot of research or like changes in the fundamental technology. Um One of the reasons why GPT-3 felt like such a big advance is because it started to be really good at um being able to kind of figure out new tasks quickly. Um And so this is like one of the kind of core experiments from the paper where they showed um what happens, like how the model's able to generalize from a handful of examples to solve a new task. Um And so this task is um or this this benchmark is in-context learning. And so what that means is like um rather than training a model on lots of new data, um instead what you can do is you can just pass a handful of examples to the model in the text that you input to it in its prompt. Um And so one of the like amazing things about modern models like GPT-4 is if you give them a couple examples of what you want them to see, or even just a description of what you want them to see, they're able to figure out what you want and generalize that to new examples pretty well. Um GPT-3 was really the first time that we started to see this

So, as the number of parameters went up to uh like went up you know from 1.3 to 13 to 175 billion there was a super linear increase in the models ability to generalize well from a small handful of examples. So GPT-3 was trained on a much larger data set. This is one of the reasons why it was able to generalize better. And so this included like a like a lot of the data that was trained that GPT-2 was trained trained on but also a lot of other data from this messy common crawl data set. So a lot of the effort that went into building GPT-3 was really around you know scaling up the the engineering side of the model architecture and figuring out how to just shove way more data into it. Like figuring out how to take all this noisy common crawl data, filter out the stuff that wasn't useful, focus only on the high quality examples and put the rest of the examples and and put those examples into the the models training set. So you know if you want to train large language models or even fine tune large language models, you know the the lesson here is a lot of the the stuff that's worth spending time on is is is filtering the data. This model was trained on like 500 billion tokens

Um So unfortunately GPT-3 data set and all the subsequent data sets for most of the like commercial language modeling companies are not you can't get them. Like these companies even if they're open sourcing the models often times they consider the data set their sort of proprietary advantage. So you can't just download the data that GPT-3 was trained on. The closest thing to an open source equivalent is from a company called Eleuther. And it's a data set called the pile which is about 825 gigabytes of of sort of English language data with like lots of different sort of uh like high quality and diverse subsets. So, it's not just one data modality, but it's data from all kinds of different places on the internet. So, GPT-4 kind of started the era where not only did they not release the data set, but they also didn't really talk about what is in the data set, unfortunately. Um so, the you know, there there isn't actually like a way to know even how the data set was constructed for the most modern models like GPT-4

But, um yeah, where's the data? Um but, we'll talk in a little bit about like some of the at a high level like some of the different types of data that the models were are being trained on these days, um because it'll give you intuition for um you know, where the capabilities of these models come from and where to go look if you if you're having trouble with the model working on a particular type of data. Um so, one of the point I want to make here is that not only is the amount of data that these models are being trained on increasing rapidly, also the amount of compute being used to train these models is is increasing exponentially. So, if you go back to you know, the original GPT or GPT-2 in 2018-2019, um these models were trained on somewhere between 10,000 and a million petaflops, roughly. Um these days, models are being trained on well over a billion petaflops. And the um you know, training a state-of-the-art language model from scratch is getting to the point where it is uh compute completely infeasible from a cost perspective for almost anyone other than um you know, very very well-funded startups, large companies, and government organizations. Like, the rumors are that the next generation of models are going to cost something like $100 million to train. Um so, another reason why training language models yourself is not really something that you need to to super concerned with at this point. Um but, fine-tuning is still a helpful tool to have in the toolkit

Um, so there a concept that you'll come across in the machine learning world is called the bitter lesson, right? So, like if you think about the history of GPT-2, GPT-3, GPT-4, um, you know, I think what researchers wanted to happen subconsciously is they wanted to be like the genius breakthrough, um, the change in the algorithm that made something that was infeasible become feasible. Um, and the bitter lesson is that you know, um, no matter how much effort you put into research and algorithms and clever ideas, often all almost always you're going to get beaten by someone who just figures out how to stack more layers and shove more data into the algorithm and and uh, and spend more money to train it. So, um, this is um, you know, like I think one of the kind of like uh, age-old lessons in machine learning is that simple things work well, um, more data and more compute wins over more clever ideas. This is we're starting to see cracks in this a little bit by the way, um, which is that the rumor is that the latest models from OpenAI are not meaningfully bigger than the last generation of models. Um, and so the like the algorithmic advances are starting to matter again. And the reason why they're starting to matter is because um, OpenAI has essentially run out of data to train these models on. Like they they don't have orders of magnitude more data than they did when they trained GPT-4. Um, and so they're needing to rely more on algorithmic advances

Um, so you know, when when we tell the story so far of like how these models were trained, um, you get the impression that these models were trained on like basically the entire internet, which is true. Um, but there's another part of the story, which is that there's other layers of data, layers of more specialized data that these models are trained on that are really important to the capabilities they have now. Um, so if you just train a model on the internet, then what you get is a model that like kind of knows how to predict data on the internet, um, but it doesn't necessarily know how to solve tasks. Um, so the uh, uh like the at the at the time that GPT-3 was created, the mindset that people had was like um, right like let's try to create a model that's general purpose in the sense that I can give it a handful of examples and it'll figure out how to generate uh generalize from those examples. Um, but by the time that ChatGPT came around, the way that we interacted with these systems is really different. Like if you interact with ChatGPT, you're not providing it a bunch of examples of what you want it to do. You're just asking. You're just saying like, "Hey, can you write a document for me? Can you um summarize this? Can you uh you know, come up with ideas for for names for my website?" And so the mindset is now zero-shot

Like the model should be able to do what you want it to do without you needing to show it. Um, but instead just by asking. And so how did that ability come about in these models? Um, how did the ability to follow instructions come about? And so the answer is um fine-tuning. Like a massive scale fine-tuning effort that was designed to help models um be really good at not just, you know, predicting words on the internet, but actually following people's instructions. Um, and so the way that this worked is um OpenAI and subsequently most of the other developers of these large language models have curated um much smaller data sets, but very very high quality data sets of examples of instruction following. The way that this works is um you know, uh like if you if you have a bunch of examples of tasks that people want the model to do, which you know, if you're OpenAI and you have and you released GPT-3 and you have thousands of people using it, um you basically get for free. Um, if you have that example the data set of instructions, what you do is you basically pay people, um pay contractors to solve the task in those instructions. And you maybe pay a lot of contractors to solve the same task um so that you can quickly figure out which answer is like whether the contractors agree on what the answer should be or not

What that leaves you with is a data set that is much smaller than the whole internet, but still reasonably large, tens of thousands or hundreds of thousands of examples, where you have um a really clean mapping between what the user wanted to do, like the instructions that they passed the model, and then what a human said uh an ideal answer would be. Then you take that data set and you fine-tune your base model on that data set. Um and that produces um that gives them all the ability to take similar instructions that it's that it sees in the future and uh solve the task that is contained in those instructions. Um so that's uh that's kind of like the first additional layer of of training that happened to these models after training on the internet. And this is like a really critical part of what makes these models good because um it makes it so you can interact with them in more natural language. You can just ask for what you want and have the model produce it as an output. Um then there's another like really important layer to how these models are trained, which you may have heard of um it's called reinforcement learning from human feedback. Um so the idea here is like it's really hard and expensive um and limited to ask contractors to describe exactly the ideal output that they want

Like if you imagine um for more complicated tasks, if you ask a lot of people to write their ideal output, um not only would it take them a long time and would it be expensive, but they also might all come up with like totally different things. Um and so the the uh like the folks at OpenAI and other organizations came up with a a um a cheaper way um to gather feedback from people and use it to to like further train the models. Um and the that way is like rather than asking people for what the ideal answer is, instead they have people rank different um different answers. So they'll give people two different possible answers, answer A and answer B, and they'll have humans just click on one or the other. Like, this answer's better or this answer's better. Um and so, this is allows them to um collect preferences from people at a much larger scale. And uh inject those preferences into the model. Um so, when the the trade-off that you make here by like gathering this simple sort of lower fidelity data is that the training process itself is much more complicated

Um but, the advantage that you get is that you can collect human preferences on things that you might not be able to easily describe as a in in the form of a task that a human can solve. And so, the impact that this has had is that it's made the models um significantly better at following instructions. Um and it's also made the models significantly better at avoiding um some of the like kind of traps and problems with these models that people complained about when they first came out, like uh jailbreaking and uh uh you know, going off the rails and you know, saying crazy stuff. Um so, one thing that you might ask is like, what is ChatGPT? Like, how does this fit into all of this? Like, how was ChatGPT different? Why was that such a breakthrough? Um the interesting thing is that like, ChatGPT wasn't really uh technological breakthrough at all. Again, coming back to the bitter lesson. OpenAI took the models that they were already using. Um they did a little bit more reinforcement learning from human feedback on data in the form of conversations. Right? So, the model is good at recognizing conversations and what a what kinds of conversations human preferred um over the others

Um but, really no kind of core change to the technology other than um you know, a change in the format of inputs that the model was expecting. So, these models are are now expecting data in the form of conversations. Um but, it turned out that just reformatting the uh like, the mode of interaction from you know, needing to write an entire paragraph of text and have the model to respond um and being able to interact with the model in a turn-based fashion as part of a chat made a huge difference in how usable it was for people. Um So, I think like maybe one other point I'll make on GBT is uh you know, like you can think of GBT as sort of having a lineage. Um these models are not really trained from scratch anymore or at least they're not trained from scratch very often. Instead, you take the like you can think of this is kind of a base model which is the model that's trained on the internet. So, this is the model that has like tried to compress as much of human knowledge as possible. And then it's fine-tuned over and over again on different subsets of data that give the model different capabilities

Um so, those capabilities are instruction uh data sets. Um so, data sets that where humans solve the task for you. Um reinforcement learning from human feedback data sets. So, data sets where humans tell you their preferences. And then a third type of data sets uh which is data sets of code. Um and these code data sets are also one of the keys to why these models are so capable. Um for even for tasks other coding. Um there's papers that have found that like training on code is one of the things that makes models better at um at forming like sort of logical logically uh cohesive sets of thoughts

Um one kind of implication for how these models are trained is that uh a lot of the recent work on fine-tuning um reinforcement learning from human feedback and instruction tuning is focused on like what you can think of as alignment, right? It's focused on helping the models avoid doing things that people don't want them to do. Um so, you know, again, if you remember back in the early days of chat GBT when when uh being first released their their API and a lot of the the negative press around like how um journalists were able to get the the uh the being chat to you know, um have express its really dark thoughts and um uh and you know, like go off the rails a little bit in a way that seemed scary to people. Um that doesn't really happen anymore. And one of the reasons why is because of all of the reinforcement learning from human feedback that these these companies have been doing on top of the models. Um but these these these additional forms of training, these alignment-oriented trainings that are aimed at helping um avoid bad outputs from the models come at a cost. Um and one of those costs is that um they like some people find that they're not as capable um as they used to be at some tasks. So, it's just worth worth knowing about. Um so, there's been open-source attempts to sort of reproduce some of these reinforcement learning from human feedback data sets

Uh one thing that you can do is you can steal a lot of the training that these models have. Um so, if you just ask ChatGPT to answer the question for you, um you can that creates a data set that you can use to train other models. Uh questionably legal, but still um pretty widely used in the research community. And um there are open-source models that have been released. Um OpenAssistant was kind of like the first major effort that attempted uh to reproduce the uh ChatGPT's ability to kind of follow human instructions. So, um kind of rapid intro to transformers and LLMs. Um if you want to learn more about this, I recommend these resources um to kind of like go a little bit deeper on some of the stuff that we covered here. Any questions? Yeah

Oh, yeah. Uh yeah, really quick question. What exactly is a parameter? So, you hear this phrase like, "Oh, it went from 1.5 bi- billion parameters to 175 billion parameters." Does that mean that like they took a piece of data and effectively converted it into 175 billion like uh floating-point numbers in a in a matrix for every single thing? Yeah, so um the way to think about parameters is like, you know, again, if you if the mental model for um for what neural networks are doing is it's just taking a vector of numbers and it's multiplying it by a matrix a bunch of times um with some other kind of functions being applied to uh to transform in between. Um the the a parameter is one of the parts of that algorithm that is learnable. Um so the weights of these matrices um when you start training the model are all random. Um but throughout the training process, they uh you know, you apply some basic calculus to adjust those the numbers in each of those matrices a little bit at a time until those matrices um are like well-tuned to your task, like where multiplying by those matrices transforms the input to the output in the way that you want it to to uh to be changed. And so a parameter in a neural network is any of the numbers um inside of that neural network that are learnable by that algorithm. And so um kind of a a good sort of rough mental model is it's like any of the um any of the numbers in any of those matrices

Um sometimes, depending on the architecture, sometimes there's other things that are parameters. Um sometimes some of those those uh like weights in the matrices are not actually learned. Um they're they're hand-coded in some way or another. Um but that's I think uh that that's how I think about it. Can you give a an example or some intuition of how the transformer network differs from sort of a vanilla classic forward computer vision network that works very well on that type of problem? What's the big ins or what happens when you go from there to the transformer network? Yeah, so that is so fundamental. The big innovation in the transformer um like if you if you think back to the name of the paper, the name of the paper was attention is all you need. Um so why is it called attention is all you need? Um there's an idea in uh deep learning that existed before the transformer, uh but it was really uh like popularized by the transformer called attention. Um the way that attention works is like if you think about a typical kind of feed forward neural network, um, the there's multiple there's many layers that are stacked on top of each other

Each one of those layers you can think of as a matrix multiplication. Um, and so each of those layers, uh, you know, if you think about that this like stack of matrix multiplications happening on top of each other, the inputs to one matrix multiplication are the outputs from the previous matrix multiplication. And so, um, the the in order to transform the the output correctly, um, at each layer, so each of these matrices, um, all of the information that it needs needs to be contained in the output of the previous matrix. Um, so if you're going like from layer one to layer two, um, or if you're going from your inputs to the first transformation, um, you can only look at the inputs. If you're going from the first transformation to the second transformation, you can only look at the output of the first transformation in a standard feed forward feed forward neural network. Um, and so it turns out that that's hard, right? Because you have to get it right. You can't, um, you can't accidentally throw away any of the information in any of the layers in order to get the final answer correct. Um, so attention, the core idea of attention is that at any given layer of the network, um, at any given step in this transformation, you give that layer access to all of the previous information, um, to and in particular to all the previous inputs

And the model learns to, um, focus its, uh, focus like the the model learns to incorporate all of that information, um, in a different way depending on the layer. Um, so the model is able to like take any individual input, um, and incorporate that information into its outputs. And that just, um, you know, that these models are equivalent in the sense that they're all capable of learning any function, but, um, it turns out that uh, attention and transformers are able to much more efficiently learn the functions that you're trying to get them to learn because of this mechanism called attention. Hi. Yeah, the the instruction tuning uh this the skeptical part of me thinks, "Okay, they're no longer sharing what they're doing on the data side. They're hiring thousands of contractors to answer basically pre-formed questions, and then uh you know, and the answers maybe are picking the best answer." That sounds like a hack at best or uh quite brittle um and not not uh not the same thing at all really. And am I Is that intuition wrong that it's becoming more and more brittle if you do it that way or or is it uh the language model still doing some some real work there? Yeah, um I guess my intuition is a little different, which is um I think of it as like you like what are you doing when you train a machine learning model? Um you are um teaching it to follow a pattern. And so the um whatever data you put into the model during training, those are the patterns it learns to reproduce

And the um and like the you know, if you think about like a a model that is not trained very much, like a model that's pretty much random, there's a lot of like entropy in the outputs that it will produce. So, you can write a sentence um and you put that sentence into the model, and it can produce pretty much anything. Like it produces like something that's close to a random sequence of outputs. And so, a lot of the purpose of training a model is to reduce that entropy. Right? It's to um it's to make the model respond in a much more specific way to a given set of inputs. Um so I think of the like these further processes of training on instruction data and training on preference data as like constraining the types of outputs that the model will produce. And so reducing that output space and that is very productive because it means that the models it's much easier to get the models to do what we want them to do but it is also a constraint and so it does like remove some capability. Okay, so my question is suppose we are I'm training a domain data set on top of this one of these LLMs

Is there any risk of overfitting that you know base LLM and then the second follow up question that to that is that is there any guidance to the minimum amount of data set for that domain that I need to get an effective output. Yeah, so anytime you fine tune a model there's always a risk of overfitting and it's something you have to be very careful out in fine tuning and it's one of the reasons why fine tuning is a lot more difficult than prompt engineering and techniques like that. How much in terms of how much data you need it depends on what you're trying to do. You know, you can these days you can get decent results even from like hundreds of data points but you know, much more reliable would be to use like thousands or tens of thousands of data points. Okay, I'm going to keep moving cuz we're a little bit behind and just I want to cover one more thing and then maybe we'll take a quick break which is the anatomy of an LLM application. Okay, so we've talked a little a lot about how LLMs are kind of these like magic functions that can take text and follow instructions contained in that text and produce outputs to solve problems. But there's a lot of limitations to how LLMs work today. The first is that despite all this additional training, it can still be really difficult to um to get the models to do exactly what you want

Um, and so you know, here's an example on the right where I was trying to get ChatGPT to correct the grammar of some inputs. And uh you know, what's like the interesting example at the bottom here when I typed like, "Who is the top person in the world?" and ChatGPT, you know, changed that not just to correct the grammar, but actually also to change the meaning of the sentence sentence to, "Who is the most influential person in the world?" right? So, this is not exactly like if I was building a grammar correction bot, this is not exactly what I would want it to do. Um, so it's it still can be hard to get models to know what's in your brain, to know exactly what task you want them to solve. Um, and so that's why there's um these kind of worlds of prompt engineering, task decomposition, and fine-tuning um that you need to know about if you want to get models to do tasks reliably. And so, we'll talk more about that um in a little bit. Second kind of big limitation of LLMs is that they don't know everything that you want them to know, right? They're trained on tons of data, but they're not trained um up-to-the-minute with the latest data. Uh they, you know, up until recently they were only trained until 2021. Um, so they didn't know how to answer questions about current events, like about the uh the way the world is today

Um, today they're I think they like OpenAI updated them last week to be trained up to at some point this year. Um but they still don't know things that are not just contained in the general internet. Like if you have information that's specific to your company or uh you know, specific like ways that you want the model to write or facts that you want to know, those are not contained in its training data. And so, you need a way to um put that data to give that data to the model so it can use it to solve the task you want to solve. Um, this is the process of augmenting models with tools and information retrieval, which we'll talk a lot about today as well. And then finally, you know, it's like never been easier to get models to like kind of work like to work most of the time or to work a handful of times enough to produce a really cool demo. Um, but it's still really, really hard to go from that prototype to a reliable production system. Um, and so we'll talk about that as well in kind of discussion of LLM ops and test driven development

Um, so putting it all together, like I think there's kind of a um, a pattern that's starting to emerge for how these LLM applications are built. And so uh, I would divide this into like three pieces. The first is the part that the user sees, which um, the user puts in a request. Um, that request is something like a question that they have for the system or a conversation that they're having, um, a document that they want to summarize, whatever input they need to provide in order for the the algorithm to do its job. Then what's happening behind the scenes is they're interacting with an LLM application. And this LLM application has different components. Um, the first is the instructions. Um, and so the instructions in the middle here are like basically the prompt that you give the model or um, the sequence of prompts, the chain that you give the model that tells it the rules that it has for how to solve the task for the user

Um, those instructions interact with two other systems. The first is the model itself, which could be, you know, GPT-4 um, or another proprietary model or it could be an open source model that you host yourself. And then the instructions also interact with external information. Um, so they interact with like all the other data that you need in order to solve the task. Tools, um, retrieval systems, vector databases, and what OpenAI calls functions are the different ways that we'll cover to uh, for your LLM app to use external information. And then even further behind the scenes, if you're building an application like this, um, you have an improvement system. And the improvement system is what you are using um, to take the requests that the user is um, is making and the sequence of steps that the app is following to solve the task. Um use that to better understand what your users what problems they need you to solve and make your application better at solving them

So this includes things like analytics, like looking at the logs of what your users are doing, testing and evaluation to validate whether the changes that you're making are improving things for users. Um and like a a bucket of things that we will call LLM ops which like is basically kind of the nuts and bolts of how to do these improvements like managing prompts, fine-tuning and hosting models. And so this is more or less what we're going to cover today. Um and um Uh yeah, I think like maybe the last thing that I'll show today or or for right now is a um just a a really simple like batteries included example of how to do this and I would think of this is like kind of a baseline for the stuff that we talk about later. This is like this is the way that you would do this if you didn't want to think about it um and that's to just use the like brand new open AI assistance API. Um so the way that this works and can you still see this if I flip to the other thing? See code? Okay, cool. Um so this is like a this is a a chatbot that you can use to interact with the um like all of the information from the previous FSDL course like the pre-LLM FSDL course. Um and so we're going to we're going to cover later today like a more in-depth version of how you could build this yourself

Um but just to give you a preview of like what this might look like to build in the future without understanding any any of what's going on under the hood, um this is how you would do this in the like new open AI API. Um and so really this is like consists of a couple steps. Um the first is that we have some data and this data is basically just um uh a pointer to the lectures from our 2022 course. Um and so we have, you know, the the uh the base URL for where the markdown for these lectures lecture notes are stored. And then we have kind of a list of each of the lectures. Um so the first thing that we need to do is we need to like give OpenAI access to this data so it can use the data to answer questions. Um and the way that we're going to do that is we're going to just call this function called upload data to OpenAI. Um and again, there's like this is really really simple

There's nothing here that is LM specific. Um OpenAI is taking care of all of the technical details about how to about what happens to this data and how it's used by the model in order to answer questions. Um so what we're doing here is we're just downloading each of these documents um each of these lectures. Um we are rewriting them locally which is just to, you know, have a local copy and to change the file name. And then we're opening each of these documents and we're calling the OpenAI clients to create a file. So um OpenAI now, after we run the script, has a file associated with um in its system associated with each of these lecture documents. Um and so it has access to them in the Assistants API. Um so now that OpenAI has access to each of these documents, the only other thing that we need to do is we need to tell it how the assistant can use those documents to answer user questions

Um so we define an assistant. And this assistant like really couldn't be simpler. It's like two lines of code. Um so we give it a name. In the this case it's AskFSDL. And then we give it a uh a set of instructions. Um these are the instructions for how it does its job. And so uh you know, here we're just doing a really simple set of instructions

You're a teaching assistant for Full Stack Deep Learning. And your job is to answer questions for students. Um put another line in the prompt here, which is um basically trying to encourage open AI to always find the information in the lectures. One kind of common problem with these LLM applications is what people commonly refer to as hallucination. Hallucination, I think it's like pretty loosely defined right now. Like people say hallucination to mean basically any kind of failure that the model makes. The way I define hallucination, like a more technical definition for hallucination is when the model is supposed to respond based on information that it retrieved from a corpus of knowledge, but it instead just tries to respond to like make up an answer from from the internet data that it was trained on. And so in order to avoid that, we add this other line which says always try to find the information in the attached files using the retrieval tool

And so if you can't find anything, then don't attempt to answer the question. And so really the only other step here is we need to ask open AI to create an assistant for us. This assistant uses their new GPT-4 preview model and it has access to all the files that we defined in the previous step. As well as it's able to access those files using what they what open AI calls a retrieval tool. So a retrieval tool is like all the details abstracted away of you know what's going on under the hood of basically everything else that we're going to talk about today. So this is the batteries included version of like I don't want to understand the details, I just want open AI to handle it for me. Yeah, there was there a question there? Just one question regarding your future assistant instructions the model uses the retrieval tool but not explicitly because nowhere is it defined what how you should call retrieval tools so that the model knows it should use the retrieval tools. Yeah

This is kind of implicit, huh? Yeah, and this is actually like I've been having trouble with this in the assistants API. I think they they maybe need a more explicit way to do it. I'll show you in a just a second like how this fails sometimes. Um but yeah, I I I think like in OpenAI usually for their function calls, they give you a way to describe the function. Um which helps uh the the model understand when it's supposed to use that function. Um the retrieval tool doesn't really seem to have the same level of control about when it's used or not used. Um so, yeah, it's one like and I think um we'll see in just a second that this like this this was released by OpenAI last week and it's in my opinion like no nowhere near ready to actually use in production. Um and that's one of the reasons

But hopefully this gives you a sense of like maybe next year when we teach this class, we won't need to cover any of the details of uh of you know, retrieving data or um prompt engineering at all. It'll just all be taken care of by the higher higher-level libraries that we use. Um so, the last piece of this is just, you know, like how we manage the chat itself. Um and so, what's happening here is we just have like I'm just going to take input from the user, in this case just in the terminal. Um the example that we'll show later will actually be a Discord bot that you can interact with, but for now it's just a question from the terminal. And then we create what's called in like OpenAI world a thread. Um that thread is like a sequence of messages that is consumed by the model to produce the output. Um and so, we create a single threat uh a thread that contains of uh consists of a single message from the user um containing this question that I I'm going to type in in the terminal

Um then we uh create what's called a run. And so, the run is like how you take this thread um this sequence of messages and attach it to a model. Um attach it to an assistant. Um so, we when we create it when we run a thread, what we're doing is we're passing that thread to the assistant that we just created. Um and that assistant is processing the thread and using that to create an output. Um so, next thing that we'll do is we basically wait for this thread to finish. Um so, I'm just polling every 10th of a 10th of a second um until this thread is in a until this run is in a finished state Um so that I know that the model has finished producing its output and I can retrieve that output and see what the model said in response. Um and then the the last thing that we do here is we just pull out this um uh we just list all the messages in this thread

We pull the most recent one and we print that out to the user. Um and then this is like some additional details about how we just make sure that the model tells us which document it's using to answer the question. So any questions on like this kind of OpenAI assistance way of creating a a chatbot? Yeah. Um I'm just curious if you can I'm just curious if you can comment on um the state of LLM apps after OpenAI, you know, Dev Day. Just because I think a lot of work, you know, over the last couple months have been relatively simple like apps where a lot of the work was put into maintaining your vector database, like how exactly your, you know, chunking your documents, like all of those things that seem to have been solved at least with GPTs or at least like in a much easier out-of-the-box state. And I think like the more complex LLM apps, maybe like a Copilot from Microsoft, there's like a huge barrier between where the community has been building and getting to that state. So I'm just curious like if you have any any reactions. Yeah, I think kind of like just a general thing in computing is like people start with really low-level stuff and they build abstractions on top of the low-level stuff to make it easier and easier to, you know, solve the problems that people were solving over and over again with the low-level things

And so I think in general like the idea of assistant the assistance API makes a lot of sense and I I do think for a lot of applications people build, you're not going to need to manage your own vector database or understand the details of chunking or any of those things. So I think like this or something like this is how most of these applications will get built in the future. Um, still important to understand how it works under the hood for two reasons. The first is because I guess like spoiler alert, this doesn't work. Um, the the in my opinion this is a premature release by Open AI. Um, I think they pushed a little bit too hard to get it out for Dev Day and it doesn't actually like this this chatbot is terrible. It's not going to actually answer the questions that we want it to answer anywhere near as well as the one that we're going to build in a more custom way. Um, and so you know, if you're trying to build today, I think you should still use the lower level stuff

Um, not to say that this won't get good eventually. Um, which I think it will. Um, and then the second is because you know, as you start to do more and more advanced things, it becomes important to be able to like look into the black box and uh customize the way the things work under the hood. There's a ton of degrees of freedom that you have about each of the steps in this in this in this infrastructure stack and tuning any one of them can like produce meaningfully better results. Um, so yeah, uh I I think I think eventually we're going to be using higher level libraries like this, but it's not ready for prime time yet. Um, so if we actually run this, let's ask a question. So I'm going to ask the first question I think it should be able to answer which is um what does the first lecture cover? So again, this is an earlier version of the course. Um, one thing that you'll notice when I run this is that this is really slow and it doesn't stream

Um, oh, that one was actually reasonably fast. Um, so it it's able to like find this source and it's it tells me that it's the first lecture is course vision and when to use ML and it covers um the like an introduction to planning, developing and shipping ML powered products which is like a pretty good summary. Um, couple things that you'll notice here. One is that like this wasn't particularly fast. It doesn't feel nearly as fast as using ChatGPT. And also, it didn't stream the outputs. Like if you use ChatGPT, you're used to things coming back one token at a time, uh one letter at a time, which makes the makes the experience a lot better cuz it feels like you're getting an answer a lot faster. Um these are limitations of the system currently

It's slow and it doesn't stream. So, you know, one of the more technical reasons this is not ready for prime time. Now, let's um I'll show you like a bigger limitation here. Um so, let's run the chat again and let's ask a more complicated question like um let's see. Um I guess what is fine-tuning? So, yeah, again, very slow. Um and so, it says I it seems that I'm encountering technical difficulties accessing the specific details about fine-tuning in the full-stack deep learning course materials. Um which, you know, we know it's not experiencing technical difficulties cuz in the previous question, it was able to find the first lecture. Um so, it's uh I think it's it's like for some reason OpenAI is not really figured out how to chunk up the information in the right way to put this uh in the context

Um the bigger problem though is that it's like if I was building the system myself, um there's all kinds of things that would cause this and I would have a way to debug what was going on. I'd be able to look at like, hey, is it not retrieving the right documents for me? Is like the content of those documents misformatted somehow? Is it Is the prompt not being constructed correctly? Um all black box in the OpenAI system. And so, um it's really really difficult to know where to go from here in order to fix this. So, um again, I think like OpenAI assistance is not really ready for prime time, but I wanted to show you because this is like, you know, this is this is like 10x less code than what we'll talk about later this afternoon. And um this is the direction things might be going. So, like you can think of the rest of the day as um, basically us rebuilding this OpenAI assistance version of uh, the chatbot, but in a way that actually works. Um, so I will uh, yeah, I'll take one more question and then I'll stop there and we should take a break until 11:00. Yeah

So, I built a chatbot uh, using embedding drag approach. Um, so it did really well. So, I gave some documents. I got really good response. But when I ask a random question which is not irrelevant to the documents, it's still trying to answer it. For example, when I ask US president, it's it's it's answering the right question uh, right question, but I don't want it to answer because I it I want only look into my documents to answer. How can we do that? Yeah, um, so this is a really common problem, right? Like the I think um, for uh, so like if you think about the process that people used to train these models, um, the process really does two things or you can think of it as doing two things. Like one is it's um, teaching the models how to understand natural language and how to follow instructions

Um, so it's teaching models like how to reason and how to interact with people. Um, but then the second thing that it's doing is that the model um, it's not just learning the underlying patterns of how language and communication work, it's also learning the specific details about all the information that it was trained on. Um, and so if you just interact with a model without it providing any information, it will answer for you based on the information contained in the weights, um, which is derived from the information that's contained on the all of the internet. Um, so sometimes that's desirable. Like if you want to just have a model that's able to answer general knowledge questions about the world, but for most applications that um, I would argue most applications that most businesses are building today, it's undesirable, um, because you want the model to answer based on specific information um, that is relevant to the task that you're trying to solve. And so uh a lot of times what happens is that the model it's like really hard to get the model not to just answer based on its general knowledge. Um it really wants to just you know reproduce what's on the internet to answer the question for you. Um even if it doesn't actually have that information in the in the data that was retrieved

So I think the best way that people that you can do that now is by adding um adding something to the instructions to try to you know convince the model not to use like not to answer the question if if it's uh if it doesn't have information available. So that was this line in the prompt here. Like you should always try to find information in the attached files by using the retrieval tool. Um if you can't find any relevant information don't answer the question. Um but you saw in this example um it didn't find relevant information because of what it imagined was technical difficulties. Um but it answered the question anyway. And it at least acknowledged that it didn't find this information in the in the retrieved materials. Um but it still kind of did this thing that we might not want it to do

And so it's um it's very difficult to get models um to follow instructions that are that run contradictory to the data that it was trained on. Um and that's I think like one big limitation of these systems today.