SBTB 2023: Bassem Yacoube, Using LangChain and LlamaIndex to customize LLMs...
Recording: SBTB 2023: Bassem Yacoube, Using LangChain and LlamaIndex to customize LLMs...
hi everyone my name is basam yakub I'm a Solutions architect with octl and today we're going to be talking about how you can build your applications we're using LL 2 endpoints uh on octo Ai and the Lang chain stack um so this is very quickly the agenda so I'm going to talk briefly about what is the oct AI platform uh then go over a couple of rag Concepts and then we'll show a few demos about uh doing grag using Lang chain and the Octo AI llama endpoints uh and then I'll kind of tell you about uh our texgen Innovation program uh that we're just starting at octo right now um so very quickly at octo AI we are all about you know making it easier for developer and practitioners to run tune and scale your applications so we just want you to be able to get an endpoint for your model whether that's an open source model out of the box foundational model or you know bring your own model you can run it on our platform you can tune it and we already have tuning for a stable diffusion out now and we are adding tuning for llms in the next few weeks and you can scale that in production to you know several replica several nodes to support your workload okay um so like I said we have an image gen solution now for stable diffusion 1.5 and sdxl uh we have three llama 2 models and a bunch of others that I'm going to show you in a second um and we have built all the cloud infrastructure so all these models are already optimized and running on our Cloud um so you know you just go sign up and and get a an API endpoint that you can start using in your application right away um so what's the text Innovation program so basically that's uh a new initiative that we're starting uh we're looking for customers or users or developers that will work with our product and engineering team and you know we will help you solve your problem and you can help us shape the product shape our you know offering of of our text generation platform uh there you know it comes with a bunch of perks uh like you get fre weeks of uh unlimited usage on Lama 2 endpoints and you get to work with our you know product and uh engineering team okay so let me show you what is the Octo AI platform very quickly here so uh if you have your laptop or your phone you would like to do this right now it's octo ai. cloud so if you go to octo ai. Cloud um it will ask you to sign up you can sign up with your Google accounts or your GitHub account it just takes a minute um and then you should see a screen that looks like this so we have example models and these are basically the the templates that we already have on the platform um so the first F page you can see um some image gen models here and uh our llama 213b and the one that we're going to focus on today is on the second page uh the Llama 27b you you see we also have like a bunch of other models like llama 27tb we have uh text embeddings using instruct we have whisper X for audio text to video with zero scope and so on and there is also a way for you to bring your own container your own model uh you can you can read more about that in the docs uh we also have a very active Discord channel so you're welcome to sign in and you know start asking questions and interacting with the team um and finally you can you can talk to us directly in the chat box uh this goes to our customer success team directly um so these are this is how you can sign up and uh get support and you get $10 free credit which are good for almost 10 hours of uh the a1g uh GPU instances okay so if you want to follow along we're going to be creating a l 2 7B model Okay Internet is a little slow okay um all right I'm not sure why this is not showing up for me I already have an endpoint but uh for those of you who are trying to create one I I hope it's it's working for you okay so I'll go back to a couple slides and then we'll jump into the code so uh llama 2 as you know meta released it about a couple of months ago uh it's currently the you know strongest open source llm uh we have optimized those uh three variants 7B 13B and 7B um and we have them as endpoints in our Cloud uh as you just saw our performance testing shows that you know across the board we have anywhere from like 20% to 100% increase in through in in tokens per second generated uh for for the different variations of the model okay so very quickly uh what is rag so it's retrieval augmented generation uh basically the llm M's you know have a problem with uh the training cut off or the data that they haven't seen before so we want to provide them more background data more context data when we're doing a query um to to give them more information that they can use to answer the question so on a conceptual level it's like a retriever generator component you can think of the retriever as your inspector that does you know a presearch in in the data source in the knowledge base and then retrieves the information that's relevant to your query and then you have the generator which takes all this relevant context information plus the query and generates an output so the very common example you've probably seen that before is chatting with a PDF so you want to be able to upload your document and then your model will be able to understand it and then you issue a query and the model should be able to give you a response from the information in that document so continuing with this example the different components will be something like this um so the document will be chunked into different uh text chunks and there are you know various algorithms and tools uh to do that and then we're going to generate the embeddings the embeddings are just a numerical Vector representations of that text um there are embeddings models that are different from the llm models uh so two kinds of models are needed to to create the rag application and we also have embeddings model on our platform um so I'll show you that in a second once you have the embeddings then you usually uh store those in a vector DB or a vector store um there are different kinds of those uh there's one like chroma DB that's embedded inside Lang chain there is llama index comes with with one and there are also some that are you know completely external products that um are cloud-based or on Prem and you can scale them to you know very large uh number of of instances and users uh like pine cone and Melvis and so on um so once the user user issues The Prompt The Prompt gets embedded through the same embedding model um and then it makes a vector search query to the database it retrieves the relevant text to that query and then the query and the retrieved information from the database are sent to the llm which then weaves the The Narrative and and creates the response okay so let's see a few examples of doing this okay um so this code by the way um sorry I should have okay so so all this code is in this repo it's github.com octl hackathon 2023 rag um so the code is available there all three examples that I'm going to show today and and it's going to uh this this reer will be up uh so you can follow up later if you want to try those yourself so the first one uh I just wanted to check if my endpoint is working and my oo token um so when you clone this repo you you're going to see an EnV file so in this EnV file you're going to put your endpoint yourl and your your octo AI token and the way you get those is okay let's see if this is working now interesting that's very interesting okay good all right oh maybe there's something wrong okay just moving on uh I don't know what's causing this issue but once you have the endpoint it's going to look something like this and you can generate a token uh from your profile settings on the site as well well should look something like that so this is my llama 2 7B endpoint um and the first python script here the chat main uh so all I'm doing is just uh creating a lang chain object and passing a system prompt that says you know these are the instructions wrer response and just using the the Lang chain prompt template uh and passing it a question question so we we already have Lang chain providers for llm endpoints and for the embeddings uh end points so if you do pip install Lang chain you you automatically will have the the Octo AI endpoint and the Octo AI embeddings object so here I'm I'm directly just instantiating this Lang chain object for the Octo AI endpoint uh giving it some parameters and it's going to use the URL and the token uh needed so you run this um you're going to get something like that so this is this is not doing grag right now this is just the model uh responding to a general general information question so who's leard D Vinci and it says uh you know he's a Renaissance artist and Paul meth Etc so now I'm sure that you know my point and and my token are working so the next example is is going to do rag so okay so here we're using uh the Lang chain the llm model the embeddings model and the vector stores uh I'm using feice for this example feice is also a package by meta that does uh Vector storage and and Vector search so here um I'm pointing it to a PDF file this PDF file is just a uh you know free open source novel Sherlock Holmes novel uh by Arthur conand Doyle and it's just 15 pages uh for the sake of the demo so it's going to read the PDF and allow me to ask questions uh about the PDF so so I uh I ran the code it created the embeddings stored them in theice storage and now it says you know ask anything about the document so let's see if we can ask a couple of questions about this document so who's the author of the book okay so it says based on the text provided the author is Arthur coloy which is correct what's what's the book about okay let's try that again okay so again based on the text provided the book is a a mystery or detective story which is correct okay and let's try another one so who is the [Music] murderer okay okay so it couldn't guessed this one but obviously it shows that it's it's reading the information from the PDF so so that's very you know simple example of how we can do uh rag on a PDF file um the last example I'm going to show here is uh a little bit more complex and uh it's the one in the in the web retriever folder of the GitHub so here I have two data sources uh our own octo AI documentation and the kubernetes documentation and I created um a web crawler that just crawled or the the URLs uh for the Octo AI docks and the kubernetes docs um and then I used um you know the same framework Lang chain and chroma DB and created the embeddings so I already have the SQL files for Chroma DB for for the information in the Octo AI docs and the kubernetes docs and and now here the code file um I'm going to be calling uh two models in parallel so I want to test the Llama 27tb against the Lama 27b and see what's the difference you know in quality and performance and you can do this for a lot of you know other situations if you want to test against chat GPT or any other model um so again you know using the the Lang chain llm providers uh for the Octo AI endpoints and passing the the two endpoints uh end point URLs and my authentication token um and all the code for doing the embeddings and storing everything in the database uh is is here as well but uh I'm not going to cover this right now for the sake of time okay so we're on this we're going to see something like that so we have the response from Lama 270b and the response from Lama 27b so I asked it a question about how to reduce cold starts and I pointed it to our documentation octo AI dogs and so uh basically went through like all our dogs and found the the different solutions uh to reduce document uh reduce cold starts so things like using volumes using warm-up script smaller container size pre-built images and so on um and you can see there is a difference in quality I I usually find like 70b to be more more concise it it usually like gets to the heart of the problem if you will um we are going to announce the option to to run 7tb on multiple gpus U so the response time is going to be better than that so here I mean obviously if you're running on a single GPU 7B is going to be faster um but yeah um we are providing multi-gpu for for 7tb in the next couple of weeks so hopefully you'll see you know faster responses here okay um so the last thing I wanted to show is um how we can deploy this into AWS as an application so now we have a container that sorry we have the code that you know calls the the Llama 2 endpoints and the embeddings models let's say we want to publish the into an AWS uh application and have it hosted in the cloud so um very very quick way to do this is is using AWS Lambda functions um so the way to create an AWS Lambda function for something that is as big as this uh this by the way like once you install L chain and chroma DB and you know all the other requirements it's going to be about 2 to three gigs in size so it's it's not going to fit in the regular uh Lambda function so I'm using the container image type of of the Lambda functions so I'm building an AWS container uh based on the U Amazon Linux image and then you know just copying my code and requirements and U installing the requirements here and if you have noticed uh my code has a Handler function which is the entry point of the Lambda um and it's going to uh return the answer as Json so these are you know the requirements of the Lambda function function um after doing this you can use something like the the Sam AWS Sam framework uh to build this container and deploy it for you as a Lambda function so uh after installing Sam you point it to that folder and and do a Sam build so this is going to basically look at that Docker file and and create a container there okay so it created the the different layers of the container here and this this build succeeded uh you can test it locally by doing something like uh Sam local invoke so this is going to run the code from the container that it just built okay and should ask the same question uh how to reduce cold starts let's give it another second okay so here we got the response as as Json since this is going to be uh a Lambda function but you can see we have the the first response from the llama 270b and then somewhere here we should have yep the Llama 27b response so basically the same code has been run uh in a simulated Lambda function so now if you want to deploy this in AWS so you do something uh called Sam deploy SLG guided and and then it will yeah keep asking you a few questions about your stack name your uh AWS region and so on after you answer this these questions it's going to basically push this container coner uh into the the AWS ECR the elastic container registry uh just to show you that the container that was generated so if I go here and okay so these are the images okay so if you run Docker images you can see here uh the the images that were generated uh right now when we ran Sam so again if you if you do the Sam deploy this will push the to your ECR registry um which will look something like this so this is my container image here um and then the last step would be to copy this URI and go create a Lambda function and when you create a Lambda function you're going to select the container image um give it a name and post that U here and hit create function so that will use the uploaded Docker image to create a Lambda function it's going to look something like this so this is my LMA 2 Lambda and just to make sure that it's working you can go in your test Tab and run a test so now this is running the same code uh which uses the Octo AI llama end point and Lang chain uh on the container in AWS Lambda function okay okay so looks like the the call succeeded here as well so so now the Lambda function was able to talk to my 7B end point and get a response and also called my 7B endpoint and and here's the response so now this is your application deployed in the cloud you can easily just uh create an API Gateway in front of this Lambda function and give the URL uh to your customers or front end developers to build an application all right so this uh concludes my demos here I wanted to just recap you know the benefits is now as you can see you know you can create models uh very quickly using the endpoints on octo AI without having to worry about infrastructure or how are you going to get the you know gpus for your models um you can move to production easily uh by you know selecting the number of replicas for your endpoint and you can easily containerize these applications and deploy them to to a public cloud like I just showed you uh with AWS so you can really start building your applications in minutes versus you know days or weeks uh so happy to take any questions but I want to put uh this slide on in case anybody here wants to sign in uh sign up for the texgen Innovation program uh like I said you you get you know a llama 2 API endpoint uh with no access limit for four weeks and you get the opportunity to work directly with our engineering team as well so looking forward to having everybody join that program thank you