SBTB 2023: Chris Fregly, Implementing a Multi-Tenant Generative AI Chatbot.
Recording: SBTB 2023: Chris Fregly, Implementing a Multi-Tenant Generative AI Chatbot.
all right so a couple cool announcements actually before we get started um just submitted the final draft or the final manuscript for this new book that's coming out um it'll be out uh for reinvent the big AWS reinvent um oh and I'm Chris fley work for AWS I've been there just over four years uh principal solution architect uh I focus on gen startups and um deal quite a lot with sasses and uh retrieval augmented generation rag I'll you know go through and actually Define all these terms as we go but uh new book is out so check it out um I asked Dean to review it but he was too busy so uh and yeah yes he works for the yeah for the competition too so I understand uh I'm going to start with a demo as crazy as that sounds um and I also want to mention and and I'll show this here in a bit that everything I'm showing you is available on GitHub all this code is available it's actually part of this uh blog post as well um where uh and so this demo will be a multi-tenant chatbot that's using retrieval augmented Generation Um it's also using the new Amazon Bedrock service which uh there'll be a lot of cool announcements coming out at the reinvent conference if you're going to Vegas also if you're a startup or uh sound like a startup I can get you some pretty uh cheap passes to reinvent so uh just come find me afterward and uh that's in Vegas I think the week after Thanksgiving so you know little uh tidbit that conference has always happened after Thanksgiving and it completely ruins every like Amazon employees Thanksgiving because all we do is work and get our uh slide decks finished and demos finished and so yeah so back to this so uh all of this code is going to be or is available currently uh and we're going to actually deploy eks uh fully you know production ready uh and I'll show the architecture and stuff in a bit but this is using eks the elastic kubernetes Service uh it's also integrating with Amazon Bedrock for the generative AI portions and it's multi-tenant and all the fun things that sasses um use and require so let me just get to the demo here okay so we have two tenants and I'll show you let me zoom in quite a bit here and this is using anthropic CLA so some of you might be aware that like Amazon just invested I don't know 500 billion or some huge amount of money into this company anthropic um that's not true it's not 500 billion but I can't remember what it was um and it's also going to use Amazon Titan so two models within Bedrock one of them is from our partner one of them is uh our own model Titan for the embeddings which I'll show you where those come into play when I cover the architecture um all through a single API which is bedrock which is the Amazon Bedrock API okay so tenant a does not know uh like anything about the book that we just wrote so tenana knows about Sage maker kind of old school um you know doesn't really know anything about Bedrock so um and that's because yes so the part I didn't show is I have uploaded two different documents one for tenant a and one for tenant B so uh tenant a knows about Sage maker and then tenant B is the new hip cool um or like generative AI tenant and knows about Bedrock so if I ask what is Amazon Bedrock to tenant a they should not know it's going to show the isolation here they'll come back confused sorry I don't know all they know about is Sage maker okay and then we've got tenant 2 here or tenant B and me do user one ATT tenant B okay and I'm going to ask the same question what is Amazon bedrock and this should have a nice friendly answer Amazon Bedrock managed service blah blah blah um I could and so what's happening here is there's separate state right that's being maintained per you user this is all happening um within Dynamo DB right like Amazon uh like Dynamo DB for the data store it's all happening on eks it's all separated uh and isolated let me find uh this part here so I could show the kubernetes cluster behind the scenes zoom in a little bit I'm starting with the demo that then I'll finish with the boring architecture slides all right come on now all right we'll get back to that but yeah I was just going to show the actual eks cluster in action I think I could show it maybe through here I could show um yes out of curiosity does anyone here use eks right now all right oh and I'm not sure if you're supposed to see this new UI because it's coming out at reinvent but uh and so what do we have here clusters I can click on the cluster name and I could I should be able to see the nodes that are involved node groups uh relatively small node group just three nodes uh you know just for like demo purposes but um you could see the version 127 all that stuff so and if I can get to the Pod I don't know why that's struggling there um I could show you what's on it's basically each tenant has its own namespace so namespace isolation you can also do uh separate account isolation but that's that's quite a lot more um that's a bit more complicated so right now we're doing it at the tenant level which typically we see a lot of our customers doing um but you can do VPC level isolation you could also do the account level as well all right so jump to the slides all right who here is using generative AI in some form even chat GPT yeah does anyone write their paper yeah yeah has anyone written their their school papers or anything cheated yeah like you don't have to answer that yeah I've been trying to get my nephew who's a freshman to use it more and more and he's all scared to cheat and I'm like good God man yeah like you're only a freshman you shouldn't be worrying about that so I'll skip some of the I mean here I just showed chatbot the virtual assistant um there's also summarization which is probably the most common that we see it seems like yes every Enterprise wants to summarize something usually you know customer support type things uh but I've been working actually since I joined the startup team just recently I've been working with a lot more uh like multimodal use cases images videos uh water marking um you know and then there's like code generation as well too so GitHub Code Pilot yeah Amazon has something called code whisper uh the idea here is increase productivity all the good stuff and of course yeah we need multi- tendency uh and specifically for these uh different tenants and so this is what I just demoed which was the third bullet point there which is retrieving tenant specific documents um and what I did was I had uploaded and this was last night uh I had uploaded two different documents so a sagemaker PDF and then to tenant a and then a uh a uh like Bedrock uh PDF to tenant B and we're using retrieval augmented generation commonly called rag and the idea here is that we're using the actual large language model to just reason through and to be able to call out to retrieve data from the actual data store right in this case I'm using um a local data store called uh feice FIS SS Facebook uh AI something similarity search something um and let me get to that here yeah so uh we are we have indexed this data that's what I did last night I actually uploaded these documents those documents were then indexed uh by feice itself which is just a library and it's just storing it on disk I'm going to show the architecture and then show the next step to improve it and actually make it use something like Amazon open search or pine cone one of those other services um okay so and then then real quick you know why I use eks for this um 100% of my customers use eks uh for their generative startups uh it's just here to stay right uh so yeah it seems like some of you know eks we've got resource isolation uh it's relatively cost-effective we can do automatic scaling um and also uh so actually not eks but Bedrock supports many Foundation models so why Bedrock 4 generative you have data privacy this is running in your account right so you aren't uploading your data to a SAS uh you know potential uh competitor of ours this is actually running inside of your Amazon account your like AWS account uh you've got the Integrated Security all of the organizational security all that fun stuff is all there uh it's a single API like I said this is the same API that's calling anthropic one of our partners as well as is calling our Titan model for the embeddings and so here's some of the foundation models I mentioned already that uh we're using anthropic and their model is called CL is called Claud we're also using um Titan which I didn't even put Titan on here but it's uh the Amazon model that's the names of these models tighten tighten text tighten embeddings you can also if you wanted to use Sage maker uh and you could use sagemaker jump start which actually I've come to love I didn't really use it that much before and it's super easy to get these models up and running and you're not you know fighting through all these API calls it's just literally a sagemaker jump start uh deploy and these models come up and you could start making predictions right away uh and jump start works with hugging face works with llama to actually f um yeah sorry Bedrock yeah also supports um uh llama 2 as of I think Monday of this so was that two days ago I believe they launched it so I had to add an meta down there uh so that's Bedrock but if you want to use uh sagemaker jump start uh pretty much all the popular models are available flant T5 Falcon Falcon 180 billion all the good ones okay and this is the actual libraries and Frameworks that used uh so right like besides K which is the obvious runtime we also use streamlet for the chatbot UI that was that fancy UI right here yes every streamlet app looks alike if you don't uh customize it and we're using this Library called rag API which is really just a fast API uh like implementation that works with a library called Lang chain which I'll get to here in a bit uh and then feice is the vector database really it's a vector database library and it runs locally within each pod uh and so that's the part that that I'll explain how to scale that uh in the architecture uh review here in a bit who here has used Lang chain yeah yeah Lang Chain's awesome uh so this is an OP Source framework uh it's really many things and what we're using it for is to or orate and to orchestrate between the uh chat history so saving the chat history out to Dynamo so all of these questions that I ask uh get uh basically combined into a single you know so I can keep extending these questions I can ask about Bedrock I can ask about more specific things about bedrock and all of these question answers are stored in Dynamo uh that's our no sequel you know similar to Cassandra uh and also uh L chain helps sort of orchestrate all this helps actually orchestrating the retrieval augmented generation which I'll go into here in a bit and then also uh of course the prompt engineering aspect of this so um yeah what is Lang chain their uh latest sort of tagline is building context aware reasoning applications and this is a a you know mouthful the context aware part is the fact that I'm not relying on just the actual llm the large language model to know the entire world that's just not practical right so what like Lang chain gives you the ability and there's right like other Frameworks uh llama index there I think there's there's probably about 15 or 20 of these uh a startup I've been working with called grip tape also uh yeah that does something similar and has like Enterprise support and all that as well but the context where part means I'm going to not I'm going to first you know uh maybe do a web search for the the concept that I'm looking for I'm going to then uh present that and I'm just going to pull bits and pieces out of it right or I'm going to look at the PDF that was uploaded scan it and just look for the pieces maybe before and after that have the word Amazon bedrock and pull those in and so that's the context and then of course reasoning is really what we're using these language models for these days right the initial days we were expecting the actual large language model to know everything and yeah obviously can't know about news you know new things that are happening that day uh and so but we could still use the llm to do reasoning and to try to figure out what does it need to do to find the next bit of information okay so context aware reasoning and of course we all know application so that's Lang Chain's uh like primary mission right now it seems uh last I spoke with those folks uh this is open source uh the famous MIT license it's composable it's actually really really well architected um and tons of support there's you know hundreds thousands of committers I'm sure by now uh and it's support a multivendor so we actually contribute to Lang chain of course um you know mostly on the Dynamo DB Side open search which is our like elastic search similar to that um and then also for the new Amazon Bedrock Service as well so yeah surprise surprise yeah like AWS actually is contributing to this open source project uh this is really at a high level I just took this right from the documentation uh because I was in the Uber and pulling up right when I was building this but uh so Lang chain really is quite a lot of things uh the fundamental is yeah obviously chains and so you can chain together right that's where the reasoning comes in because part of it uh can you know Federate out to one chain to get data from this source and then uh you know go to another source and then reason through all of that data uh there's also agents and so the term agent is a little bit overloaded these days uh but think of an agent like something that uh knows you know how to get data from some Source or run some python code or do something a a little bit more complex than just um you know calling into the llm okay um and then memory of course that's where the um chat persistence comes in and recently more recently this is becoming more and more invaluable these days is the evaluation side of Lang chain and actually I didn't even know about this um I I took I like literally took a week off last week to go to Chicago where I saw Dean at cucon came back and this was part of uh Lang chain and this is huge so the fact that you can start to not just evaluate your models but actually do the evaluation across like the entire chain right so you have these chains that could be fairly complicated agents you know doing their things memory coming in and out of persistent store and now you can actually put some some uh eval metrics around this okay all right here's an example I guess you know stepwise here we have a user going into the application um it's going out step two would be going out to the relevant uh knowledge sources step three takes the output that was retrieved that data that was retrieved from the knowledge Source passes it into the llm and then steps four and five really this is where things can start to get more complicated but the most simple would be just to take a look at what the llm came back with uh and then display that to the user uh and so this seems really simple but but people get really really crazy in a good way with their prompts so the actual um you know so you might ask find this piece of info reason through you know find this uh database pull in all this data summarize it for me and generate a table right uh or or some some visualization for example so this is not limited to just llms but typically when you're dealing even yes even with your multimodal models which is across images and across videos and and audio there's still a language component to it you still have to ask some question you know using natural language or give it some type of instruction okay and typically when you're dealing with these multimodal I don't have this in like the rest of the thing but while we're talking about multimodal you want to keep an eye on what is the actual underlying llm the language model that was used to train that multimodo model um more recently we're starting to see the Llama twos and the Falcons being used but even just three months ago it was you know the old uh first generation uh llama model which will limit the you know reasoning um like aspect so keep an eye out for that okay here's an example of just a simple chain simple prompt how much time do have nine minutes and so like I said people get get or can get pretty crazy with their prompts and uh this is really what I probably spend about 70% of my time uh working with customers trying to figure out the best prompt this is the prompt engineering part of it each of these models has their own characteristics certain models respond better than others to different ideas or or different instructions uh different parameters uh typically by the way if you say think step by step that gives the model this language model model time to actually think and uh which sounds really silly but if you you know think of it that way uh if you don't put think step by step at least in sort of the last week's uh generation of these models they sometimes might just respond with something surface level right or wrong um and so if you give it time to think which might take another few seconds then you actually get the response that you're looking for the like correct response uh and then the other thing too that uh customers seem to not know is that like you can ask these models to site their sources and so just say site the source for for this particular um and it'll come back with uh that like metadata which is really cool so here's a really simple chain it's just plugging in you can essentially parameterize um and it'll fill in uh oh what happened there try this side I'm on a 2019 MacBook I'm waiting for the new M3 to come out before I replace it so all right it's doing something okay see if this does anything let's give it another minute and then I'm pretty much done and then I could take some questions I guess um oh yeah I never showed the actual architecture for these okay yeah I don't know what's up here I can share the deck after but I pretty much talk through how it all works it was there for a second okay yeah we're back okay so there's single prompt and then there's uh multiple prompt okay so there's a lot of flexibility with Lang chain supports a lot of different uh sort of modalities uh the way that you want to use it here um at the end of the day so my big joke is we're stuffing The Prompt right and so that's really what like a lot of this is doing here is where're we're like defining templates you know yeah I call it case and then you're you know slamming in the actual uh values um into these templates so multi- promp L chain you can specify multiple um prompt templates and then fill them in as needed um okay so let's let's switch gears a bit and talk about the embedding vectors because this is important for the retrieval augmented generation uh side of things and what we're doing is we're taking documents like these large PDFs right not just simple things but actually large PDFs and we're chunking them so that's another concept that you'll have to sort of get used to in this rag world is chunking so you're you're taking it either Page by Page or smaller chunks um and creating these embedding vectors and so these you know vectors are in this this crazy high dimensional Vector space and what we can now do is do what's called semantic similarity search is right so these are old Concepts nothing here is like revolutionary these things have been around forever it's uh it's you know yeah like 2023 was the year of the vector store I think uh 2021 was the year of the feuture store uh this is now the vector database era um and it like gives you the ability to do searches not just for keywords right like a keyword search like we're used to but actually do semantic or like meaning search okay but this all depends on like not just the actual Vector database or the vector store itself but it depends on how you ingest these documents how you chunk them up right you so uh like you don't want to chunk too small because that has downsides you don't want to chunk too big because now you'll be pulling back too much and these like llms I should mention really only have space for you know maybe uh like 4,000 like tokens or words um yes open AI just I think last Monday uh just announced that they have 128,000 what's called context window and so people kind of get relieved by that okay now I can pass in like a 200 Page book or something right the downside there is if you pass in too much you're really I mean a this is going to require a lot more input in like output which is expensive into these services like open AI um and and yes stuff as well too but also it's going to take longer to sift through there's also studies that show that the model really doesn't do well if the data is in the middle uh if you you pass in a big chunk and you know the word Amazon bedrock and its description is in the middle of this hund you know this 200 Page book really it seems to do better on the edges on the top or the bottom uh these things depend on on the model itself but it's always best to pass in the smallest amount that's needed for the context okay so here's the high L architecture back to eks land we have multiple tenants here we have the famous Ingress controller um we're using ISO uh and you have the UI there's your vector store this is the actual data ingest side of things so you have raw data typically we see customers using S3 for this uh ingest it chunk it convert it into these vectors yes all of this stuff is just a single API way to convert these things in into these vectors this isn't that difficult um and then when we go to retrieve so this is where it starts to get a little crazy here actually yeah this isn't that crazy there's the chat UI that's running inside of a pod uh there's this sort of server side to the chat UI which is going to be Lang chain exposed through this rag API right now it's using feice locally so so every pod has its own local that's not going to scale so we know that um there's what's called a session table and that's how we can link multiple uh requests um and then there's the chat history table and then there's uh the Amazon Bedrock so let me dive a little bit deeper and let's see I think this is basically the same thing but just a little bit deeper showing the session IDs traveling back and forth um probably the most interesting thing here would be step five which is that the user input that is currently on the screen my my current question will get combined with all of the previous questions that were in that that single uh session right and all of that gets so we're using Titan uh which is exposed through bedrock to convert these into these you know huge vectors um and then that's what's actually being being uh sent over to Claud to then do uh this uh rag approach okay so excuse me here's the current architecture which is uh using that local Vector database just stored on disk not very uh you know scalable U but here you see the tenants are isolated by Dame space the chapot UI pod um that's running with the database alongside of it there's some other security things I'm not going to get into here uh and then this is uh one possibility here would be oh it looks like I didn't get the animations right let's see if this comes together here yeah so we could remove feice that you know local database storing it on disk we could replace it with either a Bedrock agents which is similar to The Lang chain agents where you can if you think of like a Lambda that can basically do anything right can call a Weather Service can call anything um also uh there's this concept called uh a knowledge base where you can uh basically just upload your PDFs and um yeah the Amazon Bedrock service will do it all for you the other oh yeah I think I skipped one of the options here which was you could also use Amazon uh like open search which now has uh a vector database uh Vector store capability within it uh this is what we see a lot of customers using because they are very already using open search for other use cases for their uh just keyword searches okay and Lang chain on AWS you can integrate with sagemaker endpoints for your own custom models uh you could use uh the uh like Dynamo DB integration for the chat history that was something we provided and Lambda support of course Lambda is the shim of all of AWS right uh and actually I should say like a little tidbit in the four four years that I've been here I did not realize how much lambdas are actually used internally for our products so I'll just leave it there but Lambda is a very popular Service uh that's like behind the scenes even services like eks um use like lambdas everywhere so yeah Lambda agents can call out you know send emails and that's it so here's the links to everything blog post GitHub thank you