SBTB 2023: Ben Church, Creating Context: Loading your life into a vector database

SBTB 2023: Ben Church, Creating Context: Loading your life into a vector database

Recording: SBTB 2023: Ben Church, Creating Context: Loading your life into a vector database

[Music] um anyway just want to throw some props to the keynote uh some other props is due is for everybody who's attending today that's all of you in the crowd um without the attendance and the volunteers who put this on um I think our industry would be a lot more boring and would not progress nearly as fast so thank you for coming now you didn't come here to hear me say thanks to you I know that you came here to for me to talk about Vector databases now before we can do that I think a short little intro is in order uh who am I uh I haven't created a programming language I don't have a PhD in physics and I'm just from a small area on the west coast of Canada on Vancouver Island it actually took me three flights to get here today um I work at a company here in the Bay Area called airite you've probably seen their logo around um and I'm just an engineer I just build things and that's what I want to tell you about me is I love to Tinker I love to use this new technology and I love to teach so today what I'm kind of going through is myself being a masochist and find all these potholes and building these llm applications so you don't have to right I fall in there and you can step over me now I made the title to this talk months and months ago when I was kind of cooking it up loading your life into a vector database right we're going to be building an application where we take all of our data from all over and put it into one data store so we can start using it in our llms but as I was going through through this I think there's a few more titles I could have chosen right could have called this stubbing your toe on token limits right I could have called this RG Against the Machine retrieval augmented generation you know bad puns is what I'm after here and we're going to get to this but I could definitely have called this stealing my friend strawed data so I don't have to exercise like I promised my wife I would making this talk right but all these silly titles aside there's a through line through this there's something I want us to acknowledge and carry through the entire of this session and bring with us you know afterwards and that through line is you know new systems have new possibilities it's what gets us excited about llms and the things we can build with them but more importantly these new systems have new shapes they have new constraints we have to work around and new things to consider that you don't know about until you start building with it right so in this talk we're going to be talking about the constraints of a modern llm based system we're going to talk about the considerations that come afterwards and how you work with this new technology after that we have to think about what are the second order effects when this becomes standard when the best practices become known how does that change our industry and we're going to be look at this through an extremely simple application that I'm sure everyone in this audience has seen before you know along the way we're going to talk about vectors and Vector databases and architecture uh context stuffing token limits graphql you know the types and introspecting schemas um and I am I'm positive there's going to be someone in the crowd who's mad at me for you know using graphql I'm sure there's going to be someone upset with me for using JavaScript and not python or not scholar or not Swift uh sorry Chris um I'm not using postgress that's going to be an irksome point I'm using a weird variant of node.js called bun and I'm using my employer technology like even though they're open source and awesome I'm sure someone's going to have a bone to pick and to that I say I don't care this is my talk this is not yours so enjoy come along with me okay so before we get into the building we have to kind of get on the same page we have to share some context so I'm curious just like a show of hands in the crowd who here has built something even a toy application using uh an llm okay good good that's a good amount and how many people have used vectors and Vector queries and Vector databases oh perfect nearly just as much all right so for you that's going to be a refresher uh for everybody else this is going to be some prerequisite knowledge to start building with this so the first thing is what does it mean to be a simple Le llm application what it doesn't mean is query caching or query optimizers or personal data filtering or any type of UI what we're going to be focusing on is a subset of nearly all of these types of applications which is retrieval augmented generation this is the part of the system that goes and grabs context for your user right this is your business data this is recent information that your model has not been trained on where you take a question you turn that question into Data you use that data with a query to finally get your answer for your user for your system there's a Twist though we're going to do retrieval augmented generation with graphql right we're going to take a question it's going to go into a simple application we're going to take that question ask an llm to turn it into a Gra graphql query we use that graphql query to then get the context then we feed it back into a final prompt to answer the question there's kind of like a two prompt approach here um there's technically a third one but I'm gonna wash over that it's for an example record okay so right away we've kind of arrived at our first constraint in the system right all these systems have constraints and the first one is unstructured to structured data the questions you get the inputs from your users they're not like they used to be they're not query parameters they're not URL slugs they're not forms they're just a question and because of that it's very uncomfortable for us traditional developers to work with it I can't go name equals form. name anymore it doesn't work I need a way to go from this unstructured input to some structured context and this is where vectors come in now often this is refer referred to as embeddings in ml and AI I'm here to tell you it's the same thing they're just making up words right um what a vector is it's just an array of numbers now that array of numbers in this case we don't really know what they correspond to it's long it's uh basically unparsable by a human but we do know is it's a indication of some kind of similarity some kind of um contextual value to the model that created it so we don't know what the internal weights mean we do know if we have a vector from a model and then another Vector from the same model we can see are they similar right if we take the query what are common household pets we imagine that question Vector is going to be very similar to dog cat parakeet and a little less similar to salamander and definitely not similar to kodo Dragon right it's a dimensionality but we use these vectors for similarity searches so I can go hey they've asked this question give me all the documents similar to it given this threshold right nothing too complex there nothing too scary and Vector databases are just purpose-built databases around the concept of vector indexes they can also be plugins like PG Vector um but they're optimized for similarity searches now there's a few different algorithms to consider when you're choosing your similarity search but we're not going to dive into that just know when you hear Vector database it's a database with a vector index and some extra nice features now because vectors aren't unique to llms and AIS they weren't invented by it but they become more important because of it and because they become more important Vector databases become more important and because they're coming up at the same time we're starting to see these tools add very specific features for the AI use case and that's an important thing to consider when You're Building these applications right you don't have to use a vector database you can roll your own but I would caution any small team or a new product think about using these when you're making an RG system because you don't have to reinvent the present and you certainly have less of a chance of Reinventing the future because as these systems grow and build you're going to go along with it they're going to keep adding features as we discover best practices okay and now for anyone here uh one more show hands I promise my last one anyone here work in analytics and moving data into a data store there we go okay we got nice quarter of the audience so this is really similar to U analytics database use cases right when you want to ask questions you need the data in one places when you want to do Vector comparisons you need the data in one place so consider this when you need to ask questions and get context you need to start considering how do I move all this data into one place and how do I have one entry point to query this data right uh you might have multiple llm clients that all need ways to interact with your documents and your vectors okay there's a context let's start getting into building it right let's make our way first step we have to load our data now this is where I'm going to irk some people by starting to say airite but if you believe me that we need a central data store that means we need a way to load things into it and in the past I think this has often been cobbled together shell scripts I've done enough splunking in my career where you know you're like I just need to get data into this database and you write like a small little bash script or a small little python script and you and you leave it and you put it on Cron job and you know two months two years later it breaks and the person who built it left and you know there's no tests and it's really painful so I don't want to dwell too much on how to load the data but I want to give a very strong recommendation to consider using a data platform doesn't have to be airite though again I think think it's great but the purpose is you want to be able to load data from a bunch of different stores into a small subset of destinations and you want it repeatable you want it tested you want it robust and you don't want to waste a lot of time on this writing and supporting oneoff scripts is a pain so for loading the data uh we're going to be loading data out of straa Air's going to grab the records and then we're going to load it into we8 which is a vector database that I'm using and why I chose we8 we're going to come up in a second finally we8 is responsible for vectorizing the inputs so it's got access to my open API key or open AI key and then it'll transform it into an embedding and to give you an idea of just how simple this setup was for me it's set setting up we8 Cloud now we8 is open source you can deploy it yourself but I'm lazy I went with a cloud offering it was free perfect you know I'm a little bit cheap it was two clicks create cluster save cluster done the next step in here is okay if I've got a data store I need to move data into it in air bite we call that a destination we have a large catalog of destinations we8 is one of them all I had to do is Define my API key for weate my API key for open Ai and hit save now the source part in this example is actually just slightly more complicated we have a large catalog but in some cases where you have a more nuanced API or maybe something that's more bespoke or obscure you have a custom connector Builder this is kind of like a wizzywig for building these um extractors these data extractors so for me I had to make one for straa and I think it took about five minutes I had my API key for the data and I just had to define the roots I was quering the pagination save and we're able to start syncing so this total process of getting a vector data store up and then getting my data into it is approximately 20 minutes if you know your tools and if I need to keep adding more sources it's another five minutes of investment every time that saved me uh probably a few weekends and definitely some of my wife's time and enjoyment with me going out mountain biking all right now this is a time I have to thank a friend of mine Melissa I started this talk and I was like you know what if I get some Strava data into this that's going to incentivize me to bike more and then I'll get more fit and exercise and I'll get a six-pack and all that and uh it didn't happen I I stayed on my computer I stayed on the couch and as I needed the data for this I'm like I need to go find it and thankfully my friend Melissa who is a tremendous mendous athlete and has um some arguable T takes on who she gives access to her personal data gave me her API key and I got to use it for ingestion for this okay so thank you Melissa now step two here is once the data is in we're able to query for context and arguably this is the most important part of these retrieval systems so in the spirit of time I'm not going to open up the wv8 dashboard but I will open up my terminal at some point here I promise but with these in the database I can start querying I can start asking and I can start asking for objects like activities but really really important I can start asking for slices I can start asking for specific feels right and this is where wee v8's use of graphql is really important so why did I choose we8 why do I keep mentioning graphql and why am I a fan now graphql has a buil-in ability to get the schema as an API client right it's self-documenting you can introspect these apis and with one call I can get all of the fields available all of the objects all of the input and output types all of the mutations I can as a model as an llm understand the API without me having to write any code right and it comes along with first class descriptions so we can start to annotate objects with what they are how they're used what the relationships for again all very important context-rich information useful for an llm now to C you go back it's worth stating there are alternatives o data is one um open API is another grpc or trpc I think there's a type version of it these all fit that use case as well I just think uh graphql fits it best and if you want I'll be around after if you want fight me on that so there's another reason is inside graphql we have complex filtering um these llm models which we'll show are bad at filtering if you give them too much context they don't know what is useful what is not a lot of the time and they are extremely bad at math they're very good at going one plus one is three so you need to give them the ability to ask for facts without trying to generate them themselves that's where aggregation in an API is important summing counting averaging medians Etc you don't want your llm to do math for you because you will be wrong often finally this use of a very flexible schema that you can view solves two of the most important constraints the first constraint that you have to consider when you're making these applications is token limits there's a limit to the amount of content you can jam in by default with I think 3.5 Turbo with chat GPT it's 4,000 tokens thankfully last week before the talk they announced four turbo which has 128,000 token limits and made a lot of this possible without doing some really uh uh some minimizing right but you have to be aware that there is a limit and the limit set by the model it can be low it can be high like 128 but there point is there's a tension right so between token limits and meaningful vectors think about it this way if you want the most context the most um useful Vector you need to create it from a very fat object right like the full pet the name the uh species the owner all of that into one vector however when you're using that object when you retrieve it you actually need the thinnest slice possible right maybe you just need the name maybe you just need the age maybe you just need the owner but you need some way to say I can retrieve this and then I need to slice it down and worth pointing out here this is the same reason why time series can be difficult right I can't say hey what is the average rainfall of all time you run out of context limits right you need to be able to thin slice and and sum these things up and this L leads to the second constraint imagine like if you imagine a world where there is no token limits even if that's true we still run into a too much context problem the more context you feed these models as that context increases your chance of it hallucinating and telling you wrong things are going down wrong paths increase your answer quality decreases so have this weird Balancing Act of relevancy that you have to be always on with these systems so that's your consideration here is when you're thinking about this you have to really think about how can I get a flexible and accessible schema and query language right how can I give the most power to query back to the model okay now we're getting in at the 20 minute Mark 2/3 through building it putting this together I've got some slides but I'm actually gonna jump out of the slides here and I I hope everyone in the crowd just cross your fingers because when you start opening a terminal this is where things go terribly wrong um bear with me as I mirror display range there go so if you didn't know this uh screen is called decimator which is that's interesting okay so here's my editor here's the terminal uh the part of the code that we're looking at is just demo and I'm going to fly through how this works and then we're going to run it so go down to the ask question this is all of 80 lines a lot of it's logging but to refresh your memory on how this is supposed to work is we're going to get the question as an input we're going to transform it into a vector we're going to get the graphql schema from we8 as well we're going to use that with a prompt to have it generate us a query that will retrieve an example record we'll then use that example record and the schema again to do a another query we're going to create one more prompt and say hey given this schema can you and this question can you create another graphql query that's going to ask the database for all the context I need to answer this finally when we have all the context we go on to our final prompt which is just with this context and this question and some of this like coersion logic can you answer it oh is is it okay let's go theme is that better okay great sorry about that okay so going on to the final answer and the final prompt along the way I'm logging these two files so on the left here we have logs they're checked into source code so you actually see them get marked as they've been updated as we run this and this is where one of those other contention points goes I am using typescript but I'm not using node I'm using bun just because it's got some nice watching functions so let's kick this off and watch this go so as it's working through we're going to start stepping through uh what's being computed so right away the question that we have is how many virtual ride activities have I done in total relative to all other activities that's the vector version of this question you can see just how huge it is and then we move on to the introspection we have this reduced schema where we start to see the types of objects it has access to and how to query them we see the example record query get generated this came from the llm it wants to get a whole record of an activity we see the response this is the example record it was given we see the query it generated to get all the context you'll see we got the huge Vector piped in as a filter and it's just looking for types finally we see the response there's a couple virtual rides and we see the final piece here based on the query provided there are two activities now this is the funny part is is if I save this again and it reruns even though I've set the temperature to zero and I've said make this deterministic give me the same result every time it won't so let this go through one last time and we'll go back to the slides and I'll show you some of the other queries we've been able to get it to generate great we got virtual rides coming in let me go back to the response let this load okay I got the five minute warning so I'm G to sneak back over there we go okay back to our slides back to the share and there we go okay skip through so that was the that was the application it made multiple graphql queries without any intervention by myself we went unstructured to structured data and we got a final response I ran this a few times over the course of the week and started giving it Mo to use Aggregates and sums and groupings and you can see here this is a bunch of ways it did query our Vector database you can see it summing you can see it uh going by lat and long when we start asking about which rides did I do in New Zealand it it was pretty cool and impressive the other things that were not impressive is like I said before it's bad at math right here it's off by a thousand before I saw it off by a factor of seven uh it it it varies a lot and it's non-deterministic right here we have the answer to the question you just saw but it says 43 it says 8 it says zero so you have to be aware of this now we went through a lot of constraints and how these work and my only wish for everyone in the audience is for your next project if you are doing retrieval you have to take this with you which is llms are limited by their tokens they are bad at math they are non-deterministic context and meaningful vectors are at odds with the token limit you need a strategy for pulling out the most amount of context in the least amount of text is a very core principle and your data models your apis and your air messages are all much more important in this Paradigm you need to think about what air messages you're sending back how you're exposing types how you're exposing your schema it is now first class not a kind of side effect thought finally I want to leave everyone here with some thoughts for the future right so if self-describing apis are now first class what does that mean for your errors do we need an llm on the other side to start communicating back right does that Loop of query air query ER query error becomes so important that it's no longer valid to say just look at the docs right and the other thought is if llms are both important and struggle with concrete facts does that make the data engineering profession more valuable not less right do we need to be much more aware and attuned and invest more into pre-computing a lot of facts and doing a lot more data wrangling than we're even doing today finally I think we have to think about what do llms mean for the world and specifically are we Frozen in time now if it's easier to work with programming languages where you have co-pilot that's been trained on your programming language does that mean it's now harder to pick up esoteric and new languages is it now harder to pick up new Frameworks in libraries you know did everything just get frozen solid and did culture get Frozen at the same time right does Tom Cruz with generative AI end up living forever and just become something you inherit and that leaves me with the final alternative title to this talk will Tom Cruz live forever God I hope not but probably okay so thank you thank you for your time um I appreciate and I'll be out back if you have any uh questions for me thank you uh five minutes for questions here yeah oh okay uh two questions one just uh because I'm dumb why why the second step of sort of hand it over the llm creative actor then go back to graphql and create create another round of vectors there was this what it seemed like a two-step in there I didn't understand what that one was for and then the more broader question which I'm gwing on in general is how do I lash a non reliable probabilistic model into a deterministic workflow and to your point like how if I can't rely on this thing how do I put it into a workflow chain if it's unreliable gotcha gotcha so uh second question I understood was on um if they're unreliable how do I make them more reliable or how do I depend on them yeah how do I use it great great and what was that first one just first question was just simply like there was another step in there where you yeah I took the query handed it to the llm it sends it back to graphql graphql pumps something out then you pumped it back into graph you ran it through the llm and dumped it back into graphql before you created the final query why that middle step gotcha gotcha okay so uh I'll start with that first one the reason why we had that two-step is we need to first get our context and because we have unstructured to structured data that that requirement our best way to do that is through an llm right we have to say hey someone asked me this I need to make a graphql query but I don't know how to tie them together can you and then once it gives us back that query we have to take the query go to our own data store our own apis pump it in get the context out and then you have that second question of okay now that I've got the context I still have that question can you actually answer it now so it's more of like a a flight control kind of thing and then that whole unstructured to structure data um the other point on these are untrustworthy characters how do you trust them this comes down to you have to be very um on it with your monitoring you have to have some kind of query optimization but also query monitoring you have to be thinking about how can I keep tabs on this system how do I know that I'm still getting the correct outputs and you have to do a fair amount of QA when you're building these prompts um so it's mostly a monitoring problem and then secondary it's asking the right questions here I was asking it to sum for me I probably shouldn't have been doing that or if I'm doing that I should give it probably some more prompts to be using the summing of my API and not its direct summing that makes sense yeah there we go any other questions people are taking pictures of this I love that because I spent the least amount of time on this one we'll be at uh QA table or Q&A table number two out side afterwards so okay great hey thank you again everyone take [Music] care