LLM + Graph Database for RAG

Bay.Area.AI: LLM + Graph Database for RAG, Andreas Kollegger

Recording: Bay.Area.AI: LLM + Graph Database for RAG, Andreas Kollegger

thank you everyone um for for coming by today thanks for the the previous speaker that was really great stuff I'm going to dig into r lator i want to actually check out whether we can plug in near for J into that whole uh framework that sounds like it'll be a lot of fun to do so wherever you've gone let's tag team on that later on let's let's get that going I going to talk to you though like in terms of like llms and the entire universe what we're doing with Gen I'm in the world of rag and on the database side here for J is a database so that's what I'm going to talk about today LM and graph databases for Rag and really the entire talk comes down to this this whole idea about having graphs inside of a rag workflow is putting rag in context heard a little bit of context in the previous talk it's always a part of what you think about when you're building rag Solutions and what I mean about it is something very particular in terms of graphs this is another you've probably seen a million variations of this diagram basically at this point this is the basic rag flow somewhere there's an application that takes input from a user sends it to the LM gets the answer back you saw a version of that in the previous slide pre previous presentation same idea the extended version of this which you also saw a version of in the previous talk is when you go ahead and you don't just take the user's input maybe do some prompt engineering and then pass it to the LM you actually talk to a data source of some kind first package up information from that data source to provide a bigger context that hopefully is relevant to the user's question then pass it back to the llm and then the user gets an answer okay that's the full rag round trip everyone has seen this a million times right has anyone okay yeah we're all on board here okay this is all the easy stuff this is where it starts for me to get a little bit less well explored because we tend to fixate on the llm side fine tuning the llm getting the right llm also pick picking the right model and even on like the the database side to be honest like what does the model actually look like but when you think about actually building end user applications it it isn't just those two things that matter it's not just the llm and just the database and maybe just the orchestration framework you want to think about the full stack of information where it's available what's available and what is relevant because as we know from the memes the person who's asking a question of an l M the way you phrase the question changes the answers you get the same is true when you're doing Vector search how you phrase something might end up getting different results and so you actually want to end up building solutions that takes into account that there's some user that maybe you know something about and maybe that user actually knows something about what they're asking and then of course you're building an application that hopefully does have some of that information about the user so taking that into account while you're doing your round trips then of course the database itself knows something about some domain and then the LM itself knows either whatever found on the internet or whatever you have trained it on uh that's what its expertise is around you want to take all that into account when you're building ENT to end applications for sure and I'm going to get into some of the Nuance of that as we talk through I'm a software developer at heart so I'm not a data scientist or an ml person so you're gon to have to excuse some of my software biases as I go through this but for me this is the part that I can control I can control the application that can control the database people in this room some of you control the llm build the llm that is not my world that's a separate conversation I'm going to focus on this middle bit and when I think about that middle bit um immediately I start to think bucket like what kinds of information I have because depending on the kind of information I have what I can do with it changes that's what I'm going to talk through next there's these three sources I talk about and you've heard these terms before I call it pure text more generally this is unstructured data that where this is where you're talking to your PDF okay great you've got some unstructured data there's information in a bunch of a corpus of text whether it's PDFs text files whatever it might be maybe even just multimedia files videos and things as well fine that's the pure text world of information that's available when you're building an application that you think of that as being your only extra source of information what you're really doing is building a system for information search it's information retrieval in computer sciency terms it's still information retrieval that's the r part of it right but very particularly this is just searching for information you're hoping that for the user's question there's a good mirror of the answer in the data somewhere and so you're just going to do a search and there's different techniques of course for doing that search including including of course Vector so if you had business data you could ask questions like okay what is Apple's primary business hopefully somewhere in the text it says something like oh Apple's primary business is whatever it is find that you pass it back to the L come up with an answer way on the other side and this is less commonly I guess where people start but for Enterprises people have heard of things like data warehouses and data lakehouses and Lake warehouses all the different variations you already have data not unstructured data but structured data in a database somewhere and you also would like to query that the approaches you have for querying that are different than when you've got unstructured data right you already have query languages around and so the approach of course you take is rather than hey let me take the user's question turn that into a vector embedding do a similarity search you start with okay let's see if we can take the user's question and generate a query SQL or for us Cypher query language is a graph query language from that natural language run that get an answer again hand it to the llm get that packaged up to something that looks like Shakespearean Pros right ultimately that's what you want shakesperian pros of how many iPhones Apple sold last quarter what's awesome about this is if you know techniques on the left hand side you know techniques on the right hand side you can combine them and then you can get kind of The Best of Both Worlds you can do some Vector search stuff you can do some text to query language stuff and you can combine these techniques for querying across all the different data that you have accessible to you kind of out in the wild this naturally occurs if you happen to be running a WordPress site you've got a lot of text which is really great for Vector search and then you've got a lot of structured data around that text as well you actually want to combine those two available data structures the unstructured and the structured structure um in your queries and so this ends up being opening up actually it's not only that it's a different Source the kinds of questions you can ask end up being qualitatively different it's not just searching for something or turning a a natural language question into a query and running that it ends up being information Discovery where you kind of know a little bits and pieces of what's going on and you're trying to bring it all together into a relevant context your pack packaging up into a prompt and then passing to the llm you're combining things at this paint when you bring this all together in one kind of unified data model that's what ends up being a Knowledge Graph so for me this is my definition of a Knowledge Graph um there are a million definitions of knowledge graphs probably everybody who's ever done anything with the semantic web or any kind of graph whatsoever will have their own version of what this is for me I think about a knowledge graph is just being an information architecture um for data using graph structures basically that simple and of course within gen the whole goal of this the reason that you might want to do that is for placing all their information you might find within a context that matters both for getting better answers but also for ranking those answers so both expanding the context and also then ranking the context the access patterns you end up coming up with when you're working with a Knowledge Graph broadly the umbrella term is graph rag graph rag as with knowledge grass and everything else will mean lots of different things to whoever is doing this kind of work but the umbrella term is assuming that you've got something that looks like a graph here's the different access patterns you can use for accessing that graph in a rag application so graph rag the techniques under the hood of course as I've already kind of talked through if it's just text you're probably going to start with Vector search that's the reliable way to go you always start with that on the data side it's the text to query language generation start with that and in the middle it's a Well it kind of depends you actually have to do a little bit of you know thinking here not a little bit there ends up being a lot of art here driven by the kinds of questions you're trying to ask that fail to be answered on the left hand side or the right hand side you start to look at how can I combine these two things to be able to answer those questions so again kind of in a software developers mindset this is test driven development for me so for in in this context this is really eval driven development you can continue to evaluate what the questions that you can ask are actually what your possible questions are and when you run into boundaries not only are you getting good answers but like bad answers but answers that are not possible at that point you're like okay let me change my approach to see whether it's possible to answer that question in some way using a combination of these techniques to make this a bit more concrete I go to example and that I have some some notebooks around uh pulls from the sec's Edgar Database The Ed database is the public database where SEC filings become available to the public um this is the stuff that if you're a publicly traded company you've got a file with the SEC it ends up in this database also if you are a large management company that's you've got a large portfolio of Investments you also have to report what you're investing in so what the company is how much the dollar amount was and how many shares those kinds of things so using this data set using actually these two form it's the the form 10K and the form 13s the form 10ks end up being really great for just text so for the unstructured data we just go ahead and pre-process all that and as you all are probably very familiar even that step as easy as just say but it's the devil in the details there is just ridiculous if you go off and look at the like the raw data you get from the SEC it is just a big mess of awful XML and trying to pick out the bits and pieces that you want to is non-trivial that's another night's talk that maybe I'm not going to give and hopefully somebody else is going to give I'm going to ignore that let's assume somebody else has taken care of all the data preparation you got the relevant text out that becomes what we're going to do Vector search with the other form the form 13s uh also has text but kind of care less about that text at least you know I did when we put this all together what I cared about was simply the the data of like who's the company who's the management investment company and how much did they invest in some public company take that we actually ended up generating a giant CSV file out of that that became our structured data that we then linked to the unstructured data that ends up being the overall data set the philosophical approach I have for doing this data modeling is different I think um and generally this is true for for graphs it it is very different than how you're used to doing relational data modeling if you're used to doing relational data modeling you end up with a relational mindset trying to anticipate all the things that might be necessary in building out a really beautiful schema and then slowly over time that schema gets destroyed as you have new queries that can't run or that run slowly or whatever that is right with a graph you don't take that approach with a graph I recommend taking approach of saying let me build the smallest thing possible that is useful the minimum viable graph that I can ask some questions of and then at the point where I run out of questions that I can ask then I elaborate on that and so start with a small graph pick up out interesting information enhance that information find out if there's some Dimension that you can basically add an index for and then connect it that is the tiniest graph you can create in this data set it ends up being take that text do the classic text splitting create a bunch of chunks and this is not actual uh 14k data this is lurm ipsum apologies about that um my copy paste skills are poor so that was easier to do the natural thing to do with text of course is go ahead and then do a vector embedding for all of those store both the vector embeddings and the text into the graph at this point you already have a legitimate graph it doesn't have edges it doesn't have what we call relationships in in kind of Knowledge Graph land but it's already a graph because you've got it in nodes these nodes are backed by an index you can do the vector search so that's already pretty nice but to make it slightly bit graphy without getting too crazy with designing a whole graph around this the easiest thing you can do at this step is just making a linked list you know that the chunks came in order preserve the order why throw away information the easiest way to do that is just say okay great this chunk follows this chunk follows this chunk have a series of next relationships across the chunks so this is the the pattern that I you know had a couple slides ago create some records enhance the records connect the things repeat repeat repeat until you get to a place where you're happy even at this stage there's two things that are nice about this um for doing just Vector search you still land on chunks as you normally would but because you're in a link list you have an arbitrary kind of Chunk windows that you can pick you can go before and after this just by following out the link list grab that stuff up hand that back into the context and off you go as you know and I think as as the previous speaker mentioned how much chunking is the right like how big are the chunks how big are the windows completely depends on the data set what we have actually here in the this example that uh worked on our chunk size is completely arbitrary we're like I don't know what's relevant here let's do 2,000 characters that kind of thing um typically with this pattern you also you don't do any chunk overlaps if you're going to have a link list because if you want overlaps you just increase the chunk window oh right the other kind of quality of this is and this goes all the way back to my earlier slide about keeping the end user in mind uh the point of this isn't just to help extend knowledge available to an llm but to give good answers to a user and so for the user they might want to know not only how's the llm turned this into Shakespearean Pros what was the original text not just a pointer to the text but here's the original text that was there and in kind of boring UI terms can you go to the next chunk and previous chunk now that you've got them in a nice length list you can just combine a sort of Chatty interface with a more traditional let me just page through the data interface it's up to you to design like what is appropriate for your context following the kind of iterative pattern here the next step that is remarkably useful but often underappreciated and you'll see it in most of the kind of orchestration Frameworks there's some some version of this don't just have the chunks realize those chunks came from a document originally all the chunks could have metadata about that document but really the metadata belongs to the parent document so goad and create that parent document and then in the graph just recreate the structures so that from each of the chunks you can get to the parent and here there's a parent relationship that the section relationship points to the head of the list so if you're navigating from the form back down to the junks you can get to the head and kind of go through it and along the way this is a great opportunity to actually do a text summary take all those chunks run a summary through the llm store that with a vector embedding on the form you now have at least two different options for doing search you can do on the chunk level granularity or because you've got a summary some questions might be better answered by that summary and of course might be wrong because you're trusting in El to generate it and depending on all the things nevertheless you've got new options available to you here so there's already some nice strategies that become uh possible turning over to the structured data side of things um this is also a very simple graph but reasonable for the data set that we have uh the badly named manager nodes are the investment companies and then the company nodes are just the publicly traded companies but so these management companies have invested some amount of stock they owned some stock in these public companies so just go ahead and create that in the graph as well again this is just reading from a CSV file in the in the data set that I have the demo um there are values you'll notice both on the nodes but also on the edge itself the relationship itself can carry arbitrary properties um so here it's the what the number of shares are and what the value those Shares are that gets put into the edge and kind of you know maybe surprisingly there's no reason to put a vector search on like the names of the companies you could do that but you don't really want conceptually similar company names you want textually similar company names that's really a full text search not a vector search so you do a full text index on on both of the the names for the companies and the investment firms this is on the structured data side this already you know opens up again this is now the third I guess option like access patterns that you can have you can already start to query this from natural language this is what I alluded to earlier in in the the deck where this is take natural language turn that into a a query run the query get those results for us you'll notice here in one of the one of the dots is a pattern matching so in the graph query language in in Cipher which is now as of last week in ISO standard so great for us now there's SQL and the unfortunately named gql so sorry graphql we're going to have a a name battle out in the wild um but let's just call it Cipher for now the the premise of that query language of the graph query language the entire idea is do pattern matching and it's pattern matching across records that if you had a relational database you could still do this kind of stuff relational databases and also SQL the query language is really optimized for within a table let let me do operation there you can do a join to another table but it's really awkward to do many joins and express that in a nice way a graph career language basically turns that on its side a little bit and is optimized for going across tables is one way to think about it as soon as you start to think about how records how you do queries across tables as the primary way of accessing what results is that you end up describing patterns of records that you're looking for that's what the query looks like and I'll show some examples later on here the simple pattern would be find me patterns a manager who un stock in some company and then you've got like a wear predicate that you can add for like what's the name of the company or what's the name of investment firm or maybe you're limitating it by the the amount of the dollar investment or something like that but it's kind of SQL like at that point okay uh the one more step of elaboration that I ended up going through with this particular data set because I have addresses for for the each of these entities pull those addresses out because now we get to ask questions about near nearness which uh again is awkward to ask about in Vector search unless you happen to have a document somewhere that talks about companies that are you know collocated um really easy to do in a geospatial search it is just like vector search but like with you know two Dimensions so you know X and Y are Latin long still basically Vector search right so kind maybe it is still Vector but you pull those out into separate nodes now you can do both pattern matching through location if you'd like to do that or you can do nearness like within 50 km or you know whatever the criteria might be so that's yet another option for doing queries on the data set this lets us get to the point finally where we've got some structured data that we can do some interesting things with we have unstructured data that we can do kind of typical rag things with connect those together we know that these companies have filed those forms make that final connection and now all the different individual access patterns that are possible you have the great pleasure of trying to decide how to choose which strategy at which time for the given user's question that is amazing because you got many options that is annoying because you have so many options and like doing a perfect mapping is a non-trivial task the thing that ends up happening there you've got two kind of main strategies for the strategy I suppose you either reduce the kinds of questions you want to ask and have them be here's the things that we're going to allow and have those be really good or if you want to open it up to any kind kinds of questions and don't want to shrug your shoulders you're going to have to do a lot of like round trips of like strategies basically going round trip with the llm and kind of an agentic type workflow saying hey for this question and given these tools that are available these agents that are available what's the right strategy for answering this question and then there's a bunch of subflows that end up happening there any of which I'm happy to talk through all of which are awesome and annoying because there's so many things to do okay this is the summary slide of the what would be called the the the knowledge graph construction that I just went through from just junks to elaborating on that slowly growing a graph around it each time going through this same kind of process create some data enhance the data connect it to things either connected to itself or connected to things that already exist keep going like that again until you can answer all the things you want to this goes on as much as you'd like what is again both awesome and sometimes frustrating about a graph is there's no natural boundaries to it as much as you'd like to put into it as much as appropriate and useful for you to put into it you just keep glomming things in that is amazing it's a Temptation that you should resist is in the same way that I recommended you start with a smallest graph that is reasonable I also recommend that you resist growing the graph bigger than is necessary for what you're trying to get done focus on what you're trying to get done for for the end user but the possibilities end up being I didn't do anything like this yet right you can still do like name density recognition from all the unstructured data go ahead and like pull out the people places and things into the structured data as well connect things through that in this particular example and many other sort of legislative like examples there are often cross doent links that are explicit great pull those out as well have the chunks themselves connected to each other and then maybe do some postprocessing do some page rank on that now you know which chunks are the the most influential chunks in the kind of Corpus right all that information the more you add to the graph there are a few things that are interesting about that like I've been talking mostly about the different access patterns that become possible also within the access patterns you get new opportunities for ranking basically like running the page rank is a good example of that if you end up with if you have a million chunks and you know you're getting uh your top K is like 100 they're going to be hundred of the same thing that's maybe not the most useful but if you then like take a thousand and use page rank on that actually maybe you're 100 top 100 is going to be more useful then right um because you want not only what is most similar but what is most relevant and useful not just most similar most similar itself is just one criteria for picking what's helpful for answering a question um add in the user data keep user history keep the user memory as some of the Frameworks like to talk about and along the way give the user a chance for feedback record the feedback and again use that for enriching the graph so that over time as a particular user uses the applic that you're building for that user it continues to improve possibly for all users it continues to improve this is to some degree The Mechanical Turk of of labeling of course let your users curate the data just by using the data and saying that was a terrible answer chat PT does this why shouldn't we all do this as part of our apps right thumbs up thumbs down is enough to kind of help things out okay um I'm going to switch over to some notebooks to give a quick look at like what the code looks like quick preview of what the the the cipher looks like for doing queries with the graph query language um before I do that I'll give you a chance to take a look so it's the top bit there is The Notebook that I'm going to look at in true demo fashion style um I was sitting on the couch there you know kind of trying to run through my notebooks and realized that I had broken the data model and some of my careers are broken um I'm going to run them anyway and we'll see what we get some of them will work some of them will not work I apologize for that it is Live code um and that is the nature of the Beast before I do that though I do want to take is it okay to take a couple minutes right now to get some questions before I get into the demo is that cool okay two one the audience and one for the recording so going to give you a sandwich please don't separate it okay cool thank you um Factor databases uh usually have some tricks for uh High dimensional data um do graph databases do the same thing like how to transform High dimensional data into lower Dimensions so for what do you mean by transforming exactly at this point uh lower dimensionality data yeah so projections specifically so um with a graph you end up with kind of both strategies a lot of that ends up being a data modeling strategy for like how granular you want your nodes to be like whether they're very fine grained or they're big chunky nodes um because the chunkier they are then when you're doing pattern matching rather than Vector search that affects the granularity of like what kinds of questions you can ask is is that what you're getting at like for the no or there specifically the vector indexing uh is has few tricks to enhance the performance because if you deal with all the dimensions in in Vector uh Vector data Vector meting you're going to get really complicated uh index but they transform that into lower Dimensions uh losing it's a lossy transform but it's still fast and I'm wondering if neo4j or any kind of craft database does it have anything like that up up its sleeve so so so I guess that's what I mean and like it's not as as directly applicable like in the when you're doing the graph modeling so like the graph query Time Performance ends up being um directly proportional to the number of relationships that you have to touch and so the bigger the bigger the nodes are the fewer relationships you have to touch to satisfy a pattern and so that ends up being faster to execute the and I guess the other dimension that or the other aspect of that is that the overall data set doesn't affect the query performance qu performance is entirely perform determined by like locality of the data so like the number of like people that we know between us doesn't change then the query time for finding like how many people we have in common doesn't change what does change is if you've restructured the the graph and have a lot of extra nodes in between us that impacts career Time Performance so that's what I meant by having like kind of either you can kind of flatten out a graph and have like what amounts to like a triple store like representation of things right and it's like very fine grained concept to concept to concept to concept to concept that is slow and slow and slow because you got to kind of reassemble things and that takes more time so by so for us we have What's called the property graph model by bundling things up you can perform better but then we get to kind of the relational world of like okay how much do you want to be bundled how much you know and how how much how well that maps to natural language questions as well does that make sense okay question it seems like you you've created an architecture where I could present both the ability to do a cipher query as well as just allowing the user to do a prompt right a query and just a normal query yeah do you do you see a a scenario where the the llm could actually generate the quer the the cipher query as well oh yeah so like in when I switch over to the notebooks that's one of the things that we do so whether you just do it without any training at all and you just say hey here's natural language generate a cyppher query there's enough examples of Cipher query out in the wild that um even GPT 35 turbo does a decent job of generating Cipher it is important to pass in the schema at least in the prompt like hey given this schema and this us your question can you generate a query for pattern match level queries usually pretty okay um as it gets more complicated then it tends to break down and is definitely helped a whole lot by F shot learning so if you have like a bunch of examples given this question here's the quer to run and I'll show that that improves it a great deal but out of the box it's not bad yeah and and is there the possibility where the llm could actually validate the cipher uh query itself it's like so some type of the ability to to go back a few times until until the query until the cyer quer is correct what we what we tend to do with that is actually so once you've actually had the query generated then for us it's rule based we don't go back to the llm say hey does this look like correct Cipher what we do though is is actually well okay there's a couple extra things about making sure that it's not a destructive query all those kinds of things come into play you have to make sure security is there fine but once you run the query and you get the result back you never hand that straight back to the user there is another round trip with the lolm saying hey given this question is this a reasonable answer because it could be syntactically correct but semantically useless right and so that happens as well um and so you've got to put in extra you know round trips for that thank you uh okay so my question is uh so say like if I have a book like sh BS book right so how do you um extract the uh the graph structure like which note link to and which not and do the chunking so because I imagine that uh natural language is more complicated than the the financial form right because Financial form somehow I I can uh observe the relation yeah and the my second question is that um what's the typical criteria so if I want to sh a gra database so I mention graph database is bunch of node and ages right yeah so it's like you can do a auto shot in or it's like a user can choose the Shing criteria yeah that's question that that that's a great those are both both great questions uh so starting with the kind of more uh call it prosaic you know text like the free truly free form text where there is no inherent structure or IDs you can hold on to anything like that um you're right like that is harder to do well than something that has some inherent structure like hidden structure I guess inside of it you do still start with just basic chunking and try to get for the text you've got is it a paragraph level or a sentence level and you let that be driven by the kinds of questions you want the user to ask rather than so a lot of the chunking strategies started with like being very sensitive to um token count right as token count goes up okay nobody cares anymore like that's not really a thing what becomes the overriding factor is not token size but like how relevant the similar search ends up being so it's very dependent on like what the embedding model is that you had that will help determine what the chunk size should be for free form text so there's that aspect of it and then still kind of for for generating the graph around it for constructing the graph around it you start with the kind of Link list that I had that's always going to be helpful you also probably have chap you probably are like something of hierarchy as well do that as well and then it's name density recognition then it's you know using the llm for a really great NLP pipeline go through figure out the people places and things pull those out and again potentially use that for page rank or other clustering algorithms or whatever might be interesting depending on what the text is that's useful but you end up still elaborating on it but it's not as nice I guess is your point and I agree with you like as as a legislation where you have like very specific citations like this refers to subar paragraph five of section whatever like okay that's super that is a thing that is awesome you're not going to get that in The Iliad or or something right um oh and then for for uh for basically graph size right like what scaling look like so uh graph partitioning is uh not easy is the fair thing to say like static analysis you can kind of figure out how to partition a graph reasonably the thing is even if you did Static analysis and tried to partition a graph you want to partition it so that it's partitioned well for queries that are asked because the whole reason for partitioning is usually spread across machines and have a lot of parallelism going on right when you're running queries that have to hop across machines that sucks and so you want your partitioning strategy to be driven by the questions you're asking that ends up being in production like kind of a non-starter like because the even like ignoring gen like the things that people do with graphs always cut across too much and so there's no perfect partitioning algorithm the approach that we're taking now that is this is being recorded so I'll say like going to be available sometime I won't have my product people scream at me okay the what we're looking at now is actually still thinking about partitioning but having it be rule-based partitioning so it's basically at the data modeling level like as you know you're getting a bigger data set you can say actually I know I've got a geographic split and like there's a bunch of data that's in North America a bunch of data that's in Europe bunch of data that's in Asia wherever it might be those might have cross connections but it's so rare that it's okay to have those in different places and so you go ahead and do that so you end up kind of determining on your own what is the reasonable split it ends up usually being Geo or time or some natural kind of partitioning that you decide and then on a rule-based level then pred decide what those rules are then it can actually get um distributed and I think and again product people I apologize if I'm over promising here even scaling within those as well you can have like scaling rules for how elastic the storage might be for like how many so for the you know for you know if Europe is going to go crazy can we scale that up infinitely or not rules end up applying there as well so at the moment uh that's not available that that is a a future amazing thing that's going to happen um probably next week right product people cool all right uh maybe we'll let Andreas show us the demos and before he does that I just want to add something in defense of the Iliad The Iliad has canonical number of paragraphs so Scholars can rever to them which is better than most other texts right just just to to add that I need I need a better Alison Wonderland I should have said no it's probably got oh damn it all right probably have let's look at the demos okay um okay right this is when we watch the how I've broken my code um oh wait I've got to change my sharing on I do that by stopping sharing and then sharing the full screen I guess do that okay ah there we go uh so that GI of repository that was on that slide there for a while that's this is one of the notebooks at the end those notebooks go through do doing the construction um there's another uh bunch of notebooks that do some experiments this is the last notebook in the bunch that is not currently in sync with the the the data model but it goes through a couple different techniques of like how you can actually access a Knowledge Graph what that looks like in code here I'm using Lang chain it's pretty much the same with llama index or anything else you might you want to use um again Ry will we'll figure that out how that works with fry get that going as well standard stuff at the top here um bring in some stuff from Lang chain connecting to the database um I add a little extra you know let's just make the responses when you get them back look nice this was set up to be with open AI or o Lama okay this is the first part um and this is Lang chain specific so Lang chain has support for near for J that makes it look like a vector store um this is the simplest possible thing you can do with new forj that doesn't take advantage of any graphin it is just treat it like it's a vector store fine that's great you get to specify how to connect to NJ nothing too to clever is going on here so this is straight up just run Vector search get responses back um for this this bif here of Lang chain that we' set up so I set up a chain for that and I pry it awesome um I'm not going to run through these so you can imagine you can run you can see the examples that I ran while I was sitting on the couch okay that works um for better or worse this is what the the vector search I talked about the the query window that you can have the context window it looks like this in in Lang chain um I think it's called a retrieval query that you can add as one of the parameters when you're creating the newj vector class um actually I guess I could just look down and see what it is here yes it's called retrieval query right so this bit up here um is the make that a little bit bigger even and get that X so the retrieval query ends up working by assuming that a vector search has already happened and as a result of the vector search what you get back is the nodes that were found what they were um and then here there's a parameter called window as longest window that I'm going to pass forward um we order them and then collect chunks around it wait where did I go here okay the pattern matching that's going on here is not super awesome it is just kind of straightforward just find stuff around what where you are collect that all up I collect the text from all the chunks put that all into like here's a bunch of junks together hand that back along with the store and and the metadata um it's not super exciting but it gets the job done that is though the part this is like this retrieval query parameter here here I'm just using it for the um the the context window expansion this is the same part at which if you had multiple agents for the different agents like agents who knew about how to answer different kinds of questions this is where their strategy can come in like you can still do a vector search but then after you've done the vector search you can do like post ranking based on a pattern where you can expand the context window based on a pattern anything you can do with a query language you can do right here so if you have the unstructured data and the structured data all connected this is the part where you kind of inject that extra functionality here I'm just doing the the chunk window okay I'm going to pry that um and I I could just let me see if this actually runs again have I lost it connectivity okay okay and so here are the two results um is one better than the other they're different for this context doesn't really seem to make much of a difference it doesn't improve or not you can decide if you like one or the other better this is a sort of decision where it so so depends on what the text is that you're quering whether having a context window is really important or not the sort of the anecdote I always like to come up here with that isn't really real world obviously is like the mcgyver you're diffusing a bomb scenario I'm like Hey how do I which which wire do I cut first oh Vector search cut the red wire the next chunk says but first boom right okay you you kind of want both of those chunks to come back you want to make sure you get the full context and like because text is imperfect people you know don't always go through nice sequential order of things so you want to make sure you've got enough here it didn't really make much of a difference you can decide which one's better um this is a bit of more expanded and actually this is the part that I think is not in sync with the model at the moment this is another retrieval query where instead of just expanding the chunk window I'm reaching out into the investment portfolios and saying okay given this chunk that I found I know that that came from some form that form was filed by some company that company has some investors so from the chunk follow the pattern all the way back to the investors package all that up into text pass it to the llm let it answer questions if somebody happens to ask about investor you can now answer those questions and and by by combining them as well so I guess importantly like so the the flow here still ends up being anchored in Vector search so if you know there's a company that uh is at risk from Chip shortages because they'll have to say something like that in their filings right Vector search will find those companies and then you can say okay which investors are have portfolios that are exposed by a chip shortage that ends up needing to go from the companies that are exposed by the chip shortage through to their investors this is when a question like a query like this ends up being valuable um same things package that up to Lang chain add that to the things that I can run um and then go ahead and let it run oh right yeah and I ran this before and that's when I was sitting at the couch thinking what have I done this does not work because I broken the data model um and not updated the queries so um later tonight maybe it'll be a race you guys can go and Fork this repo and try it out and if it's still broken then you can yell at me but I'm going to try to tonight go get this fixed um so that you can wake up in the morning for sure and these queries will be a little bit better here's the full kind of let's do the full like text to Cipher example this is a giant prompt that is basically saying okay at the beginning here it's the usual kind of thing hey you're really good at writing Cipher queries you know about writing Cipher queries here's a schema that I'm going to tell you about don't do anything other than writing Cipher queries and by the way here's some examples of what you can do uh this first example I love I threw this in because it felt like something you'd want to be able to ask if you didn't know what's possible you're like hey what do you know about like okay you know about companies what companies do you know about as a pattern match that's simply match a pattern for companies and then you know okay return just 10 of those things randomly not it's a a very poor way of I guess doing that it's not comprehensive but it's an example of something that's easy to write in a query in a query language query that a natural language query doesn't really you're not going to find that in the text anywhere so okay that's what that looks like continues like with hashmark which city in California has the most companies listed as we know llms are really great at understanding this California here oops there ends up being the California that I'm using here in a predicate there's a we predicate after here this is a bigger pattern from a company that's located at an address with the address. status California return that address you know count how many cities are there so right count the number of cities is the number of companies and then order descending so that ends up getting uh the the top companies that are in California and this goes on so this I have like I don't know a handful 10ish or so oh I do yeah right I end up so all the things that I talked by through in the early slides like let's take advantage of full text let's take advantage of doing je spatial search let's take advantage of vector search and pattern matching all in different combinations I ended up coming up with different example questions and then what the queries would look like these are the kinds of queries that particularly like even like the um like this here is the point at distance that's doing a geospatial index lookup and then just doing a a distance calculation that I've never got that to be autogenerated from a text to Cipher uh you know just ask the LM to do that without any kind of help um so that's a good one to actually prompt it with um but it goes through here all all those different things and then at the end like okay great for the how many do I have the four or five different strategies for any question I ask let's run all the strategies see what the answers actually look like uh and this is kind of hilarious as as these things always are okay so that one that I said earlier was a really easy pattern to match for okay what companies do you know about I I know that these first two are lying because I know like in the small this is a small sample data set that I have and it does not talk about um freshw works for sure I think news corporation is there alphabet is not there service now is not there page like so these I don't know whe this is being you know how this is being hallucinated but it is being hallucinated at this point so it has probably would benefit from some prompt engineering to help rain it in a little bit but because I've also broken some of the other pattern match stuff these two things come up with I I don't know um actually I'm sorry right this this bottom one is not going through the pattern match the retrieval query this is the text decipher with f shot learning because I basically told it how to answer this question this time and not it doesn't always happen this time it actually you know took me at my word ran that query got the answer and this is true all the those companies mentioned there are are in the data set um and then you can go on and ask other questions this is I'm not going to go through all of them if you'd like to I would love it if you guys actually uh try these notebooks out and tell me what's broken about them tell me what you like about it what's missing ask me like how would you do X Y or z um because the reality of going down this rabbit hole is it really kind of is a rabbit hole it really is a question of you can keep adding to the corpus questions that are possible but probably you're going to end up with lots of different strategies for answering different kind of classes of questions and so it's not it's not an overnight thing it's a A continuing evolving thing um driven by need for the most part so I'm gonna pause there and see if there's any more questions about what I've done so far yeah there's a mic headed your way hold on all right thank you um so I have a question regarding the demo um I remember there's a you know um test to Cipher query about like fighting the companies that are you know um close to Santa Clara right so um here for this um query I wondering why do you have to use two different matches well the first one is to get you know the address there that has a city equal to Santa Clara but I mean what if you simply use one you know match Clause here to compare the um address of the company to the location of Santa Clara that's a great question and so I guess it depends on this is part of the uh I guess subjectiveness that comes into like designing this stuff like here my assumption was because you can do exactly what you said because we do have in the graph itself in the graph structure there is an address that has a city that is Santa Clara so it's just a simple pattern match go from you know company that's at Santa Clara management and like and you're all done but it depends what you mean by near Santa Clara like and geographically I'm going to be at a little bit of a loss here but like you know what's within 20 km of Santa Clara you've gone outside of the city limits at that point but if you still consider that to be nearest like so if nearest means you know within an hour's drive right or you know some something like that like then it is no longer something it's easily answerable through the pattern match um it is easier to do than a je spatial match for that so je spatial index look up for that although that said there's nothing stopping you from actually popping up a level and being like here's the county that Santa clar is in and like let's find things that are in the county because that's nearby then you get into border questions of like if you're at the border of the county is a company across the the border near you or not right and so it does that make sense yeah that makes sense but I mean for the first query what I guess that there are like probably tens of thousands of AES that tast within set CL and I think I mean in terms of scalability is it going to make this query super complex you know uh when it comes to you know how things are run in the back end uh because of the number of addresses that you have on hand is that what you mean yeah so yes and no I suppose it depends on like exactly how granular you end up being and how you end up mapping like so if right as it is in this data set I just have one address node and it is assumed to basically be a city level of of granularity if you want to get down to street level granularity you could do that but at some point you do have too many nodes and it's no longer useful because now you're doing a scan rather than a pattern match and you're kind of losing the ability of it and you probably want to switch to what ends up amounting to either you go like with a pattern match that jumps up to a higher level like go to the city and then come back down um or just run the geospatial indexing because it's some level of granularity that's why the index is there it's easier to do it's kind of it's not dissimilar from like what ends up happening with full text or vector search or anything else there are always trade-offs like the reason you have indexes around for doing stuff is your assumption is the assumption is that you have so much data that an index is a specialized data structure for helping kind of sort through and find the window of stuff of what you want sometimes the inverse is true where actually you know enough about the where you want that the index does not help you as much as just going through the things you know about um and so you have you already constrain the problem set right and now is it is scanning is not as expensive as you thought it might be and doing a scan is not bad because you can if you can fit it all in memory so for the first match Clause does it only return the city level add in in this data model it does yeah oh I get yeah yeah oh I'm sorry right yeah and so I I wasn't clear about that like so when we pulled out the addresses here the address nodes do not get any finer than City level okay that the STA set that totally makes sense that was the part that I oh sorry yes got you the answer up with my question Cool other questions I'm just curious if there's um if you have any llms that have been fine-tuned to generate accurate cyer queries there are some that do better than others uh I know that I know that we have a team internally who's uh constantly trying to evaluate that I don't know what the current best recommendation is the generic recommendation always is you know uh use mixl basically or something at that scale and then go down based on what the queries actually look like like the abilities will change um we are internally not only evaluating models but also trying to do some fine tuning of own and come up with our own model that is specifically for Cipher right product people that'll be available next week so not not not yet available yet but is it's something we're trying to tackle um one of the challenges there ends up being as often like there's not enough just kind of sample data that we have to do the training super well um so it's limited in its Effectiveness I think right now for for zero shot what which llm would you recommend zero shot few shots so I'm pretty lazy I do not mess around with a bunch of different models myself right now so I'm pretty happy actually with GPT 35 I find for like most most out of the box works fine um so I don't I don't bother looking around for the best yet at the moment but if you'd like to know I'm happy to ask the folks who are spending time looking at that and I'll let you know what they recommend hi um yeah I had a question about the when you're inferencing you're processing the the text Chunk data so I think someone mentioned dimensionality reduction earlier and I I wanted to ask because it's an important question um like when when you're processing this text Chunk data do you do any any techniques like you know principal component analysis where you're looking at the textual data and um you know trying to identify you know across um what identify the components which have the highest degree of variance and then use those components to bring it down from like let's say 100 Dimensions down to two or three dimensions to speed up the queries and be able to inference on Lower dimensional data yeah so I I have to admit like I don't have a lot of variety of experience in that I know that the customers that we have who are doing this stuff spend a lot of time at that step trying to figure out what the right strategies are for how they can actually figure out what to put in junks and like what the embeded model is for the junks and also like what they can derive from those chunks at at different granaries to actually have useful graph structures around them there this whole that first stage is it affects all the everything else you end up doing like if you get that done really well then the quer complexity goes becomes Better or Worse depending on what you've done there yeah definitely like like like the other person said I think it is a lossy transform but it can definitely speed up the queries like you said because you're you know inferencing on much lower dimensional data y yeah there thank you Mak sense other questions any questions from this wonderful Annex you guys have been silent no all right making sure cover okay no more questions then let's thank Andreas for the great presentation