Bay.Area.AI: Vectors and Graphs - Better Together, Roie Schwaber-Cohen
Recording: Bay.Area.AI: Vectors and Graphs - Better Together, Roie Schwaber-Cohen
thanks you thank you so much Alexi um and thank you um to GitHub and to NE forj and to other the other sponsors um so yeah um not going to take a lot of your time I know it's laate people are tired I'm tired everybody's tired um but I want to talk to you about um the kind of experiment that we ran in the last couple of months at Pine Cone trying to kind of figure out what is a um interesting kind of modality for us to kind of combine um Vector databases and Vector rag with um graphs and when I first read like the Microsoft paper on rag I I was under the impression that it's going to be like this very very specific architecture um that follows like a very stipulated kind of recipe and I really liked actually like Direction that's coming out of the new for Jak Camp which is graph rag is kind of like a family of architectures right like it's it's a family of architectures that combine a graph with some generative process and and a retrieval process um and in our case what we're trying to kind of do is instead of thinking of like an A A an integration that takes the vectors into the graph itself we're actually using the the vector database to sort of create a focused subgraph from a much bigger Knowledge Graph that we create and then use the graph n and llm to generate and compose questions that are going to be interesting for the people who are um sort of doing this exploratory kind of activity using using the system so when I first kind of started thinking about how to build this thing my immediate notion was like okay I'll make an ontology um I'll take an llm I'll point and I'll point the llm onto my ontology I'll combine it with a with data and everything is going to just work um and no that doesn't just work um what happens um definitely without an antology right when when you're just sending out an nlm to look at a piece of text and you just tell it you know go build me a graph from that right what you get is a graph that isn't homogeneous in the sense that like the connections that form within it are sort of all over the place and what that means is that basically you lose your ability to ask coherent helpful questions out of that graph right the questions become a lot more oneoff right you get a lot more variability in the types of nodes and edges so you nothing ever kind of consolidates into something that's actually usable um there's also like scalability challenges which means you know you just create an explosion of types of Nod types of edges and in as a result of no nodes and EDG that you didn't need in the first place and kind of consolidating them is also a uh a problem as well um so instead of that we were like okay what will happen if we take a data set that actually already enforces some structure through its own metadata um that introducing introduces some stable ontology that we can work around and then limit the use of the llm to the things that we know that it's really really good at which are two things one is summarizations and the second is generating queries right and we I'll talk a little bit about how we did that in in a moment um just to give you a little bit of a taste of what the final product is because otherwise this talk will be all in the air right what you see here is some user typing in a an open uh an open-ended text question about a the Supreme Court decisions this whole experiment revolved around indexing and learning about um Supreme Court decisions um we on the left what you see is we create the normal rag path right which basically tells us for this given questions what which pieces of content actually match semantically and then we summarize those right but in parallel to that we also have some reference to nodes in the database that correlate to the same cases that we found um in our rag process right so what you see here is essentially that graph that subgraph that is essentially a a a a a lens on top of the entire graph focusing on specifically what the user is interested in and then the second step is we use that graph that subgraph that was created as a result of our initial semantic filtering to construct questions that are then mirrored in a cipher query and that CER query is executable and has a good chance of actually returning with some actual result right that isn't just disconnected from uh from the graph itself um again I'll talk a little bit about like what the limitations of this are not every question that we that we that we generate actually results in a successful um successful uh query but as you see here and many of them do and in fact when they do the Insight is quite helpful right the in a lot of these cases the question is is indeed a a follow-up question that could have been proposed by the professional um to then like and and it correlates directly to the text that they found which is again semantically uh semantically um relevant okay let's back this so how did we how do we build this thing so there are essentially four big systems participating in this um one is any kind of document store and it really doesn't matter where you store the stuff when it's just metadata just just like Json with with with a bunch of metadata um in our case we just saved it in in a in a in a instance um again it's important to note that the data that we're using in this in this case is essentially specialized right we're we're in this example like I showed we're we're taking um supreme court filings and those inherently have within them inin that particular API that we're using have within them structured information like for example who were the justices who uh heard the case who are The Advocates who um argued the case where the case was argued etc etc etc so we we are already starting from a place where we have some structured data to to um to go off of um we ALS so the three other systems are a vector database of course a graph database and LM that that does exactly what I said summarization and query gener so as you saw in the demo the the components are a semantic search op operation that again this is like the rag we all know and love right but it has one extra step and that is giving me back some handle onto the graph and that is kind of like the Crux of what's happening in this in this little experiment is that we're saying there's going to be a an asset in this case the case IDs which match between these two modalities and they form our our join bridge between those two databases and that's how we're able to make some sort of an an uh get some sort of an Insight onto the things that we've indexed semantically and then leverage that Insight on every every consecutive query that we do on uh the graph um this is the overall structure of like the flow of this this whole system so we take the the raw data as it comes in we parse through that data again to to just get those entities that we know are structured within the data like Advocates cases justices and so on and so forth we also do something a little bit um additional to that which is use good oldfashioned ner en enrichment to extract additional entities and again associate them in a pretty strict manner back to those cas entities right and that just gives us more opportunities to form more interesting questions about the same exact um data without opening the aperture to an llm that will start you know adding um types of nodes and types of edges Willy nly instead it's very very limited to that subset of what the ner model can extract um and it just it just so happens that it produces better results down the line um again in parallel we're doing chunking and embedding like again in any any any other rag process um and then when the application Phase um hits we do our semantic search and then that gives us back our document IDs or our case IDs and then we do graph filtering and then something pretty interesting happens and this is where like our work with Neer really took took shape is based on that subgraph we can now start doing some analysis on the graph itself to give us the best information to then pass on to the component that will do the cipher generation because if if there's any if there's one component here that's a little bit iffy right in the sense that it's very um fragile to to like the the inputs that it's getting and the quality of the output um it's the llm Generation Um and its ability to generate actual cohesive and uh meaningful queries um yeah I think this is already covered um just to close off before I'll take some questions um the stuff that we want to keep um kind of looking into with this um experiment so as you saw um we do these sort of intermitted steps that are that are divided by some human interaction so we start with a semantic search and then we get back results we choose one of the so one of the the cipher queries and they they give us back some insight that process in and of itself could be quotequote automated with agents basically allowing these agents to perform the cipher queries themselves even correct them if they're getting back no results or bad results um and try again um with better quiry Generation Um so that's one area that we want to explore um of course using richer data sets that are much more highly interconnected is something that we we're looking on and looking at um using much bigger ontologies and pointing the llms to them so far hasn't worked really well but but with Alexis's help we're going to crack that nut as well um and yeah I think that's basically it if anyone has any questions thank [Applause] you as usual raise your hand I'll bring the mic to you U this is more like a hypothetical question so uh so I was so so I've worked Knowledge Graph like almost like six years so so I would I would ask the question as like imagine you have unstructured text like the real the internet so what do you think would be the effort like to like say take that unstructured text and build your own ontology based on understanding topics and and then like I don't know and use llms to like create a Knowledge Graph web scale Knowledge Graph out of unstructured text like the actual the internet just for the entire internet though I mean it it could be a smaller smaller set but but I'm saying in terms of understanding topics and creating uh like say uh objects and sorry property graph I mean nodes and yeah splitting them that that the thing with anies is that that's exactly what they're like they're they're a set of of um terms right that help us understand a domain right so you can expect that for every domain that you want to cover you're going to have to create some ontology for that domain right and I think that's like one of the biggest challenges that you know the the graph Community has had in in in general right like how do you generalize any kind of knowled canonical knowledge graphs um and there's been you know obviously like many many attempts in the past and none of them really stuck because it's like all like you know people have their own opinions about what and it needs to happen it's really hard to reach consensus I think the question is how do you reliably you like create a mechanism that can create an ontology on the fly but also not let it diverge in really weird ways right so like I want the llm to understand my data on ological level but I it right now it can't quite do it because it kind of like fritzes out and kind of goes it gets lost right even if you give it like very particular things to focus on um it doesn't quite do it a great job at at that just yet um so I think that it's still a lot of room to research other questions no let's thank Roy again [Applause] point