DevReal: KG-enabled AI Workflows, Chris Rec
Recording: DevReal: KG-enabled AI Workflows, Chris Rec
[Applause] well I would uh I would love to make this as um as interactive as possible so if anybody has any questions please feel free to raise and you can talk about it but um my name is Chris I'm one of the co-founders of Yow that a um we're a company that's building knowledge graph infrastructure for a variety of generative AI Solutions um lately that that often means a lot of multi- agentic uh type Solutions powered by uh knowledge graphs um before I get into it though um how many people here are doing rag of some sort or thinking about rag maybe and how many of us are using Vector databases is anybody using something like elastic search or knowledge graphs or like interested in knowledge graphs for rag yeah there's been a lot of talk recently about um using knowledge graphs in the the rag workflow um and I think like a lot of you we started from a pretty similar place um maybe like a year ago um when my co-founder Tom here and I and our other co-founder Chio was working on uh Building Solutions uh for a variety of customers in the like for example legal domain or the uh the finance domain uh areas in which there are a very high bar for accuracy um and repeatability um we were trying to build a lot of AI Solutions using um existing open source Frameworks um Vector rag naive Vector Rag and we found that it was fast and easy and really powerful and it got us to about roughly about 80% of the way we needed to get there for uh accuracy um but for really um really like let's say often the traditional domains or domains with a very high bar for accuracy and repeatability uh 80% is unacceptable so we were thinking about ways to really um improve um one the the reliability the repeatability the visibility the controllability the accuracy um of our of our rag workflows um and that's when I met Tom that's when Tom started telling us all about these really exciting things with knowledge graphs about really like having a a lot of control of the structured semantic representation the data uh the idea of making very uh small and and and modular and granular representations of your data uh so that your rag systems could could focus on only just that little bit of amount of data that needed uh to answer certain question um I'm glad to answer any uh questions about like our approach um our thoughts in this space but in the spirit of De ising uh Tech and AI uh I came prepared with some code and demo and I would love to kind of dive into that and and rip in any question you guys might have um so what I have today is uh a demo of the platform that our team put together and it is a graph studio is what we're calling it um it's meant to help users um both technical but also non-technical uh come with unstructured data in the form of PDFs or text files or also structur data in the form of like Json or CSV and use either schemas if you want to or not uh to go from unstructured data into a structured graph representation your Daya you can then begin to query uh it's part of your right pipeline um so if it's cool you guys I'm just going to dive right into it kind of give you a tour uh and then we' love to kind of address any questions you guys have towards the end but if you have anything just please shout out uh we can handle it so um this is the Doc Page uh I'll start by giving you guys a quick tour of the UI and then I can show you the SDK code as well but um everything pretty much starts uh right here it's pretty basic like uh homepage kind of dashboard for your for your data um everything like most like rag systems is based on the concept of uh of chunks so when you upload data to our platform everything gets split chunked embedded stored that way we can use it for graph creation um so chunks can be um strings they can be objects if I can find something in here that has an object um if you upload for example like a PDF uh gets turned to text we chunk it we embed it we attach like metadata that you provide us that we uncover um or for example if you upload a CSV or Json object one of those objects will be a chunk itself also gets embedded uh and then used so everything here is based on the concept of chunks um so when you upload a document for example um and since we're in Amazon we're going to run the example of the Amazon 10K demo that we have um so what I did I first uploaded like the 2024 um Amazon 10K document uh and then that doc was uh split chunked embedded stored uh to be reviewed uh next uh there's this concept of schemas that we have so I wanted to emphasize that while a lot of use cases um are really conducive to having schema controlled graphs uh in the graph creation process there are also a lot of interesting use cases in which you kind of just want like an llm for example to just figure out what relationships matter um in the uh in the data that you have so for those use cases there's a lot of really interesting open source tools I know neo4j for example works a lot with like uh the F at Lang chain they have an llm graph Transformer tool which is really good at pulling out just interesting relationships across all your data turning into a triple format and then storing it in like um or database um but in this case if you want to maybe for example um tackle a problem with more of a schema driven approach uh we have this concept of schemas so a schema is kind of like our version of like an ontology ontology basically tells you like all the different types of relationships you care about how entities are are organized and related to each other in your graph um and in this very simple form we basically just tried to put together a Json object so um you have your entities which represent your nodes in a graph you have relations which represent uh edges so the connections between the nodes and then you have patterns uh and patterns are maybe another way of kind of saying uh triples you may have heard of it but consists of a head a relation and a tail Chris works at Yow um these are the types of patterns that we that you want to be able to build into your graph so we use this schema for two things one for extracting data that matters to you from your raw unstructured text uh and then we use it to constrain the types of relationships that are built into your graph and this can be super powerful because sometimes you want to make sure that the semantics relationship and of the actual entities that you're querying on are exactly as you expect them to be so you can have more consistency and predictability in the ways in which your agents or or your users or yourself interact with that data so I can go back and kind of show you how we actually use these schemas and the graph creation process in the UI is quite simple so all you have to do is start by selecting the types of data that you want to use to create graph so I'll just call this the the Meetup graph we're going to select a schema and I'm just going to pick the one that we talked about before I'm going to select the chunks I want to use in this case all I'm doing is selecting a um uh a set of chunks that are labeled with that uh document title and what's happening under the hood is we are pulling out all those chunks that have been embedded we're then per chunk identifying those types of triple patterns which of them exist in those chunks and then we're using those to create triples we combine those triples and do a graph so this graph is being created under the hood right now I'm going to let this finish up and while it's creating I'm going to jump into one I just created a little while ago um this one here we go so I want to I'll jump back to the other one uh in a little bit but this is the UI that we've created for exploring and interacting with your uh your graph so this type of like graph Network visualization experience should look pretty familiar um if you've used any sort of network graph uh this is pretty basic um but what I think is really important and exciting about the approach that we've taken is we've tried to really bring the uh the information from the the raw data itself the Providence from that data itself directly to um the network graph to so if we're in here and we look at something like um let's see AWS so AWS contributes to revenue sales uh Global sales of of what is this storage compute and database what we try to do here is we try to show you that for any given node or a triple uh you can very easily narrow in on the types of relations you're highlighting uh you can see the individual triples that are being um extracted and then shown the graph but then what we also try to do is show you exactly where this data came from so we have the raw chunk text here as well so that when you find any sort of triple or or node that seems relevant or important you can always trace it back to the source from which it was extracted this is important because for people that want to build explainable systems or uh generate answers that are much more um like user friendly uh you have Source you have Source data that tells you exactly where it came from so you can have much better kind of generative uh experiences um another thing we try to do is um maybe the extraction was not perfect maybe there's more data you want to add in here and you want to also attribute it back to the source so you can do something like let's just say highlight North America we can call that a there's not really an entity here for that let's highlight infrastructure and we're going to call that a revenue stream ad node and what you should see up here now now is a new node has been created that can then be automatically attributed back to the source from which it was extracted so we start with our best shot at pulling out all the best all the data that is relevant to you and then we try to give you all the tools that you need and BR your Narrows back in and is able to map to your understanding of the world um another couple interesting things so um I saw in here a couple different nodes called like sales for example if it look that up some of the things that you might notice if you ever try to work with graphs is that um entity resolution or entity co- reference is often a challenge so um maybe you'll find the word like sales capitalized or you'll see like infrastructure sales or like sales organization and to you they may mean all the same thing but to like an llm or to another model which did the extraction they could be quite different things so um there are a lot of really awesome companies doing interesting things in the realm of of um uh of cor reference resolution entity resolu solution but uh what we try to do is kind of give you a way to be able to create rules and come back in and actually narrow this into your specific understanding of how your data works so if you look in here there's like three different types of net sales right lowercase net sales uppercase n uppercase s net with lowercase s like they're all the same node really so what I want to do is just call this one the winner um I care that net sales is the say net sales here is the specific um like uh entity I want to uh save and make sure they all are the same like node all I do is hit merge hit save as a rule and now if we go back to the graph we should only ever see let's see one net sales let me see if I can find it there we go now we have one net sales um node that is basically going to be referenced for uh across the entire graph so this is now also um I'm going to come back to here also highlight this uh this is also going be saved as a rule so uh one thing we try to do is make it a bit easier to um make sure your graphs are kind of learning getting getting getting smarter over time so uh this Rule now that it's saved in the workspace is able to automatically detect things like net sales that are like of those different um syntactical structures and then always call them net sales this is just an example of a type of rule we want to really start to enable these types of graphs to like learn and get better over time question yeah yeah yeah please so would you yeah yeah the question was like can we use any other NLP processes to be able to automatically detect duplicates for example um the answer is definitely yes I think the types of like pipelines and and rules and and types of C customizations we want to be able to add in are are inclusive of those things we want to make sure that data is right the first time what you're seeing here is basically um us kind of taking our first swipe of the problem we don't want to infer too much upon the user maybe they do care about the difference between like Capital net sales and lowercase net sales but um ultimately there's a lot of different kind of post-processing or pre processing workflows that you can you can Implement to be able to do that so what I just showed here is um an example of kind of adding in a rule that does post-processing on on the type of data but I want to show you the SDK code shortly that kind of shows how you can plug that in so before chunking you would right so what's happening now is now that this rule is in here uh we chunk the code uh the question was are you doing this all before the the graph creation during the chunking process the answer is we store data as a chunk exactly as it came in so we can always give back the right data um when we do the um the triple creation for example that's just the language the graph speaks so we want to make sure that the graph is following the rules that are in your workspace so when it's turned into a node or an entity that's when we apply rules and that's when we perform postprocessing so in this ex this example here is is not rdf this is a property graph that we're using but there's we've worked with a variety of different like types of solution sols rdf is one of them as well we love working with like Cipher and like the Neo for J guides too um there's a lot of different typ of solutions but rdf is a great there a great one it's just not this example yes I can show you an article I wrote that has that exact type of use case so I'll flash up the blog afterwards I'll make sure I give that to you guys afterwards the question was can we have some sort of example of a knowledge graph enabled rag workflow versus a non-graph enabled workflow so yes I'll I'll share a blog post afterwards um okay so the Meetup graph has been created um this is this look very similar it's the same data um one of the things we offer up here as well is exporting into Cipher so um if you want to take this out and then drop it into your like a neo4j or DB you can do that just it's just a cipher command you can run um we offer a um a public graph um endpoint as well so you can basically take any graph and turn it into publicly accessible data so if I just open up like a incognito screen you should be able to see that like you can you can basically have readon access to this data um you can't like edit or or merge any data you can just basically poke around and play with it but like this was really important and relevant to us because even when we were trying to like maybe work with C certain customers or ourselves try to get people to tell us hey is this right is this look like the right type of like graph you want to see uh can you just double check this real quick um it often came down to just like staring screenshots or um like credentials and this was just a much easier more or like way for us to be able to do this so this is one option that we offer um and now we can get into some of the querying as well so um we in addition to just doing the graph creation we offer a query interface um all of this again is is API enabled and I'll show you the example after this but um you can take pretty much uh any natural language question or structured question and and ask it to this graph so if we say something like umop okay so if I ask a question natural language like what kind of services does Amazon offer it's going to basically perform um like a semantic search over the data that we have stored in this graph uh spit back a set of triples that are most relevant semantically relevant to this data and then conduct and then create a natural language answer so uh a lot of what you see in this in this UI based demo is like is is is just that it's a demo so like having asking a question and getting back this data is like really relevant especially for like a non-technical user but oftentimes in production or when you're a developer right you want to have access to the raw data and then generate the answer yourself or maybe perform that post-processing on the day to make sure it maps to your understanding of the world so everything I'm showing you here is API enabled as well but um as you can see this is the type of data you're going to get back when you run this with an API call you're going to get back a set of triples a set of nodes and then each triple is going to have reference to the data from which it was extracted so you I think you can start to see how this can work into your rest of your rag pipelines um instead of maybe doing a just a vector search you can do a vector search and a graph search or just a graph search get back that data get back that raw chunk data with context as to how it's related to other Concepts in your knowledge graph and then feed that into your um to be able to generate an answer in addition to natural language querying we offer structured querying which is which works much like any other type of like structural query language you might use so in this case if I say for example business segments and revenue streams and I want to know um what contributes Which business segments contribute to revenue streams and uncheck this and hit this this is basically just going to run a filtered query to get back any node that is or any entity that is a business segment and what contributes to a certain Revenue stream again just a regular structured query okay so that's how it works in the um the UI I'd love to dive into some code to kind of share a bit more about how this works it's like an SDK or API en workflow uh before I do any questions before jumping into this yes sure yeah the question is about how the schema is constructed um is a question about how we actually built it or like what's what is the format of it yeah good question so um there's a couple different ways that we can build schemas um right now we I think I'm just so used to working with this I often just kind of drop something into uh like in a Json format and just and dump it in here a lot of times when we were getting started with this we just used like chat TBT gave it a format and said hey build me a build me a schema that works with the structure and fixed it from there uh but then we also started working on our own kind of generative approach as well so I'm glad you brought this up um we want to be able to use questions from natural language or things that people care about with their data and then generate for them a schema that we think represents their uh their their data so I'm G to add a couple questions real quick okay so if I can ask some simple questions like this and this is actually kind of calling back to some of the first use cases we had in the space if we wanted to take from questions then automatically start to create a graph and this is exactly how you can do something like this so if I ask questions like where who does Amazon serf and where do they operate we find meaningful relationships and entities in this data just inferred so like I can drag these over and say um a company a customer company serves a customer and that's the type of triple relationship I want if I also want to say a company uh operates in a country I can do that as well and what happens under the hood is this gets constructed it into a um Json schema and then we can just like save it as like a Meetup schema and now we can then use this for creating graphs in the future cool and I'll emphasize as well like schemas are definitely powerful but not the only way to create graphs if you're trying to really explore your data to it too yes so when we do our extraction we do it on we really get pretty grander with like this like the the chunks of day that we're extracting we also extract mostly the triple level so the rest of this is very like um it's very like it's helpful information and also if you look at like the types of descriptions here these often actually act as like um uh prompt injection Solutions where this is information we can give to the element to say like hey like look for information like this this is how you recognize this type of entity but when we look for data we want to pull out we really care about like relationships between the data and how that relationship ties back to the chunk so when I'm looking through each piece of data that I've selected for for creating a graph I'm looking for representations of companies of customers and then companies serving customers in that example so there's a lot of tools we use under the hood some of them are llms some of them are LLP Solutions um but there's yes you're using basically individual patterns represent in this schema to extract these entities and those relationships on the like per per chunk per page basis cool all right so um we also have have an SDK and um it's quite simple to use and the use case I have was basically showing how we can create a um a graph of um GitHub commits and GitHub PRS in a certain repo um for uh into a graph and asking questions of it so um importing our SDK and all we have to do what I'm using for this example is I'm basically using like the boto 3 um repo so everybody should be pretty familiar with that um so I'm just I added an element here as well come back to this as well but this is basically just a quick prompt to summarize some data uh to make it a little bit more mean semantically meaningful on the graph um okay so all I'm doing is I'm going through the um the GitHub API and I'm pulling in PRS I'm pulling in user data from each of those PRS and then I'm also pulling in commit data and user data from those commits for each of the PRS so if you think about the type of like graph structure we're building we're thinking about like a repo a repo has a um has PRS PRS have commits and users are commit or submitting is uh PRS or they're contributing uh commits uh those are the types of relationships you'll see the kind of triples that we're constructing the bottom but all we're doing is doing some simple parsing we pulling out the data we pulling out commit data and then PR data and then what we're doing is we're using this information to then construct the elements of a so a triple as you can see up here we're pulling out is like a triple um a triple model relation model and then a node model so we said before triples consist noes relations so with every triple we need to basically Define a head node a tail node and then a relation so in this example um I'm creating a node here for the repo so I'm basically just saying this is the boto 3 repo uh for this I'm basically creating a um a node um or a a PR node so I want to I care really about the um the type of the pr in this case um let start running this too while we're at it so we can prove this works um and then um in addition I'm pulling out the commit data as well so um I have a commit node uh down where is it have a commit node down here and you can see what I'm doing with these triples is I'm basically defining um the structure by saying ahead is the pr node relation is submitted to and the tail is the U rebone node um uh actually this should say um like is part of uh let's this one uh this one a PR was submitted by a submitter um a commit was committed by a committer so these are the types of triples that I expect to then see in the graph now this is the difference between doing this by the schema constraint approach versus kind of just using the API to do it ourselves um I maybe don't necessarily know exactly the types of relationships I want or the types of entities I want when I'm kind of building this or maybe I want more flexibility to be able to create this graph the way I want to um in this approach you don't have to conform to a schema we can just infer the schema for you as you continue to add and update data um okay so this Rand already um one cool thing I'm doing here that um I wanted to call back to there's if you ever look at some of the commit body of the commit text you guys should know it's a little bit messy a little like hacky sometimes so instead of like trying to show like like like literally some code like in the in the name of the node I basically just asked like LM to to generate me a summary of like the commit note and then just called that like the title of the note so you'll see that in the graph as well um okay so the triples have been generated they're basically sitting in an array now and I can print this out for you too so what see here is the triple object which is a head which has like a PR has the name of the pr it has chunks that you can link in here which I haven't done but you can definitely just like upload the data and make sure it's mapped to that um to that that node uh some property data which basically just metadata about the about that node um and then the body of it and then there's the tail node uh right here so uh this is the it's should M pretty much to the structure that you've seen in the uh in the UI and what we're going to do is we're going to create a graph just by running a uh a create graph command so I'll just call this the mea GitHub [Music] demo and that's it so if we go back to the console we'll look for our Meetup graph one we just created this is what we've created so um this should actually just be pulling in like the top I think limit is like I know like 20 or mid somebody actually just pulling in the open the open PRS right now but what you can see on the side here is the exact type of entities and relations we Define right commits PRS repos and users um boto 3 has all these PRS um a given PR has a commit and then the commit has a user and those are the types of relations that exist across the entire map so now that we have that data in there we can actually start to query this a bit as well so um what I can now do is ask questions in a variety of different ways um like you saw before you can run an unstructured query which basically allows you to ask a question in natural language so uh this is something I was running earlier um let me show you the other graph I was running this again so all right this one's a little bit different there's a bit of a bigger graph um I'm not sure if this guy Tim Tim finnean or finegan finegan is here or if he like works here at all but I was using his name and his data a lot for this demo so I really want to know maybe if if I'm working at like if I'm working at GTH working at Amazon or if I'm like part of the open source team I really know I want to know maybe what a certain employee is doing or like um I want to know a bit more about who worked on a specific PR I just can't remember the name of um I can then use like this type of query approach to be able to learn a little bit more about this data not just in a structured way but also using unstructured data as well so if I ask a question like this just like which PRS did Tim Fin and submit what this can then do is basically run the unstructured query go in find the related triples spit back and answer um much the same like we just demonstrated um the structured query Works quite the same it's quite a bit faster it's fully deterministic and very reliable so now we get back just all the same data but I had to know going into this that I care about Tim vinan I care about the submitted bu and then PR and user um uh entities but what happens if I don't know that information what if happens if I'm trying to build a a system by which somebody can just ask a natural language question I have to go kind of figure out maybe which entities to call upon or I want to ask I maybe know like which kind of graph to be able to talk to um there's a couple different examples I wanted to show you here so in this one here these are kind of a couple different hybrid type approaches so um in this example uh what I'm able to do is basically uh infer the types of entities and relations based on the schema so what I'm doing is I'm basically saying hey go get me the schema ID for this graph and the scheme ID should contain the PRS the the users the repos the commits for that scheme ID I want to then detect which entities and relations actually matter for the type of question I'm asking so I just just wrote a quick prompt here and I said uh go through these schemas go figure out uh based on this question I'm asking which entities and which relations from the schema matter give them back to me and I'm going to use those to run a structured query so what I did is I basically took the output I can print I'll run this to you guys and see how it works but um it basically gets the schema it finds the relevant entity types and relations and values and then what it's going to do is then use that data it plugs it into a structured query here this very much the same as we ran before and it runs that query and it gets back the same data but this time instead of just printing out the the raw output from the um from the the system we're feeding into an LM and asking to summarize the information so we actually get is a bit more of a structure uh kind of well structured more detailed answer with information about like all the that's contained with the all the properties so when it was updated but the body of it actually is um so this is one type of like hybrid approach but another one that is like interesting and I think like particularly like exciting about doing work this type of work with graphs is this concept of completeness so I have a blog post I'd love to kind of share with you guys as well at the end um I'll just flash on the screen but one of the cool exciting things about um doing this type of work with graphs is that you're not necessarily limited by like having a top K like relevance uh you can retrieve um as much data as you need as long as you know the structure of your graph you can go in and retrieve as much data as you need passing your LM and and know for sure that you retrieve the data that you need to be able to answer a question uh completely so doing this though from a natural language question into a structured query to get this data out is a little bit tricky but this is the pattern that we started working on to kind of like be able to solve for that so let's say like we go back to this graph and who's our guy Nate there's a guy in here named Nate I've also been working with quite a bit all right so Nate here has been pretty busy right he's been working with let's see he been working on a lot of different types of PRS several different types of commits um and let's say for example like you know your VP of engineering or you're like working on open source project you want to just know like who's been working on a specific commit or um I don't know what what one of your teammates has been doing so if I run this type of query for the unstructured uh this is essentially a um a probabilistic query um it's it's it's it's running that top k type response getting back the triples and trying to summarize an answer what you can see here is that ring getting back nine uh which is actually incorrect I kind of this up earlier there's 13 um there's 13 uh PRS here that that Nate is related to um one other thing I wanted to point out is that we are just using like the word Nate we're not using the actual like node name here we're trying to actually infer who this person is based on the similarity of that of that name so um two challenges here right is finding the right the right Nate and then making sure we can get back all the data so one type of approach that we have here in addition to the inferring the scheme of before is what we do is we get back all this data here uh from the U unstructured response if I print this out instead of just the answer you'll see all of the raw data that we were showing in that query interface earlier so answer is just one component but you'll also see the uh the graph the uh data that makes this up so these are all the different uh triples I scroll over here these are all the different triples that make this up and then within your triple you get an ID you get a node you get an entity type all the data that we needed to then actually create that structure query so what I'm going to do is I'm going to go through that answer I'm going to pull out the the labels for the heads and the tails the values and the relations and basically going to say this is all the information that appears relevant according to my my um natural language query I then want to use use that data both the labels and the commits and the users uh the values I just want you go get everything for me go go find it all just tell me like what this person's been doing and then I want to use this information to then generate a summary so what I got back here if I just triple out if I just print out all the triples that have been created uh this is basically what came out from the initial part but this is all of the like data that seems relevant to that type of question but brought back in a fully complete and structured way so now what I'm going to do is I'm going to feed that answer into a prompt that just says here's the question here's the response we got in the form of the triples and I want you to then answer the question based on this context and what we should get now is all3 so now we see that we have a way to be able to kind of like start from unstructured find out which types of entities or relations are are relevant and important use that to get a more complete answer and then work that into the rest of our ride pipeline to be able to do some postprocessing and generate our rers so from this you can also just pull out other chunks of data you can do your own postprocessing on this information do lookups to databases to other like vector data Stores um this is meant to integrate directly into your existing rag pipelines uh but give you a lot of confidence that you're querying data and the semantic querying semantically rich and structured data uh but also able to kind of unlock that that completeness factor as well um so we doing some interesting benchmarks in this stuff um I'll jump over to our blog so you can see that as well um it's just Enterprise rag on medium and you can see some of the types of uh research we've been doing here um there's an article a bit on the um the completeness type argument uh there's a lot of interesting things we're talking about here when it comes to like multi-agent to multi-graph type questions um a lot of the exciting patterns we' think about lately is like what's it look like to have like a graph per agent to ground each agent's um like world view in the context of a semantically rich um Knowledge Graph um that's a really interesting approach um a lot of kind of multihop U multi-section type uh workflows as well um definitely some interesting things in here and then that article on um graph versus Vector versus a lot of other type of like use cases in here as well so uh if you have any questions please definitely check out the blog post but uh before that glad to take any other questions if you guys have them yeah sure I mean I I would love to kind of is there a specific type of question or use case do you have that can maybe try to run another query that kind of shows you what this looks six as well yeah sure let me let me show you this real quick so um I think your question is kind of hitting on um yeah great it's structured data but you it's pretty predictable what type of like maybe text or entities relate like exist in there but what if I like don't really know what type of like uh data I'm looking for I can ask a question like this like um uh who worked on S3 stuff let's see what this comes back with so the the value of this and I think the power of doing both like structured and semi-structured data in or structured and unstructured data in a in a graph is you can search on that unstructured text so if I look at this right here right we can see like the data that came back so um the answer is Tim finnean and this this person um so here so what we did was when we as we asked this question like who worked at S3 stuff we basically performed a semantic similarity search to discover like which which entities or which um which PRS are related to that term S3 right so we came back with this node uh from there able to then like understand that that ticket was submitted by or maybe this one commit was worked on by this other person and we able to pull together this answer so um I can share with you some other notebooks as well if you actually go to our documentation and then check out the uh example section in our docs there's other ways that kind of show you how we're doing like this type of tech this type of search over um all un structure data uh versus like semi-structured but um what I just demonstrated is that we can start with something that is unstructured start with something we don't know about and then use the structure of the graph to kind of get other more relevant Tech context that's been defined by by by us in our schemas if you have a question I'll raise your hand I'll bring in the mic to you so we record it it's a very long room so I'll get a lot of exercise be name thank you yeah so for that example so I interpreted the question that just happened and I didn't understand if your answer was different than how I interpreted it but it sounded like the question was what if the data that you have is not structured in any way like just it's an essay or something it's just one Big Blob of text then can you infer the do we have ways to build a graph not from a a structured format and I didn't understand if the thing you just said in response was an answer to that or if it was like how you do a an unstructured how you can take an unstructured query and turn it into a query against a structured graph yeah you're right that's exactly the question I answered so to answer your question though so yes that was the original question that was the was that the original question okay okay yeah you nailed it yeah you caught me um yeah so that what it just demonstrates how to start from my structured querying and then work you back into more structured data but um I don't have the the code I don't have the The Notebook up right now but I can pull in um this is actually an example I worked on not too long ago so um this is basically to demonstrate how like completeness Works um there's a lot of different ways to pull out this type of triple data uh from unstructured text the first way I showed you is like our engine up here so if you basically run that create graph and you specify a schema you're you're relying on us to use that schema to then like use these triples go through each individual chunk pull out text from those chunks and say hey great this is a this is a triple that matches the schema um there's other tools like I mentioned that are pretty powerful like LM graph Transformer right uh that we talked about before um that if you haven't used like the that Lang chain tool like take a look at this um LM graph ter just let you pass in like PDS for example you can specify different types of relationships you want to see and it'll rip out triples for you that you can then import directly into neo4j um but the other example I want to show you is kind of just a bit more rudimentary um if I so here's a really simple like a really simple prompt I wrote that kind of just shows you like how you can use an LM for example to pull this text out so in this example this is actually another Amazon 10K example um I use like llm Sherpa to basically if you guys haven't heard of LM Sherpa it's like it helps you extract like text but also structure from PDFs uh but I basically use LM Sherpa to then find me sections that are like Financial proceedings or I'm sorry like legal proceedings or like legal the legal aspects of a 10K document because I wanted to build a graph of the like legal things Amazon had to deal with over the course of like 2024 or several years so this is all the data right have like 10K documents from Walmart and from Amazon in here um we kind of show here about how you can use um how it compares to doing Vector rag with pine cone so definitely check that out and then if we actually want to do the triple extraction what I'm basically doing here is I'm taking that PDF I'm turning into text I'm chunking each one and then I'm going through each chunk or I'm going through top K chunks of retrieve from Vector database and then I'm saying using this context right like build me a triple that looks like this and I want it to come back in the form of like a head a relation in a tail so you can trust this to an llm you can trust it to your own um to your own models under the hood you can trust it to to us you can trust the LM graph Transformers there's a lot of different ways to do this but what pops out from here is a list of triples that we can then put into that same format I then serialize them and then put them into the make sure they map to the model that um that Y how uses and then use those to create the graph um structured data also very easy to work with because of that but unstructured data is just a matter of deciding um which relationship exists in there you can use a lot of different tools to do that other questions here guys on the far end you've been quiet if you have a question I can run to you any more questions sure let me bring it you mentioned um 80% on um accuracy on legal before you started using these techniques so what what's the accuracy that you reached what's the highest accuracy you reached in legal documents in legal docents yeah actually Tom you might be able to answer this question if you want we we actually just started running some some tests on this stuff just this past week to kind of compare both doing like uh rag against like grass versus Vector databases versus variety of other things also you guys have used rag ass before is also a really great tool you can use to kind of answer questions sorry I'm putting my co-founder on the spot but the question is like how do how does accuracy compare with doing this these types of like use cases with with graph rag uh we were asking specifically about legal documents but the higher level research we've done is more about like kind of graph rag versus other rag Solutions as well yeah nice accent by the way um yeah so it's not really a fair comparison because I can get 100% on a document if I spend enough time on it right there isn't really like a benchmark data set that we can compare against we were using ragas for like a larger academic set and ragas has things like faithfulness uh similarity complexity you can use recall and precision however you want to manage that uh as a general set across like 100 academic documents for about a thousand questions that have a golden data set we're probably looking at a lift in faithfulness of about like 45 to 55% uh and then accuracy you're probably looking close to 30 to 35 but again that's like a generalized Benchmark that I wouldn't really use like we can put those numbers up if you want you can potentially get published with them and we will and we have but uh when it comes down to your specific problem it's like it just depends how long you want to spend on it and I think the important thing about the structure of grow s and how you retrieve them is if you spend enough time and you iterate you have explainability and you have you can iterate and change and update and Stream in like we spend a lot of time on streaming so the answer is 100 but it's not really a fair comparison and um this is not necessarily about the legal use case the um there's another uh article we released it talks about doing like multi- multi agent multi-graph multisection of multi-d doents type retrieval workflows which is definitely pretty interesting but uh this type of uh benchmark ran was actually just using that 10K example to then figure out like how many correct answers can we get of all the legal proceedings that occurred and uh do we do it with varying numbers of of documents mentioned in the single graph how does it compared to doing it with with Vector rag as well um and some of our findings are in here thank you Chris I think that's all the time we have for the first talk because we need still to get to the second talk by m so let's thank Chris again and Tom right thank you thank you guys thank [Applause] you for