AI Agents and Knowledge Graphs for Intelligent Querying and Insights, Tomaz Bratanic
Recording: AI Agents and Knowledge Graphs for Intelligent Querying and Insights, Tomaz Bratanic
okay so uh my name is tomash and today I'm going to talk about AI agents and knowledge GS so unfortunately I have a bit more slides so I should go first cuz now maybe you get a little bit disappointed cuz lean was so good uh but uh first we'll go to do a little bit of slides and then uh I'll have like a quick demo as well at the end so um as mentioned my name is tomash U I've been with NE forj for like eight years now and uh I write a lot of blog post and for each blog post I try to produce the code so because like if you just uh look at articles like white papers and you don't actually see the code like as a developer they're not like really useful so I really try to produce the code and as mentioned the dbo demo was also my code but let's go straight into it right so what are llm agents and like how are they useful right so like in a typical Rec pipeline how see this basically everything is predetermined I user ask a question we go to our knowledge base F some information pass it to an llm llm summarizes the answer and we get our uh answer back we go we give the answer back to the user so there's like nothing agentic here right but uh I mean the thing is like if you ask five people about agents you'll get like 10 different definitions right but how I see it basically is an agent is when LM has the option to decide what to do like how to fetch the data or how to construct the data uh and uh yeah generate like the final answer right so in here in this example we have like one predetermined step and then the LM can decide it either it wants to go to step two or step three right so it's very simple but it has the option to make a decision and uh basically the most popular framework I've seen so far is basically the react agent so basically we want the llm to be able to do this reing right so decide what they want to do and then we also want to give them the option to actually go and do the thing they want to do right so here kind of the reasoning and acting comes into play and uh this has been like a very popular U architecture and I feel like that uh the tools is like the perfect Implement implementation when you want to do agents because uh you give the agent a couple of tools uh it can use uh for specific use cases and then like you give the input to the agent or the llm the llm decides okay can I answer the question directly or do I want to go and use any of the tools it can be like multi one tool multiple tools and uh then the final answer uh is created right so basically for me an agent it's like if you just give like an llm two three tools that it can use for me it's already an agent right so some people I say you want to add memory to the llm stuff like that so that's like the add-ons but like as soon as an LM can decide what it wants to do for me that's an agent uh and uh mostly what we see in practice is that you can have a couple of predefined steps uh that you know you want to execute every time and then along the way dat L can decide of some of the steps it wants to do right so this is like one example from Lang graph and from Lama index um and then um what does this have to do with knowledge grass so I'm lucky that LE did a nice introduction uh to knowledge grass so you kind of know what we're talking about uh but like for like 15 20 years now it was kind of known that knowledge graphs are really good at representing structure data right because you can have entities such as U or like nodes such as Lew entities or like um digital entities and here for example we have Mona Lisa which was painted by Da Vinci and so on but like now in the context of llms it's much more um mainstream or like the use cases have Shi shifted a lot more to unstructured data right because everybody wants to talk with the PDFs or documentation and the nice thing about knowledge gr is that it's very useful for the structur data but you can also add unstructured data as was also shown in the demo so here they have like one example of in a legal domain where you can have like a contract and then you can have some structured information about that contract like who signed it where are those uh companies from but then you can also append the unstructured information to the contract as well in this example it would be closes and U uh knowledge graphs are mostly domain agnostic so you can represent any type of data that you want and here I have like a de Ops Knowledge Graph where you can have like teams microservices and in this this case like your Asana task or like J task that are linked to the microservices of the team could be your unstructured data right and the nice thing for me is that you can have explicit links between uh structured and unstructured data and that really uh empowers you to then do better the the like application Downstream so that's kind of how I see is basically that knowledge are really good at uh representing or storing both structured and unstructured data right because in Rec applications most of the time people are just focusing on unstructured data so like PDFs and documentation but there's like this whole world of structure data that is kind of ignored or like not really looked into as much as it I feel like it should uh so I'm here like to advocate for like using more structured information in Rec applications so now let's bring it all together like we want to use AI engin with knowledge graphs and how how can we do that right um so um the idea is to give the llm tools that it can interact with a Knowledge Graph so it can either retrieve information from the knowledge graph or it can also store the information to the knowledge like sort of like a memory of the conversation or in interesting facts about the user and I mean you can do a lot of things and uh at first when uh GPT for came out we were all very excited about generating database paries like either text to SQL or text to Cipher in uh newj space so like the idea is you give the llm user input database schema and then the llm generates a database credit that fetches U relevant information from the database right and this approach is like very flexible you don't really need a lot of upfront work because it's very flexible so it's kind of also domain agnostic and like you give the llm complete autonomy about um what it can do that because if it can generate um database cred is basically it can do whatever it want with the database so we kind of give the lln complete autonomy over the database now one problem is that it doesn't work right for the production users like for demos you'll see it's kind of nice and U for like toy examples toy uh questions but like when you go to production generating database queries is usually kind of works maybe most of the time or even not that right so for like use cases when uh you're just like drafting database queries and you have like a subject meod expert and actually validating those database queries that's a totally fine approach but in a Rec pipeline where you have like non-technical users who don't really know what's going on under the hood uh like generating database queries isn't good enough um at the moment at least uh cu it can struggle with complex schemas or questions and like this like the whole thing of like mapping use like because users usually don't use the same vocabulary as the database people right so then how do you map this to um vocabularies so there's like a bunch of problems that needs to be solved and like you can get like kind of good results and then when you say like when you get like 60 70% accuracy that's kind of good but like in a like pipe line do you really want to give that like to your boss and say you ask the same question three times and you get like two or three differences then probably you uh I mean it's a dubious uh idea right so like then like what's the solution right if we can't really give the Ln complete autonomy over the database I'll do questions at the um at the end so if you can't really do like that direct database quy generation what do you want to do right and like before I mentioned that you can give the llms tools to interact with the external word and in this case you can give llms tools to interact with your knowledge graph right and the idea is to move the complexity away from llms and uh switch it or like put it into the tools right so you can have like subject subject matter experts who Define those tools and because they are like deterministic you know that like if you get the same input you'll get the same output every time that's and you can also Define what types of outputs you want so that can also help you design better applications uh down simil like chat interfaces right and because I said that you want to have like predefined tools so how is it basically mostly there are like two types of tools to interact basically with any database it doesn't have to be just graph database but like one the first one I call it like Cipher or credit templates where basically you define like your database credit so here in this example we match an organization by its ID and then we return its partnership subsidiar suppliers um because we use the same dboard graph and uh in this case right you just get you just say to the llm if you want to find information about the company like you give the description of the attribute and then we do a little bit of mapping to the database but essentially the LM decides if you want to find information about a specific company and then it just passes that company as the parameter to the template and then we get the same exact same information every time when we interested about the company and then the second approach is a little bit more involved so you can also then dynamically generate Cipher or query statements so it's not just you have like one static um database query very populate parameters but like based on the tool input you can like dynamically generate um Cipher statements and U it's like dynamically generated but it's deterministically right so you know exactly uh for the same input you always get the same results right so again here you can see that basically the llm are just uh tasked to populate the tool parameters and then the actual retrieval logic uh it happens inside the code I think because it's code it should be consistent every time right so that's kind of the idea how you can go about moving the complexity away from LMS to generate database queries and put the complexity inside the code so you get much better uh results and like one example could be like a movie agent so you can have like a knowledge graphs about movies and then you could give the llm tools to find like let's say information about movies or people or like another uh tool could be to recommend movies and then finally because we can also store the information to the knowledge graph it could be that it could act like as a memory tool so like if I user says I already watched Top Gun you can store that information inside the knowledge graph and then you can use that information to produce better recommendations next time right and as always knowledge glass basically it's uh domain agnostic so before we had like a movie agent and now we have like a biomedical agent and U so like the approach is very domain agnostic but it's uh very nice I feel like because you can combine U structured information that you have also with unstructured information so now let's do like a quick demo uh second so again we'll be using divot as the data provider but now we'll have have like 100 million notes uh in the graph and new for J is where we hold the that information and then like the Lama index will be used like as a simple agent interface uh and here's the database schema so basically again we have the organizations and people and news but it's slightly more involved than in the lean example so let me do the now let's just see if it works okay so basically we defined a little bit of utility functions because when you're dealing with seduction information most of the time you need to map values from the user input to the database right because uh let's say if you want to ask questions about NE forj uh you need to find like the note that represents NE forj in the database there are different types of approaches like you could use Vector search keyword search combine the two uh here we just use a simple keyword search and then as I showed before we'll have one Tool uh uh like a cipher template that will be used to find information about organizations uh which is this one and then the the second one will be to get the latest news about companies or their the related entities uh so it should be quite simple and then we put the view open AI put it all together uh with Lama index and let's see what happens so now if you ask who the board members of NE forj the LM decides okay I want to call the get company information tool that's available to me I will populate the parameters just company NE forj this is what it gets from the database so it's not really nicely printed and then U the information contains which are the buard members and it can generate uh the final answer to the user right so this was the first tool and then the second tool was to get the latest news so again you can use like what's the latest news about newj it gets that information and then generates the answer so there's like a newj connection event going on but as I mentioned um because we have information about new forj but also it's suppliers competitors so we can combine the structured information we have in the graph with unstructured information so we can say what's the latest news about new forj partners right so in this case we will identify uh Neo forj in the database will Traverse the partnership relationships and then find uh the latest news about those uh entities that are basically partnered with NE so in this case we are combining the structured part of the knowledge graph with unstructured part which are the Articles right and Ne part with Microsoft and as we can see snowflakes and uh they're doing a bunch of cool stuff so um and this is like I'm sorry because he has such a nice demo from Le before so might be a little bit underwhelming uh but that's the demo I have and now um basically we can finish up so basically at the moment like when you're doing agents you don't want like full autonomous agents to let them just do whatever they want to do but you want to constrain them with like tools that were defined by subject method experts and then how I see it like anytime you can switch the complexity away from LMS and change it into code you definitely want to do that right because with LMS most of the time you get like 60 30 maybe 90% accuracy but like if you can change that 60% accuracy to 100% accuracy with the code uh you definitely want to do that so um that's it for me and now I can take some questions hi I just want to go back to your slide of the inaccuracy of llm this could you please explain the uh the xaxis ahuh okay so um the the most important metc here is p p one so basically how you kind of usually U evaluate database query generation is you don't actually evaluate the generated query but you evaluate the database that um the data that you get from the database so if the data that you get from the database matches what it should be getting then we get like one and if it's not it's a zero right so the passet one means that 50 % of the time we got exactly what we wanted and the other 50% we didn't get and then pass it three means that oops we ran the same query uh we we generated the query uh three times and now because llms are non-deterministic maybe one of those three times it gets it correct so like pet three means that out of three rri was at least one successful so like we get from 50 to 60 right uh and then the jacket means that um even though the data wasn't uh exactly U identical is it still similar right and um I mean the thing is with database credit if you run the like llms is that if you done the same evaluation multiple times you'll get different results but it's like usually in the 50 to 65 C 70% accuracy right and as I said it's good enough for like drafting database queries but not for rec applications thank you so on the slide 18 inv mention a solution a predefined toolkit uh sub matter experts can help Define how and which information to can you a little bit explain exactly I assume subject matter expert is a human how this has been done I mean yeah so I mean because sorry before subject matter expert mean they're very relative right means they subject matter expert within quote within a code sometime how the quality of outcome that result Al of this sort of approach I mean the quality definitely improves the thing is that because we have to kind of foree the use cases and prepare the tools we definitely limit the scope or flexibility of our agent but in uh for like because we limit the scope we we get better results right so the we like you're always kind of trading with like agents do you want them to be more flexible and less robust or less flexible and more robust right and I feel like the less flexible more robust is the way to go at the moment um what is the thank you uh what is the um right or or the upper bound of tools uh that you recommend uh use uh using um and is there um an approach like reranking or something like that for Tool usage you order to scale it are there any insights about this yeah I mean so this is not really yet like a mature space and it really depends on the llm that you're using but like I've seen some studies where they say uh more than like 15 20 tools the accuracy drop significantly but then it also depends because with tools you you they really should be how I say semantically orthogonal so they shouldn't be very similar because if an LM can choose two tools for the same use cases you kind of confuse it right and then you can get different results so you should be careful that you don't really have semantically similar tools uh so I would say I want to use more than 10 tools but as you said like you can um dynamically select tools so you can have like a vector search and like use the description of the tools uh and match them with the user input and find the top five 10 uh tools that you want to pass to an LM so you can have like a base of 100 tools but at run time you just P the top five 10 similar tools to the agent so in that way you can scale I have a full question on that so are is this toolkit entails like if we are implementing within one Enterprise and E system does this toolkit entails a domain specific different tools like in an Enterprises HR marketing or different different uh like use cases so every tool will have a specific domain specific use cases and will it be AI enabled models based or it's an independent independently driven like how does that work I mean so because the tools like you have the code right you can do whatever they want you can have like nested llm agents so like the tool can have additional llm tools if you want uh so the tools can be as complex as possible as you want them to be but like my I I I advocate that basically uh you don't want to use uh too much like if you then use again llm in the tool right then you you can again um how you say um come up with erors of the llm right because here be show that you don't want to use Ln when it's possible right but you can if you want but then again you'll get the non determinism of llms and then maybe the error uh rate um uh is greater that but uh for me I would say like you definitely want to have like very domain specific uh tools uh so they're not I mean as I said you reduce the flexibility because it's very domain specific but you get uh accuracy which is nice my question revolves around the uh popularity of Neo forg within the last two years um ever since the uh the democratization of open AI um I feel like everyone is starting to move towards knowledge graphs so my question to you is what's the next um I'd say um not revolutionary but the next big um thing with neo4j when it comes to adding new features and uh new things that you're working I don't know if you could disclose that but it would just be interesting to hear yeah I mean I can tell you what I'm working on so basically uh as I said we you you can have subject meod expert defining the tools but I seen that like there's like a a bit of overlap with in the types of tools that you want to have like because like how you see it at the moment like you can you can like do some aggregations or or you want to count things in the database or you want to list some information and I feel like that can be very generalized so instead of having subject matter expert Define the tools we can have like some generic framework that provides the tools for you uh obviously it's not exactly what a subject matter would do but like to just to get started uh I think it should be good enough so I'm my job like my hope is to get like because uh knowledge graphs can be scar right because you need data modeling you need like subject meod expert and like how uh can you do it like as with Vector database you just click to buttons you do two lines and everything I mean at least the the first steps it works right so how can we make like generic tools uh to pass another and that just work out of the box have any questions back here okay what is this Cipher about and is it a programming language like Python and why your preference that's like a SQL for gra databases so that okay I am hear for the first time okay uh I think Michael you had a Michael had a comment no yeah exactly uh he can tell you exactly I just wanted to say some of the things that are coming for NJ for J so first of all there's a new NJ python graph package which can do knowledge gra construction and um graph retrievers as well uh we will add knowledge gra construction also to the data import of NJ so that you can basically pass in you know folders of files on a three or G um Google Cloud buckets and it will automatically construct knowledge graphs we just released uh relational database report uh starting with post Quest but you're adding many more sources as well uh there is co-pilots now in J we just published fine tune texture Cipher models and data sets as well which also integrate some of the data sets from tomar and uh we're working on uh integration with near forj with snowflake so that you can do graph compute on your snowflake data as well so that is also now available in Early Access and uh what we want to do next year is to look into can we actually integrate agentic uh tool definitions into the near or platform so that you can basically then have an UI and apis that then would use these basically these tools automatically for an uh natural language exess so that you can basically just by using the apis everyone could use the tools that you defined for your databases and for your schemas as well and the last thing that we've also been looking into is actually use llms to support generating graph models because sometimes you have other unstructured or structured sources that actually have only entity table information but no relationship information and then llm is actually quite useful there's a whole area of foreign key detection with llms so to actually generate graph models from isolated uh entity sources as well so that's some of the things that we working on or have already released thank you Michael okay have you seen any it seems as though uh agents have a far easier time with a simple database schema have you seen any movement to like have people convert an unusually complicated database into something simpler so that it can be queried encountered this recently myself I mean uh I didn't see like simplifying schema but I've seen like simplifying quties right so you can take a user question and then you can break it down is this a simple question it is like like a multihop question where you're actually answering more questions than just one so kind of you can do the break down and then like you can answer each of those questions separately or you can have like a flow where you kind of generate the cipher for each of those questions and then you ask LM to combine ciphers together so it's not like simplifying schema I don't I I've not seen that so much but I've seen like uh breaking down questions into like chaining uh different uh steps and simplifying the database cred generation I've seen that a lot maybe one more question and we'll have uh I'm curious if you have tried to get the AI to write the tools for you as one question and then as a second question it seemed like the graph you showed with the the bars that was to see if it could generate consistent but I'm curious if you feed the erors back to the L and give it multiple passes at generating the query whether you've tried that and whether that works uh so let's start with the second one so uh one problem with like doing multiple passes is that you don't like you don't really know if the information is correct or not right you just know if the credit return some information or not and you can say okay if there's no information let me ret but if it's invalid information you don't really know right so in that case that that that makes it tough so you can do multiple passes but only if no information was returned because you you have no way of knowing if it's valid or invalid if you knew that you wouldn't need to do multiple places right uh so that's it and U what was the first question ah yeah yeah yeah so that's basically uh you can definitely do that U so basically you just get them to write python code in this example definitely possible and in I mean it's then depends how much uh human uh uh like feedback do you want like do you want to like provide the types of questions you want to be answered and then the LM generates the tool or it can be more like generic CU what I'm doing at the moment is going more generic route and like having like a rule based tool generation and then the llm can just Define the descriptions right so it's l llm does the prompt engineering but the actual code is predetermined so you can do that or you can have the llm generate tools as well uh but it's not we haven't tried it out yet thank you very much [Applause]