DevReal: Vasilije Markovic

um thanks a lot for the chance to talk um I'm vasili originally from Montenegro a small country in the Balkans and the only thing we have there is a lot of history and beautiful nature and so if you want to kind of remember all your ancestors let's say there is like seven eigor or something like that back in the tree if you loaded them to the vector database and you wouldn't retrieve the right Igor that you need right because the eigor the son of eigor or the grandfather of eigor so what we try to actually do and what we are kind of interested in is like how do we actually apply these ontologies how do we load and represent the data so we can show to the llms you know what happened at which time uh what are the contextual bits and tidbits that we can actually interrelate inside of the graph datab bases so we can understand how can uh llms answer the best and and give us the accurate responses so what you see here quickly is the effective um proposal or like idea of ours is that effectively if we try to just ask rag something load to Vector store probably is not going to work if we add cogni in between which is an SDK graph rag um store that uh can also be Auto optimized and has a lot of other features improvements on Microsoft graph rag we will have the increases in accuracy and this is all fun and interesting on paper technically but uh let's kind of see a small example of a project I did yesterday for hackathon and I'll kind of just show you how these things can work with these agentic Frameworks and how can we actually create atic memory on the fly so um in a sense um what I have here is um crew AI agent uh this agent has tools uh one of the tool is cognit tool which searches the agent memory that's loaded into our cogn store and then effectively um this is uh two agents or actually a main agents a manager agent that can uh research about real estate or let's say Apartments we want to rent in a part of town and then he has two sub agents one is a location analysis specialist this um location analysis specialist is going to go and check is this like part of town something where we want to live and the second one is the one that's going to actually analyze the home itself so this agent might check um the home and the size of the home is the furniture okay what do we really like about it so all of this you know with the creai framework it has a set of tools we can add to it it can search the internet it can do everything but let's say we already have some type of data that we wanted to load to it and this data was um effectively the information about the part of town that where some apart departments are we just generated this with ch GPT so this data would be um you know information about the prices pretty much scraping some website um then we might have an image of an apartment uh that we might want to load uh and analyze and see if this is actually going to be something we want to recommend to the user at the end and then we have some type of a user profile so this is a typical SF engineer who's into rock climbing and cryptocurrencies you know and and does all of the software engineering in Python and knows cloud computing and he's uh and he's into fitness of course so um with this um all this let's say different type of data we could even have some audio and and many more things we might need to um process it store it and interconnected in a certain way for it to be actually accessible to these agents and we saw that the agent framework has a main agent and two sub agents that now need to go and research and use this data somehow to understand what's going on so um with this um demo we can actually try and load the use cognit to load this data that we have inside of the graph store uh we can use neo4j for example we can use others although neo4j of course works the best thanks Alexi uh but uh yeah in that sense uh in the background we're also using a vector store in this context lens DB and now we are making a lot of calls to the open AI to to decompose the this data into the graphs but also we have some set of deterministic steps that can actually let's say uh improve the quality of these graphs and not just uh lead to more hallucinations so the process is now running we are decomposing all the images text yo Files about the personality type inside of the graph and then as we do that we get this graph visualization file so that's here so we can see that we just now loaded three different things um this first thing is uh the people that we added the user profiles so Jane John uh chunks uh names of the chunks everything colored uh prettily with d3js of course and then on top of that we have let's say the analysis of that room so you saw the picture we have the table location uh about that there is outdoor area wooden flooring so all of these things would be Dimensions that might be of interest to the user down the line especially if he has some filters and then here we have uh information about the neighborhood the property all of the let's say scraped information that we added um about the actual um apartment that it is in South Beach or that we use Zillo for that and and and whatnot so now uh what we would do with this data is we would give it to the agents to actually research so you can imagine this being a small snippet we could do this with tens of thousands of files populate these graphs and then agents would do searches and actually locate based on their preference sets that are set by the user just the apartments in South Beach with wooden flooring and things like that so that uh then becomes much easier to to navigate and um you pretty much can then contextualize only the things you need inside of the llm prompt so things would work so I'll run the agent a bit I don't if it's going to do anything well like this was a hackaton last night so let me try um [Music] python um nope um I broke something um but yeah sorry about that I would uh fix it another time but effectively the agent um would uh now run and it would have access to the tools um that would uh search uh the cogny data store that was generated after the first run and then uh it would use that in the context when making decision um we did this with two Frameworks and I'm not sure why this is not running now I should have tested it a bit more but effectively um the Assumption here is that the agents can also store their outputs back into the graph so we have another set of layers with their observations States and U decisions and then that the user interaction preferences and everything else can again be stored in the graph so what we can then have as this graph Vector representation is uh a lot of sets of data in different graph layers that as we navigate retrieve uh we can uh get to um kind of updated context that would represent that current state and and give more meaningful answers so that's my lighning talk I'm not going to bother you much longer it's been long evening I think and and good talks before uh but yeah that's um if there is any questions happy to answer than appreciate [Applause] it so essentially cogni it it kind of creates these relationships and um like does it do you feed it back into an llm or let's say can I take it put it in my pocket give it to an llm in a different place and be like now you know everything and it's kind of like I can implant memory yeah you can trans you can move your memory elsewhere so that's the point and you can search for it of course with and while here so once you create the memory on the fly like let's say the the data changes so how do you like do you just rerun the job again or is there more efficient way of doing it great question so we had a mechanism to effectively um where we delete the data or update the data based on the hash so we would rerun uh the data and we would try to calculate the hash of the file or the information that we added and if the hash pretty much changed we would replace that piece of the graph uh we also have like this non destructive way which is just adding more data on top that invalidates the data below so this let's say onology that then just gets propagated back to the graph so we could give it a rule set that says like oh this is no longer the case and then please ignore that part of the graph in the future which can work so multiple ways um and currently we're working on the ontology part more more closely should be live in like 10 days or so so how much would you say it's more like how if you compare this solution to a normal Vector database solution how much of a difference would you say there is like between just finding the relations between the data and cuz even with the vector database like we can find decent relation like how much of a difference would it make that we implement the solution that you're saying yeah so we ran it on Hotpot QA heart problems uh on a random set of 50 Questions where we saw that it was around 60% for a typical Rag and we got here with a human eval so human went and checked 87 with automated LMS an evl and F1 scores it was around 8 to 3% awesome thank you all right let's thank vasilia and all the other speakers uh let's also