Bay.Area.AI: Graphs and AI: Making it RealAlexy Khrabrov, AI Community Architect, Neo4j
Recording: Bay.Area.AI: Graphs and AI: Making it RealAlexy Khrabrov, AI Community Architect, Neo4j
this is the the talk about graph rag with the knowledge graph so let's kind of just do very quick show of hands who heard about knowledge graph before all right and who heard about graph rag before right that's a pretty good audience right this is more than than you usually have so this is good um so about me uh on top you have all my handles so I'm Chief scientist everywhere it's not made up I've been the chief scientist of public company uh called Nitro which was an early uh leader in smart documents so we applied machine learning to PDFs when it was not sexy and they very boring and it's still not sexy but it pays Bill the bills right so contracts uh all kind of business documents leases uh very interesting area so we use spark and uh kind of early approaches in LP to do that uh I'm at Chief scientist everywhere where that basically exists on Twitter and telegram I'm chief scior on the web where I blog on my medium and I'm Chief scientist on link so I'm now ai committee architect at newj which is a Dev devel function you my colleague is here I'm extremely excited to be in devl because we get to play with code uh and we connect graphs open source AI people and also alliances like AI Lance I'm actually a co-founder of the AI allance uh which I set up ATM helps set up with meta and 100 other companies um and I personally brought in data bricks any scale length chain llama index Snowflake and a bunch of others and universities like jpan Berkeley Dartmouth Northeastern and essentially all of the folks are knew in the community so it kind of comes very handy when you know a bunch of people if you need to do something like this you can actually invite a bunch of people who understand why they should do this and they come and my really favorite one of them organizations I help built is open source science you can see that on my t-shirt this is a URL open source. science and that's at non Focus non focus is a python data stack home it's a nonprofit Foundation who heard about non Focus before very few people who heard about pandas or Jupiter or napai or scipi right this is all in non Focus right so check it out theate to if you can uh and so we connect open source researchers with uh open source developers with Scientists right so accelerate science to open source and I have a pet project there which is a map of science buil with neof J so if you want to help us we we building you know map of all open source ever usedful science with papers projects and so forth so talk to me if you're curious about that so I'm also founder an organizer of communities like this one um and I run the conference called scale by the bay which I found 10 years ago hopefully we'll bring it back next spring and that's basically like this but times 60 right so we have three days full of talks on three tracks uh data engineering software engineering and AI scale so and finally I'm a photographer you can not just make you know carry my camera don't mind me uh I make a lot of pictures I publish them uh and you can probably find yourself in one of them and I also like kids cats and dogs and I have them in the ratio of 4 to one uh okay so brief inro to new who heard about new before well this is really an advanced crowd I'm really appreciate you know appreciating you guys so just kind of to catch other folks up new forj is a graph database it's actually a category defining database were no commercial graph databases it's it still feels like a startup it exists you know for more than a decade and uh basically it's a database right primarily it's a database and if you notice any serious it business has to have a database all the data should be somewhere so the biggest companies like Oracle now data brakes snowflake they're all databases it's not you know there is a reason for this and the the cloud companies build their own right so so Google uses big query this public version of the internal big table Facebook has very similar stuff you know who who's the like the old adash goes who's the closest to the data has more power right and actually it's interesting like inside of big companies whoever is closest to the data stream wields more power in these companies because because they can process this data understand it and do something with it so we store the in a graph and we do analytics on a graph and who who here knows what graphs are right who has Computer Science Education who is a computer scientist okay who is uh you know not a scientist let's see if if adds up not really we have tertiary logic here but uh what what happened is it's it's really phenomenal I found that it's not a given anymore right we have such an influx of people in a programming a Engineers a lot of them did not know what graphs are so if you know graphs it's it's phenomenal but that's true we need parameters and graphs I'm not going to do one here but I'll do one uh kind of I make a point so everything now majority of data is stored in relational databases who works with Rel relational databases yes my SQL right anything sqls everywhere so so what what happens is you want to connect persons and assets right and in order to do that you has basically have a table you know every table has a unique ID every row has a unique ID usually person or asset will have unique ID and then you basically need a table right person asset which will just list IDs and so if you do a join right like this is how we connect persons to ass through that through that uh table and and so now you have this structure which is effectively a graph right so you can say a person all is an asset and instead of having this intermediate table you have a relationship this is how you naturally like this is how you naturally think of data right person owns so if you think of this it's very strange that we got to store our data in relational tables because this is not how we think right this is not how data is actually organized it's not naturally stor in the tables so uh we have this graph and now we have at newj uh database which can efficiently store this uh graphs at scale and really it's very hard to do it at Enterprise scale and we I think the best company to store it at at at high volume but we also can query so there is a language called suffer that it looks very nice it looks graphical right so nodes are represented as this parenthesized labels and then you have relationships right and there's like a little arrow so it's almost graphical and so you instead of SQL queries you can actually issue Cipher queries and now it's been standardized as gql graph query language which was the first query language standard in 40 years after SQL so there is basically now I think understanding that this is as important as SQL so and basically to sum sum it up right we use property graph model so you think of your data as uh objects with properties right and they have relationships we have graph data science Library available which runs all kinds of algorithms on this graphs so you can find shortest paths right you can find clusters clicks and so forth uh and it's used in many critical use cases right so it's very naturally used by Financial companies to find fraud because all kind of actions all kind of human interactions are naturally um used modeled with graphs and a lot of three agencies are using this for the same reason so there is there basically any social network you have natural application for this so now what has to do with graph rag so three source of data in graph rag so we have unst structure data right we have a bunch of text your dark data your company internal data PDFs and Excel tables and a lot of stuff which is not on the internet and nobody knows how to find stuff in it including slack right and when you ask you know what's Apple primary business then you you basically have like internal Google you can drop everything in Lucine or elastic right and somehow it will actually give you not necessarily the answer it will give you a bunch of documents which may be have the answer by way I must give credit to my uh colleague Andreas scholger who basically built all these slides and stuff right it's my third week on the job so you know I'm shamelessly using everything I can find you know mediocre artist copy great artist Ste so this is direct stealing from Andreas and I think he did a great job here uh so now pure data right so we have actual data like I said stored in databases so uh we have structure right people spend a lot of time carefully curating validating data usually these databases have schemas constraints unique IDs you know they have to be well formed emails has to be Anil and so forth right it's a structure data and now basically answers from this data look like numbers so if you're a SQL analyst you ask a SQL query it comes back with either a number or a string or a tuple or a vector something very rigidly expected right now the fun stuff happens in the middle right want both why not have both right so we need some how to connect these things right and this is not easy this is where llms fall on their faces and a lot of stuff goes wrong because one thing is fuzzy one thing is rigid how do you connect them right and so interesting question which investors impa by chip shortage like this sounds like an interesting question which is hard to answer it's really hard to answer by any of these traditional techniques so we want to do this right we want to attack this so we have basically a way to to use graph rack so we're going to use a Knowledge Graph so Knowledge Graph is an architecture so generally speaking uh my view of this is that knowledge is something that only humans can decide right if you think of knowledge uh it's not what computers do computers spit outputs they output strings they blink with pixels they just do mechanical things and you can interpret them as content right so essentially Andre Kath said like there will be no program languages anymore you will be inputs Giant llm and will directly visualize things on the screen with pixels like it may be text it may be image they can like rip out complete infrastructure of the internet and just go directly from your sensor input your keyboard your audio input to pixels so machine probably will learn how to do this like it will just show you what you you want it may not know at all what it's doing it's optimizing the pixels that you're going to see and making make noises and whatever like smells in the future right it maybe will plug into your neural link and create images in your brain of the kind you want but it's just doing whatever it learned in the process kind of knowledge is only something that humans can say it is knowledge so we always need humans and humans are not going to replace AI is not going to replace humans at another talk which I you know her at Stanford indust conference uh my colleague former colleague from IB Zu who is a chemist uh computation came and said AI um cannot be a stakeholder AI cannot uh have ownership right so similar thing like humans are stakeholders in humans own knowledge so I think it's very important that knowledge graphs are created for humans they can be automat automatically created but humans are going to use them humans are going to look at them Knowledge Graph is for us to organize our knowledge so ultimately it's something which is a technosocial which exists for humans it can be used you know by machines it can be built with machines but we need to understand what's going on so and so what is graph rack graph rack actually uses a Knowledge Graph to do information trial meaning search query and Discovery because it works with a Knowledge Graph it in this way de hallucinates LS it grrs them in human knowledge right so there's implicit understanding that our human knowledge uh is there and H going to find this useful so how do do we do this so we have Vector search we have graph queries and in the middle we connect them right so we do search and Potter matching so and also remember now we are moving to this compound systems with agenic architecture we don't have to do just one thing we can do a vector search can get a bunch of vectors then we can go find things in these vectors and we can go like maybe they nodes right maybe where the IDS of nodes we can go in the graph we can Traverse the and find interesting things we can go to the upper like subass or super class we can find a person we can find the manager we can find kind of people they are and we can repeat this right so this is a compound system uh so here's an example uh we have you know C see Edgar financial data it's a very well-known database uh strangely by programmers who are not usually Financial people who heard about Edgar a few people right so this is something publicly traded company have to file right to basically uh update the public that they're not going to fail and so all the critical risks and the kind of state is there and for 30 is what investors have to F to file and so it's a lot of fun basically to merge these two things and they're extremely dirty they're kind of messed up so they unstructured data but they have some structure which we have to obey right so they are metadata there are names of people who invest and so forth so how do we model this so the key strategy is a minimum viable graph like you have in the startup minimum viable product you build a minimal viable viable graph and you do it in these three stages you create you create the notes and relationships you enhance them right so you you can put anything in a note it can be a bunch of embeddings it can be you know a lot of different things it also the relationship this age can also have any random data in it right so you basically anything you know about every note and their relationship you could put in the graph and then you connect right and then you look like how can you connect this nose are the relationship between them just like we do you know as humans so here's an example so what do we do with this uh Adar so we take all these forms and we chunk them and you will hear about chunking a Hope from other folks talking about rag but that's a very typical thing so we for now we just do what Rags do what Vector Rags do we chunk them and chunk them and we embed them so we replace chunks by embeddings we get a bunch of embeddings now we steal a vector rag right and we Now link them it's already interesting right like it's not just random chunks not floating in soup they now have a link We Know they're connected still not much right like it's still kind of very mechanical uh so we create them and we have a schema right or it's already interesting we have some kind of formal linkage but now it becomes really interesting because we know they all belong to a document right so what we do we create the summary of the document which becomes a head of the list and then we create a link from every chunk to the summary right so now we can always go back from the chunk to the summary and if whatever happen to be in the chunk is kind of similar unrelated we still can find that that's related right so that's already interesting now we take four and 13 we really go to town because that's a really fun thing for graphs right it has all the managers and it Leist what stocks they own right to see that there is no conflict of interest and they own stocks in companies so we just put everything in the graph and also they have to list addresses so we can really have fun here because we can put addresses there and we can have like poor men's uh Geo search GE special search we we kind of we can actually use Google Maps right to to put them in fun proximity so this is this is really fun and now we can put it all together so this is kind of the combined graph and now it really looks like a graph right so we have a lot of fun stuff in it so and then this is basically a summary of what happens here right we started with a bunch of forums and now we have this really rich graph which has chunks of things it has summaries of fors it has managers companies and addresses and it's all linked together and we built a bunch of indexes nothing prevents us from thinking this is all a bunch of documents so we can just put it all in the full text Index right we can have your TF IDF whenever we want but we can also have a GEOS speciaal index and we have our links so this is not an end yet I'm going to show some code but I'm going to stop here this of the slides uh I'm going to maybe get a few questions and so these are the links so the top link is the GitHub uh repository with all the code which I'm going to allude to uh the second link link is the shortal to the slides as Google Slides so you can see them uh all of the videos from 2013 from our mop are going to functional TV which has more than a thousand uh talks from the mops and conferences so you can find these videos there in a few days because Matt is extremely efficient uh and he was doing this from 2013 so let's give a round of applause to Matt he's recording this always uh then we have an amazing graph Academy so everything I said if you need to dig like like they now s delve into this uh you can go to graph Academy which new forj AMS so when I joined NE forj this was the easiest and the best on boarding because they have this amazing resources they're really really good they also do course with deep learning.ai with Andro Ang on this right so I highly recommend you go and puruse this you can spin up Cloud instance of NE forj for free you can get your Neo desktop for free so there is a super easy curve this is my email you can email me and as I said I'm Chief scientist everywhere and one more thing so because we are devs now I want to really convene a new committee called Dev real. so this will be what devs should do to help developers learn this stuff and I think uh we did some interviews with the speakers we're going to publish them there we're going to crosspost all kind of deal content we're going to invite cool kids from everywhere whatever else to to share their knowledge right so this is going to be useful resource there's nothing there yet but it'll be very soon so keep an eye on it so now uh let me see I I can take questions now um or maybe you know I'll in the interest of time I'll I'll show you some code and I'll just take questions in the end okay so that's the um that's the that's code so this is very interesting right so this is um the repository right so so what it does I'm just going to focus on one notebook and the Beautiful Thing by the way this is you know vs code you can run your notebooks in you know I was out of python for a bit like I'm super amazed how far the ecosystem has gone it's really pleasure like you know if you if you like run if you know shell and python like in IDs it's super amazing so um yeah so this is basically notebook you can run it on Jupiter but I prefer to run this code uh so what happens so the uh previous notebooks here they do this minimal viable graph so you all the Edgar database is prepared right all the hairy stuff is taken out and so it's it's right here in the repository so it basically pares it it creates everything I said in a NE instance so this is by the way what NE instance looks like it's a beautiful code and beautiful setup so it's uh the new desktop get it for free it runs an instance of local forj database it has uh the the CER browser where you issue your commands and all the history it see it has all the instances and notes conveniently extracted uh here and it has this beautiful visualizer called Bloom so you can see and that obviously all of this is scalable right so you can actually drop your graphs in this and explore so I'm connecting from vs code I'm connecting to that instance and I already have all this stuff in it so basically use chain another beautiful thing about open source NE plays with everybody it turns out this is so useful wi integrated flank chain llama index Pine con which Roy is going to talk about zillis with Jung is going to talk about we8 and everybody else so we canr basically with all the measure rags and open source packages for LMS because they find it useful so we're going to productize our Lin integration in you know few weeks it's been basically it's out it's going to be officially supported lamb index and so forth so this is this is really really great time for open source AI so every all of this stuff is supported right so basically what I do right I set up my things I talk to uh open AI llm I have my credentials the usual stuff all right and so Vector search so first of all uh you can treat uh new forj as a vector rack because all the text is just a bunch of documents so you want to completely ignore all the all the links you just um treat it as a vector store right so I have now Vector store search right and so there is a company that called net app so if I do you know if I do this search on net app it tells me where is net app headquarters headquarter in San California if I uh and and then I can ask more questions I can ask what's it primary business and it gives you an answer this is the vector rack right and then uh I can ask about Investors and it says it's not mentioned in the provided content because the documents themselves do not say easily who is the investor can the r cannot find it this is a very primitive R I'm sure Pine con and and Mills can find it but this thing cannot right and what are its products and Vector search will tell you what its products are and this is all remember is coming from this form so this is not bad some enterprise software now really fun things happen so what now you can do like look at this it's a thing of beauty you have your text query but now Lang chain has the notion of retrieval retriever which basically now that I got a bunch of objects back what I'm going to do with them and because it's a graph rack what comes back on notes and relationships and now we can match so what you see here this is a cipher query right and basically this is a representation of a sequence of chunks it's a kind of FR X for chunks right so we have nodes and then we basically rank them right and and basically uh we use the the uh Cipher query in addition to do something with the result results right so that's query window now let's do comparison so we'll say who is the most important what is most important to net app's business right and you get two different answers the top answer is Vector only the second answer is Vector and the uh the query the window query with some rearrangement of the nodes right and the the first thing is very Bland it's like what you see in marketing present ations right data management cloud storage Cloud operation markets right well the second thing is also kind of business diversifi customer base and Partnerships with leading Cloud providers it's already kind of a little bit more but you can still it's a wash right it's not like exciting so now now you mix text and data right and you basically add uh you you add more you add more um Cipher to this right now now we're basically merging what we know in the graph with so we're not just ranking the vector output using nodes we're actually using Cipher query and that should be more fun right so let's compare that who is that app stop investors so the vector search comes with nothing Vector search with ranking come still with nothing because the first thing was nothing but the the knowledge gra obviously knows about it because we build this right so it says basically top investors one guard group black Rock and Prime cap management amazing and now the final C kind of um most amazing question so what we can do we can generate Cipher from text right so we have text to Cipher generation so we basically say we want the query for the graph which will do this right and Cipher as you have seen pretty involved so the thing just goes and creates a cipher query which optimally reflects what you want to find in the graph then goes and finds it so we do that it's kind of enormous right and now we compare everything right so we have Vector we have Vector plus ranking byos and then we have Cipher and text to Cipher so most important so we have this kind of wishiwashi two things and the vector and and graph says Cloud Le data Centric software Services it's kind of more interesting hybrid multic out okay that's fine but let's let's look at the investors right which was failing so the vector Rec found nothing the rank found nothing the vector augmented with graph get gave a few right it gave one guard Black Rock and Prime cap management and the suffer we found all of them onear black rock Prime cap capital International investors and State Street cor right found all of them so and then there is some GEOS special search which I'm not going around all right so then you can do a lot of fun things so basically that's you know the end of my uh little demo so uh what's really exciting right like you can really play with all this stuff you can look at the uh graphs and we don't really know what's the right way to do it so Roy told me like these things work these things don't and I don't have the answer the super fun thing is there is a lot of things to play with right like you have just strings it's kind of voodo magic right and you can kind of AR I'm really excited about this way because it can optimize these queries so I talked to Michael he says we can do that so I'm really now hopeful we should connect and like this as you have seen this query which has a prompt and a chunk of the query language hopefully we can optimize it as a part of the system right but in general if you are a computer science geek if you love graphs like now you can bring all your forgotten graph algorithms you can dust off your skills and you can bring them to bear on this stuff so this is super exciting and basically it connects to everything computer science is good for right and now we have a super fun database to play with so I invite you all to to try it out and check it out and talk to us if you have any questions thank you so I'm going to bring back this uh question slide so you have more time to remember the URLs maybe somebody can help me with the mic uh to bring it to people uh well you have a camera oh maybe Yan can do that all right yeah so please wait until they bring you the mic so we can record it on yeah yeah do you have any strategies to prevent the data from becoming stale is there a way to you know ingest a new invest report and if some of the investors have changed being able to find these noes and just update those and not re rebuild the graph from scratch is that something that is possible this is super interesting right uh again uh I uh have this pet project of map of science which I want to update regular with new papers right like same question so uh the graph itself is not natively time stamped right so basically but you had no so you all you in New investors when you investors come you will just drop them in the graph in the next query will pick them up right so this this is not a model we should train the knowledge graph is consulted upon every query so in that sense there will be no stess right like the new documents basically you know it's it's your your pipeline it's not the setup of this right because as soon as your documents drop an enter you scrape it's how you set up your you know like fulfillment filling of the rag you you pick up a new document you parse it you put in a vector rag you drop notes in the knowledge graph and the next query will run against them yeah so I think it's the data Engineering St hello um my question is probably not something you would encounter a lot in not a scenario you would encounter a lot in a machine learning based application necessarily but I'm but I'm curious about graph databases and like how like what optimizations or what support exists for like doing aggregate style queries on Craft databases so an equivalent of that question would be how many investors does Net apppp have as opposed to who are the top K so like running aggregate queries um are there any optimizations on graph databases which make them faster than the current yeah I mean the the key point of the graph database it doesn't C like the notion of Agate query in that sense does not exist because an agate query is your convoluted SQL which does three million joints and then it fails right like after actually like a dozen so in this case you do not need this if your data you can follow 12 links right and it will basically match again like I need to catch up a little bit on the performance bottle next but the main idea of the gr database that if you have deep relationships it will basically follow them right to the end of the Earth this this is not a limitation how long the chain of this will be right it's not constrained by by like like SQL by number of joints because it kind of blows up in memory right it's structured differently it's structured specifically to to have this queries run efficiently yeah thanks for the talk the question question is um what's the current status of the text to Cipher with language model and what could be any suggestion to make it very robust for example I saw like providing few shots or maybe providing the schema yep excellent question again catching up like first of all uh so like a lot of this texto CER is developed as examples right by devels now we have an engineering team which takes what theels do and prioritizes it and what's really fun about exer we have people like totally volunteer right we just did a podcast so I recommend you like find Neo podcasts and so Alexander erle does this Neo live and a few days ago he hosted uh a developer who just came up with amazing text to CER implementation like really deep using Laura like effective right so so what what we're going to do again you know don't hold me to it I think we're going to productize one of these versions so I know that our engineering team is working on doing this and so the more the more how they do it they basically pick best of breed right and we kind of put put it out and then we firm it up and we productize it so I believe they will be officially supported texto sare implementation you can just take uh but there is a bunch of them in the community you could use right and and so including like if you take the one which was released like a few days ago with a podcast it's fun because you can see how it's done you have great education resource and then you can go and use it good question hello uh this is more like a opinion question for you uh so this is something that I tried like a year ago uh with Neo 4G and uh llms so what is your opinion about using uh like doing this text to Cipher for uh factual correctness checking because uh not graphs are factual databases and you can always conver into Cipher and verify that things are right and then create a data set out of it and then do things with it so I think that's the best use case right because you know first of all graph rag is not a specific architecture in my mind it's a family architecture so if your customer like NE customers have knowledge in their graphs if you already have a graph you do not construct it on the Fly Right you actually have one and so what you could like we just built one meticulously right we kind of went through this we don't create it on the Fly you can create a graph on the fly from your documents if you need to right kind of extract entities from a prompt and make a little graph but if you already have a graph like we did with the investors right then we ended up with a create an query just running the old fashioned query right we don't need any LMS for this just fetches all the investors right like we use LM as a kind of wrapper to just rout it to suffer so if you do have an knowledge draft obviously that will be correct so I would like to see more of this kind of to me super interesting how can we create a lot of these graphs using the LMS then validate them using experts right and maybe human of loop maybe do spot checks like it's very interesting question high confidence validation at scale can be like you know maybe high confidence can automatically accept and like some low confidence we can feed through humans right but that would be super interesting now you can you have some kind of graph which you believe is knowledge right as I mentioned before and then you run queries against it and the Y help you to ask the right questions and if something is not in the graph you keep adding it to the graph and you increase your knowledge and now you're sure your knowledge is correct I mean that would be one way to do it so yeah absolutely uh thank you for the presentation i' I've used neo4j and I was really impressed with the balloon tool M I'm surprised you didn't uh show it in its uh full animated form so I put my data into the graph and I was able to just play around with Bloom and see a lot of relationships cuz I could see hundreds maybe thousands of connections at a time and and look at different relationships so I got a lot of familiarity with my data just having that visual graph just having balloon second thing and this is a technical question when we create our Vector embeddings can we create those within neo4j or do we have to create them externally and push them in so I think in this example we do this in in the python script right we feed them but we put them in a graph right so it's a good question like do we have functions we have a lot of external functions but I think it's you know it's up to you what to use foring and you it's a good question I will check right because you can probably configure some functions in you for J what what happens when you add a piece of text have some hooks right but I think yeah I don't know specifically it's a good question but you know you can do it easily one way or the other okay I was doing it right then thanks all right thank you for your bloom again I'm sure you know more about Bloom than me because you know you played with us a lot and I just started recently I hope to catch up to your level soon right but please come and show us Bloom right like this is you know maybe next time right we should do some demos because it is amazing we just did this Presence at eii conference we had booths with demos and people did play with bloom a lot so I guess we'll probably have to uh stop here because we have two more speakers so so let's see it's 7:20 so we can do one of two things we can just chug along right and have two more talks we can do a five minute break get some refreshments and come back but only on the condition with you do come back right like how about that who who is for staying who is for breaking who wants to do a break okay let's do a five minute break and come back all right but when I call people back please do come back I know this is may not be easy so the next talks will be pine cone and zillis and they're going to especially kind of how Vector Rag and graph rag work together from the primarily Vector rag providers so this is going to be super interesting so I really recommend that you come back in five minutes cheers