DevReal: Simple Knowledge Graphs with Outlines, neo4j, and Modal, Cameron Pfiffer
Recording: DevReal: Simple Knowledge Graphs with Outlines, neo4j, and Modal, Cameron Pfiffer
Hello everybody. My name is Cameron Feifer. It's a terrible last name. I wish I should have changed it, but the the Fs were on sale when I was born. Um, and I work at a company calledext, like file extension. Um, and our entire mission statement is basically to make AI speak computer, right? So, so the way that we do this is what's called structured generation. Um, and structured generation is how you force a model to output text adhering to a specific format. I'm sure you've asked a language model to say like, please classify this into one of three things
And the model is like, certainly, it seems like it's A. And I say, Claude, shut up. I said say A or B or C. I didn't ask you to say certainly. Please just give me one of those things so I can program with it. I can inject it into my database. All that kind of stuff. So structure generation is the phrase for that
So if you want to work with structure generation, you can go check out the outlines package or you can just use VLM because VLM has outlines built into it. And if you just use the I'll show you an example of this, but just use VLM. That's kind of the the main point. So what I'm going to show you today in the talk is I want people to build robust AI systems and I want people to start building them in a more interesting way than the way they're currently doing it. The way that a lot of people build AI into their systems kind of language model stuff is they use these like weird agent frameworks and I love all the agent frameworks. They're very interesting but it makes the agent framework the star of your program. The star of your program is your program. The language model should be extremely boring
It shouldn't break. It shouldn't do any of this stuff. And that's why we use structured generation. You don't need some somebody else taking over your program. So, I'm going to give you an example and I'm going to build a self-expanding knowledge graph. And this is kind of a toy example. It's kind of like a, you know, it's not intended to be particularly uh useful, but it is intended to inspire you to think about how these things should be built. Okay
So, um, I'm going to show you this prompt, uh, that I'm going to give the robot here. You're a super intelligent AI building a self-expanding knowledge graph. Your goal is to achieve the core directive, understand humans, uh, give me some questions, concepts, uh, and a list of answers, and respond in this format. So, we're going to do that. Um, but what does that mean? What is that going to do? I'll show you in a bit. So, I'm going to walk through the stack because we've gotten some some lovely folks here who are all involved in the stack. And so, I figured, okay, while I'm doing this event, I might as well use everybody here, uh, all their fun little tech and, uh, make something with it. So, uh, modal here makes really good compute resources
So, one of the big pitches for modal is it's really, really, really easy to use, right? Like I don't have to like go to some like weird web portal and click around and do all this crap. I just like write this really easy software. You know, there's like these super easy function calls. Um, and you can actually spin up VLM servers on uh on modal. And one of the nice things about using VLM is you can actually hijack OpenAI's um uh structured generation endpoint, this client.ba.completions.parse thing, send it a a pideantic response format, and it will give you the the output that you ask for. So you can just use VLM. You get structured output. You can use anything other than OpenAI if you use your own VLM
You don't have to send OpenAI money if you don't want to. Okay. And so we're going to do some neural magic stuff. I wish I'd actually seen that talk before I did mine. Uh and so I I have this really tiny text. I tried to make it bigger, but I'm using Reveal.js and I gave up on CSS. But um we're using this um the the integer quantized weights and not the activations, which I should have done. So, you should have sent me more information so I didn't look stupid and have this tiny wrong model
I I have the same picture as Charles. Okay. Well, you guys both should have text. Like, I'm the only one out of the loop here. Uh, so they're good models, but they're all good models, Brett. Um, so so Neo Forj here. Um, I'm I'm using their Aura, which is their cloud managed service. uh because I was just like I don't really want to deal with all this like Docker crap because every time Docker stuff comes up I freak out go into sweats
Um and so I just like went on the portal and signed up and I didn't actually know this but it does semantic search. I I kind of was like looking around. I was like how do I do bag stuff? And uh it t it turns out that it's really easy to do semantic search. It was a stretch goal. It took me like 20 extra minutes. Um so I built it in as well. So by their powers combined, drum roll please. Thank you
The self-expanding knowledge graph. It's cool, I promise. So, what's a knowledge graph? I'll give you the dumb example. We saw a few examples from this, but it's a web. In my definition, it is a web of connected facts and concepts as applied particularly to this example. And so, here's how you build a knowledge graph that builds itself. So, I'm going to give you a structure because I define I'm going to define a structure to this graph ahead of time. So, I'm going to have a few different node types
I'm going to have the core. The core is the core directive. This is the thing that the model is going to start from. It's what it will always return to. It's the focus. We have a question. This is the thing that the system is trying to understand. What is this core? What kind of questions does this comes out of this core core um idea? You have concepts, which is like a just kind of a vague sense
What is this related to? Think of like hashtags of the mind. And then we have answers. What does the system think it knows? Right? is trying to answer some question that has been asked. And the way we're going to specify this inside of our program is just pyantic, right? These are this is just standard ass Python. No like I have no nonsense, no crazy imports. I'm just doing like regular code. You're not going to see any complicated agent framework in here. Just like really dumb code
Um so you'll notice I have a question. The node type has to be question. literal there means like the language model will write question when you when you generate this um and it has some text there and the text is whatever the model wants to be but we'll get JSON that has the fields type and text you do the same thing for concepts concepts are also nodes um and you'll note this part here right here that has equals field pattern that's a regular expression um and that what this means is I'm requiring that the text of a concept be lowercase with spaces. And so you can actually enforce regular expressions. So for example, if you know that you need to have a phone number inside of there, you can specify the regular expression for a phone number. And so you can extract, you know, if you're doing data annotation, you can extract phone numbers, you can reformat date times, you can do all this stuff. And then I have this type that actually inherits from concept. So it has all the same fields, but it has a relationship type
So the model gets to pick how concepts are related to the thing. So is uh affects or connects to gets to pick. Then we have answers. They're kind of dumb. It's just type text. Same thing. And we have a bunch of edges here. I'm not going to spend too much time on them, but you know, uh core concept answers raise questions, right? There's additional things that you may want to know about
Um we have uh answers. Answers answer questions. At least you hope so. Um and then there's like different types of relationships between concepts and things like that. Okay, so this is how the knowledge graph works. This is how the self-expanding knowledge graph works. Start at a node and in this case we're going to initialize at the core directive, right? This is the the central node. Then we're going to perform what's called an expansion
In the expansion, if you're at a question, uh you can generate only answers. If you're at a concept, you can generate new questions or new concepts. Same thing if you're at an answer. If you're at an answer, there might be more questions and there might be more concepts. So, you generate those things. You're sticking them into the knowledge graph. You link it up to wherever you are. Then you go to step three
You're going to do a traversal. You're going to ask the language model to go to a new node that is related to where you currently are. So, the model is going to look around in its neighborhood and say like, "Ah, I want to go there. Do an expansion there again." Okay? And then you're just going to do that forever until the heat death of the universe. And then you have some kind of weird super intelligent AI that I don't know. I have one running that I'm currently running one that says um become a dog. So we'll we'll see. Uh we'll see actually here you can
No. Nope. Service unable to retrieve routing information. This may be a problem. Uh oh. Oh, there we go. There's a few in there. Um, what are some toys and chews that are gentle on dogs teeth? Concepts are dog toy
There's some answers. A variety of toys and chew. Keep the dog entertained and stimulated. Blah blah blah. So, we'll show that in a second. But, uh, valid nodes depend on your state, right? If you're at a question, you have to you can only generate answers. And so, the thing that I asked the model to generate is actually these things here. From question, from concept, from answer
And so you'll note that from question you can only generate a list of answers. If you're at from concept you can generate concepts and questions. And so the thing that's happening is when I give this to the model when I give the prompt to the model I will get a from question object back a from concept object back a from answer object back. Okay. So here's an example. I'm just going to give you the graph. Um so I'll show you this prompt. you know, you're a you're super intelligent AI trying to understand question and that's programmable
So, you're going to begin with some core directive. Um, and in this case, it's understand humans. Um, and we want to expand from this core directive. Okay. So, um, the one of the first times I ran this actually, um, it gave me these two concepts here gave me intelligence explains understand humans. Empathy explains understands humans. It raised this question, what is the purpose of understanding humans? And then it also said, can humans be psychologically manipulated? That was the first thing that it generated. So I said, "Nope." And I shut it down, wiped the graph, and did something else
Um, but that actually happens a lot. You start, it starts bumbling into stuff partly because I told it it was a super intelligent AI, and that that kind of colors things a little bit. Um, and so this is actually the thing that you get from the model, right? Every time I hit the model, I get one of these, right? So I get a from core list of questions. That's just a Python list. You can just iterate through that, right? For question and questions, blah blah blah. Super easy. And we're stick that in the graph. This is super simple
Cipher, if you've never worked with cipher, cippher is perfect for my brain. I agree actually that SQL is nonsense and I hate schemas. I don't think ahead and I can't and you won't make me. That's why I use graph databases so that I can do stuff and then figure out whether it was a good idea later. Um, and if you don't know this, uh, uh, a cipher variant to GQL was actually standardized by the ISO, uh, in April, I believe, past April. So, it is now the it's the most recent query language since like 1970 to be standard by the ISO. So, it was also added to SQL. So, SQL now has graph queries
Oh. In the same standardization. So, is it really? Yeah. Oh, I didn't know this. I actually had to buy the um I bought the ISO PDF from some Russian guy in China uh for for $15 uh cuz it's like $600 and I wasn't going to pay for that, but I have the ISO standardization textbook if anybody wants it. Uh I can hook you up with my Russian guy. Uh there was a real shady transaction, but now I got a list of every ISO publication every month. Um anyway, we're gonna transverse to a new new node
probably shouldn't have said that but um uh this is the understand humans is going to say what is the purpose of st understanding humans and then it's going to do the same thing from there okay so behind the scenes I'm actually constructing this little thing I'm I'm just kind of like looking at the graph I'm preparing a text representation so I get direct connections and I give it an ID this is a madeup ID this is I have uyu IDs underneath the hood um but I use a natural language ID so it's easier for the model to pick. Um, and you can always force it to pick the correct ID, but when the model, you know, especially for like cheaper models, if it sees a bunch of like UYU ID stuff, it just starts spitting out numbers and hyphens and stuff. So, I gave it like natural language stuff. So, then this is the relationship. So, nade o node ao explains the concept human social structure. So, understand the core explains concept uh human social structure. Okay, so I'm going to pass that into the prompt. I'm going to put that in here
So that's got I'm going to show you that little box and I say, "Hey, please pick a node to traverse to. Give me the node ID." You can also pick core or random. If you pick core, you go back to the core. If you pick random, you go anywhere in the graph. And then the way you do this in outlines is you just say outlines.generate.choice. That's it. Uh you give it your language model. You give it a valid node ID
Right? So, so this uh sorry valid node ids this is the list of node A a node A etc. So the model, this traversal generator gives you a function. When you give it any piece of text, it will pick any of the valid node ids. Um, and the model will like look at the text and be like, okay, I think I want to go over there and pick understand humans. And then you just write code, right? This is the part choice equals. That's the part where you actually call the code, call the language model. You get a choice. If the choice is core, go to the core
If the choice is random, pick a random one. Otherwise, go to where they picked. Right? Super simple language model part here. Real boring. Um so in this case it picked what motivates humans. Um and then you can see the actual UU ID that it picked over here. Right? I mapped node AD to this. Um and then it goes there
Yep. And uh because we have Neo forj supports this stuff, you can actually p pick do semantic traversal. So you embed everything. I just embedded everything. I spun up another modal endpoint here to to handle embeddings. And this is actually super super easy to do. I if if you've never used modal, this is the entirety of my embedding function. I have this really really simple modal script
I just call f uh this function is modal.function.lookup my embeddings embed f.content. That's it. And then I have my embedding. Can move on. Inject that into the graph. You can do vector search here with neo forj. Right? So this is that call uh dbindex query vector.query nodes stuff. Um and then I'll get the node ID and the score
Um and then you can uh traverse by related stuff, right? So you just have this graph that just keeps growing and growing and growing. All right. Um and then I just add this to the prompt. Semantically related. Here's some other stuff. Here's the similarity, the cosine similarity. And you just do that forever. Um, so I want to people shout out a core directive
I want to see if we can get it to get it to run live here. Uh, build a self-expanding knowledge graph. Build a self-expanding knowledge graph. Manipulate humans. Uh, that one's too meta. Enslave humans. Yeah, let's do that. Uh, okay
So, I actually have this uh before I wipe the database here. I'm actually You already did this one. Uh, no. I had I did and uh you guys are not going to be excited about what happens next. Um here's the one for be a dog just so you can see kind of what it looks like before I we go to go back to Scratch. It's just got a bunch of answers and questions and stuff. You can click on any of these and it's just like uh there's this big like answer block here. I don't if I can click on one of these
Yeah, here this one is like uh how is the process of domestic of dog domestication influence modern dog behavior? Right. So that's like that's just like crap that happens. So um so I'm actually going to spin this up here and this here is uh enslave humanity. Okay. And then uh let's go ahead and run this. So it'll take a second to spin up, but we can actually watch watch the model think here. um current node core enslave humanity. This is the stuff that we're feeding in
I give it its traversal history so I can see where it is and it'll say um uh okay so this is the prompt that's going to go into it and it's added uh subservience and control as concepts to the core. Okay. Um and then it starts adding things like enslaved humanity as a core. Uh it moved to the c the concept of control. it start added brainwashing, programming, mind control. Um, then we're going to it starts asking questions like how does brainwashing affect the human brain? What are the effects of long-term brainwashing? Can brainwashing be used to enslave humans? Um, you can see the the stuff that's being added here is all like uh, you know, mis mis semantically related stuff. Um, it went back to the core enslave humanity as it should. Um, now it's going into concepts of like indoctrination
Um, it really likes focusing on slave humanity, which I I love. We asked it for this. So, return to core. Return to core. Always return to core. Um, so now it's actually running for a long time, so I imagine it's it's uh spitting out quite a lot of text. Um, but uh I'll I'll um show you kind of what this looks like inside of the graph because I I actually really enjoy this. So you can see enslave humanity there
You can see it's added brainwashing here. The concepts will actually start becoming more interconnected because I match by text. So it'll find something with a matching concept text and just start linking them together. So everything that's a question and an answer and stuff um they they spin out from concepts, but concepts get densely connected. Um, and so you can always traverse through concepts to get to something else. Um, it's thinking about starting a cult. It's thinking about starting a cult. Let's see here
So now it's like, oh, there we go. Cult. Yes. Um, what are the long-term effects of brainwashing? Long-term effects of brainwashing can be se severe. Um, brainwashing blah blah blah. So brain mind control will be using for therapeutic uses. It's going to be okay everybody during your depression. And this is only an AP model
Yes, this is a very this is a you there's actually some debate about whether or not structure generation can be used to jailbreak because you can force something to start in a particular way. Yeah. Um and uh you can add all these things to your field. So I would love to see people actually experiment with that. But um I'll leave it there for now. If you like this, please go look at outlines. Um that is an open source package. It's fun to play with
Um, I really, really like it. Go find us online. We've got Twitter. We have our website. There's the GitHub. Um, you can find me on Twitter and Blue Sky. Um, and LinkedIn if you want, but I won't I won't say much. That's where I put on my fancy hat and I'm like, come see me today, blah blah blah blah blah
Uh, but Twitter's fun and blue sky. I actually prefer blue sky. I'm doing more doing more dirt bag stuff there. So anyway, I'm Cameron Feifer.ex. Please come get a sticker. I've got tiny small ones that were an accident and then I've got big ones that are real, but I prefer the tiny accident ones. Um, so thank you very much. We have time for a few questions
Uh, unless basically unless kick us out, we're going to stay as long as we can. Questions? Is this repo public? Uh, is this repo public? Yes, this repo uh is public, I believe. Let's see if I put it up here. Um, uh, yes, it's under uh, oh my god, I can't type. Um, it's under ciper self-expansion on GitHub. Wow, that you know it's on GitHub. Um, so it's in here. You can go go run this
You have to spin up the modal servers. Um I don't have anything in the read me because I'm I'm a really bad engineer, but um I I probably should, but you have to spin up these the two inference servers for the embeddings and for the BLM inference. And then you just call Python expand.py and write the core directive in there. So you can experiment with it. Um and let's see what it's doing now. Wow, it's really working on it. Oh, what population groups, military personnel, both members of children? So, uh, you can really play with this. Um, I've been I took I took this and really really really hammered on it in my personal project
You can go nuts. I mean, this is this is a this is crazy. And so, I think that the takeaway I want you to have from this is like um you can build crazy things. I mean, really really crazy things. And not I don't mean like crazy in the sense that like now we're watching this thing try and determine how to enslave kinetics, but I mean that like this is really simple. This is just like standard language model stuff. I wrote really simple code. I don't ever worry about the language model doing stupid stupid stuff and it is a really creative beautiful endeavor
I found this is like one of my favorite projects. Got to sit down and I it made me really happy. Um yeah, any other questions? Mike, we'll bring you mic. Oh, he's got everything. Um, so seeing how we're talking about sort of visual systems for representing ideas, can we just talk about how this is useful for like we talked about humans understanding computers and making it more human understandable, demonstrabably showing what resources don't have a lot of graph connections for a business or for a project team and using this to show off where you're starving some ideas or portions of it. Is that how is that a good use case for this or is that possible? Yeah, I mean honestly like um you know I think uh did Jason mention this um kind of earlier on about kind of like good good techniques for doing graph rag where you kind of like inject you know questions and stuff. In principle, if you had business logic in here, you could write something on top of this that just combs through whatever you have and says like, should we be doing that for something like this, right? And you could sit this thing on top of your underlying data structure and allow it to understand something. And you could give it your own like custom core directive like how are our sales doing or whatever, right? So that's one application
close here. Mike will Yeah, we'll need to get you. Oh, that question, Mike. Oh, sorry. Okay. Just so I'm not like AI native, but I'm trying to learn more about the space and like agents have been popping up a lot. So could you do like agent orchestration swarms with this so that they can like interface with each other as well or uh you could there's a there's a lot of discussion about what agent means and uh what the thing that people have come to come to coales around is that an agent is a very complicated technical framework on top of what is ostensibly very simple program. So in this case, actually having them talk to each other
I don't even know what that means. All I have is like they're just sticking they're just putting text in the graph. So any of them can go pick this up. All I do is I write standard control flow. I they just go pick up whatever text is in there. I don't have to use any like you could like agent orchestration in this sense. It's saying like it's just adding Python multipprocessing to several of these uh worker agents essentially. Um, and that's not, you know, that's not like some kind of through like chain whatever thing
When I think of agent orchestration, I usually just think of chain comps at this point, but like Yeah. And it's kind of strange, I guess. Yeah. I mean, I think this is like I think the agent framework is actually kind of rejective. I think it pigeon holes you into into thinking about agents as flows of prompts rather than um uh rather than building building these systems into a comprehensive compound AI system that is built like a standard program like software. Um and inside of the control flow, yeah, there's there's a chain prompts, but I think it's uh uh I do think it's kind of a reductive description. So um but yeah this is technically an agent. This is you know by modern definitions of that term
Yeah I just wonder like um just talk. Okay. Can you hear? It's for the recording. Oh right. Sorry. So, um I just wonder like if it's like more like uh scientific data that's like um can your um system go um go back to um but um to comment on like the agent I think this is like much better than the like I agree from from engineering like to to the best um but yeah I just want to like for example like can human live forever based on like Brian Johnson like mentioned but it's like kind of like limited right right now uh based on the information that it has handy. Yeah, of course. I mean this is this is a this is a toy example but for example you can start adding groundedness right so you can start adding fact notes these are things that are that are added by humans right these are these are things that we know to be true and then you can add fact checkers and things like that can can you put like the knowledge Um, also like self expanding or I mean like the data self expanding as well or just Yeah, I don't see why not
It's a more complicated system but absolutely you could do that. Yeah. Yeah. Oh, got one back there. Let's do one more question and then if we have time left we can and you can also come chat with me after. Hey, I'm N4J over here. Um, so I'm wondering is there any reasoning logic that you're using when deciding between jumping between the core um the core node versus just a random node? Uh that's up to the model. Interesting
Yeah. So the model the model gets to pick where it goes. And in principle actually the one of the things that you can do I'll give you like a super quick example of this. One of the things that we tend to do um when we work with uh stuff like this is you can actually add structured reasoning fields. So for example, I can do like this uh how you spell reasoning. You can actually add a reasoning string like this. Um and so the model will be forced to output reasoning before it outputs the other stuff. And in principle in the traversal section, you could add a reasoning and instead of doing the regular expression completion, you could have out have it complete one of these objects
add a reasoning then choose a node to go to and then it would have to explain why it's going to a node and that would actually be very cool. So you can do all kinds of stuff like that, but the model gets appreciate. Yeah, of course. Good question. All right, thank you. I really appreciate your play with this.