Context Engineering: Connecting the Dots with Graphs | Stephen Chin , AI By the Bay 2025
Recording: Context Engineering: Connecting the Dots with Graphs | Stephen Chin , AI By the Bay 2025
All right. Thanks for the amazing introduction, Mary. Um, so she she actually does a better job introducing me than my own bio. So, thank you very much. Um, and very pleased to be here at AI by the Bay. So, amazing conference. Um, I'm going to be chatting a bit about connecting the dots with graphs. Um, given that Neo Forj, um, was actually named after the Matrix film
I think it's appropriate. We have a lot of Matrix jokes. So, we're going to have copious amounts of Matrix jokes. And, um, a big thank you for my co-presenter, the copy bar. Alexi, hold up the copy bar. I think he's the best part of this conference, right? >> No, that was Cody Bar. >> Cody Bar. Cody Bar
Sorry. Sorry. Cody Bar, the coding copy bar. Okay. So, I think that um this is how we've all felt the past couple years if you if you think about how our lives have transformed with AI. So, um you know, we're we're locked in our cubicles. We're we're slaving away trying to get AI models to do what we want them to do. And um I mean I think the the pideantic presentation was was amazing actually showing you know real principles of reliability and type safety and all these things but a lot of those have been kind of thrown to the wayside as we've become prompt engineers for the AI trying to craft more complicated prompts trying to get them to do the things we want and then struggling against things like hallucinations and getting the AI to be more reliable for the things we want to do
And where we want to evolve this as an industry is to move from being prompt engineers to being context engineers because now we have agents which actually require more context more information to do their jobs properly. Um they actually need proper typing proper information so that they can have structured responses from different data sources from databases from um vector embeddings from rag systems. And we need to be able to to kind of raise up the the level of information so that the actual interesting signals are heard above the noise. So the the AI models have now very large context windows especially for the the latest models. But they do a very poor job of pulling the right information out of the context unless it's moved up to the top. So a lot of context engineering is compressing information, getting more information that's relevant to the models, but then moving the things up to the top which actually are going to help us to to get the correct answers and achieve this. And um hopefully by the end of this presentation rather than being um prompt engineers, we're going to be information architects and superheroes like our like our friends in the Matrix film who who saved the postapocalyptic world. Okay
So, um let's talk about how we accomplish this. And um if you look at context engineering as a discipline, it encompasses a lot more than just prompts. So, context engineering is also about passing in the right context from different systems doing retrieval. So, um retrieve log to generation or rag is still very relevant for context engineering because we need to pull in those enterprise data sets and things which the model has no information about. We also need to know the state and the history of the um transactions and the conversation which is going on. So we can advise on previous answers or relevant conversations which have happened. And in general we need um access to memory both short and long-term memory so that we can provide additional information and working state for the agents which are collaborating and we can also give um context on the conversation so they can answer longer running queries and longer running conversations as part of the workflow. and then um also provide abilities for the um the agents to give structured outputs because now they're not independently running and just um chat bots
They are hopefully part of larger enterprise systems where you're integrating them with other agents with other workflows with other applications where they're providing um actual useful information and doing jobs which help us to build more complicated and um reliable applications. Um and speaking about memory, there's two different types of memory which we need to concern ourselves with. One is um short-term memory for the agents as they're working. Um so this is kind of the the current context for the agents. We want to do as much compression as possible on the information which we're handing the agents so we can get more information in the context window. um bubble up the relevant information into the um um into the cont at the top of the context and also integrate tool results. Um one of the challenges if you have too many tools which um have filled up a lot of the context window for the model then it can be quite difficult for the model to answer correctly. So aging out old tool results or um kind of reordering things will help the agent to do a better job of answering
And then we also need long-term memory so that we can track and we can understand um episodic um information kind of have a structural and semantic relationship for the long-term context and then start to take the conversations which the AI is coming up with and then have that be part of the workflow. So if it's making planning steps for how to attack a longunning problem, um making sure that it can actually follow the plans, follow the instructions and it knows what the relevant information is and that kind of rises above the noise from all the database results, the tool results and different things which are filling in the AI. Um, and hopefully we achieve a state where now we're we're kind of like Neo and we're plugged into the all of this information and matrix and we're actually getting the most relevant information for our model as early as possible so it can successfully accomplish its job. Okay, so why does why does context matter now? Why is this kind of a a big deal for us today? And things have changed quite a bit. Um, so we need to get the right information to the models. I think a lot of the failings of different projects is based upon not having the right context and not having the right information for the AI. Um Gartner predicted that 80% of general AI projects were going to fail by the end of this year. Um they're probably not far off because you can see that like a lot of projects stall at different points because they're not able to get to production
They're not able to hit the reliability metrics they're trying to get to. Um now we have technologies like DSPY and BAML where we can do dynamic prompting. We can do more structured types. So that also requires better context. We have reasoning um for internal context engineering. Um now we need to make sure that we're going from human in the loop to agents accomplishing things so they need the proper context so they can collaborate together. Um, it's always better to spend more time testing our models and making sure the system works um rather than just spending an inordinate amount of resources and um um energy [clears throat] at training more complicated models. I think as you've seen the successive model um growth has tapered off
So from you know like GPT 35 to 4 like there were huge advances. um the very latest models um just give about the same results with you know maybe 10% better on specs and benchmarks than the previous generation. So we really need um to build applications which are capable of using both the latest models and also a generation back for efficiency and performance reasons and um we can get to a point where we're able to explain the results better using technologies like knowledge graphs. So how many folks are familiar with knowledge graph technology in the audience? Okay, we we have a very astute audience here. Great job, Alexi. Um, and just for those who aren't that familiar with knowledge graphs, um, basically it's a it's a way of encoding data in a graph. So, it's if you were going to go to a whiteboard and you were going to draw a picture of what a problem looks like, you'd probably end up with something which looks a lot like a knowledge graph. Um, it's both good for humans and LLMs
So LLMs are very very good at reading, interpreting, querying knowledge graphs and I'll have some demos to showcase this as well as it's something which as humans we can read, we can interpret, we can reason about the the relationships, the structure inside of um knowledge graphs and it can be a digital twin of a whole bunch of systems of your organization for um people graphs of um supply chain systems for like more complicated business use cases um for financial systems. So you can basically model anything in the real world as a digital twin with a supply chain or or with a knowledge [clears throat] graph. And the the basis of a knowledge graph is you have nodes, you have relationships and you have properties. So in this example [clears throat] um so two people and you can see their their names on the properties Dan and an um live together although it's they know each other. Um, apparently Dan lives with Anne, so she probably owns the house. And then they both have a car and again, um, it seems like Anne is the one who owns the car and Dan drives it occasionally. So we can see who wears the pants in this household. Um, and then the type of car they have and then we have some embedding information as well
So a common technique when you're integrating knowledge graphs and AI is to attach the embeddings as um properties on top of the graph and then you can start to traverse and get vector results traverse to graphs and vice versa which is very helpful. So in this case the nodes um show you the entities the relationships are expressing interactions and then the properties give us a bunch of information which we can index for performance and um using knowledge graphs together with LMS kind of gives us the best of both worlds. So LMS are very good at language reasoning creativity like they're they're able to do things like um in the earlier pyantic example of actually playing word games against each other and guessing words which is fun. Um but knowledge graphs are really good at kind of that domain specific um knowledge organization um structure which helps us to solve more complicated business use cases. Um so if we want to evolve from just basic rag we talked about rag being a key tenant in context engineering to graph rag. Um some of the things which this gives us as benefits are now we get more relevant search results bubbling up to the top. Um the way vector embeddings work, they'll give you similarity results back, which can be very helpful. They're not as good at pulling back um multiple um angles of relationship
So whenever you're trying to answer a question which involves multiple parties or multiple people u maybe situations about temporal data over time, knowledge graphs do a much better job of finding the relationship between multiple entities as opposed to the um the vector similarity searchers in those cases. um context. So whenever you have factual specific information on the subject, knowledge graphs are great for those use cases. Um explainability, so the ability to provide detailed information on why the AI is reasoning in a certain way. Um vector embeddings and rag tend to be black boxes at least from our standpoint because we can't understand kind of the choice of the vector embeddings and um the word vectors which it uses. But then it does quite a good job of similarity results just without any visibility to us. With a knowledge graph, we actually have a um a portion of the knowledge graph we can analyze and understand why it answered in a certain way and we can start to do things like um role- based access. So like if we were doing a um example of a healthcare use case where we have a doctor and we have an administrator um the doctor should have information about the the patient diagnostics and like the conditions and the pre-existing conditions whereas um they shouldn't have access to your home address or your phone number
Whereas the the admins of the system or like someone you call for for help on your account should have access to that personal information but not the sensitive medical information about your diagnostics. And you can annotate that on the graph. You can um control what information is presented to the LM. And then the basic flow is the user asks a question you're doing a search against a knowledge graph. Then you pull back a portion of the knowledge graph, pass it in as context to the LM and then the LM uses that as grounding to enrich the answer and to give a better response to the user query. Um so this is great for storing learnings, for visualizing confirmations, conversations, for analyzing the context so that we can get um identify different opportunities for improving the flow of our application. [clears throat] And um just like in the Matrix movie, the the most important part of this is is training our information and actually getting our models um with a really good set of data. So, I'm going to do a short demo and we're going to see if we can't train a model um with an build a knowledge graph and then get it to answer some questions for us
Okay. So, um I think everyone can still hear me. Now, um for this demo, what I'm going to use is I'm going to use um Neo Forj Aura and it's this is just a free instance of our online cloud service. So any of you could sign up and and create a quick instance like this. You can see I have a an instance running and it has 135 nodes and 418 relationships. And what I what I did right before the presentation is I loaded up two documents, a a bomb and a a VEX document. And let me show you what those are quickly. So this this bomb document is a a bill of materials, supply chain bill of materials
Um so you can see it's a cyclone DX document. It has a bunch of check sums for the um security things and then um different libraries and the versions of them are referenced. So these would be our software dependencies for for a given project. And then I also have a VEX document. Um VEX is a security standard for um disclosing vulnerabilities. And again this has some vulnerabilities listed. It says the severity. It says information about them
And these are these are kind of linked together these two documents because the vulnerabilities relate to the dependencies. Um but they don't have to be like you could have additional um vulnerabilities which don't apply to your project. Okay. And then um I'm using the knowledge graph builder which is a um basically lets you load up local files. So I previously grabbed these two documents and loaded them up. Um it used the LM to build a knowledge graph and we can take a look at the knowledge graph as a preview. Um this is a open source tool so all the source code's available and you can do the same things with our integration with um um with lang chain llama index um and other open frameworks. Um we also have our own Python library which um does a lot of this stuff and you can see that okay so there's a bunch of nodes here which have information from the documents
Um there's also I think down here is some of the vex stuff. So, national vulnerability database is NVD. Um, come on. Got it. Okay. And here's the CV number um for the particular defect. So, you can kind of see some of these nodes actually represent the information of the document. And what we're going to do is we're going to ask the LM and what what it's going to do when we when we chat with it is it's going to first do a vector lookup in the um in the database for similarity
So it's going to try to find things which are similar and then it's going to navigate from those similar nodes to find related and connected nodes that it can pass in as context to the LM. But if we ask it about something it doesn't have knowledge about, what we'd like it to do is be grounded and not answer. So, um, let's ask it for what it knows about vulnerabilities in pyantic, which I don't think those are in the document. I didn't check, but probably probably it should tell us it doesn't it doesn't know anything. Um, okay. So, nothing in the documents. It asks us, is there anything else we can help with? Now, we did see Jackson, the Jackson library list in the document. So, we'll give that a try and see what vulnerability information and stuff it can pull out
And um basically, it's it's pretty quick, but it's doing, you know, first a vector lookup and then it's doing a um um a graph lookup. And you can see that it's pulled back the CV number. So, that was the node we clicked on earlier. Um it knows the specific version that's affected. So, version 210. Um, and then it talks a bit about the fact that it's in the um the Sneak NVD databases um Red Hat, BugZilla, and other places. So, pretty good. Like, it's given us some good analysis from the document
We're going to see if we can improve on this in a bit, but you can see that without a lot of effort, we've got a grounded result. This would be something if you were building an application on top of it. It's it's more authoritative. It's actually giving us um correct information um which we could use for a um corporate application. All right. So, there was demo number one. Now, um you know, I think we now have some tools where we can we can battle the agents. It's kind of funny that in the in the um Matrix movies, the agents were the bad guy
in in our world. Well, I don't know if you can characterize AI as good or bad, but hopefully they're, you know, they're they're doing things which are good and um solving problems and then we don't have to send agents like Neo to to destroy our our agents and systems um or at least, you know, kill them off when they when they run rampant. Um but the next topic I want to talk about is memory retrieval. So, um kind of how we can look at memory retrieval from a graph context. We we talked a bit about the importance of memory retrieval in context engineering. So it's very important that you're bubbling up the right information from the conversation to the um agents and this is a architectural diagram of how you can do memory retrieval in a graph context. So in this case we have a knowledge graph which has the information from the conversation um that's either being added or retrieved by the agents in the system. And um we're basically retrieving information about um memory short-term and long-term about conversation history
We're pulling back that into the agent loop and that adds to our contextual memory that we're passing to the application. The reason why graphs are a good fit for this um and there's a lot of companies which have built their memory systems on top of graph technology like um mem and and zep and others is because graph memory represents the relationships in forms uh or represents memory in terms of relationships. So it's kind of like um similar to to how we store and organize memories and then you can search nodes over time or different embedding properties to get the results. Um so this is a projection of the nodes on an embedding space. So we can now do vector similarity search to be able to look things up in vector space. So this is very helpful to get initial results for similarity. But then we can also use graph algorithms like the K approximate nearest neighbors or community grouping algorithms um or page rank algorithms. So there's um dozens of different graph algorithms which you can use data science algorithms to apply on top of your knowledge graph and then use this based on your domain or the information how it's stored to pull out the relevant information for a given context and then you can use this to enrich your results back coming from your LM
So it helps you to get kind of that additional temporal state and memory context coming back. Now um this this is kind of like superpowers for your LM, right? So, we're we're basically giving our our agents the ability to um to control time and motion and um doing amazing special effects like the bullet time effect. Um actually, one of the um conferences I used to run was um Java one and we actually built a physical bullet time rig in the expo hall using um Java and of course um software running on Raspberry Pi is where we did a time sync on all the cameras. So, that was a lot of fun. Um but let's go through a quick example of how you can um organize some of this information in knowledge graph. So let's say we were trying to figure out when the last time I presented this together with Sid who's one of my colleagues. Um so Sid's our community manager in India. Um you know I'm VP of DevOr in the US and actually the last time we were together at a conference was at the GDS conference in Bangalore
So now we've we've kind of pulled back people relationship. So we have two people involved in it. we pulled back an event. So a temporal event on when things happened and um let's add to this the memory of that time um in the past. So now we have kind of three different angles. We have a bunch of nodes in this. We have an event anchor and we also have a temporal state for this. And with a query like this, graphs are very good because they can represent those different states and now we can um do algorithms to pull out the relevant context based on this information
Um and there's a few basic patterns for how you do this with graphs. So the first one is um explicit retrieval queries. So we can have queries which use cipher the query language for graph databases. Um it's now a standard. It's an ISO standard as of last year. So GQL is the ISO standard for quering graph databases which is a subset of cipher. Um you can also do use a text to cipher approach. Um so I'm going to show a quick example of this of how you can use text to cipher and you can use an LLM to query results and this allows us to do some um more specific results and get out more specific information
And then we can do start to do aentic traversal as well to navigate and crawl the knowledge graph and get more specific information out of it. So let me show you all the second demo. Now for [clears throat] this one we're going to do something a little bit different. So um we're going to use the same knowledge graph which I set up previously and um we're going to use the basic cloud desktop user interface. And previously I um set up cloud desktop with the Neo Forj cippher MCP server. So if you were building your own application you would have your agents calling the Neo Forj MCP cipher server. But I think this is a quick example that any of you could do it at home. And I configured it with the same credentials as um the LM graph builder
So exactly the same information which we loaded up over here into the um knowledge graph builder. So this knowledge graph is what we're going to ask Claude about. Okay. So let's um go and open a new chat with Claude. All right. And say what do you know about the um Jackson vulnerability? Um just to make sure it actually looks at the graph database, I'm going to give it keywords. So in the um in the description of the um of the MCP server um I specifically added in graph and database as keywords so that it would know to look those up. Um and now we're doing a multipass um series of steps
So there is basically two main operations for the MCP cipher server. Um one is it can query the schema. So it can get back the schema of the database and then that gives it the structure to know how to navigate the relationships. The second what it can do is it can do cipher queries and it will um in plain text ask for information that gets converted to a cipher query and um then it can um navigate and pull back information from the database. And the difference between this and the previous demo we tried is now rather than it kind of blindly pulling back a subportion of the graph and trying to answer a question from it, it can get back information incrementally and reason about it and then ask additional questions. So you can see the flow here. So it started with the relationships then it went to vulnerabilities. It started to get document chucks to get complete the information
It got CV and severity statuses. It got all the information from the CV and CVSS databases. And then it gives us a a more complete response. So it gives us the vulnerability type, a full description, the attack vector, specific severities in different um vectors, fix recommendations. So basically you have to be at this version or higher to to have a um a fix for the particular Jackson vulnerability. And then some references. So you can see that this is a a much more complete response than we got just by doing the um the vector with graph augmentation approach and gives us a lot more information and context to work with. Okay, so hopefully I've shown you that you can do a lot with graph technology to be able to, you know, empower your agents, give them more precise information
Um if you want a lot more graphs and a lot more information how to build graphs yourself, a great resource is um graph academy. So this is free training which my team puts together. Um it has courses on graph rag agents with with Python with TypeScript. Um also courses on cipher knowledge graph fundamentals all those things that you need to understand how knowledge graphs work. So, um, basically this is this is our attempt to educate more people on graph technology in a in a basically a vendor neutral way. We want more people to use knowledge graphs and be successful with them. Um, also there's a lot of great content you're seeing here today at AI by the Bay, but you can get even more with um an online conference we're running in April, Nodes 2026. The call for papers for this is currently open
So any of the speakers in the audience, please [laughter] please submit. Even attendees, if you have a great use case or you're just interested to share what you're building with LMS, you don't have to be a professional speaker. So it's a it's an easy format. We do everything online, you're just chatting with the audience and sharing things you're built. We actually want more real world use cases rather than just um vendor talks. So um please join us for Nodes AI in April. And um last thing is you can get a lot more of this information. So if you want to if you want to kind of get to the end of the Matrix movie and beat the architect, we need a lot more knowledge
So um we also have this resource graph.com. It's a communityrun site all about graph rag to help people learn and be successful with it. And it has a lot of the current research. We basically look at the latest research papers coming out of industry and we do human summaries of them so you can explain how they work, what use cases are applicable and start to apply some of the latest research. So there's a lot of new techniques coming out for applying graphs to memory, applying graphs to retrieval. Um even kind of thinking about graphs as your agent brain for larger systems. So um you know help us build the next generation of agents which are more reliable and grounded in knowledge graphs. So thank you all for joining my talk and I'll be happy to answer questions outside afterwards
Thank you. [applause] >> Um we do have one minute uh for one question if anybody wants to ask here. >> Okay. >> Awesome. Thank you. Um I saw that you're doing arbback. um now that we're moving into more different types of Starbucks like peback and that are you guys looking into that or how would that change the knowledge graph? >> Um so I I think that like what we're doing as a as a um an organization is we're trying to make the knowledge graph layer which other people can build on like other startups can build the technology on. So a lot of the stuff which which I showed here are examples of what you can use knowledge graphs and apply them to
But we really collaborate with other startups who are are building memory systems on top of Neo forj who are building um um frameworks for doing um um retrieval and other things and we we're kind of the core which um allows people to build more complicated applications. >> All right. Thank you so much. Um, so now I'd like to invite you to go to the hallway or the interview room if you have more questions for Stephen and then uh we'll have like a 10-minute break in between and before Eric Meyer will be in this room. So, thank you. Thank you.