Context engineering in ever changing environment - made simple | Linghua Jin, AI By the Bay 25
Thanks everybody for coming. It's a lovely day. Um today I'll be talking about context engineering everchanging environments. I'm Lin Hua. I'm from Coco Index. Right. Let's sit back and uh begin with a story. So, Alex, our intern got a task to build AI agents to chat and plan over meeting nodes
First, she needs to index meeting notes so AI agent can query with structured information. The task is simple. You read the files. You split the files into meetings because you likes to stack these meetings into single files. like you have one-on-one with your boss, your manager and all that that and then you parse the meetings by AOM. You write it into into database and when you write it into database, we use the file name and we use the offset as key. That's it. Well, people love to have meetings and enterprise
You just have one meeting after another. So to keep the index database in the database up to date, Alex has to set up a chron job to run it every hour. As input keep growing, she quickly ran into problems. First problem is wasted computation on the same content. The corpus of enterprise is usually really large. If 1% of the content is added each time, the most of the computation is wasted. Because of this, the chrome job may not even finish in time. So we may need to add some cache to solve this
Suppose this is problem is solved. The problem two is document updates introduce ghost data. In this example, someone renamed the document for example from team A into team one doc, right? And the existing existing key that is in the database is related to the old old file name called team A. So after team A gets renamed, the old key cannot be referenced anymore. So all the keys related to key maybe become ghost. So you get ghosted the data then your AI agents reading the file systems you have all of these it gets confused. So for this there are like three typical solutions is you delete everything before inert then your production agent systems is having a downtime. The second you maintain two copies of index and you do this like frequent index swap but then you need to handle handshake with production AI agent systems right then the last one is okay let's do some file diff with last time and let's let's delete the ghost records so let's say Alice implemented the last option because it has the least handshake with the downstream then what if but the problem is not resolved because what if after meeting one someone decided to update it with some additional thoughts like meeting one becomes longer, meeting two, meeting three gets pushed down then your offset offset has changed right because all of the offset after meeting one is changed now meeting two and all the other meeting nodes have new offset now and then all of these previous one become ghosted the data again the thing about this is where Alice is doing incremental updates with this imperative programming against the database and it just gets more and more complicated with the incremental updates and it's very errorprone and even more to handle down the road in like pipelines you usually have like extra data sources you do filters you you do a lot of fan outs these days for example your meeting notes is long sometimes you need to chunk it and process each chunk etc the problem three is fixed interval for the batch job usually is too slow to catch up AI agent seems dumb because it cannot it doesn't know the latest content we have a TPM and she wants to chat with the agent immed immediately after the meeting for the next week's planning
It doesn't have the context. So, Alex now has to use like a some event driven task, subscribe to change streams and only process what's changed. It's just starting to get complicated, right? And now Alex pipeline job becomes more complex than this oneoff thing. You have to do manage dependency on cache, delegate ghost records clean up, event driven task for instance change clean up and all of that. So now like despite all of the data changes you have requirement changes and code changes and your AI agent is already deployed in the production. So let's say we ask Alice to add embedding columns for the output column output table for semantic search. So Alice added a few lines to compute the embedding columns, but she has to worry about the schema updates, right? Like alter index index table schema, add new columns, run back field job to reprocess all the files, compute new columns without rerunning LOM and without messing up the production table and now she just want to smash the keyboard and leave and we're done today. So but as a recap as you see like gluing components is simple but to deploy the agents in production system the bar is getting higher because the real real world is dynamic
The data is changing the code is changing and the bar is just become higher to do it. You have extra deployments. You have to do additional skill. You have to have additional skills to like to to know all of these complicated stuff and you have to do the long-term maintenance. Think about who is going to maintain this complicated thing. When Alex left after the internship, it's probably going to be you. So, um some headaches. So, let's it's time to take a break
So, let's just look at the original source code, right? It just have actually the business logic we need. You read the file, you split it in meetings, and you pass by and you output it into the into the database. So this reminds me like the serverside rendering. So let's step even further back and let's take a look at some comparisons. So at the beginning of the web applications people just use serverside render. So as it's like similar as batch job you have to request to run it run everything from scratch but when there's more and more interaction with the system like AI agent is also having more and more interactions with your file systems and all of that the latencies and and and cost becomes a concern right so people don't want to wait like I click a button I wait in front of the like the uh computer waiting for the screen to refresh right so people start to use jQuery which can increment mentally update the DOM without render which like what was Alex was doing there you have there and you have there but it's just it's a lot more complex when it comes to handling data and scale so then everyone loves React because React simplifies a lot of things because you only need to worry about declaring the state but the React handles like this virtual DOM how does it output to the to the like to how do the render right so it's it the framework itself was handling the updates for you. So we are proposing a persistent state-driven data pipeline for the AI agent world where you think about it is it's about like the external system you have external system which is like render instead of render to the web to the browser you render it to the external database you connect to internal source and that is your state then you declare how do I want the state to look like right and then the system continuously do this render for you it can be batched it can be streaming it doesn't matter but it will do the incrementally and choose the best strategy. So let's make Alex job h 100 times happier by this state um persistent state driven programming model
The state connects to the source the engine keeps the target and source in sync with latest transformation logic and then Alex only needs to focus on the business logic. Note that uh here like we change it a little bit from database observed because then is like the imperative part to this output DDB collect. So we have this collector in the system and it collects to the system. So that's how you declare the state will change and so basically Alex don't need to worry about if this I end if then I delete I only need to declare this is my source data this is my transformation that is what my output look like and she will be done with it. So introduce Coco index. Coco index is implementing a state persist persistent state-driven data transformation framework. It connects seamlessly to databases, APIs, web and so on and lets you process both structured and unstructured data. So user will write simple transformations in native Python and Coco index will run it as a resilient with RAS engine
Um and it syncs updates to external system with fresh data that is always ready to serve syncs as we're at when you update the model the external system will automatically be propagated. You don't have to worry about maintaining this complicated data pipeline anymore. Koko index has a smart incremental engine downstream that will continuously process this and reflects the changes for you. It has to be explainable. So because of the declarative state and data flow driven. So coco index provides this explan explainable components to it open ecosystem source and targets with standard interface. So users are not bounded to for the framework itself to implement native connector for you and it saves users infra team on planning all all of this ongoing maintenance work data refreshing persisting back field tolerance scaling up GPU coordination and all of that. So coco index can be used for like context for AI agents, search, ETL, feature engineering, notification, alerting system
Basically, how it connects to the world from state to the target. And we'll just do a quick example of like how does what does it look like to build with Coco index and let's quickly walk through example of building knowledge graph with Coco index and Neo4j. So let's use the meeting notes as example. you have this continuous up updating meeting nodes. What does it look like when we want to build a knowledge graph out of it? So first let's quickly go through the models. So we're using property graph. So in this property graph you have three type of nodes. Person, meeting, task
You have relationship. So person attended a meeting, a meeting can decide a task and and person can assign to this task. And we use Neo4j which is most widely deployed uh graph database. So when we look at look at how to implement this it matters there are two piece it's pretty simple actually first is structured extraction what are the structure we have from this document and second is how do we map it to neo forj so for structured extraction part we just define a python class we have a meeting we have list participant we have list of task you read the files you uh split the file into meetings you transform the file with lm then does structured extraction and and once after that you have like structured block of individual meetings and then you do two collectors. First you collect meeting notes. So meeting notes collect basically this is a meeting second meeting right and then you collect something called a relationship. So this relationship collects attendee and the meeting that's it. The next step is mapping
So with coco index you have to write cypher query. So what you do is you just do export. So you collect the meeting notes, you do export to Neo4j, then it generates the Neo 4G meeting notes for you and then you do the relationship export then Coco index automatically generate the notes and relationship for you. So that's that's it. Then you have the knowledge graph and finally this this is what does it look like. So you have different people attending different meeting, assigning different tasks. You can view who attended what meeting, what are the relationship in between and all of that. And there's like this um oh it's a little bit blurry but there's like this data flow component to it let you happily understand what does data look like step by step
So finally to summarize um it saves a lot of these kind of work for you uh where user we are basically proposing a new programming model where you user only needs to declare the transformation but the uh system handles all the like reduce computation handle update delete CDC and all of that and also in addition to that you don't have to write cypher queries u when you're building for knowledge graph uh so I hope I made it in time for the next speaker uh thank Thank you so much for uh your patience and support. [applause] >> Thank you, Lua. >> Thank you so much. Do we have the next speaker? >> Yes. Um we're waiting for the next speakers. >> Okay. >> They haven't made it yet here. >> Thank you
>> Maybe we can take a question or two until the next speakers come. Just you know if you if you're not >> Yeah, sure. Sure. Sure. If anybody has a question. >> Hi. Uh so for the incremental computation is like very useful like what but what can be done if let's say we we we change like one very small part of the document let's say but you still want to generate the embedding for the document as a whole like how will incremental computation can it even be done for in in that kind of use cases okay >> yes it can so if you have a small compon uh component in the document that changes other components are cached so only the updated piece will be changed I think that's I don't see other hands up but I'll be around so you can uh grab me and we can always uh chat about exchanging insights. >> Thank you
>> Thank you so much. [applause]