Graphs as the Backbone of Scalable Agentic Apps | Laura Funderburk, AI By the Bay 2025
Recording: Graphs as the Backbone of Scalable Agentic Apps | Laura Funderburk, AI By the Bay 2025
So thank you everyone uh for introducing yourself and for letting me know what interests you about uh graphs. This is very much uh a data architecture talk for sure. And uh what I want to talk to you about is some of the things that I've been working on for the past uh past year. So I'm a developer advocate and community lead at a makerpace and our focus is providing educational content uh around agentic applications. For the past two years I've also been working on writing a book focused on haststack which is the graph-based framework that I'm going to be exploring today. And one of the things that I noticed so that there's two things that I noticed um you can get the repo in the QR code. So, you know, in 2023, Langchain exploded. It became a super popular framework for building LLM applications
And then very quickly, I think people started to realize just how heavy it was to actually maintain this. The main idea behind lang lang chain was this idea of the LCL runnable and the chain, which is typically something that is quite difficult to to debug and observe. And on the other end of the spectrum, you had folks like Haststack from Deepset based in Germany. They back uh they they they were started or they started their framework before the LLM boom happened in 2023. And back in the day, their focus was on fine-tuning the BERT and Robera models. They fine-tuned the Robera the BERT model so that it could uh answer questions in German. and uh they released the farm package which was focused on being able to build these Q&A systems around a language model. GPT exploded in 2023
Everybody lost their mind. So they had to adapt to keep up with the field. So they they they revamped their package and they released their 2.0 version. And their 2.0 version is essentially a directed graph. They used to be directed a cyclical graph based framework. uh they added the the loops and uh for the past year I've just been working with these two frameworks and just seeing what it is that I can build. So my my background is in data science and mathematics. So I'm I I love structure
I love data. I love finding good ways to process data efficiently. So I liked langu I liked haststack for that. The reason I like langraph in specific is it's a different graph-based framework but its focus is entirely on the agent runtime itself. it making choices, it making uh calls to a tool, it deciding what node to call next. So, I'm going to be talking about these two types of graphs and how I find a really cool way to combine them both. So, as we all know, um you know, back in the days of classic data science, we used to be the data consumer. Uh we would typically spend a lot of time and I think we still do building these uh complex data pipelines so that we could gain insights from our data
We would take the raw data from I don't know SQL databases, PDFs, HTML files, CSV files, JSON files, IoT devices, collect those, set up a data infrastructure so that we could gain insights and use those insights to make decisions. Nowadays, we're seeing the emergence of a new data consumer, the agent. Now, we want those insights in natural language, but the agent tends to be much more picky about the data that it gets. Um, you know, the promise of agents is that they're revolution revolutionary. you know, we can update, we can automate uh a lot of our tasks, but the reality is that it comes with a lot of pain. You know, these uh these systems tend to be brittle, inefficient, and flexible and oftentimes stuck in a silo. So, what I want to talk to you about today is uh how to incorporate a hybrid graph architecture to get uh the best of both worlds. Well, Langraph is very strong in its ability to control um uh state, and I I'll show you what I mean by state in in a minute
uh it's not very good at the tool layer and I kind of want to make a distinction between these two levels of agentic development. We have the tool layer which is focused entirely on developing good highquality debugable scalable deployable tools and then you have the orchestration layer which is the the agent brain that is using those tools to actually make decisions. And so we can sort of think of um the the two kind kind of graphs. You know we have haststack as a factory assembly line in charge of extracting the incense from that data and processing them. And then you have the state machine graphs which is like a project manager basically has knowledge of the tools that are available and then decides which tool to use based on the task. So the central pattern I want to explore is uh using graph-based orchestrator like langraph focused entirely on the agent workflow and then uh we're going to be focusing on using these graph-based pipelines using haststack. Now the key pattern when we're using two graph base frameworks is who's calling who, right? Which graph is calling what graph. And uh the key um pattern that I'm going to explore is not necessarily calling a graph as a data structure, but a graph as a deployable micros service that another graph can call and use
So the idea of the first type of graph is the processor. So this graph is in charge of taking a bunch of raw documents and chunking them, embedding them, getting them ready for a vector DB. The key pattern that I am explor exploring is this idea of interoperability where you know this this graph right here. Uh I think the key innovation that they explored was this idea of deploying the pipeline itself as an endpoint. You know in the classic days of data science we would deploy the model as an endpoint and then we would use the model to make inferences. Nowadays the tool itself is the service. And so what we can do with this is we can we can build these um uh these pipelines through the use of their components and the the the font is very small but essentially what they do is they give us these uh components as uh specialized units that you can connect into a pipeline. Each of these components have a super strict input and output parameters
And so for instance, this specific uh component is going to take us input uh a query which is a string and it's going to return a list of documents. So these input output contracts make it extremely uh easy to know what is going into what component at every time. Um then once we have built our component we have a series of existing components that we can use for things like rack. Uh we can essentially break the entire rack process into two key pipelines. The indexing pipeline which consists of data extraction, data cleaning, data chunking, data embedding, data storage. That's one pipeline, one micros service. Then you have a second pipeline for retrieval which consists of embedding your query, retrieving, augmenting, returning a final answer. This is a separate micros service
So you define these components. You can either use pre-existing ones or define your custom one using the input output contract. Plug them into a pipeline. Once the pipeline has been uh evaluated, you can then serialize it as a YAML object. Um the serialized uh YAML object can then be exposed as a rest endpoint with uh Hooks which is another framework they offer. Uh you have the option to either leave it alone as a rest endpoint or package it as an MCP tool and then uh you have the option to to scale it with something like Kubernetes. So this essentially means you know this entire thing about uh agentic workflows. I think what's really tricky is there's so many different moving pieces
You have the tools themselves which I think uh I think they they deserve much more time than what we give them and then you have the agents themselves which aren't really supposed to be doing much of the heavy lifting. They're just supposed to be choosing the right tool, executing it at the right time and then providing that answer. So I think part of the reason we're having so many troubles with agentic systems is replacing uh the the the bigger burden of data processing to the agent itself. The agent doesn't know how to do that properly. it knows how to use a tool, how to answer a question. So what I'm proposing is separating the tool layer as its own uh its its own universe where you you you focused entirely on developing highquality tools that you can deploy and scale and then give those nice scalable tools to an agent to use. Now the the second type of graph is the orchestrator. So the orchestrator uh its only purpose is to hold state and decide which node or tool to call next
Um the orchestrator doesn't need to know anything about how the pipeline was built, what components it has. It doesn't need to know that. It just needs to know it's a tool for X job and it's going to be executed at Y time. Um the key pattern here is observability. So each uh we can define uh I think one of the things that's really nice and also not so nice about langraph is you really do have the freedom to define whichever agentic architecture you want and as an example so the use case I'm going to show you so this is a uh Yelp navigator so what it does is uh it's connected to a bunch of different uh haststack pipelines in the back end and uh I just ask a question I set up two different architectures. So, Langraph lets me do crazy stuff like Oh, I wish I could Let's see if I No, it doesn't let me do that. So, Langraph lets me uh create different types of architectures. Yeah, I'll just go back to V B V2 where essentially I can decide which flow I want
And what's going on in this specific case is um um I'm setting up a clarify note which is essentially going to just take a user's uh question and clarify the intent. Once it has enough information, it delegates to another another node which is a supervisor. The supervisor has no access to tools. It just has access to nodes and is its main purpose is to decide which node to call. And then we have these uh three specialized nodes. Each of those has a pipeline in the back end as a tool. So for this Yelp navigator, I have a search tool that it's only getting um information about the business, a details tool which can get information like the website. So I have a crawler in the back end as of my pipeline and then a sentiment tool which is focused on extracting business sentiments and summarizing
So this idea of the the second graph as a state is much more fluid than the first graph that I showed you that this second graph really can incorporate loops. We can have uh input guard rails, output guard rails, we can have um iterations in the middle uh such as human in the loop or llm as a judge. So this type of graph is much more fluid and much more flexible than the first graph. And I think that's a good thing. So the first the first type of graph the tool needs to be rigid. So because we want good data, we want to feed the model or we want to feed the system high quality data. The second graph can be a little bit more flexible because it enable us to um define more complex patterns that a rigid pipeline would not necessarily allow us to. Um so the key the key elements of this second type of uh graph are are going to be the state
So we typically define the state as a data class uh which inherits from uh langraph's message state and it essentially keeps track of what's going on within the process. So for instance in this specific Yelp um navigator I have my search query which consists of a string a search location in a detail level. So for this specific case um my clarification node whenever it receives a human question it should detect what is the human asking where is the human asking it from and then what kind of detail are it looking for something general detailed sentiment etc. So all of this is to say is this state is is not uh fixed. This is something we can define depending on the use case. Um it can also include things like the the output of the pipeline and then some agent output. So we kind of have the freedom to define our state but its main goal is to track explicitly what's going on within the agentic workflow at every time. Uh we have the tools
So the workflow that I'm exploring is I I design um a pipeline through my graph-based approach. I deploy it through I I serialize it and then I deploy it as a rest endpoint and then this REST endpoint can be called as a tool via the tool decorator. So by this point all I'm doing in this in this architecture is I'm just making um API requests to my pipelines in the back end. And so the tool is just going to be the interface that allows an LLM to call my tools. uh we have the notes which are the uh workers of the graph. These are just functions that accept the current state as input and then perform a task using the tools. So the nodes are always sort of keeping track of the state and we are the architects of that state. And then we have the edges which essentially define how those nodes are going to interact with one another
[snorts] Um so in this specific case if I go and ask a question I don't know uh best coffee in Bay Area. So when I go and ask a question, it goes to the clarify node, then goes to the supervisor. It determines to trigger the search tool. The tri uh the search tool is being triggered. And what's happening in the back end is I have my haststack pipeline. My server is running in the back end. So I can see that it was successful at identifying the tool and the query has been executed. And then once the process finishes, it returns uh the top coffee recommendations
So it's just Espresso Station. It gives me the rating, the categories, the website, Westlos Bay Area coffee, Bay Area Cafe, and Bro Bay Area. I don't know if these are good. These are the ones that it recommended. Um but one of the things that you were able to see immediately is this kind of decoupling uh architecture gave me observability on the agent layer, but it also gave me observability on the back end. And I was able to see that my pipelines were executing executing correctly. I was able to see how many results were found etc. And if I want to go change something in my pipeline, I just have to go to my components, change the logic of my component, reerialize, rebuild, and then the service is ready
Okay, so um I just showed you what the Yelp navigator did and it looked quite sleek. I just asked a question, I got my recommendations. Now to show you just how complex the graphs can get. So this is the the Yelp navigator orchestrator that I just showed you. So we have our supervisor and then uh three tools that are using pipelines in the back end. A summary tool which doesn't necessarily use a pipeline. It just receives outputs and then provides a summary and then a general chat in case the user wants to ask something other than Yelp reviews. So these pipelines can get um quite complex
Uh so really as complex as we want just as an example. Uh so if I go to building natural language pipelines and I go to so this is a repository book I go to chapter 4. Chapter 4 contains uh indexing uh and hybrid. So if I take a look at an indexing pipeline. So this specific pipeline is uh routing uh content from different kinds of uh files. So I can I can route content from HTML, PDF, CSV, etc. I can then uh process uh those documents. So for instance, if I'm getting if I'm getting sources from two different kinds of document types, I can process them
I can then join them and then I can I can um perform ranking. So this this specific indexing pipeline is Yeah. So just to show you just how complex it can get. Um you can really customize these um as much as you can. So the three that I'm using here are just a query converter. So this query converter contains a named entity um recognition extractor which sees a natural language sentence, pulls the entities such as location, people etc. It then feeds those um uh located entities into the keyword extractor which can then trigger uh an API back end. So what's really nice about this is nowadays we sort of tend to think that LLMs are are replacing everything about NLP
You know you can use an LLM to detect locations. You can use an LLM to classify things but they are not necessarily good at that job. So uh this type of approach ensures that we can use tested models for entity recognition or classification embed them as part of a pipeline and then each of these pipelines can then be deployed as its own separate microser for another LLM to use. That other LM is just an orchestrator or a tool user. It doesn't have to be able to do all this. It just has to use the tool. Um so yeah so the the the path is fairly straightforward. We defined and build the pipelines for the components
We serialize them, we load them and then we initialize uh the Hook server. You have the option to just serve them as REST APIs or as MCP servers. And let me see how am I doing for time. Yeah. So just to show you as an example. So this first one uh has an endpoint exposed uh at business search. Uh it takes a input a query such as cheese shops in Madison. And then the key of this pipeline is to extract the entities from natural language queries and then search for businesses in Yelp
So before we we actually get to providing a query to an API service, we can process that natural language query and we can even um control how much complexity it handles at this level. So if if I were to receive multiple queries or multiple requests in one sentence, the pipeline logic would be able to handle uh those as a batch or as a stream whichever one we use. So the idea is we put a simple natural language query and then we return a JSON output with our results. Um and this is this is something that I I kind of want to talk about. So uh the way I designed the system um pipelines two and three which are for business details and sentiment analysis tend to receive pipeline output number one. And um this is a really nice opportunity to perform context engineering. Um so my first iteration was a little naive. I kind of just gave output one the whole thing to pipelines two and three which is like a huge blob that they may not necessarily need
But uh what the nice thing about it is is I can I can kind of focus on how much context I feed to the next pipeline at this level. I can sort of choose through the through the pipeline definition. You know I don't want pipeline 2 to get the whole blob. I just wanted to get the key information it needs to get it job done. [snorts] So instead of me performing context engineering at the agent level alone I can also perform it at the tool level as well. Okay. So uh some of the key things that I found with this specific architecture is you know um and oh yeah so scenario I don't know why I lost my uh there it is. So scenario
So for instance, if business search uh is slow or doesn't work out, you know, if if I was handling the entire thing via an LLM, my entire system would fall apart. If one node fails, you know, the entire system is not working. With this architecture, I kind of separate concerns. Each tool is its own separate microser that I optimize, that I test. If one of those microservices fail um instead of the entire system going down, I just have one point that I can go and debug and fix. Uh the nice thing about dealing with microservices this way is you know for instance a user gets a bad summary and this is something that happened to me quite a quite a bit while I was building this specific system is you know clarification node would would um would pass you know it the the the state in this clarification node that detail level reviews this belongs to the state graph the lang graphbased graph. So it got right that I wanted details. It searched the notes
It got the the details. Um and then you know maybe my my third pipeline wasn't fetching the results that I wanted. But the system didn't break. You know the summary said you know what I wasn't able to find information but here's another recommendation. So the system didn't break down. It just it just showed a failure note. I I can just go and fix that specific pipeline and then the system is isolated uh to one this specific tool. Now uh one of the things that I also found with this specific approach is you know the beautiful thing about working with strict decks such as haste is you're sort of forced to build good input output flows
you're forced to connect data input output sockets and you you this strictness is beautiful because it means you don't have to worry about um well you do but when when it comes to things not working you have a specific point to pinpoint with a more flexible langraph type based graph because you're you're the architect of how you design your nodes your agents your state you know there are no I think as of this point if you know any but as of now you kind of have the freedom to choose how you define your state. My first uh iteration uh of this application I built I built it such that each each node was both acting and deciding. So the very first architecture and I really wish I could show you but essentially I was I was letting uh each agent choose the next agent based on the information it had and um there was no uh no error. I mean the the system worked. it was returning responses like if I go to my v1 version right here and I really wish I could let's see all right so this is v1 so so lang graph and state lets me build whatever architecture I want no matter how bad it is this is a pretty bad architecture so what I'm doing here is I I start and I'm using the same tools in the back end that's the nice thing so I use the same tools but I can play with different architectures so in this specific case I go and ask for a clarification it always triggers a search note And then the search node is in charge of deciding whether I should go for the details tool, the sentimental tool or this summary tool. And then once this whole workflow ends, the supervisor approves at the end. And you know it if if I go and ask it a question, I'm going to go I'm going to go ask for the same thing. Um so best coffee in the Bay Area
So, it clarifies, it searches, um, it summarizes, and it kind of looks like it's doing the same thing. It's also giving me a bunch of recommendations. The supervisor approves, I get my final summary. It looks like it's the same. So, what what what is the difference between V1 and V2? So, uh, this this goes back to this idea of context engineering, you know. So we we we focused on this idea of robustness by defining strong tools in the back end. But uh the the downside of having all this freedom with langraph and defining my own state is I'm also responsible for managing my own context. And so uh if so one of the things that I found here and that I wanted to share is you know the the first the first version or the first architecture every tool uh was dumping it as JSON output into one big agents output dictionary
So this means I was bloating my context unnecessarily. Um the flaw is you know the supervisor was sort of at the very end. There was a bunch of context poured in between the notes and then making it to the very end. And I had a supervisor at the end who sort of timestamped and said yeah every every request that was asked it was it was clarified. So this is good. Now the flaw with this is you know every node was acting both as a as a tool executor and as a decision maker. So it uh each node had to have all information and so I have massive context bloat. Uh the second so if if you take a look at how I define this
So the monolithic state I have one single state object that specifies the messages the user intent the level of detail the workflow uh uh control and then supervisor approval. And then I had this um graph that had a bunch of conditional edges. You know if agent one says search go to search if it says sentiment go to sentiment. If it says detail, go to detail. Um, now V2, uh, I kind of learned to separate things a little bit. So agents are only responsible for executing a tool and then the supervisor is the one that decides which node to get executed. Uh some of the things that I found you know just cleaning things up a bit is having the state definition uh uh separate from the supervisor than for the executors and then just defining a a simpler workflow where I have a central supervisor making the decisions. Um now why why this was important? So I had it I had I had I ran a test on both architectures just to see how many tokens I'm using and I kind of found out that my my first version was consuming a bunch of tokens but particularly for the sentiment analysis and uh detailed information notes and sort of that second architecture where I have a central supervisor whose only job is to choose a node and each node has the is is only in charge of executing a tool uh chopped my my tokens by 50%
which if you're thinking about optimizing your applications is a big big deal if you're making queries thousands of queries uh per day. So uh just to summarize so the final uh graphbased blueprint that I'm proposing is you know you separate the agent uh workflow from the tool workflow. uh the focus is on defining discrete testable scalable tools first then provide those tools to the agent once they're ready to go. Um if you are going to use langraph to define your state one of the things that I found was very useful is get the supervisor to be in charge of only choosing the tool or the note that's going to be used to create separate states for each of the different workers. have a state for your supervisor, a state for your workers, and then um yeah, I I kind of found that uh if you're going to have agents executing tools, their only job should be to be execute a tool and provide a summary. The other thing that I kind of found was really interesting about context engineering is it happened not only at the agent layer, but also at the tool layer. So, how much context we're passing from one pipeline to another is also key to reducing or minimizing the amount of tokens you pass while maximizing the quality of the results. So um this is it for me
So this uh compresses pretty much the work that I've been doing for the past year both as an instructor at a maker space but also writing a book on this topic. So if you are interested in and sorry that it hopefully it comes back. Um yes so you can get the GitHub repository on the QR code. Uh so in chapters three and four I go very much in depth and five I go very much in depth on evaluating hybrid versus naive rack and how to systemize the process um of this. So thank you so much for your time. So I'll open it up for questions now. [applause] Do you have a Do you have a question? Just raise your hand if you have a question. >> Sure
>> Yes. Uh which one in specific? >> Which one in specific? >> Yes. So so I think Neo forj tends to be if I understand it correctly more focused on the knowledge graph type of graph where the focus is on entity entity creation and and and um uh semantic meaning. Um so the only other one that I worked with has been Ragus through their knowledge graph generators but typically tend to focus more on extracting um key insights or key uh key portions in the text and then creating notes and edges from those documents. So I think I I've mostly worked with RAS and I' I found it's a really nice way to evaluate uh RAG applications. But uh if you tell me a little bit more about if if I've worked with others. Yeah, I think I think those are the three. Yeah
Yeah. Okay, we can talk after. Yeah. All right. If there's no other questions, thank you so much for your time and for coming here. Oh, yes. Yes. Question
>> Yes. [applause and cheering]