The Missing Protocol: How MCP Bridges LLMs and Data Streams | Viktor Gamov, AI By the Bay25
So for those of you who uh just came in, thank you for joining this this presentation. We're going to talk about MCP. I don't know how many talks about MCP uh you already heard today or you heard before. Uh but this one's going to be important. And on the very beginning uh I was thinking okay how I can start this like a cold cold conversation instead of like jumping into the slides immediately and I was like what about um going into Google trends and when I opened Google trends this morning do you know what I saw >> exactly yes like is cloud flare down like what seriously um no it's good thing is that otherwise the demo would not be kind of like working because of the the AI and the LMS and everything is available now through uh internet um remember u you probably don't see this because of the table but um this is somewhere in March 2025 it's not even last year so even last year if you were at this conference the no one was talking about MCP and now since the March 2025 like the things just exploded and people start talking about this um and I put the rag um retrieval augmented um generation thingy here so So it's kind of like a going slow because I will try to compare those two things but it's not don't expect like full-blown uh comparison. So MCP becoming a very important topic for conversation is more important the topic uh as a piece of technology. I hope we do have a here practitioners people building apps building agents anyone um I came to you from the world of streaming data and uh streaming agents and things all things Kafka is kind of like a um near and dear to my mind. So you should expect something how you can apply this MCP and in in Kafka world um miss protocol how MCP bridges LMS and data stream
So this is what we're going to be talking about and I'll try to make a statement about certain things um open for conversation. There's nothing settled in the set in stone. Like I said, things are still developing like if you were following the development of this like agent coding and different LLM evolution for the last couple months, it it become an exhausting because you have to do like over and over and a lot of things changes and if you miss like one month it's just like whole uh year in normal human world like okay there's technology just jumped like very quickly. My name is Victor Gamov. work at a company called Confluence. I am uh the principal the developer advocate there. Wrote a boot about Kafka Java Champion. So expect some of the Java and JVMs conversations
Any Scala people left in AI by the Bay conferences. Okay. Just >> I know they trying to they're trying to you know make the as painful for me as possible. Uh I'm not a big fan of Scala [laughter] as you notice any Python people here who runs the Python in production are you still okay okay okay for like LLMs and agents and stuff I don't see same people only few people who who runs this everyone runs agents in Java in production let's agree about this of course >> so um another important slide for this presentation uh this video will be recorded slides will be available all things uh that I do or we do together with my colleagues also can be find on this website also if you follow to aiass assistant.engineering uh you can find all the videos there as well. Now I'll give you three two one. Okay I will show the slide at the end one more time. Um so um before we start talking about actual uh nuts and bolts and uh the meat of this presentation so I will give you like small background where I'm standing so we can have a same like a common understanding what what we're going to be talking about. So the thing what we call intelligence these day is nothing more just API call uh by the end of the day uh how the way how we interact the model we interact with this model through request response through some sort of API and as you can see here there's a lot of things missing in this conversation when we start with the user interaction or user prompt is something that we asking from the model uh there's a lot of things missing um for example if you will ask give it first request will say hey my name is Victor uh and next request you will ask uh what's my name? It will not remember because it is rest
It is a stateless and LLM doesn't or at least they tell us that LLM uh don't store any information. So um the coming up with the idea of having the memory something that would be constantly updating our context of our conversation with uh this LLM is actually helped to establish um this kind of like um a little bit more intelligent conversation. So it will remember something that you said or something that uh you enter before but it's still not enough because the uh the context window will become kind of less and less concern over kind of like last couple months because of the models evolved and they have a better um ways to to handle this context window and the context window is also growing still not enough. Why it is not enough? It's not it doesn't know your your data. So we in a kind of like enterprise uh AI world where we not only doing like a chat GPT and asking like what's uh what's my psychotype or what's my color of my hair and what kind of like costume we're going to fit with this. We actually care about how this LLM would interact with the data that's you know the our business uh is relying on. So there's no there's no um there's no way how the LLM would know about that data except if you will provide dispersal ways. Uh and it cannot do anything despite of popular opinion
We still kind of like a far from the point where uh the lamb will do do something for you uh by itself. It's still just API. Um does anyone understand why it doesn't know your data? Because of this. Once again that's that's how our conversation happens. If we're not pushing the data there, they would not know. Um, and in this particular case, I will talking about models, not the applications say like it's not CH GPT, it's a GPT model or it's not a cloud desktop, it's going to be the Sonet 4.5. So, do we understand the difference right? So, it's a application that has all this rich functions and our model. Now, how we can uh how we can change the situation uh when we need to deal with the data
So um the the community came up with this pattern uh of uh augmenting our prompt with the additional context that can be retrieved from some place. So we do have our data performed in a special format. So it will be easy consumable by LLMs. Uh in this particular case it will be stored in um in a format of vectors. So take we take our data we turn it to embeddings and we store those embeddings because uh mathematically finding kind of like a relevant information or related information much easier if you will be applying some of the vector uh distance between those those data. So we're going to go with the the the searching this database for particular information that we care. We integrate this together with our question. So if I need to get information uh of um say I'm a airlines chatbot uh that requires getting some of the user information in order to respond
So saying hey what's my um what's my current uh the amount of miles I do have. So we need to get this information from the enterprise database put this into the search vector extract this information and that's how this information can be be part of the request and put this together in one uh one piece of context that we submit to LLM. So this is how LLM would know the the the context of what we're talking about and this is kind of like a way to have extended memory and the teaching um what kind of information would be sent to LLM. There's a couple things here. So um the first thing is that the process of bringing this data will require some time in order to create those embeddings. It's kind of like indexing process in in any other system. So you need to get your data. You need to pick up the proper model for uh for indexing this creating this embeddings store this in a database in vector database somewhere where you will be able to query this effectively uh or your application in this case
Um and after that we uh in inject this into the context. Now is it like old already? Like I said the since March of 2025 like is it old already? Yeah, maybe it is. But you know in if it works for their use case it's actually a good thing. Um you already figure out the infrastructure how to run those systems. You already figure out how to um store this. Here's the thing though. Usually those uh the the rag systems they usually kind of like ETL style and the batch oriented. So you need to um establish some sort of procedures to get the data from your um enterprise uh data stoages and perform embeddings and store this in in the vector store
And another thing is that we still cannot do anything. So we we we went a little bit one step further to um to get the smarter um a smarter application smarter agent and uh but still we cannot do any any type of reaction. Before again let's let's go into a little bit deeper about the agents what we're going to how we can find how we can find this AI generated look at this look at this range I haven't seen these type of tools in in the real life but hey you know AI knows better um so what makes an agent so first of all it's some sort of like a small app that does some some sort of logic so it needs to serve certain um certain things so its agent can be autonomous meaning that it will be running it background doing some of the things or the agent that will be reacting on certain uh certain things that happened in real world or reacting on um certain requests. Uh it calls uh calls LLMs uh its agent that will somehow integrate it with with LM to get extra intelligence and also take some sort of actions. um LLM by itself cannot do anything and agent can talk to LLM and describe what is a available for this LLM to invoke and describe uh what we call intent. So uh and LLM will make a decision that wants to call this tool or not and after that this agent will actually go ahead and call this tool. Now uh isn't just the micros service but we adding like LLM sprinkles. It's if you look into these u agent idea from perspective of deployment and uh how those units will be uh delivered
Yes it is it is microser and uh yeah the corporate asking you find a the similarities between or differences between uh the agent and microser. Um yeah they sent me. So what is cool uh what is tool calling? So essentially we have a user input uh that we integrate uh with some of the information that is available uh for agent to to invoke. Usually it in the frameworks it needs to be designated with certain annotations depends of your language of choice. Um and uh this information would be sent to LM and there's kind of like a loop or conversation between this application um between the agent and LM in order to figure out is actually tool call required and the decision so we express an intent uh LLM does decision and after that tool will be executed a response will be sent back to um to LLM. LLM will provide us final final response. So um so we have a data data is uh uh prepared in in rag format and now we have a tools um and the the people in entropic that were start thinking about this. Hm
What about uh this process of uh getting those tools and um the setting things the the connecting to external instance can be can be some sort of some somehow um the standardized instead of having like each individual agent like in enterprise world write integration for every system how about we going to find the ways how these agents will invoke something standard interrogate this getting capabilities and uh send the data back. So instead of AI agent would be integrated with these uh the the systems we actually go in with the um webc. So the problems with the rolling your own um tooling integration. So you need to write your own code every time you changing um the one edit capabilities. Uh there's no resource integrated. Uh there's a rag but you need to do a lot of things to do this and also discoverability of these tools is not is not available. Now now 10 minutes in we we're going into the subject of this presentation. So um entropic uh introduced this uh idea of model context protocol is the way how our agents our our applications our microservices will be uh talking to LLMs and providing capabilities for LM to interact with external world
So um it will not lm would not going to call your your database or web service. Your agent application will be running uh the calling this for you uh for for LLM and send the response to to LLM. So we will have two components in this in this game. We have a client of MCP that will be integrated in your agent integrated in your um uh the the desktop application if you'd like. Um and after that it will be uh talking to uh MCP server. MCP server will provide certain um APIs that will show uh what is available. So for example uh one MCP server will provide the the methods how to talk to database. So MCP server by itself would know how to talk to this database and to outside world it will just like show API um that this server uh will expose or same thing for for Kafka or any any other external system
There's even like integrations with APIs like like Figma where you can um just the the tell how this the Figma system can be integrated with a lamp through this MCP. Now with these um with these components this is how the interaction would look like. So this is your agent this is your microser this is something that you're writing that will include MCP client for your programming language strictly speaking you don't have to have like dedicated client but it's also discussion between API versus SDKs. So MCP uses this JSON RPC protocol which is HTTP based and it's well researched but for some reason someone decided to dig it up from the times before 2010 even and start implementing these things with with with I don't understand this uh but that is what it is. So there's much better protocols already exist on top of HTTP that already doing a lot of cool things but hey you know JSON PC for it is um so it can be microservices it can be cloud desktop it's one of your applications that you will be u talking to your LLM they also have a capabilities to have a built-in um MCP client and uh uh one application might have a client that will be talking to different servers depends on the depends on the context um between this it just request response and uh between clients and server just request response like like I said it's a JSON RPC uh there's nothing fancy there um request response also can be sent through server sent events and um the the if the client and server runs on the same machine they can use standard input output to exchange information um so the multiple clients can talk multiple different uh servers and so far and so on. So for example use case where you want to schedule something you have access to calendar and uh calendar will expose the information about calendars through MCP server and uh say my the geospatial search that will you know finding things based on the the text description will be also hidden by MCP server and the way how the the agentic application or the agentic loop would look like. So our user express an intent to have a something booked uh tomorrow. So this information would be passed to LLM to extract uh intent what this user wants to do with this
So uh based on this prompt we get information that tomorrow meaning that we need to understand what's the availability of this person um in the calendar and uh we'll call uh the APIs get information and also uh book a restaurant. So in this case it would make certain assumptions about location but the there might be a conversation between like what's what's the vicinity of this restaurant like where you want to be I want to be the closest or it's going to be a little bit drive and things like that. So this how this um the the gentic look would uh look like with uh introduction of MCP server. So before that uh before that you would have to talk directly to APIs. you will have to talk directly to uh the calendar provider API provider for uh Google maps API and this integration needs to be baked in your agent. Now in this case they would know how to talk to MCP server. Usually vendors who are uh providing this calendar API they also give you MCP server to to deal with. Um also the built-in features of the u MCP server is also discovery
So you can go and learn um what kind of tools available there's a things are very well uh described like you know that's uh that's a discretion of the creators of MCP server but in general people understand that it would be used as a lamp so description of the tools and the discovery would be uh very important say if we're going to look into um into this like request set up the coffee for me dom um and there's a there's some prompt system prompt you are helpful scheduling agent with access upon request geospatial API that can be defined meeting places and uh and you have access to complete calendar. So this information we send to LLM and LLM will uh respond us back with intention that okay yes looks like this tools available I want to use those tools and those are parameters that I'm interested now this will go and call this tool get the data send this back to LM make a decision by you know maybe calling another um um another MCP tool to do some some sort of action um and only thing that is required here is just URL so we're going to put the URL into into place. So we connect to the server and our a jointing microser will um deal with this. Now so the MCP also gives us few things. So the context management uh remember with the rag we need to uh bring the data into special format. So this u the the vector search capabilities will will get the right data from our databases. With MCP you have this out of the box uh and uh the part of this context uh management you can ship a lot of information in context. Um we can talk about how this can be negative like at the end if we have a uh we'll have a time to talk about this but essentially uh let's let's let's go to happy path right now
So uh we can call external tools we can call different systems and also it has a API discovery. So with this uh how this would look like. I do have this uh my um let's see if this will allow me to do say um if I can do this where's my uh what uh do you know about my my Kafka clusters like that's I think uh the good way to start a conversation with the clot and the way how this would know that uh it will extract information extract my intent from this initial request and um it will analyze it with saying that okay so let me think about this it's thinking right now it's still thinking still thinking and u now it's uh it's it's it has some some thought process so um user asking about kafka cluster and in my memory this is not exactly what I wanted to see I wanted to see how it will not use memory but rather use some information and however to actually get information about specific cluster I need to use confl tools. So how it would know uh in this case I do have my MCP server is configured and the way how uh this system would know my um discovery where is it sorry we're going to go with extensions no we're going to go connectors uh everyone uses the different um naming and these uh these are tools that available in this MCP server. So for example, it can get create topics in my Kafka cluster. It can get I'm more interested in list environments, list connectors, list clusters. So when I describe this intention of getting information about Kafka cluster, u it would know what kind of tools to call. Um so if I go back to my uh conversations, what we see here, it will figure out I have two clusters
One is AWS, one is GCP. And okay. So okay but um do they have any data because you know I could use this API and go and look inside my coffee cluster but I don't want I have intention to learn what is in my cluster so my MCP server pretty capable of providing this information to LLM. So LLM will uh format this response. So let's figure out that okay data because this LLM already knows that data would be stored um in my Kafka. So data will be stored in topics. So now it need to invoke list of topics and getting this information. So that's example of this uh the the cloud integration with MCP
So if you are in the Java side for example uh you're using the um uh using stuff like uh spring boot um in spring boot and spring AI there's a great integration to deal with this type of situation. So in this particular case u this is example of agent that will read the data uh that user submitted to my to my application and I will get the summary out of the Kafka topic. So this application doesn't know anything about Kafka except there is MCP tool available in configuration specifically um this so inside the springi specifying this is going to be a connection to my server. So what I can do in this case in my demo application that's my system prompt and in the system prompt I can actually write the rules that will generate uh stuff that I actually need in this particular case I don't interested I'm not interested in actual output of this but I actually interested in this uh tool will make some action specifically submit a message to particular topic in particular format so I want to have a JSON format so I express the intent in uh in a few words. So if I would run this, let's see uh where is it? Uh if I would run a save but run. So what it will just run this one more time. Um it figure out that I have a tool it get the information about the server get all the capabilities all the tools and send this information to LLM. Uh oh what is happening here? probably something that is not required
Uh maximum maximum token size. So wait, I need to wait like a few more seconds because looks like uh those sessions are competing and uh uh let me kill this guy. Um and wait a few more seconds because of the you know because of the APIs. Um uh I will quickly check if I do have uh slides left before we bounce off. So uh when not to use the uh MCP. So um it's a very good question and it's also always a trade-off. It's always the balance between u the what kind of information you want to expose. Usually if you interactive with external systems maybe it's a good idea to use MCP but internally if you have a capabilities to create this u rag system that might be easier um and the uh agents the writing agents would be much easier
As you can see, my agent Spring Spring AI agent um is let's see if it's still running. I'm basically using nature language to in order to um to um um I'm using natural language to describe the temp and write my logic of my agents. So yeah, so um the agents are cool and this is some again you writing some some microser. Um another thing is that probably um this is where we're going into how the agent will be kind of involved with the dealing with the legacy data. We probably will go to route where we have aentic communication. So that's why we have the protocols like A2A um and things like that. They they also can communicate through the streaming and uh let's see if this oh actually works fantastic. See perfectly timed demo
Now, as you can see here, it's it's a respond of this the agent, but let's see if it's actually delivered the message in inside the Kafka topic. So, I do have this LM summary. And let's see if it's a if it's a fresh one here. It says, let me see if I can quickly refresh and see if the data um actually came from my uh Yeah. So, I didn't submit any data for for this. uh there was only idling and system monitoring was submitting this activity. So it was able to read this data from from the topic and also produce data as you can see in the format of the JSON. So in this in this particular case this this agent was able to call LLM call Kafka cluster get information from Kafka cluster tell LLM how to get the summary of this data and after that use the tools to produce data back into into topic
So hopefully uh this this presentation will give you idea of how to deal with the MCP and how you can use this and how the like streaming data also can be integrated in this particular case. Um with this uh my name is Rick Gamov and as always have a nice day. I am available for enhanced interrogation. My flight not going to fly until 11 p.m. So I will be here and happy to answer any type of questions. And with this I'll see I'll see you around. [applause] I will answer questions while I will be switching um to uh to another speaker. I guess that's what >> Yeah
Thank you so much, Victor. Um for any questions, please raise your hand and I'll hand the mic over. >> Um so, um I'm just curious earlier before the talk, you made a comment that uh you watched >> No, no, no, no, no. You you you made a comment about using Java in production versus Python. Yeah. >> So I'm curious about more context about and whether it matters anymore because now we'll all be just like chatting with whatever program. >> Yeah. Correct
So what you want to look for your agendic framework is that um you know how um how well it's integrated with like different models uh what kind of like capabilities available there. So, Spring AI and lang chain 4j is like two leading frameworks that you know I was researching in the last couple month that um have all this capabilities easy building memory easy building integration with the tools. So um the Java still kind of runs strong because you still need to have integration with uh like CI/CDs the the the how you can deploy those agents different libraries for monitoring and all this kind of people running Java in production in in in many years. So I would I would fight everyone who say that that you know your agents are not required the same tenderlo and care uh than your microservices or whatever you're running u in a different language. So what I'm trying to say you know use Java for production because there's tools available and uh this like LLM integration is just a sprinkle of the frameworks and it just like a developer experience better versus you calling directly rest APIs and whatnot. Does this make sense? Yeah. >> How exactly does it work with Kafka and Kafka entities? So like consumer groups, topics like commit offsets like how does it work in the agentic workflow here? >> Yeah. So the MCP server handles this part
So the MCP server provides the tools for dealing with the consumer consuming messages. um you have a different you know you need to express different intent if you want to go a little bit deeper on the you know doing with consumers and committing offsets but in general in order to read the message from from Kafka topic it will use the same API but this API would be wrapped around MCP call so my my my client will call MCP server that will read data from Kafka this information would be included in the context and will be sent to LLM get the response uh we did the we did the summary after that calling another tool that will be publishing data back to copper. So everything is is wrapped around these tools that I showed in um in my cloud in my cloud desktop application. >> Correct. >> Yeah. But intent would be intent would be described by LLM. If LLM knows how to deal with this, it will be sending some like metadata wrapped around. So the the framework knows how to call um this particular thing
>> Yes. >> Yeah. So uh you mentioned that you're not suggesting using MCP if you want to access local resources. >> Did I stop recording? Okay. So let's talk about CSS yes. >> Yeah. So is there any particular downside of doing that or or it just not necessary? >> Yeah. Exactly
So here's the here's the problem of like when you start using the MCP and you start seeing that all the things are like awesome like you can use different uh different database you can talk to your database in natural language instead of talking SQL. Um you um you kind of feel very excited about this. But the problem is as soon as you starting adding more tools, you're actually polluting your context and LLM would be having like a very hard time to figure out what I actually need to call. So that's why kind of like a context management. If you use this like a very scoped for example, I use MCP very limited set of tools. When I using this for a coding, I'm using only like MCP servers that will do search of documentation. I don't rely on u MCP to do g stuff or for example even though there's a g g server mcp because I know this will uh eventually will pollute the the context window with all this information about the tools will be shipped together with your actual context. I don't want to do that
So you need to be careful and find a balance between where you want to use MCP and how you can get excited about this MCP. Um so that's to to the matter of the gentleman that was asked about frameworks. So this is where your your your frameworks your like uh the agentic uh client or agentic application framework will come into play like how you know what including request every time what including in certain conditions and things like that. I hope it makes sense. >> Yeah. >> Thank you. >> Yeah. Just uh >> this again
>> Oh yeah of course. Um yes go ahead. >> Yeah. So it's just back to um the CFKA part. So I'm not sure I've totally understood. So um it means the MCP server is uh subscribing to the >> so the Kafka topic and just >> you know someone so you can take the phone and say hey um can you uh give me information what we have in my Kafka topic? So for that this person that you're calling to needs to know and have all this kind of like a console tools or some of the API calls to go inside every topic and give you this information saying oh yeah I see this like there is a 10 messages in one topic and 20 messages another topic. So this is what exactly MCP server does. It provides information and expose the tools to interrogate the Kafka topic but your LLM actually responsible for formatting the response and giving the you know some of the some of the interesting information
So instead of going for every Kafka, every pulsar, every database, every rest service, you write integration your yourself, you just use MCP and your um your LLM and your agent system, they will be talking to your external systems through MCP by invoking tools. Does make any sense? One more time. Uh >> asynchronously >> uh no it's it's a it's it can be synchronously it can be uh synchronously so it's not necessarily about the style of where is it um yeah so instead of writing integration for every database by itself so you have a Kafka client new forj client mongu client you using MCP server that will just provide you interface and the the LLM and agentic tools will just connect