RAG + Apache Iceberg™: Structured Retrieval Made Easy | Josh Reini & Vino Duraisamy AI By the Bay25
All right, getting started. Thank you so much for introducing. I am Vinatami, a developer advocate at Snowflake. I've been an AI and data engineer in the past, but currently a developer advocate. And Josh, >> yeah, I'm I'm Josh Rainey. I'm a developer advocate as well at Snowflake. I spent uh a career in in data science and then spent a lot of time doing eval and now get to get to work at Snowflake >> and for today's talk it's all about donuts. So if you do not like donuts or if you don't like ordering donuts you're probably sitting at the wrong talk >> or maybe this that's the only reason you came
Well, all right. So, when you are building a doughnut ordering assistant, right, what do you think are the capabilities you like what questions would you want to ask for that assistant? >> Or or maybe like what's your favorite doughnut order? Crowd participation. Anyone? What would you want help with? Oldfashioned. How many? >> How many? >> Yeah. Okay, cool. Sounds good. Anybody else? Do you want to try this one? >> Yeah. >> Yeah
All right, >> I'd like an old-fashioned donut. Um, so as I ask this request, the first thing the agent is doing is it's determining the intent of what I asked. It says, "I'd love to help me get an oldfashioned donut." And maybe when we want to place the order, I guess we don't have a location in uh Oakland, so maybe we want to place the location or the order in Atlanta, let's say. Uh, so it's analyzing my my request. Um, it's going to search and find the right doughnut. Uh, check to make sure that this doughnut store has a old-fashioned doughnut. Um, the thinking seems to indicate it that it doesn't, but let's see. Let's see how that goes
Uh, so it looks like it didn't have an oldfashioned um, but the thinking indicates it has some like similar donuts. Uh, still still working on the request. So, it's suggesting a few different donuts that we might in instead. Do you have a a favorite of those three? Cinnamon twist, maple bar, or raspberry filled? >> Cinnamon twist. >> Cinnamon twist. Cool. Uh, so it it knows I' I've chosen the cinnamon twist and it's just going to help me place my order. Uh, we're not ordering real donuts
Sorry, I won't charge you later. Don't worry. Nope. Cool. So, in addition to kind of just ordering donuts, it can also help me decide what donut I want. Um, so maybe I want to know like what donuts are most popular. Um, and to do this, um, it it has access to data about, uh, what doughnuts are being ordered at different locations. Um, so it can query that data and figure out, uh, what's most popular based on, uh, what customers are really ordering
And it re recommends uh the strawberry frosted frosted as the the best seller. Um, so maybe we go with that donut. Uh, let's take six of this one in Chicago this time. Um, and I think you get the flow. So, we can both order donuts and help uh decide which doughnut we want based on real data. Um, and now I can pass to Veno about how how this uh demo all works. >> Well, true. Yep
If you thought, "Guys, you just built a chatbot that could help me order donuts." Really, that's not exciting. Well, that's not all, right? The data set that we're working with here is in the form of iceberg tables in an open format, right? So, again, like donut was the hook, y'all. >> Sorry, who doesn't love donuts? >> Fit into the [laughter] >> right. We just love donuts. Great. but also like sort of like take a step back extend it to a broader architecture of how do I work with the data that is sitting in an open format in any of the you know data sources and how do I build an agent on top of that that is capable of answering questions on that open data and again it's not just about building an agent that can answer questions right it's really in the enterprise setting an agent that can be trusted so you could publish it to your data consumers like if you're a data person or an AI person building these apps for your employees internally or externally. How do I make this agent trustworthy? And how do I make sure this agent is able to learn over time? Like how do you improve this agent over time? That's the focus of today's demo. And as you can see here for this demo, we have the data coming in from iceberg tables open data format in an AWS S3 bucket
And then I'm going to create a catalog linked database in Snowflake to essentially connect to an open data that's available anywhere. And Josh also has a couple of, you know, doughnut menu reviews and other data sitting in Snowflake. And he's going to really build this agent that is capable of understanding structured data and structured data through I guess cortex analyst that snowflake offers and cortex search which is you can think of it a hybrid search including vector search at the end of the day to give you that agent that is capable of answering questions across all of these data sources wherever you have your data. And as you can see right like I was talking about in an enterprise your data is really everywhere. Some proprietary formats some you know transactional databases and some in open formats in a cloud storage your data could literally be everywhere and you were still able to work with all of your data. The best part is that you do not have to move any data from everything into Snowflake. You could work off of that using catalog linked databases. Essentially a link to your data external to Snowflake and you could work off of it
So what do you do is essentially build this unified data foundation that the AI expert can really work off of or make use of. And how do you do that? And as you can see here, Snowflake offers Snowflake is an AI and data platform that gives you everything you need. But the focus of today's demo is really the data and cloud storage in open format and how do we really leverage that to build our AI apps on top of it. And that one specific feature that allows you to connect to any data outside of Snowflake to work with is this catalog linked database which is a birectional connection from Snowflake to your open data. It could be in any open format specifically if you have iceberg tables. That's the focus for today's demo though. And that's pretty much it. But then you have all of the data
You build a unified foundation using that catalog link database. You can't just dump this data over to your AI counterpart and be like now build with it. But you need to add some sort of semantic layer on top of it. Add the you know business definitions, KPIs, metrics and whatnot along with really like the data dictionary of the data that you have, right? What are the tables available? What are the facts and dimension columns? And really the agent will have this additional business context from this semantic view to be able to answer your question. Can you do an I mean can you write an agent without like a detailed or right semantic view? Probably. But would it be trustworthy to be able to answer all the questions right with high accuracy for user? Probably not. So this semantic view is pro is your way of really packaging the data in a meaningful business context added to it before any AI app can use or make sense of it. Now in the demo let's see how we really packaged or built this unified data foundation right as you can see this is the donor data that I had in my AWS SDA buckets they were in the form of CSVs but for this you know purpose of the demo I wanted to have a bunch of iceberg tables so just so it's in the open format which is available and can be queryable from any other open query engines as well and you can see iceberg table data metadata donut orders
Great. And these tables are for me managed by an AWS glue catalog which is one of the rest cataloges that iceberg supports. And this look great. So this is the location of the data and this is the glue catalog. And what I told you earlier was that from snowflake I will be able to connect to this data in S3 without having to move it in anywhere. So what I can do is to connect it is a three-step process. one create an external volume essentially point to the specific S3 bucket where my data is great and the second step is create a catalog integration because in addition to the data because iceberg tables also have the metadata right available how do I access the metadata about these tables that is in an open format I want to connect to the glue catalog through a catalog integration so now snowflake is pointing to the underlying raw data and the glue catalog So it has all the information that it needs at the end of the day. So in the end when I create a catalog linked database this is just another type of database which is linked meaning the data underneath for that database lives in the S3 bucket in the form of iceberg tables not snowflake native data format not proprietary to snowflake right so you could work off of any data external to snowflake using this one and if I were to go find this so this is if you can see the doughnut iceberg data is the catalog linked database that I created
And then under that you see the donut veno DB which is the same as the glue catalog which is essentially this specific glue catalog is now replicated not in a real sense but like a point in time I mean a pointer to this glue catalog is now available through this donut pen DB schema. And then now you have your iceberg tables which were in open format in S3 available for you for further analysis in snowflake tables. Now with that said we have only made the data already and how do we think about the semantic views remember because it's not just enough to pass the data off to the AI team but we also need to add additional business semantic context and then I had created this semantic view and it's essentially like what are the tables available and then what are the you know time dimensions packs and other dimension columns and what are the metrics I might potentially you know run on these specific tables and whatnot. And you could even add relationships between these two tables because if you know your end user might ask some questions that would involve joining a couple of tables or you could even for example add verified queries like semantic views in snowflake allows you to add verified queries. What is a verified query? Like if I'm a data analyst we keep asking getting questions from our customers like hey can you quickly pull this data for me and when you know what are the type of questions that you constantly get you could put those bucket of those queries into verified queries. So the agent would have at its disposal. These are some of the questions that I could expect from an end user when it happens. With all of that said, what we have done so far is essentially built the AI component of it, right? I mean the data component of it
We made the data AI ready, but now Josh is going to take it over and tell us how to actually build the AI agent. Not just build the agent, but how do we improve it? How do we make it trustworthy and more? >> Awesome. Thanks so much. We know. Uh so now our our data is AI ready um and we can start to actually uh build what the agent looks like. So there's kind of two big components that we want to figure out when we're building our agent. Um so one is the tools aspect. Um so uh what tools do we want to give our our agent to work with different types of data and to take different types of actions
Um so the first tool we want to add is our uh cortex analyst which is going to act over our structured data. This is a text to SQL uh tool. So we can ask questions in natural language, convert that to SQL using that semantic view that Veno showed Veno showed us how to build. Um and then execute that SQL so we can get uh data back into our agent. Um and that's going to help us answer questions about like which donuts are the most popular. Um we also have a a search service. Um, so this is going to be over our unstructured doughnut menu so that we can determine which are the right donuts uh in our menu that are available to be ordered. We saw that when we tried to order the oldfashioned doughnut earlier that oldfashioned donut was not in our menu and we were able to figure that out using the search service
We also have some custom tools available to our agent. Um, so the first is the take take donut order custom tool. So this is what allows us to look at the conversation history and determine all right so we've uh mentioned a couple donuts like what is the actual uh doughut order going to be and then the last set of tools are based on the doughnut customer reviews um so here we have um some unstructured uh reviews coming from our customers about uh you know I really like the strawberry glazed or the raspberry fil doughnut is uh terrible or uh whatever what have And we have a SQL view that sits on top of that data uh that's uh constantly summarizing. Uh so we get a a refresh summary um both overall and in the last 30 days in this like recent uh tool about what customers are actually saying about our donuts. So now um we can ask uh additional questions. Um, so here here's another example of a question I tried. I asked uh what the most popular doughnut was in the morning. Um, and it told us that uh people are ordering strawberry frosted
Um, and then Boston cream is another favorite. And as I'm looking at the thinking, um, it doesn't really seem like it's able to kind of uh correctly filter in the morning. Um, but it's kind of hard to tell, right? I'm just like looking at the answer. Um so one way I can uh kind of dive deeper into what the agent is actually doing is looking by looking at the traces. Um so here I have pulled up a trace of exactly what the agent is doing underneath the hood. Uh what planning is it doing? What tools are being called? And you see it's taking uh two different cordex analyst calls. It's doing some SQL execution and then it's uh generating an LM response at the end. I can see the inputs and outputs of all of those different steps
Um, but importantly on the left, I can also use an LM judge evaluation. Uh, so rather than manually looking at these traces and trying to figure out uh what's going wrong myself, I can let the LM judge tell me. Um, so the LM judge determined there was an issue here with logical consistency. So we asked a question where uh we were looking to determine which doughnuts were most popular in the morning. Um but when we look at the explanation here um it says the agent acknowledges internally that it can't filter to morning hours uh but it doesn't actually communicate this limitation back to the customer. Um so this is a clear clear uh gap um and an issue that with the agent that we've identified. So now that we've identified uh the issue let's turn and see if we can improve our agent. Um so to improve the agent I'm going to use uh cortex code which is a uh CLI tool spec specifically optimized uh for working with data and working with snowflake
Um and I've just pointed it to the the uh donut so or the donut agent so far. Um and if we think back to the the failure mode that I've identified, we identified an issue where the donut or the the agent can't answer questions about specific time ranges. Um, and if we think back to the Venos's portion of the demo, um, we can understand that there's probably an issue with the semantic view, right? The the Cortex analyst tool doesn't properly understand the data and the features of the data to be able to do that sort of filter. Um, so let's see if we can optimize the semantic view. And I've I've added two particular skills to this uh CLI um so that it can help me optimize both my agent orchestration and the the semantic view. So let's improve the semantic view so that my agent can answer questions about what's popular in the morning, evening, etc. Um, so what Cortex code is going to do, it's loading in uh the configuration of my donut uh agent from uh Snowflake. It sees that it's it uses this underlying uh donut semantic view and it's uh learning about it describing what the semantic view looks like and getting the definition of the semantic view
I'll zoom in once or twice so you can see better. Uh it's learning about the data. It's quering the data so it can actually understand what the structure of the data uh looks like. It's learned that it has a time stamp column that's going to be able to uh actually use this this time of day data. So now that it's learned enough about the data, now it's going to start to modify the semantic view. So we can actually add that time of day information back into the view so that we can actually answer that new sort of question. Looks seems like it made some syntax errors the first time. So it's going to going to uh try again
Uh and to do that it has access to the snowflake docs uh so it can actually understand uh exactly how this should be set up and what the syntax should be. Uh so it it came up with a new insight now um about actually how the syntax uh should be configured. So it's trying again. So, we're we're still working on on figuring out how to add that time dimension. >> Making a lot silly and stick, but >> it it does seem like that. Yeah, >> absolutely. >> What UI are you destroying? Uh, >> so this is just a terminal. Um, so Yeah, just regular terminal and then uh I just launch it with cortex code very similar to cloud code but like optimized for data and and how to use stuff like um and then I've added two like sets of skills
Um so these are kind of like folders of markdowns that just like tell it how to do particular things. Um so on the agent optimization uh flow it knows uh kind of the procedure should be like create a test set a test set uh try the test set figure out what the failure modes are um and then make some changes in the orchestration prompts to improve against that test set and then semantic views optimization. Similarly it knows kind of what the procedure should be uh to make those changes and optimize the semantic view. So it looks like it has now updated the semantic view with the timebased dimension. Um and it's just going to test the agent and verify that it works. >> Sorry. >> Uh no. So this is all in Cortex code uh which is just in in my terminal
Um so it seems like it's doing well. It's doing a few more tests and then once it gets done, uh, we can actually go back to the the agent like UI that we were looking before and just like make sure it works. Uh, so it seems like it it worked well with the the morning popularity question. It's just testing it with an evening question u because that's what I I asked for in my original task. Yeah. So, while while this is uh finishing, I'm going to just uh pop back over uh and just kind of describe uh how we can think about our evaluations. Um so uh at the end of the day uh what what we found is that agents work really well uh when their like goal uh plan and actions are aligned. Um and that's because when you think about how the agent operates, it first needs to discover the user intent of what the uh user is actually trying to do
Um it's going to articulate a plan um on which tools it needs to call and when. Um and then it's going to actually make those tool calls. um and it might need to replan along the way. Um so if we can start to define some metrics uh so that we can evaluate how well that goal plan and action are aligned um that can help us diagnose failures. Um so one evaluation we used in particular that we saw a second ago was that logical consistency metric. So this is looking at the alignment between all three, the goal, the plan and the action. Determine how well all of these are aligned. Are there gaps where the system instruction says one thing and then the agent make takes a different action? And that's exactly the sort of example we saw
So, uh the the user asked a question, it wanted to know uh what donuts were most popular in the morning. Um, and then our logical consistency metric was able to tell that the agent made a uh a logical error when it uh didn't communicate properly to the user that it didn't actually have the ability to to filter down onto morning uh data. Um, and uh we we've have a a paper that I can send you the link uh to as well. But what we found is that using these LM judges um to measure these metrics, we can get to really close to uh human detection accuracy um by looking at those agent traces and determining both when the agent makes a mistake and exactly what step it made the mistake on. Um so this is really useful if you have ever tried to like manually annotate agent traces. It takes a lot of time. Um, so if we can use LM judges to speed up that task, um, it's very helpful and can actually iterate improve our iteration speed as we're improving agents. Uh, so let's go back to Cordex code and see uh, if it finished
And it says yes, it can help uh, discover uh, what times of day uh, or like what doughnuts are most popular at different times of day. So let's try it in our agent again. So let's start a new thread and ask the same question. What donuts are most popular in the morning? So now the agent is thinking, it's planning, it's figuring out which tools it needs to call. It's executing SQL and now we were able to filter into the boarding. Um, so now we're super happy the agent now has this new capability. Um, I think with that I'd love to move into questions. If you have any questions on kind of the whole end to end story or like how anything's set up, we'd love to take it
>> Will you >> uh yeah, I think can we share the slides? >> Can you slides? >> Oh, uh yes. >> Well, yeah. Question in the front. >> Uh okay. Great presentation. I didn't know Snowflake had a g a cortex code or a cursor code like product. So I think that's what one of the interfaces that we hadn't seen before that you were showing. >> Yeah
So just confirming that >> um >> is that agent GPA uh that ven diagram you were showing is that a consistent way to judge that's what it it looked like you had or snowflake had established a metric uh called agent GPA which is a combination of those seven to 10 different evaluations. Is that what we were seeing or can you teach me how to understand that? >> Yeah. Yeah, definitely. Um, so I guess to answer the first question, so the the Cortex code CLI is new. We just announced it at Build. So it's in preview now and like will be more widely available uh soon. Um and then on the GPA question, um so this is kind of a series of different metrics that are looking at the alignment of like these three different uh like steps of the that the agent is taking the goal, the plan and the action. Um and the way you could think about this is that each are like detailed criteria on what the trace should look like if it is logically logically consistent or if the actions are adhering to the plan and things like that
Um so we won't like compute a composite metric but instead allow you to have the individual metrics because that's what tell you tells you actually like what went wrong like >> how did >> Yeah. Exactly. >> Or acting. >> Exactly. >> Yeah. So, okay, good. This is helpful. It's a good follow on to our previous conversation about >> uh agent evaluation
So, which of the three buckets is the part of the agent that we saw and maybe it was where we saw it was in Cortex code where it was just making bad. It wasn't thinking very intelligently about where to put the as in a or whether a semantic view has a comment or not. You could see the agent struggling. >> Mhm. >> So is is that a goal metric, a planning metric or an acting metric or am I am I thinking about it the wrong way or >> Yeah, there are two different agents here, right? One is the donate body weight assistant agent. That's the agent we were trying to in for the logical uncertainty part. But then the cortex code is another AI powered assistant that helped us improve the semantic view. So cortex code sort of lives and operates separately and that's also an AI agent under the hood
That's >> it has a low GPA. [laughter] >> We we definitely could >> it's in private preview which still [laughter] means it's in beta. We just wanted to you know show you and see what's coming. So it's fair that it is >> I think your your question is still important right like uh the cortex code is also an agent right? So like what is it doing wrong? Um and when we saw it made make those kind of like silly syntax errors I would put that in like a uh this like five bucket. So in between like plan and action uh issues with tool calling issues with like adhering to the plan. Um, so I think like tool calling in particular is where I would expect to see like a low score for Col. >> Awesome. Yeah, thank thanks for the question
>> We have time for one more. >> Yeah. Awesome. Um, it might not be direct to what you were presenting on, but one of the problems I trying to get my head around is a lot of the queries that end up happening like you've enabled access to sales data or to inventory or or geo data. Uh, and then some questions though will become like frequently asked questions and I don't know how to categorize that kind of response like should I have you shouldn't have to do that whole thing over again for an almost identical question that you you just looked up but I haven't figured out with all these multi- aent operations how to automate the process of creating like a frequently asked questions response. Yeah, this so this is a great case for like verified queries and veno you can say more for sure. Um, and so we actually have a new like Texas SQL model called Arctic Texas SQL R1.5. And one of the cool things that I really like that this model is able to do is it can determine if it's looking at uh one of those verified queries, it won't it'll like skip the thinking phase and just like uh generate the SQL from that verified query
Um do you want to like say anything more about kind of how you like maintain and create those verified queries? >> Yeah, firstly I think it's part of the semantic model that he talked about, right? So you as a data person know what type of queries that keep coming up from my consumers and then you can preemptively like add a query repository at the end of the day. So the agent instead of thinking and going on the whole loop would actually like a cache at the end of the day, right? like you just look at the query repository and see if there's anything that I could directly answer by just running a SQL instead of having to do this whole thinking reasoning and acting on that. And second is this is probably particular to Snowflake. So we also have something called a semantic view autopilot which is people at Snowflake are running all sorts of queries against the data that they already you know are using. So when we do create these verified queries, how can we make it AI powered? Right? Instead of me, a data person having to write these verified queries and provide it to my agent, the agent can itself go look at the query history against a specific database and see what are the most frequently asked queries and then sort of use that as a query repository for itself to sort of be smart about you know how to add for the frequently asked questions. >> Yeah, good. Thank you. Yeah, thanks for the question
Think out of time. Awesome. Well, thank you guys so much. I really appreciate it. [applause]