Agents Need More than Vector Search | Prashanth Rao, AI By the Bay 2025
Recording: Agents Need More than Vector Search | Prashanth Rao, AI By the Bay 2025
Hello everyone, my name is Prashant. Um, so I'll start with a quick introduction to myself. So I uh my name is Prashant. I am an AI engineer. Uh, till recently I used to work at a company called Kuzu, a graph database and I love working with embedded databases. I've been a longtime user of DuckDB and Lance DB and I've obviously used Kuzu and SQLite before this. To me, it all feels like these embedded databases are all from the same family and I'm an active open source advocate for various frameworks and tools. Uh, and many of those founders are actually at this conference
So, it's great to be here. I have a lot to cover. So, I'll breeze through the outline of what we'll be doing. Uh we'll start by describing what we mean when we say agents in 2025 and we'll discuss the challenges of big data in the age of AI and why Lance exists and followed by an introduction to Lance DB. We'll demonstrate an example scenario and illustrate the benefits of using Lance DB for multimodal data and modern AI workloads. And we'll also touch upon some ideas uh of how LanceDB can be used for experimentation when building agents including tool calling and other downstream tasks. So on the topic of agents, I'm sure a lot of us have seen or heard this one before. Uh 2025 is a year of agents
But I think a lot of you would also agree with me that um and even Andre Karpati here who said we are living in a decade of agents. There's a lot more ground to be covered and agents are only just getting started. To summarize what an agent is at this point in 2025, what do we need uh to build an agent? To build an agent or an agentic system, we need a capable language model that can perform tasks based on given prompts. And we also need to solve uh meaningful tasks with agents. So uh to do this we keep track of state that is what the input was at each intermediate stage and what was obtained as output. Allowing the models the ability to call tools is really important. So this is what allows agents to take action by executing the tools not just reason over the tokens. And all of these pieces require context management
the outcomes at each stage uh is subject to the quality of context that each model sees and then around this you have the scaffolding to define the control flow of the application. How do you orchestrate things? How much autonomy is given to the agents and so on. Of late the term harness has been used to describe uh the the scaffolding. The simplest form of a harness is a while loop that just tracks state where the you keep looping through the workflow until the system meets its criteria and stops. But there are obviously way more complex and sophisticated harnesses that exist in the agent frameworks that we use. But the bottom line is that the agent system lives at the intersection um of all these pieces and all of them rely on data sources uh at some point whether it's public APIs or your organization's proprietary data. So this brings us to the topic of data management and the importance of it for building agents. The knowledge layer in an application can consist of data in various formats like text, image, audio, other forms that you may come across
You may also have the concept of state which is typically stored as type dictionaries or JSON in a lot of these frameworks. The agent harness is plugged into observability tools and these keep things uh track of things as time series. And as for memory, um it's yet to converge on a standard format, but you typically see agent memory being stored as text logs, graphs or embeddings that an agent system can retrieve from. So the bottom line is that the data we deal with when working with agents uh and AI in general is more varied than ever before both in terms of formats and where they're stored. Let's also look at data from the perspective of query workloads. The most common requirement for agents is retrieval where you have a combination of things like vector search, full text search, graph reversal and so on. You may have um analytical workloads that you want to run in SQL to understand your data and run aggregations and analytics on it. Uh while experimenting with the AI workflow, you may want to serve the data to agents
Uh but to do this, the data needs to be augmented using things like feature engineering where you transform your data or extract features and information from the unstructured data underneath. The knowledge freshness aspect is really important. U you'd want some form of versioning or time travel in case things break and you want to go back in time to query a snapshot. And for state management, you may want to do key value fetching operations to obtain the right context based on the current state of an application. And uh of course the context richness depends on the kinds of data that you provide much of which is multimodal including vision um audio and so on. The richer the context you want to handle, the more multimodal the data is. And to improve the models that power your agent, you may want to provide a training examples and do things like fine-tuning, training, and prompt optimization. Uh, and each of these have their own versioning and checkpointing challenges
So the point here is that um in modern AI workloads just like the data itself uh the the variety is quite huge. Many of these are online workloads like search. Um others are batch workloads like analytics. But um all of these have to be made available um at runtime to these agents. So it's fair to say that the arrival of AI represents a generational shift at how we think about data infrastructure. And if you've come across the three V's of data, uh I believe it was Gartner that published this this terminology very long ago. But um the challenges we face in the age of AI are not new. They've been around but just in a different form today
The first V that is variety. Uh enterprise data is getting more multimodal and multifaceted in the kinds of workloads that we are seeing. Uh in terms of volume things like PDFs, images, audio, video, these are formats that are 3 to nine orders of magnitude larger in size than traditional tabular data. And thanks to AI, the velocity has gone through the roof as well. It's never been easier to generate vast amounts of image, video, and audio data in addition to text data at a rate of thousands of tokens per second. So this just adds to the volume problem that was mentioned before. So Lance DB was founded to address these challenges. We need to rethink the way data structure is built for the age of AI and now agents
Traditional data systems and compute engines as we know are optimized for tabular data, business analytics or business intelligence. But these systems are typically not fast at search. They are not great for managing embeddings like vector embeddings and they're not built to handle multimodal data. On the other hand, you also have vector databases that do address some of these issues and they're highly optimized for vector data and fast online uh retrieval, but they don't integrate well with data lakes where we have an ocean of data currently sitting. They also require data to be copied from the data lake. So people are want going to want to store their data in object stores no matter what we may want. and they're not great for tabular data and or multimodal data. So you have this bifurcation uh of the ecosystem that we live in right now
And this is where Lance comes in. So Lance is a file and a table format that's now gained significant adoption in open source by some very well-known companies which I'll list at the end. Uh and the goal here is for it to become the new standard for multimodal data. the columner file format that Lance has. Um it offers 100 to 10,000x faster random access performance than parquet uh without sacrificing the scan performance that you may expect from paret. Um it also comes with native support for binary data like large blobs and mixed width data which is very common uh in AI applications and it's also designed for very high performance on pabyte scale data especially multimodal data. Also lance is a table format and I'll talk a little bit bit more about these three benefits listed here uh because they're worth spending more time to understand. But um the third benefit uh that lance provides is that it's extensible disk based index uh with full text search of course vector index and several other secondary indexes and I'll talk a bit more about this as well
All indices in lance DB are persisted to disk or object stores uh which basically means lower memory usage um as your data grows in size. And of course all of this works well when you have very fast random access which is what the core strength of lance the format is. So with that bit of background we can summarize what lance DB is. So Lance DB is a multimodal lakehouse for AI. It's more than just a vector database. Lance DB open source is an embedded vector database. it uh it runs as a process in your application and it provides uh search uh capabilities search and retrieval um because it's embedded the database sits on files on your local machine then you have Lance DB cloud which is a typical SAS product um it's serverless it's managed uh and the deployment is designed to scale to massive amounts of data uh but search is still the main focus of this product and the newest uh entrant to This is Lance DB enterprise which is what we are calling the multimodal lakehouse and it's deployed as a distributed fleet and scales to pabytes of data. It's absolutely massive in terms of scale
Uh along with search there's also other um tasks that it helps with um exploratory data analysis feature engineering and model training all designed for scalability on multimodal data and uh the standard data formats that we may be used to and I'll highlight a couple of examples briefly. So all of these products are built on top of the open- source lance format and it's incredibly easy to get started with Lance DB right away. Uh you can just pip install it uh and import it as a library in Python, TypeScript or Rust. So the reason this is uh important is uh this is an example application that was demoed at race summit last week uh by the folks at Netflix who built a video search application on top of lance DB and what they're doing here is they're searching for um the query elephants in a jungle but then they swap out the query to say elephants in the desert and the the great thing about this is it's pabytes of data that they're searching through the these are video files amounting to pabytes of information and um the end result is that you're getting the query result in less than a second. The use case behind this is um outside the realm of traditional search that we may be used to. Uh you can imagine a scenario where uh an application like this is consumed by an agent that can use the information it retrieves from the video files uh as uh context for a downstream task. U and the advantage of course is searching through this amount of vast amount of information that was embedded inside the video. So let's spend a few minutes discussing the reason uh these uh use cases are possible using LDB and the key benefits that it provides and of course tie that back into the agent layer that we are talking about
So um each of these is uh their own unique points. So I'm going to talk about them one by one but let's make this concrete with an example. So say you have this uh healthcare data set of clinical notes. So you have a patient who visits a clinic and they interact with a healthcare practitioner. Uh this is termed an encounter and each record has an ID uh like things like a summary description entered into the system of the the conversation and the typical format of this is unstructured messy text. There's you have things like names of patients um you know other formats uh other information like addresses and so on like metadata but then you may also have raw data that is multimodal. There could be an audio recording of the conversation um imaging data from clinical scans uh and even video data for example movement analyses from a physio checkup. So all of this in addition to the useful metadata that is in the text is basically what you want to store in your underlying u analytical application for potential retrieval downstream
So the first thing to note is the fact that you have very fast random access and um the availability of multiple modes of indexing. The typical workflow would be you u run a vector search lookup. So you create a vector index and what you do here is you create a new column called embeddings alongside the the table that already existed before and you may also create a full text search index alongside this like a BM25 index that supports keyword- based searches. So all of this is done in just a few lines of code. uh you have a Python example here. Import lance DB connect to a local or a cloud database and um you create a new lance table and you proceed to create a vector and full text search index in just a few lines of code. So the ease of use and the the pace with which you can get building is is the key point to take away here. So what we get out of this is the ability to do standard vector search things like answering questions based on a retrieved context which is the traditional drag scenario that we're quite familiar with
But this second benefit is where things get really interesting. When you add columns to a launch table, uh this is incredibly common in your AI and agent modeling experiments because the needs of your application continually evolve as you're building um these agents applications. So the way this is done in Lance DB is it is done in a zerocost manner. Uh by zero cost we mean that adding data to a new column. In this case, we have uh gender and zip code as extracted entities being added to this table. Uh these operations um are essentially done without copying the entire table beforehand. And this is really important because um traditional systems that are based on parquet uh and including traditional data lakes like iceberg um when you're doing a backfilling of a data transformation like this on a new column uh on pre-existing data uh you typically need a full table copy. So this zerocost data evolution aspect is enabled by the lance format uh which is essentially based on a fragmentbased design
So the the power of this is that you can essentially um write uh python udf uh and this is run in a vectorzed manner. Um, and there's this framework called Geneva that Lance DB provides that actually enables a lot of this uh for the AI engineer without having to build all the uh complicated data pipelines and ETL uh from scratch. So all of this is parallelized. It it leverages Ray clusters under the hood. Um and essentially this is used to power the large scale data pipelines that are used in these organizations like Netflix. The example here is highlighting the fact that if you had an original table that was 100GB in size, um that's a non-trivial amount of data. But the new column that you added only added 1GB to the whole table. If you were copying this entire data and transforming it and doing a back fill operation in iceberg, you would have to copy the entire 100 GB data table that was to the left of this column
But in lance DB it is a zero copy operation and all you do is write the 1GB of data. So from an IO standpoint um that is the biggest saving costsaving and time-saving aspect of it. The other very interesting aspect of Lance DB that is uh often overlooked is the fact that Lance does not have its own type system. Uh it leverages arrows type system and as you know arrow is an in-memory column data format. So um any launch table once persisted is essentially an arrow table and you can query it directly in uh query engines like duct db using SQL. So in this case I have an analytical question that I'd like to answer. Um I might want to do vector search beforehand and then do a filter to find the relevant uh rows in my table or I might want to do a SQL query uh using duct DB. And both of these are seamlessly possible uh depending on what kind of question I'm trying to answer
So depending on the scenario, LansDB adopt adapts to the kind of workload. Um you can do both online workloads which are common in search applications or batch workloads which are very common in analytics applications. And finally the biggest benefit is the fact that multimodal data in a launch table is colloccated with the embeddings and the metadata. So large supports uh provides first class support for large blob data as I mentioned and what this means is that when you have new data coming in say you have audio files or images or videos you can essentially treat these as large binary blobs and put them into their own column uh alongside the the row uh ID for that particular record. So what this means is that um you reduce any unnecessary IO fetch because the data that you're looking for uh for any downstream task is colloccated with the uh unique identifier or the pointer that you're going to use in your downstream application including all the metadata associated with that thing. So this allows for easier governance. Um in large DB multip multimodal data is just treated as another column. And um the other point to note here is that tables are typically thought to be long and narrow where you have a finite number of columns and potentially billions of rows
But Lance follows a two-dimensional layout. uh you technically can do wide tables where you have any number of arbitrary columns and within a column you may have very wide data which is blobs or embeddings or anything that is non-standard uh alongside the traditional tabular data like num floats and numbers. So how does this tie back into the topic of agents? We know that an agent is only as good as the tools and the context that it's given. And all we're saying is that the organization's data has value and as long as it's dug up and cleaned and placed into the right tools, that is what gives the agents the power to do what they need to do. So at a high level, you could think of a retrieval agent that functions this way. You begin by specifying a task such as a question that you want the system to answer. This is sent to a task planner. The planner's job is to break down the complex tasks into subtasks and each of those can be broken down into tool calls
So essentially you have a routing agent and the router's job is to uh route the request to the relevant tool. By tool I mean uh some form of retriever. And you can see here this is an example list of tools that this application is exposed to. Um you have a hybrid search retrieval text to SQL layer, a vision inference layer. uh you could potentially have um web API, web search APIs and so on. The idea here is that um when you obtain the responses from these tools, whether or not they are valid responses, um downstream you tend to have a synthesizer pipeline that gathers all this information that was retrieved. And there's a critic that judges whether all this information is sufficient to answer the question or whether the question was indeed answered. And if the process was not successful uh you send that back to the planner because uh something went wrong or the query was misunderstood
So typically you would have some sort form of looping where you try more than once and beyond a point you may want to set up a human in the loop feedback but once the task finishes that is when the pipeline ends and you gauge success has been achieved and um the agent has done its job. So there are plenty of great frameworks that I think already have been talked about. We've seen Pyantic AI's demo in the morning. Um I personally use BAML and DSPI for a lot of these pipelines. Um but the idea is that tools are exposed via like things like MCP endpoints or REST endpoints. And um essentially the retrieval layer is what is powering these tools under the hood. One point to note about uh measuring the outcome or the success of an agent pipeline. um an eval driven approach is definitely recommended to help run experiments uh that incrementally improve the results
So you can imagine that uh success looks like a certain way depending on what you're trying to do for your task. For question answering you might look for things like exact matches uh or substring matches. So you would start with a simple simplistic criteria that way but then as the requirements get more subjective and it's hard to quantify the output u you use LLM as a judge or an ensemble of judges uh to compute metrics that gauge success on the task. But the key point here is this is a forward backward sort of process. As eval continually change and the your um you find more failure modes in what what your agent does. You find out what is missing in each of these retrieval layers and that takes you back right to the source which is the data. So you then go back and augment the source data. You use techniques like feature engineering or exploratory data analysis to find more useful ways to express your data
So it all ties back into the fact that the data is the most fundamental layer at the base of the agent application. So the key ideas we want to take away from this talk is that um it's clear that the more uh systems are becoming agentic u they're becoming the like super widely used in every form like we're seeing more and more agent layers being injected into enterprise applications but all of these need a variety of data and the more and more tools we can provide that expose the enterprise data to these agents um the more valuable these u use cases become in the enterprise and it's not just about retrieval. So enterprise data keeps evolving. New features are always added over time and the capabilities of the data infra layer that serves the needs of these applications matter a great deal. Feature engineering, exploratory data analysis, serving of uh results very fast uh and training uh pipelines as well. Lance powers uh training data pipelines for foundation uh models as well. All of these can help augment the data and the performance of the downstream agent pipeline. And uh underneath every flashy agent demo that we may see, there's a lot of unglamorous work that went in from the data engineering standpoint
Uh and all of these rely on solid infrastructure. Um I don't know about you, but I find the data wrangling part fun. Like that's why I'm in this job, I guess. But um Lance DB is actively pushing the limits in terms of massive scalability performance um for all these kinds of workloads that involve multimodal data. Uh multimodal data is a first class citizen in Lans DB. Uh and as we know this form of data like PDFs, images, audio files, video files, these are abundantly present in pretty much every organization's data leaks. So the final image you might want to take away is uh what if you had a single source of truth for all the AI workloads that you are going to run. Imagine you have a lunch table on an object store that evolves over time
You can easily maintain a secondary index on the ID column uh for a key value store uh at various stages of the pipeline. the LLM may return only the ID at an intermediate stage. Uh but then you use a key value fetch to retrieve the rest of the metadata tied to it. Uh and then for analytics, you can run uh SQL queries on pre-existing data uh and new feature columns that you may want to add over time. Um you can easily create and manage vector and full text search indices including hybrid search uh methods and query them uh via retrieval functions exposed as MCP tools using this approach. And because your multimodal data is coll-located alongside the metadata uh the power of this storage mechanism is that with minimal IO fetch uh you can send an audio video or image blob uh in binary form the bite representation uh and convert them into pytorch data form data loaders uh and do downstream machine learning on this. So one exciting new feature that just came about uh in Lance DB a couple of weeks ago is the permutation API where in just a few lines of code you can filter on a massive data set of um multimodal data uh in this case images and shuffle the data to create random subsets of train and test and then send that to a PyTorch data loader for a machine learning training pipeline. uh and this is the technique that a lot of the foundation model companies that use lance are actually applying
Till recently they had to interface with the Lance table API but now we've exposed this at the Lance DB API. So essentially the the ease with which you can build these kinds of training pipelines and feed in the relevant data from various parts of your data lake um is actually the the real goal here. So ldb is designed for ML engineers, AI engineers as well as data engineers and data scientists. the surface area and the footprint is very broad and because of this uh large surface area of use cases u lb is used and trusted by many well-known companies uh most recently and wellknown of them being Netflix and Uber uh but also Midjourney Runway and many many more uh that are listed here. So that's it for this talk. I'm definitely open to answer questions about um how people are bringing data um into these systems and building agents on top. So thanks a lot for stopping by. Um we have two open source projects uh Lance DB on the GitHub repo here and the Lance format
Um and just a exciting development today is that the Lance format was announced as an independent um governance structure. So we have a separate website uh where launch the lunch format is completely open and independently maintained by a larger community of open source developers uh with people from many many different companies involved. So we're excited to see how that community grows. Um we launched DB on X and yeah uh hope to see you on Discord. Uh some of my colleagues are at this event as well. So please find us and we'd love to chat more about your use cases. >> Okay. Thank you pashant
We [applause] we do have two minutes for some question. Yeah. Does anybody have question for pashant? Yeah. No. No. Quiet. Quiet crowd. No
Oh, good. Got to go around. Oops. Here. Thank you. >> Uh, hi. So, so what gives the performance speed up compared to path K because ARO is also columnar based and so like what uh yeah I think that's my first question. >> Yeah
So um the performance speed up happens in scenarios where random access is important. So there are two scenarios in which you have uh workloads that you run um you have search workloads which require random point in the entire data set. Right? when you're searching for something you may have a a billion records but you care only about one or two of them which are randomly placed in the data. So that is what we mean by random access. On the other hand you have um uh block structure like a columner structure where you want to run um aggregation or analytical queries that group on large parts of the data. So those are touching hundreds or thousands of rows in the data. So those are analytical workloads as opposed to random access. So paret is optimized for the analytical side of things
It's meant for scans on the entire column or the entire table. Um but Lance is optimized and really good at the random access part of it. So that 100x or thousandx gains come when you're doing things like search. That being said, um there's a trade-off as well. And um Lance does not compromise the scan performance for that massive gain in random access performance. >> So does it mean that the trade-off would be on the storage side? would lance probably take. >> So, so lance is basically an alternative to parquet in these kinds of workloads and it's up to I guess the application and the user to try out the the performance gains on those particular workloads. >> Got it
Thanks. >> Yeah. >> Great. Yeah. I guess we still have one minute. Does anybody have any question for Prashant? No. I guess if you can think of any more question um you're invited to go to the interview room which is you go up there um left and then it's down down the hallway that way if you have more questions for pashant. Yeah I guess then we are right on time and in 10 minutes we're going to have Tyler from aka back here to do a demo on multi-agentic system
So yeah so feel free to take 10 minute break and come back. Thank you. Thank you Pashan.