Agents Need More than Vector Search

AI Agents need rich context and a variety of ways to tap into that context all along the agent chain. While a vector database is very useful for running semantic search, agents need more. They often need to run filtered queries and/or analytical queries that aggregate data. Crucially, agents can benefit from access to multimodal data like images, audio, or video. Most vector databases are not built for storing such multimodal data. Getting the most out of these varied sources of data can involve tool calling routers that retrieve from the appropriate source based on the query at hand. We’ll introduce the key ideas behind LanceDB and the multimodal AI lakehouse that can be the source of context for AI agents. In this talk we'll show you the power of LanceDB for building agent pipelines, with some useful ideas for multi-hop retrieval via frameworks like DSPy.