LLM Avalanche: Aakrati Talati : Function Serving in the Lakehouse: Delivering Personalized Context..
thank you my name is akriti and thank you for introducing me earlier I'm an engineer on databrix's machine learning platform team and I'm here today to talk about feature and function serving in the lake house um cool so all right so uh let's forget for a second that I work at databricks and let's think that I am building a traditional hotel booking website so what uh most likely like my website is built on top of a web application uh that takes in user city of Interest some booking dates performs a personalized search on top of it and does some machine learning ranking on it it's pretty good but it cannot answer like all of users questions so that's why in today's presentation like we'll look at how we can build a personalized chat bot for my hotel booking website by using a large language model and we'll also see how we can leverage the Enterprise data that can make the llm chatbot very powerful with the personalized context so we learned a lot about llms today and this is the last talk so I'll not take too long I promise and but llms are extremely good right like they are very good at understanding human language their intent and most common use cases that we have seen are answering questions and generating texts so um we also know that there's a lot of research that's done on Vector databases that can provide like uh approximate information for the llms however we can make llms even more powerful like and even more performant using uh by retrieving personalized and giving it personalized context information so that's what I want to talk about today um but before we do that let's take an example of what an interaction with a generic chatbot would look like so my chatbot uses uh here like I used an out of box open AI model and then I asked it to create a seven day vacation plan for a user on my website a user wanted to travel to Amalfi Coast in September I didn't do any additional fine tuning or didn't provide any additional context uh chatbot was good it was able to understand first that user is planning a vacation and then second that user needs to book some flights an accommodation good but like it doesn't know a lot of things right it doesn't know that anything about my user literally and what kind of vacation do they generally book what is their typical budget what accommodation they like like do they like beach facing rooms so that is all information that is missing and this is exactly what the Enterprise data in my lake house has so we'll look at how uh my lake house can provide all that information uh in a personalized context so all right so my databricks lake house already has this Enterprise data from from my website databricks has recently launched this functionality to serve any business data and functions at a very low latency in a real time endpoint um it's called feature and function serving and all I need to do is store my business data in the lake house and serve it using one of these real-time endpoints by the way if you are using model serving you already have all of this functionality integrated for you so let's see like I have all my data I have my feature function serving endpoint let's see how I can use it in a real-time llm application so remember we were booking a hotel booking chat bot so this hotel booking chatbot has my user data which is like users budget some Hotel data such as hotels availability and discounts um so we will serve all of this data using a feature and function serving endpoint and then use it in the llms to provide personalized context let's see how this workflow would look like for my website so when a user asks AI bot to create a travel plan for them there are many things that my lake lake house already knows first what do I know about the user I know and that the user has a budget and some preferences like they might like a quiet place or they might like a downtowny place uh they might like a beach facing room for example then from this information I can do a top case search uh using based on their preferences and then I can use a feature serving endpoint to get real-time availability and prices for their preferred hotels finally I can do on-demand calculation of the hotel price for vacation so using any custom code that user can create within the lake house so I think we talked about like all the awesome functionality so let's see some of it in action to remind you earlier we saw an example of a generic chat bot that could understand the user intent but didn't know anything about the user and it was asking like hey can you tell me about who the user is ETC so now we will use a feature and function serving endpoint to provide some of that context information and will create some tools in line chain which are basically helper functions like if you haven't used them before that takes some input and produces some output that is used by an AI chatbot so let's see the example here okay let me all right so let's define this tool that is using a feature and function serving my function serving is called uh it's it's called user budget preference and it is calling one of databricks's endpoint I'm providing it some token some authentication Etc and then I'm finally making a rest endpoint request similarly I am also defining a tool to retrieve hotels from the vector search and then finally I'm defining a tool to fetch prices and do some on-demand calculations so let's set up an agent here um let's see what's here so as I earlier promised you like we are using all these tools which we just Define a few seconds ago the user budget preference tool the hotel retrieval tool and the total price tool and then finally we are initializing the agent with the tools that we just defined and using the same generic chatbot that wasn't able to give us a very useful answer earlier um so from this AI chatbot we are also providing a prompt message that basically tells the chatbot as to uh provide some hints to the chatbot as to where it should like go and fetch the information from the tools that we had defined earlier so I am telling it to like basically fetch some budget preferences Etc the agent help agent can make decisions so agent can basically like make decision of when to use which tool uh when it is asked the question and then it will provide all the input that the tool needs and then it will use the tools output to take next decision as to what to do so all right enough talking let's call the AI bot and ask it the same question that we asked the generic chat bot earlier which was to plan a seven day vacation to Amalfi Coast around September for my user ID before I get into the details here I want to show this is the output that It produced it told me that there are some four great hotels that fit my budget from September 1st to September 8th it told me that I should probably go book hotel miralfi Etc and then it asked me for any further questions so let's dive into how it really did that um so oops [Music] back all right we use two-factor authentication we are secure cool um yeah this is what I wanted to show you like this is basically my chat bot in detail um it's basically first so the agent is making decision to call the budget server tool first it called the budget server it got the observation here which is my daily budget for this particular user is 700 then it did a vector search and found out some four Hotel IDs that were really ideal for my users preferences and then um it it basically called the hotel total price feature and function serving endpoint and got the hotel all the hotel information that it finally displayed to us so this is pretty powerful right so finally I think uh I as I promise I'll not take too much uh as we saw like data breaks lake house plus llms is really really powerful uh using feature and function serving I was able to seamlessly deliver the personalized context from my Enterprise data to the llm that I was using which was a completely generic non-fine-tuned llm here and I was able to run on-demand custom code that I had defined in the lake house at low latency in a real-time endpoint I was uh so that could have like trade secrets for example for my website and then finally uh we used Enterprise level security Behind These endpoints because all of this is governed uh in the unity catalog so I I would only share the information to the caller that the user should have information for thank you thank you all for your time thank you