Chip Huyen · The LLM sandwich -- the data layer before and after LLMs

Chip Huyen · The LLM sandwich -- the data layer before and after LLMs

Recording: Chip Huyen · The LLM sandwich -- the data layer before and after LLMs

thank you my name is Chip thanks for introductions I'm here to talk about the LM sandwich I know it's dinner time so I just try to choose the appropriate picture so it's 10 minutes wait how do we move uh okay slide so in 10 minutes so we won't be able to go into anything too deep but I believe that um LM is not the everything is part of the of the pipeline so we want to talk about what happens before and after LM so one thing is going to like um celebrate a little bit I do things that's like the popularity of LM made a lot more realized a lot of things that the MLS committee has been talking about for many years so for one thing like you can't do AI without data and now this is a lot of companies at the wake of General AI try to figure out they did a story so that they can leverage the data to enable new use cases and the second is that real-time is good I think for a long time we're talking about bash predictions online predictions and now with LM everything just happens in real time it's pretty awesome and of course another is a data distribution shift so now we see LM and enter hrgbt it's like oh our knowledge cut off is in September 21 so we can't help you answer with that and we know that the world has changed a lot since 2021 and of course the last one is a state matter because now we want to get to the context history the previous chat and it's not it's just one off requests anymore but once you like Leverage history historical information to make better predictions cool so agenda very simple before om we have context loading and after we have execution and monitoring so connects learning um certainly a couple of questions like for example like when we ask when we so I asked a friend about whether he thinks that context learning would be here to stay because a lot of questions like whether problem engineering is hiking or like how long we still need to do contact constructions context learning and he respond with that context learning wouldn't be relevant as long as human to human communication is relevant and the reason is that like for a lot of questions we need context to respond so first the way I ask you what is the best Vietnamese restaurant in the country and the context required could be which country right and if you ask into like the chat to be today it was assumed that the country here is in is the us but a lot okay for many cases so therefore context is not obvious and you need to tell so AI what you want or another question could be how many on booms has Taylor Swift released so if entering Chachi BT it might not contain like all any other AI it might not contain all the new informations on Taylor Swift so you might need to enter the newer and Booms as part of the context so you can answer a question correctly so there are many use cases um so you notice like you are pretty familiar with them um so for some like customer support like we couldn't make external chatbot so chatbot so you can talk with external users but also into the chatbot so like for example the employees can talk and ask questions about the company like hey I have need to do these vocations what is the policy for like vocations or like oh there's a policy cover that XYZ so like there's a lot of like internal knowledge that you can use a chat to help the employees discover I've got document processing summarizations uh or like storytelling or like any task involving a lot of genes and proteins so in this like very interesting paper they found out that like for a specific academic data set like 16 of the questions require context to response but of course I would imagine that for industry depending on the use case that percentage can be a lot higher and here's another like you can see this like uh people talking about context link because I choose input text in choose your uh into the prompt it might take a lot of like input tokens and here's an average of some of the tasks it can go up to like hundreds of thousands of tokens which also means that contact lens will be a key challenge for the future AI use cases so here stretchen ML and the LM workflow changes a little bit so one thing is that like for traditional machine learning you get a users using the applications and it might create operation request and then you might also like tap into internal data to get the future attractions so that I can input into the model so here's the input into the model would be like features this is a model can make predictions however for LM it's like almost like similar but like there are two keys difference so one takes the input into the model would be the prompt which could contains a context and if we need to construct the context from uh from like the data or like other historical conversations and then it's not going to be one operation anymore but like for each response uh you for each problem it would need to like pull previous conversations as well so you can see this like after the LM output a response it will go back into as part of the context for the next question and also think about like I like because for children ml it might be able to do that in a batch space but like for which for LM everything is pretty much in real time and you want to do things as fast as possible for the users um so here's an example for the contacts needed for customer support so here the use case is an external chatbot so say amount users and I want to change the shirt I ordered yesterday from Siam to like L right and here on the left you see like the kind of data that chatbot might need to respond correctly so first of all the first thing it might need is like to get to put out the custom identity so again response like hey Sarah let me look into this for you right so you need to know the who you're talking to and then you might need to look into this customer recent orders so like maybe that person never orders a shirt so you need to double check that to tell that like maybe like if that person really Auto issued and we might want to confirm that with the with the users and then you want to change the other status because if the order has already been shipped then there's not much you can do about it you can tell the users like okay it has been shipped so then you can return it and order one later but it has not been shipped then maybe there's some room for you to do about this and of course inventory because maybe it's a l it's not available anymore and you might want to recommend other items like okay I know that for this shirts there's no size M would you say l could be interesting as a shirt based on your preference in the past or like other users preference so like there's a lot of information that you might want to people on from your database to have the AI like responses or users in real user users effectively and a lot of that would need to be put in real time because you probably don't want user to wait for like five minutes um so connected can come from multiple sources uh so a lot of them like people talk about like very databases a lot and it's definitely going to be here to stay and where the database is not something new right like I think it has been around for like many many years for like almost a decade and like Google has scanned Facebook has files and their own very good regular basis so they're not new they've been around for a while and I think they will continue should be very important so they're really good for unstructured data so you can chunk them into embeddings and store them but also need like and like you also need them for need the structured data as well and you might want to access them from a transaction store and I'm streaming online streaming data like Kafka Kinesis latency but latency requirement really depends on use case and if you don't have a strict latency requirement then there's a lot of room for you to design the data pipeline for example if the users is okay with like 10 seconds then you might still have time to go into snowflakes query and retrieve the data but these are vacancies like a second or like 100 milliseconds then you might need to like store everything in fast uh fast storage like hot storage like you might need to go to my radius or Cassandra or postgres which console be very expensive um and of course there's a question of my security so you need to make sure that the employees or the people who use a chat bot don't accidentally include personal incentive data in the context and send it like outside because my violations of like compliance so um it's actually a very interesting thing I'm not sure how you guys if you have experiences but I see that a lot of companies a lot of managers I'm talking to are worried about the teams using llm without reporting like how they're using LM so like I think like there's some crazy number you can probably like look into research about it but there's crazy number of people who have no idea how the employees are using LM or just like copy and paste any documents into like chargeability and see what responses get and it's really not Kosher um anyway um and also you might want to like detect and block on the request that might contain this information so saying that you might have some kind of like uh check before you sending out the API request and say like okay does this contain like pii or personal or like sensitive data and if so like what could be the appropriate action to do that like do you want to block it or do you want to flag someone or like what what action you want to take there um so I will say this like context learning is actually a data engineering problem because it deal with the retrieval uh data governance and also like latency optimizations cool so here I want to plug my clipboard we do like we we are we are pretty much a data product so um we we deal with anytime when people have latency or data retrieval problems that's where we are um okay so like uh after LM is going to be executions and that also involves data so let's say the chatbot has successfully updated the order from like site M to say l now someone had to execute it right it had to like alert the appropriate people like who is involved so that maybe there's an actual warehouse employee or somewhere robot who actually do the task of my swapping out so I Am for sale and of course you want you much update on the databases first we might want to updates already on other databases or you want to update inventory database so they can like maybe remove onsite L and like add back once I am um and actually you might want to like monitor like um responses so I think like we uh there's a lot of research today like to train the standard level of risks that LM posts your business so one thing is definitely like brand risk uh brand risks so for example like it might output like very racist or sexist content or is this like bath mail your brand or a bath mouse or competitors which might not be a good look and of course you might want to detect like if maybe it's a LM accidentally reveal uh sensitive or personal data first of all there have been like a lot of people trying to jailbreak AI LM to make it return like phone numbers Social Security number email addresses or like other private information which you might want to detect and block and of course like uh you might want to like gather user feedback to improve the LM and also like um a very big question I get a lot when I go to tons of different companies it's like what to do about hallucinations like what if so yeah I just make up stuff and just like and it's like a very challenging problem and I don't think it's like the concrete answer and definitely not something we can we can cover in like 10 minutes but anyway uh that's that's my talk uh thank you very much uh for the time and feel free to reach out uh if you have any questions here's my email Twitter LinkedIn and also very active on this score foreign all right so we have a couple of minutes for one question for chip one question if you have you can raise your hand here you go so these most recent slides are they going to be on your blog or are they in the book it's not my blog but I can send it to you okay yeah cool thanks chip thank you so much for everything I'm sure everyone learned a lot give a chip a big Applause