Bay.Area.AI: Build RAG-based large language model applications with Ray and KubeRay, Kai-Hsun Chen
yeah and my name is kin and I'm very happy to have a chance to present here and uh today I will talk about Builder R best lar model application with Ray and the CU on kubernetes and this is sound of my metadata and you can see this is my G handle and uh I'm currently a software engineer in the recording in the N scale and primarily focusing on the cubr and uh uh in this year also work on some record stuff and I pretty like to uh work on open source since I was student and I work on like a ray CU AP submarine AP spark and the some engine stuff and uh um I contribute to land and also maintain some of the this project and I spent a lot of my spare time that's promoting the open source that's in Taiwan I meant like about like 10 people to become the Apachi sofware foundation commers and uh Mentor two people that to become the top eight cubre contributors and build a community in Taiwan to help more than a lot of people to work uh join the open source community and more than five people let become the Apachi software foundation and L Foundation commits uh at this moment uh so uh if you are open source over uh feel feel free to just reach out to me and we can talk about open source yeah and the first is that I will introduce the lar model and the r uh yeah I think in the past one and a half year uh larage model is almost everywhere because it showcase that it has the human label capability so open up a lot of the a new AI application opportunities yeah um and but one of the challenges for the large language model is that if we ask the question that it's not in the training data of the model of the foundation model uh it doesn't know it cannot give you a correct answer and uh to solve this we can use the like a f tuning or use the r uh R is a very popular solution and the idea is that we can put the contest generate embedding for the contest and put in the database and then when we send the query to Lage model We Can ad first that to retrieve the contest from the database and also append the contest that in our uh in our query so we can get the uh so the model can learn the knowledge L from the contest that we provide yeah and uh because uh we will build a we will demo a rag application that with Ray and the CRA so before last I need to introduce what is Ray and the CRA yeah uh I think AI is all around you uh for example like I T the Uber um almost every week and I use open TR GPT to help my daily work and uh our company that order a meal from the door Dash every day yeah so I think AI is all around our life and uh all of this company use ready to build their ml infrastructure solution and I would say that most of them that's deploy a r with cuay on the kubernetes yeah so now we uh we we currently know who use Ray and now we can talk about uh why do we need to use Ray and what is Ray uh Ray is an open source uh computation framework uh that's it for scalling the AI and the python applications and you can see this is kind of the compulation framework you can see it's similar to like a spark and something like Mar use before and uh and why do we need to use Ray I think the first point is that uh Ray is pretty easy to use um I think maybe most of you maybe learned like what is merid that in the your undergrad operating system course or distribut system course uh if you want to uh if you want to paralyze your program with the merid framework you need to rewrite it into a math function and reduce function and uh I would say that it is pretty hard and uh I believe that I have the uh I able to St this because I proba some paper about my before yeah so but with Ray uh Ray is pretty easy to use you just need to modify several L of code and so your local program will be almost the same as the distribute system version and I I will showcase it later and the second is that I really is pretty general purpose uh you can think like the like a spark it's designed for like a data processing so the abstraction is something like the spark friend but Ray is designed for the general programming so you can almost do anything with Ray and the ml worklow is a very special workflow because it is very very versatile it include like a data processing training tuning and serving so you need to have a computation wrun time that can cover the ENT life cycle so R can support the any worklow uh any hardware uh on any cluster and at any scale and you can see in this in this image you can the r core uh R cor is the uh core computation engine uh and uh we build the r Community builds several AI Library based on the record including like the data Trend tune uh reinforcing and the serving and we can deploy the Ray on the uh both like we can deploy on like a public cloud like the AWS gcp and Ager and we can also Deploy on kubernetes cler or Deploy on your un cler yeah uh so now uh we will introduce the first of talk about the rord in this slide uh you can think about what are the essential component that's for your local program uh I think there are three components a one is the you need to write a function you need to define the Cass and you need to define the variable and uh and as I told before like a spark defined for the designed for the data processing and the race designed for programming so Ray has a oneon-one mapping with the reg three abstraction so for example like we have a red task which is the remote function and we have reactor it's a remote class and then we have object is a remote variable so uh because it is a one one mapping it can do almost anything and you can the actor and the task can communicate with each other and you don't need to worry about like which know that this test is running on and which note is this actor running on so uh you can see it R enable you to just program in the distribut cluster maybe on a 100 or thousands of notes uh just as if let you are working on your laptop so red stogan is that it is a infinite laptop yeah so and this is the example to Showcase how easy it is for the record uh I pretty like this simple example uh this example is from a user that from a Traditional Bank in Taiwan and he is a data engineer and he need to handles a lot of Cs V almost every day and uh he use R to aair paralyze computation uh solution and you can see how easy it is uh we just add an annotation to convert the function to become a remote function red task and then we change the master to a function code to call the this is the remote is the way to code a red task so uh that's all uh it's pretty easy to scale up to like a of the CSV files in this use case yeah and uh now we know what is the rate core so we can talk about rate a libraries we build the rate Community builds several rate a library based on the ray core including like the ray data R train R T and R Ser uh to cover the end to end model left cycle and uh for the ray data is primar for like a data loading and like a bat inference or like embedding generation and train is for the distribut uh training and the r tune is for the hyper per tuning and Reserve is for the sound uh for the onine inference yeah so now currently we know what is a r core and the library that build B on the r cor so we currently can talk about more about the deployment solution uh in the ray we support the two kinds of the deployment solution one is on the virtual machine and the other one is on the kubernetes and uh cuber is the official support that's from the ray Community to support to deploy Ray on kubernetes and I would say that most of the user deploy rate on the kubernetes by Cube rate and uh um I'm not sure how many of you that have the contest about kubernetes or have experience uh if you have experience about kubernetes you may know the kubernetes operator and uh CU is a ray kubernetes operator and the goal of this project is that we want to separate two kind of the user the one is the data scientist to focus on their computation focus on the python script so they don't need to worry about how to write build a dark image and how to write yamama files and uh the other side is that for infra Engineers we want them to focus on the infrastructure stuff like the monitoring like a perinal Scana like a frame bits and like to St out like Ingress like engx yeah so uh currently we know Ray and a cubra and uh we can take a step back or we can revisit that what the some of the existing solution that uh I think most of the application that's on the kuet CER are built by in a microservice architecture so uh I think in most case I think a microservice architecture makes sense for some infrastructure stuff but it is not Mak sense for for some computation stuff uh for example because ml worklow is veryable style as I told before it cover like if you need to do data processing and sound feature engineering to training tuning and serving and a lot of stuff and uh if you need to have different system for a different sta uh you your system will be very very heavyweight and so for example um for some exting solution uh you may need to launch like a 10 maybe 20 20 or 30 like something like that pass that on your local kuet Custer to cover the entry and life cycle so it is pretty heavy way but if you use Ray you just need to launch two parts to start that you just need to launch one kubernetes Cub operator part and one rad part and the second is that because uh in the microservice architecture because your different stage in the model life cycle are not to design together so you need to have some way to uh gr L together for example if you finish the DAT processing you need to know how to communicate with your training process so uh user need to do a lot of stuff like a building a dark image and write a lot of yamama file or use some workflow arator like llow or flight to gr all of the different stage together so uh but dat scientist they don't want to handle anything about infrastructure and the third one is that uh because I think the AI application Chang rapidly so uh this kind of a complex infrastructure is hard to uh fulfill your new uh infrastructure requirements yeah so uh I think uh I think currently we I think R cubra is the I think it's the most popular Solution that's on the kubernetes to build the AI infrastructure solution and uh as I told before uh different worklow have a different system requirement for example like if we want to training use the data parison uh we need to have like the G scheding but if we use like the mod onl Service uh we may want to have like the auto scaning because we cannot predict how many requests we will receive and we may want to have the higher ability so the different stage have the different requirements and uh we can also focus on the single stage like model serving it is also differ uh for example uh before large model most of the model serving you can run on like a CPU or a single GPU because the is small and the most of the case the bottom neck is about a computation so uh the goal at that time is that maybe we want to figure out how to reduce the number of 14 point operation but for the large language model because model is very large you need to have multiple GPU to serve the model so the question moved from the uh computation Boton to the memory bottle neck uh because it is very large and the the large language model is based on the Transformer decoder architecture and it has done a auto regressive nature so it is a memory bottle neck so I think uh so I think in the past two year I see the workload move from like the uh different name reinforce learning stable diffusion IHF and the Larry model Moe and the multimodel a lot of stuff so different worklow has a different requirement so you need to have a flexible and a future proof a yeah and this is a example to Showcase that how powerful is the UniFi a WR time this is a very simple example this is from sensara and U you can see before they adop rate they use micros service the first is that they have the go microservice for model selection and then has the python microservice for the inference and then uh and then use a golden micros service to do the business logic and then they design to use Ray cor R reserve and Ray cor to uh to make then like work together so it can Ser the uh cost of like the disiz and deiz cost between the different microservice and they can share the resource between a stage so uh they achieve the 50% cost reduction and for more detail uh you can see the SAR engineering block yeah so uh currently we know the record re a library and the cubr So currently we can talk about the demo and uh uh here today we build a demo is not we build a lry model chot and uh we have already pre-processing some of the rate documentation stor then into the V database and we will do a Rea with the rate documentation so we can ask some a r question that to chb and uh uh this is the best we have a quarry sent to the we enter a quarry that's in our web application and uh send a quy to the model and get a response and response also shown on the on the web application and with r is that we have already stored some embedding of the rate documentation in the database so when we send a query we will at first to generate a embedding for this query and then use the query to search the most relevant context and then we will append the context that's on the query so it can get a better response yeah so I will talk more detail about uh uh this demo the first is that we generate embedding using R data and we also published a Blog that about how to generate a 10x chaper embedding uh with any scale yeah and you can see I think this is very informative and uh uh very welcome to read it and uh in that we what we do is that we uh we use the ray data and we div split it into multiple chunks and uh use the GPU to generate embeding and store it into a v database and the second is a retrieval when we have a query we will first that uh generate embedding for the quy and uh try to do like something like cosine similarity to get the most relevant context from the database and the third is that we will also uh put uh both the context and the query and send it to the model to get a better response and in this in this demo we use the N scale endpoint as our uh lar model back end you can also deploy with like our open source project RM yeah and this is the uh I will be do two demo first one is with N scale and the second one is with with cubre yeah let me sure okay cool and this is the N scale word space and uh you can see uh the first is to initialize the V database and this is the uh this is a notebook that's on the N scale platform and we generate the in generate embedding that with the ray data and store into the uh database yeah and we have already finished the embedding generation and now I check my postgress database yeah and uh I think uh we we can find out the uh oh and then and then we generate the response and we can see um yeah I think here is the response uh this is the question and this is the contest that we retrieve found the database and then this this is the answer and then uh we will we will verify the answer later and then this is the CU demo yeah because uh we don't have the notebook so I just run the python scri script directly uh we can see this is also embedding generation and check the database and get a respon yeah uh because uh the model that we set the temperature to Li so we will get the same answer yeah and then we can verify the answer the answer the question is that what is the default B size for a math Bess and uh this is a question about one of the great data function and we can see we get the top five context and then we see the first one the first one is here and we can see oh yeah the the best size is actually uh 1,24 so I think the answer is correct yeah and this there are some for reading uh one is our blog and the other one is the demo of this uh of this video the code of this video Yeah and uh cool uh thank you [Applause] guys question oh so which one cust view kuber custom view in the demo actually oh in demo yeah is that this one yeah that that one why exactly so I'm trying to understand obviously some of these are standard kubernetes but what's uh what's things like connectivity agent what is like could you just walk through what some of those components are and what function they're serving okay Al so uh you you want to know more detail about how does it work uh you can see like this one is um uh as in this script is for the embeding generation and at first is that it will we will load the uh rate documentation into the uh into our memory and then we will divide it multiple section because it is a markdown and then we will use some L CH function to split into multiple chunk uh and then we will use the rate data to do the embedding generation and then uh when we finish the embedding generation we will write into our post S as our V database so this is the eding generation and uh and then this one is that we check our Vector database yeah and uh yeah I think this just maybe little 5 second but you can see that it's successfully stored into the V database and then we and then we run an another uh python script is about uh uh inference and the inference is that at first is that we will send the generate embedding for our query which is our question is that what is the number of the batch size and then we J embedding for it and then we send the embedding to look at like the most relevant contest uh from the post grass and then we add the contest to our query and send it to our N scale endpoint and our N scale endpoint we use the model is the Mol yeah and this is the yeah this is the answer and because uh both end scale demo and the Cub demo we set the temperature to zero so the answer will almost be always be the same that's helpful the followup question on that was it looks like all of this is running in a kubernetes cluster so you have an operator that's managing all of this um have you looked at going across clusters presumably endress or indress gateways and from a networking perspective when you have actors talking to each other across clusters what sort of performance and latency elements come into play got got uh but honestly I think it is it depends on your model back end because I think you can see here that we we don't do any inference computation from our side we just do that from the retrieve the contest from the uh from the vector database so the first is that uh I found that it spend a lot of time on the retrieve a retrieve the contest from database and uh because I don't spend a lot of time to find the what is optimation configuration about that so I don't I think maybe uh another speaker can talk about that it's l and the second is about your lar language model latency so I think this two very very depends on like your different model and different uh different model and the different uh Vector database so at this moment uh I think uh uh currently the main the main time that spend is not on the is on not not from our side so uh I I think currently I don't have the answer so I think it depends on database and the uh the the end point that you use other questions okay the back at that get a mic and I'm going to hold another mic to record this okay uh I saw you uh how it integrated with the Python scripts in that one slide there I was just curious like depending on the platform you choose like you mentioned you can do to a local cluster or there's a couple Cloud platforms so you can do kubernetes uh do you have to change like how you write your python code like with the data scientist have to think about the platform they're on when they're writing their code uh I think I think it depends on I think in most case that you don't need to worry about that but in some case that you need to understand it for example if you want to serve a model uh and across multiple a availability and U I think I seem like in the VN or in a different configuration you need to try to find a way to tell the ray that this two different maybe this three different array path belong to a different a so that you can schedule it correctly course matal a okay and how would the user how did they specify memory requirements and things like that you say for the test and actor well when they're writing their python script like when you're how do they how does how are memory requirements resource requirements things like that specified got uh you can see like here there's a annotation and uh there's at R down remote and you can specify the memory or CPU something like that here but uh in Array the resource is a logical resource we don't do something like a SE group so uh the we specify the memory or CPU just promise that they will have at most this kind of the test actor be schedule but we don't do like a resource isolation between a different uh red test actor okay I see could there potentially be issues because of that or do you deal with it somehow or is it just up to the operator I I think it depends uh I think the first is that uh if you want to build like a US C group uh I think there there will may be a the some new overhead and uh but at the same time that if you cannot you if you don't really understand like your workload how many resource it use maybe sometime it will have the uh resource contention something like that more questions I'm on the back so please raise your hand so I can see you if if you have more questions okay I see hand there by the way I just found a lot of beer in the corner if you guys need more is that right there sorry who held the hand up please oh um I think in the whole pipeline of uh doing envelops R can be used in multiple places but I'm interested in training uh what are some of the setups that you see companies are using in in production do they use uh the same notes with gpus provisioned by Cloud providers or do they change that to run on premise and then serve them on on cloud providers um because of cost Bally go go uh I think that I think the first is that uh I would say that most of the people start to find the GPU from different not only like AWS gcp or AG but also like something like L lab Co right because I think uh the GPU GPU SC is a very important question and uh but I still don't see like the multic cloud at the same time like the same cluster multile Cloud uh and the second is that for theint um I would say that I never see that who moved from the public C to the unar Custer from my personal experience maybe on some on on some social media I see that but uh from my I I keep in touch with maybe hundred of company but I never see it and the second is that uh I think people start to uh export the new accelerator for example like the TPU and like like aw inferential and or training uh I I recently see more and more people to find it and the second is and the next question is that um and there are some multi host a multi host is some concept like in the like in GPU and TP both of have multihost it mean that you have multiple version of the TPU or GPU and is of land connect with the high speeed intercon next yeah so I observe the trend but I honestly I think the multihost is a pretty new stuff and it is not very uh will support that in the uh in the kubernetes OR in anything else at this moment uh it is not that easy to use so we collaborate a lot with like the TP and GK team to make it work I think it currently work in some stuff yeah so I think this is a trend that I observe all right we have a question in the front I'm bringing it to you um can you go to the step where you describe embedding models um I think it was R I think it was one or two or something like that uh this one yeah yeah um what kind of embedding model do you guys use is it specialized any context um Can the user change it to their own embedding model yeah I think they can change the embedding model and uh we have a Blog that uh uh like published by our colleagues that they do a lot of experience about like a different like the uh different trunk size different Ed model we do a lot of experience and to and write a summary that to see uh the the relationship with the quality of the response yeah but uh here I don't spend a lot of time to change the uh embeding model thank you all right guess last question for our first Speaker anybody here hi um yeah I had a question about so running r scale so you know for large node clusters a common problem is you know Ray actors dying unexpectedly so I was wondering um you know have there been any updates to Ray core to kind of mitigate that more and and the second thing was also about um utilization of the r client um as you might know there's a lot of new tools out there like code player who you know give you the ability to abstract away the r client you know as a result of like issues like serializing and de serializing as you push code from your laptop to the cluster and so I was wondering you know what steps have you guys been taking um to address those so far God God uh I think for stability issue I think uh uh I think this I think in the past two month uh we Al almost uh invest all of the recording in the stability stuff so I think you can see the it's become much more stable uh in the past few release and for the r client I think um because of some step uh there are some technical issue that it's very hard to maintain it so currently we don't recommend the user to use it but I think we still observe a lot of user Prett like it so uh we may revisit it but I think uh at this moment we don't have a plan but there we we have some design do that's in our table but currently uh still in discussion but I but I I think because we received a lot of signal about like a user pre like yet so I think after we fix s of the stability improve a I think because currently our PJ is the stability we spent a lot time it and then maybe after that maybe we will revisit it but currently don't have a plan thank you all right let's thank guys CH thank you very much cool thank