SBTB 2023:Isaac Yang&Yuan-Ting Hsieh, NVIDIA FLARE: Federated Learning from simulation to production
so today we are talking about the uhia flare first let's talk about the data uh everybody knows uh if we are doing the Deep learning machine learning AI data is so important without data we basically cannot do anything the model cannot be trained uh we basically just sit there and just see the computer I don't however uh when we talk about data data has some kind of uh properties sometimes a different location has different data or different machine generate different type of data in medical imaging system uh sometimes we see seens CT generate a little bit different uh Image Dynamics dynamic range uh from maybe GE so this is basically sometimes we need to look into if we need to train different data from different sources and uh uh for the p uh pity basically if uh we have a lot of people each one of them has a little bit data but they also want to train a model usually it's very hard for them they might need to collaborate together to collect all data to a centralized Pace however it may not be so uh easy to do that one possible scenario is those people may not want to their data to be transmitted outside their organization or maybe this is a Regal uh limit that they cannot move their data out of the country boundary it's uh all from our customers our customers mention different uh usage for the data but they cannot just say put all the data together in one location to chain them model and for the Privacy this is even more uh restricted uh like in hospital they have a lot of data if two Hospital want to train a model a unified model but they data cannot be exchanged because there are a lot of review or information required or committee review sorry okay so that means it's uh not very practical for them just say okay let's put the data together in one location and both of us can access it and the chain model and for the FSI is a financial service industry oh their regulation is even uh difficult to overcome because it's a really really hard to say uh two different organization exchange or share some sensitive inform HC is the healthcare so when we face this kind of issues but we are from the machine learning deep learning how do we overcome those strange or difficult situation to really really get all the data we want and the trend the model we need so this come sorry this come with the Federate learning Federate learning basically say change your model near your data set original people say move your data set to the training location so every data set come to one place and train it very very efficient but uh Federal learning say okay if your data cannot move how about just train the model there and move your model around so that all the model Trend can um through a agis some type of agis and you you can generate one single or best models and this is how NV flare uh created it's a open source SDK for federal learning it's apach 2.0 so it's open source you can just download clone it and uh submit your PR we welcome every type of PR and if you have a question you just submit issue or questions we will be very happy to answer it and uh it basically enable distributed multi party CL learning so we are just not just say MV flare is used for NVIDIA only anybody who want to use it even you are in maybe us and uh you have some partner or some kind of a uh uh client in different countries they want to train a model you can just use it no problem at all and uh it's a production scalability it's has been used uh uh we use this system to publish a paper in natural medicine uh it's on the covid X-ray uh classification example basically that case uh include the cons of I think 11 countries and the different hospitals because at the time uh uh just one country with their uh x-ray maybe not enough and people want to collect enough data and uh it's basically domain uh agnostic it's basically if you want to use py fine tensor flow fine X boost fine anything you can think of like a Psy learn just use it we our system does not say you have to use one particular type of uh de learning framework or machine learning framework and uh it has building privacy uh preserving agance it's uh uh H and DP H is outut of box we basically handle almost every uh context generation for the h and uh during the training time we basically make sure your data your model when is sent from the client to the server it's been protected through the C uh H so your server side even it's received the model it cannot fully decrypt it because it has certain H protected information however the server still can do some arithmetic addition multiplication on all the models it collected so basically it generate a new model with h protected layers and it can send back to the client client has the information to decrepitate to the pl TX all the floting point is needs and the to do the training and for the uh DP we auto box support different type of filters SVT uh Gia noise and different and we if you need to choose your own or Implement your own uh you just need to add your python code very simple python code and change the configuration just one line then your system has that DP automatically included and it's guaranteed that uh when the client send out the model to the server it will definitely pass uh pass through that not pass it will pest that filter with that operations and when it come back there is no need to do the reverse so we basically have all the uh necessary parts for industry and our customers so you don't need to implement everything by your by yourself and uh one more thing about production ready is uh we think the security is very very important basically the server and the client and even the the user that submitted the job need a certificate and the private key every communication between the server and the client must be uh M TRS authenticated so it's not just one way TRS it's a m TS and we manage the certificate and the private key and we even can sign all the jobs we submitted to the system basically every information can be signed and encrypted at any participants this means the server and the client and we also have a lot of apis that include uh we can use the API to cury the system what's how the system is working how many jobs is currently running and what's the resource consumption on each node and we also support different type of workflows and agons workflow means if you want to have the FED average fine we have that one out of box if you want to cyclick basically one client train the result and send to the next client next client take it over and train again that kind of workflow that's fine we also have that one auto box you just change one line configuration is done and uh here is the GitHub location for the uh productivity in addition to the uh security side the certificate and the incubed communication we also Target our system not only to the real production level and we also Target before the production for example the researcher they want to do a lot of analysis on how they F your algorithm work like if they are using cyclic will cylic sub serve the purpose that all the model can converge to the same one if it's too Dynamic then it's not converged or maybe it's train result is too bad so we have a simulator simulator basically you have one single node you have the job configuration you just run it through one single command and it create those lines you want and start to run the algorithm exactly as you describe so it's basically try to run the same thing in your local machine with a little bit less resources but it's okay then when you are ready you think you're a researcher and then you think this is basically the one you want to deploy you can deploy to the entire system because you already verifi it work and the PC mode is basically the middle layer no the middle stage between simulator and the real production it's a little bit close to the uh real production so it will have a better per participants resource control so it will create a lot of doers each doers you can specify the resource requirements and it will try to see if it break some kind of resource limitation and uh uh we also support multic cloud basically if you want to run it in edure no problem in our uh SDK we basically generate those scripts for you you just run one single command start.sh D- Cloud Eder then one aeder instance will be launched and the system is running over there and you basically just can get the instance talk to the server or if that's the server your server is ready for other clients to conect connect and the dashboard basically uh it's a tool or application that allow the entire F study organizer to distrib distribute all the private Key C uh certificates and the star star ke means all the configurations informations easily because everybody can just sign up and uh put the information they would like to use and uh the organizer will just say okay I can approve this one and you can download it and the pre-fly check basically uh when you have the system uh installed you want to try if everything is okay it can talk to the server it can listen to the port you open basically it's a some tools for you to debug if or troubleshooting your entire environment so uh on the left on the right hand side that's the exam covid uh prediction basically this is the uh paper we published so it's a basically tell you you the any flare is not just a toy project inside a company that serve the only purpose of that company it can be used for the entire world and uh I don't know how many authors but it's a lot and uh there are different organization and they all use MV flare to do their uh Federal learning and it's very flexible and it's a very easy to adapt uh you can see some of them are not just uh research style some of them even has some type of uh solution provider style basically their customer want to have some feature for their Federal learning and the solution provider basically try to see what's the current eff uh solution fit their go and they usually find out for the flexibility for the productive productivity ready production ready and for other uh uh uh requirements this usually fit all the purpose they want and uh uh let's talk about the security because it's a when we have the data the model trend on one client usually that model S the data and the model basically canor the model basically can tell you about the the data is s so one problem is if the model just trained on one single data point and the data model send back to the server and the server has malicious user the server can just get the model do a gradient inversion and get data the the model uh get the data back so this is very important so we basically say we would like to enforce a rule that any model that is sent out must pass through uh must go through the filter that we established or defined by the uh project admin or researcher and we also have different authorization levels that means uh each user has different permission to perform different operations some user has a higher level of permission they can submit a job they can abort a job they can kill a job they can grab those information back but some of the user may just can see the log and the user those user cannot uh see other information and they cannot see other organizations information or they or even they cannot check other organizations system on the training result we also have a a site spe specific authentication this basically say each site when it's need to run a job that a job need to be authenticated the job actually come from the job submitter the job submitter belong to one specific organization it will embed its own signature into that job so if one client say I don't want to run any job from the other organization that site can say I will reject that job so it's very very detail on all the authentication and authorization and the secure client to client basically is I can submit a job I can I can upload a model to the server but the server has no decryption key because that model is encrypted with the target site public key so only the target sites private key can decrepit that information back so this is basically CLI to client any participants involved in the transmission cannot really see the model itself and this is basically the result we we analyzed and the researchers basically check the information and see is really work and the already mation supports those paper and uh get the uh similar or uh roughly same result okay for the customization uh lot of time the customer would like to use our system with their own infrastructure if some customer say we will have a specific communication mechanism maybe we do not we do not allow https we do not allow JPC we do not allow the direct ptcp connect connection they may say the only way to communicate between two nodes is through for example radi then lot of the current existing F framework could not support that immediately or outut of box if when people want to add that feature it's become very very hard because they they need to like remove a lot of codes and add a lot of one and become very very proprietary and very unique but in our case let's go to this one it's basically say we have different layers handle different uh responsibility on the very bottom one it's a very good example this is GPC driver HTP driver TCP driver s driver each driver expose same API and uh if people want to have the radius driver they can just implement the radius driver and all the uper layer will behave the same there is no change at all so we already have this type of implementation and uh project is basically get it very good result then we we are well joking that if some people want to implement email driver it's possible because you just need to make sure the API is the same then you can use email to do the F and uh this is part about if uh the researcher would like to use totally different deep learning framework and if they have some kind of other ideas about how to do the uh training because the data is not uh um parallel because most likely the the data the label are at the same location this data set point has the label at this location so they can train the model without problem however sometimes the label and the part of the data is in a different or feature is in different location and we need diff uh uh one location on some data part and the other location with label and other part other part of the data like a two head training so that's is a vertical training vertical f for that we also have the example in our uh GitHub repo so you can easily just grab that one and see oh it's quite simple just use this kind of configuration and with maybe two or three python codes you have that vertical e and the ex boost no problem s learn no problem AR be no problem so all of this are either out of box or we provide the examples and you can just easily try it and run it okay so that me all right uh thanks Isaac for introducing MV flare so um we have other than the features we do have all these other uh new features that's coming up um so how to convert a centralized uh ml or DL to an FL setting and then uh also to the stream of llm how do we support that and how do you as I show how do you train um let's say an L with prompt learning or uh P efficient fine-tuning how do you do that and they also support uh experiment tracking you can can send bad your Matrix your losses using you know ml flow weight and biases or manai and then also as the uh customizable or building components uh we do already have some of these implementations in our repo so swarm learning and vertical uh federa extra boost and graphic neuron networks uh and these other features also coming up as well all right so let's talk a bit about um how do you uh transform um your ml DL centralized uh application to FL so right now we have already implemented these use cases and then um so one is lightning and the second one is decorator third one is the general use case now right side you see these uh different apis so let's directly take a look so each of those cases it's actually um respond to how many code you need to change because um lightning um as some of you might know so it already has a predefined structure um like this trainer and then what people would usually do is the you know the black code um for for you um familiar with deep learning you define a model you define this data set and then you declare this trainer is a lightning specific and then you just call fit test and predict and then in order for this code to be run in an F setting just need to add these highlighted lines so you import our um client API and then you patch the trainer and then um because the data setup it might take uh a lot of time so we have this uh loop to wrap around so that you know each round you would just fetch new model here and then you just run this local training uh for each round and then let's take Federate average as an example and every round you finish the locally trained model weights will be sending back weights or weights difference you know after applying those filters it be sending back to the server to do aggregation and then finally get that Global model so that's that uh for linning you just need to import your patch trainer and then this validate it's uh it's validating the the model you receive how that perform on your local data set so this is not necessary but then we add it here and for the case of decorator that is if you already to formulate your training um structure or code like this you have a training method and then you have an Valu method you just need to import the client pi and then you initialize it and then you add this um decorator and then once you add this decorator um your first argument will we're going to replace it with the global model when you call it and then in the end you just need to uh specify you need construct this object that you want to send this back to the um server site to do the you know aggregation or whatever algorithm you choose uh in the typical case people just gather all these model difference model weights from different clients uh and then in the server they do the aggregation so that they can uh they do the average of all those weights so they decide which step to take for the Deep learning application and then for evaluate you just add this decorator and then um you need to have this argument and then we will just um replace this behind uh with the global model for each round and then you will use that Global model to do your evaluation so you see here um while this is running you just get the glob model and then you evaluate on it and then you train on it and then it it'll it'll uh on the server side if you configure the workflow it'll just run uh the in the Federate uh setting and for the general DL use case uh you'll see that if you know you are not using uh lightning you're not format your uh training code in in certain function you can just do it as flat um code or simple code as you want you just declare this model here right so you uh we import the client API and then we do the net here and then again we wrap it in this while loop and this is not necessary is only if uh if your data setup takes a lot of time then you do this and then you receive the model you see um this one way you just get the global model parameters you load it to your your model and then you do the normal training and then in the end you compose this uh um object we want you to send back um using your trained model weights you send it back right so that's that for the ml2 DL conversion and we also uh talk AIT about the the example that we have using uh Nemo which is indas uh NLP Library it provides some pre-trained models and then u a predefined training workflow for LM as well so and this slides we're going to talk about the how do you do prompt uh tuning so um we we have this pre-trained large language model and then for promp tuning what we do is that we add this prompt encoder um and to learn additional uh prompts that we want uh for this model and then so for this prompt tuning uh in Federated learning scenario what we do is that each client they'll have their own data and then these L are come from pre-trained and what what what we would do is that each of them will have this promp encoder and then on the server side we just get the weights or the weights difference of these promp encoder and then average them um to get a new weights and then it just goes itly like this um we also have this other example just to Showcase that you can do you can do things like this so uh other than prom tuning you can also do uh supervise fine tune with your large model you can just fine tune the whole uh 1.3b uh model and then these are just different data sets and then we just showcase that um it actually work and then these are different Matrix um we have these uh H swag pqu why not these scores and then we we already published it in our report so you can check it out there and then we have more examples that are using different algorithm we already have fat average fed proc fed opt with different data sets you can just check them out there and then we can do some Demos in the end very fast we okay so that's just to show very fast okay this is one basically it's a uh uh okay sorry maybe okay maybe we just show this one this is a simulator it's on list Mech so basically you can just run it and the same job you can just submit it so it's a live demo it install any flare and uh round the example basically you just got the result so on your MacBook you can start to try it okay thank you