Scale By The Bay 2020: Aditya Sakhuja, NLP text recommender system journey to automated training...
[Music] all right hi everyone my name is aditya as nicola just mentioned um i and i work for salesforce i've been there for four plus years all right so today we are going to talk about uh the nlp text recommended system we built one of the many machine learning systems we have worked on and we'll go through the journey from like right from the beginning till where it is right now and that will hopefully bring in uh useful uh lessons for for folks listening in all right so the agenda so we'll talk about the problem space to begin with talk about give a bit of a background on the the enterprise problem we are solving then we'll talk about the approach we took and the metrics we care about for the approach then we'll dive into the various uh evaluation phases so we'll talk about how we started small and moved on to something which we have right now uh then we'll jump into the architecture talking about the various phases serving future engineering and training finally we'll talk about the deployment uh rollbacks uh for our model and then finally wrap it up with challenges and takeaways hopefully summing it up and something useful actionable for the audience right so the goal uh so the goal we have is to assist the customer service agents with useful answers which they can use to help the consumers who have the problems so that's the uh maybe the high level goal we are trying to achieve in a simple way right and uh the scenario the scenario we started in was that the agents who are trying to assess the customers they have access to uh for example search the traditional text search which is uh uh heavily relying on the actual text not really a lot of semantics behind it but it's purely relying on the text of the question to find the answers for the agents uh the problem is that the these questions are normally very long because they they are explaining a problem uh which the customer has about the product and they are generally not easily searchable as it is so that is the problem we started with uh so we obviously like uh needed to fill this gap and um we came with a solution so first of all what's the business metrics uh we are trying to improve right like uh first that that should be talked about before we start talking about the approach so the business metric is that we want to the the first one is uh we want to reduce the time to resolution right for the agent so the earlier the faster they're able to resolve a case they can go on to solve the next one at the same time the customer satisfaction everything goes up so that's the key business metric then we also have a soft objective function which is uh transferable into our system which is the case to article attach right so article is a representation of an answer and the attach rate is a good representation of how how close the article is representing the problem or solving or helping solve the problem which the customer has then we also want to uh track the time spent per case even though there might not be a resolution we want to uh improve on the time spent so we want to ensure that the agent is able to move on to a next one if uh if if they are not able to completely resolve the case but they are able to increase the throughput so that's another key metric right then there's secondary metrics as well so some of the secondary metrics are a number of recommendations serve which is a measure of scale then the monthly active organizations because being an enterprise company salesforce has companies running on them on us and so the platform we support has like thousands of organizations right and we want to keep a track on how many of these organizations are on boarding with us and every organization organization has a lot of users and users are agents and administrators and other other profile users so we want to also track how many of those are actually using the system so typically for us it's we care about the agents like that's like the those are the central users and then the serving latency we want to keep that low because that's very key to user satisfaction and helping the time to resolution as well actually all right so yeah guess what we built the recommended system no surprises we had that in the title as well so what's the architecture like right i mean what is a high level flow so you can see that we have a corpus which is uh in our use case right now this specific product we are tackling we are caring about the knowledge articles specifically so we consume the knowledge articles uh into our pipeline we go through the first phase which is targeting recall and the goal of this first phase kind of generation is to have get get enough candidates for the ranking layer to rank on so so from hundred thousands of articles we come down to a smaller set which is uh of the order of thousands and manageable by the ranking layer and finally the ranking layer which is uh which is a precision layer uh generate scores the scores the candidates and uh we get the scored results to be rendered all right so yeah as as i was mentioning in the agenda right we'll go through the evolution quickly just to give you a sense of how it came to where it is uh so the version zero uh was obviously without a model uh we wanted to build a proof of concept and kind of really show that this is something um which will which will add add value so luckily like the use case was uh as i mentioned like was very clear that we are trying to reduce the time to resolution um and uh the the customer base was already there so we wanted to first showcase that okay if you build something like this recommendation system what will be the experience like so that was the first cut we started we definitely pushed on getting our pilots uh onboarded because they were more like the pilots as in the beta customers and they were like more like partners right the first few are always your partners and um that's what we also focused on to learn from their experience and have a very continuous feedback coming in uh the first cut was built for communities use case where it's like a social network so questions are coming in there are posts which have some of them are questions some of them are plain statements uh but there are a lot of there's a lot of repetition there so there was a scope for using a system which could uh reply back automatically and reduce the customer or rather improve the customer experience by giving back a known answer instead of someone a human trying to respond back every time right so yeah so that was the first cut we definitely started thinking about the modeling as well and we relied on the stanford score data data set which was which is a very popular question answering data set then coming to version one we focus started focusing more on the ranking model itself uh the second part of the flow where we wanted to see that okay how can we improve on the precision that is where uh uh the value addition uh was on top of getting results back from our candidate generator so uh we we focus on getting a ranking model which uh did pairwise probability scores on the question and answer which was like the case and the article pair we did the model training in notebook notebook based on on-demand training was done uh which was like specifically for the initial pilots i was talking about and then we had static configuration right the data set for uh one thing to highlight is uh salesforce has very uh very complex data schema and uh the customers that's that's why scale source is so popular as well because it lets the customer really uh kind of adapt to the data model to their business use case but then that also has complexities for us to uh know where their data lies right really what projection and selection they really have for um uh where they're they're going to store the data so initially we didn't kind of tackle that we had a static configuration for uh like the default fields we we used to to get the data in and then we started thinking about the observability as well so the logging and metrics we wanted to get that in early so yeah in version one there were hints of these things coming in so in version two this is like more maybe like we can go with version two three but i've just summed it up in version two where some of the key pieces were which we were working on earlier they were built and we started uh supporting dynamic configuration for the customer to specify the selection and projection through a set of flow um we we also invested heavily in the model retraining aspect and the model management as we will see in the coming diagrams we'll see that model management and retailing is very very key to our success because uh it's um it's not that we are training just one model we are training multiple models and it's being a multi-tenant platform right um and then um we also like went on to support uh different languages to support our european customers beyond english and uh yeah that also was a major milestone i would say uh then yeah uh model deployment and uh with rollback so we we also invested heavily in our the the devops cycle so that it's not that we are every time we want to make a training code change we have to invest a lot of time so we kept that in mind and we have now come to a point where our rollbacks and deployments are pretty stable all right so let's talk about the architecture now a bit overview of the of the two parts like the serving side and the training side right so let's start with the serving so for us um at salesforce right the first let's let's see who the users are really right at the bottom right you can see we have the customer which is the end user like us who who will go to the support system and they will create a case directly or they will probably be chatting with an agent so that is where this channels comes in there multiple channels it could be chat voice it could be email or it could be directly creating a case so so once a case ultimately is created coming in through either of the channels that comes into our back end it is fanned out and we have our message queue system to uh to scale things up and uh then on the other side the handler starts triggering the answer engine or the or the recommendation system which we uh talked about earlier right so the two stages were the kind of generation and then the ranking right so the case handler will uh be doing it for thousands of cases parity the answer engine is okay the the kind of generation is kicked in it will talk to the search index which is one of the key sources for us to get back the candidates so in candidate generation what we do is we can we extract the key terms from the incoming question so the question is like a huge blob of text right it could be a very long paragraph for all you all you know and we want to formulate a query which can be understood by the candidate generator so one of the kind of generators is search and we have a our own custom rule-based technique to uh formulate multiple queries which will ensure we get our target number of candidates so we do that here in step number four uh and then finally five is the generation part which uh which is consumed by the ranking and the cutoff uh tier and there the ranking uh the ranking or the mod the ranking model kicks in it basically will uh do pairwise scoring of the incoming question and the articles which are given by the candidate generator right so it will do pay by scoring and then the uh the paid wise score is finally available to the agent so we pick up the top k uh top k recommendations back to the agent and this is happening in real time in a way right the customers uh have creating asking talking here asking questions and we are recommending uh the the the results or the answers the assistance is coming to the agent in real time so yeah so this is the main flow but let's not forget other personas so we have uh the knowledge base admin here and then the org admin here so these two are uh important personas as well so they are they are also users of salesforce platform right and their role is to manage the knowledge base for example so they are the administrators of the of the salesforce or they are using so they will uh create the initial corpus they will also maybe um like do the language translation in kick off that process those kind of things are done by the admin and then the org admin uh is the persona where uh who is responsible for managing the the product we are building right the recommendation system so they have access to to the setup flow and that is where they specify uh what uh what data set what uh view of the data is important for training so that's the job of the argument and only the organ men would really know about the r because every org has their own requirements all right so and then um last but not the least we have some more tools out here like the metrics ui is very key to to give feedback to the user in our case the art admin to know about how well the whole system is doing for them whether it's adding value or not so that's where we showcase the data metrics the the model metrics uh in a format which is consumable by them and then last but not least the salesforce internal user is also uh having access to certain tools for troubleshooting our own system right so uh any any distributed system especially a machine learning system is bound to uh have us need for troubleshooting at points and we have internal tools which lets us in a very safe way uh uh uh help resolve the problems all right so the we talked briefly about these two layers right so just to highlight and summarize it so the first layer is doing the key extraction based on nlp techniques so we use uh uh libraries extern open source libraries at the same time we have custom trained models to do the nlp key term extraction uh for part of speech extraction named entities noun adjectives those are some of the aspects we care about from the question uh finally the query formulation kicks in and we generate the the query which will be understandable by the candidate generator in the ranking layer as we talked about we do the pairwise feature generation and that is where we uh ultimately uh get the scores right and we rank the probability scores in order and uh whichever is above the threshold that is what gets recommended talking about the other side right we talked about the serving now and let's let's quickly look at the training pipeline uh in the training pipeline the first part is the feature engineering right so feature feature engineering data preparation comes under the same umbrella um and the first part of that also is the injection part so we have to ingest the data into a place where the training and the future engineering can happen so typically in a consumer world those the same system could act like the same database could be suitable for uh doing feature engineering but in in a enterprise setup we cannot really um have um uh the same runtime system uh also we used for feature engine we have to move it to a place where it can be we can do a further analysis and kind of run a pipeline on it so we do that first step as automated step and that's uh moving the data from our system of record into a data lake um and then we have uh quality checks after that once the data is moved over and then we do the feature engineering we'll talk about this briefly again when we look at the diagram all right and then like after the feature engineering is done then the model training uh kicks in the the ranking model will be basically trained and we use auto-tuned hyper parameters uh that is basically uh with grid search on we can auto-tune the hyper parameters and if that is not on then we rely on the default hyper parameters which we have configured at uh from the runtime yeah and then we also do model comparison because since we have uh automated pipeline now we want to have a system where the models are uh not just the the latest model is not always the one which is sent back uh to the serving stack we want to compare and find the best version uh there's always a chance that there's some regression and the latest trained model might not be the best performing one so we do rely on um evaluation metrics like the roc auc pr auc and then the other metrics like f score precision recall uh hit rate is at k is also used for accuracy measure and we compute this across all the cases cool so um the hyper parameter tuning as i was saying so this is a just an example of some of the parameters we use for uh which are consumed at the time of training and that kind of controls how the model will be trained for example uh the grid search on if it is true then we kind of have a auto tuned system uh then these parameters are not used um uh the the uh then uh then we have the sample size yes the sample size random sample size is another uh parameter we use to define how many random samples we are going to generate for the negative labels so um as we were looking earlier right we we consumed in just the data but the data we are ingesting is more positive samples like we are the data we consume in is more representing the data which the customer uh has given a thumbs up on so some of the articles which they have attached to a case are positive labels we also want negative labels and for that we do some uh uh techniques on our pipeline where we randomly generate some of the negative samples and this is one of the parameters which control that then we have uh the maximum number of iterations we want at the training time and and some other metrics are there as well okay so this this diagram is uh uh is giving you an overview of the whole flow uh and some of the things i just talked about so you can see like on the on the extreme left swim lane uh we we have the multi um uh multi-tenant platform aspect where the orgs the customer orgs they are of all size and size and shapes right and we want to support all of them uh cover all the corner cases as we start the training pipeline once the auto injection happens we get we we focus on the on the data entities we we are going to use for the training flow uh some of them are the case the case themselves the articles uh the attaches which is representing the positive labels then we have the corpus stats which is representing the uh uh some of the some of the statistics about the article corpus itself we do some precomputation for it and then the engram stats as well which is which is precomputed which represents uh the the unique diagram overlaps from between the cases and the articles so some of the some of the stats are computed here and some of the frequencies are precomputed which which are which is going to be used in the next phase basically in the future engineering and then eventually training as well in the data prep and feature engineering uh we go through some of the well-known steps um uh the like we do the initial uh uh for missing missing data some of the missing missing values we do data computation here filtering is done then eventually we have to do the quality checks as well to make sure that the after all the all the missing values are filled up uh null values are dropped we we we still are reaching a minimum minimum bar for the training to to happen it's it's very much possible that after when the organization data is ingested we reach a point where we can no longer train the model because either they don't meet the quality bar or um or they just don't have enough data so that's like a cold start problem we have to talk about as well then we um yeah so then we do the nlp feature generation uh so this is where we uh use the pre-computed data we uh the engram stats the purpose stats and uh we we do an overlap between the case uh the case and the article sections like the description title the body of the article things like that so that is that those are the key aspects which are um used to compute the overlap uh once we have the basic set of features then we do feature crossing uh to expand on the feature set then finally we do the the feature selection to make sure we we don't have features uh in the system which are not adding value all right and then um on the learning side uh on the model training side the feature weights are learned once once the future engineering stage is done then we get on to uh learning the feature weights the model is validated using the validation set and then finally we do the model comparison which is then eventually sent over to the serving stack so on the bottom bottom you can of the diagram you can see there is a training loop flow which is representing that we actually have a system which is running automatically we don't we don't have any manual intervention in a normal scenario and now the the data is ingested uh once the model is trained and it is it is again it is serving uh recommendations um though there is a state driven training orchestration which uh which basically will uh which will basically invoke back the training loop and the same process happens again for each arc and this also happens for new orgs which is coming in so the same orchestration flow is also used for uh kicking off the training pipeline cool so and then finally the service agents are able to get the recommendations from the app how much time we have okay i think we are good on time uh then let's talk about the model deployment uh part right so there are no surprises here on the first part of the of the flow uh this is a typical um ci pipeline where we uh which we actually we didn't have and to be in the beginning we didn't have this we invested specifically to improve our uh our developer throughput and also um ensure that the deployments are stable so as you can see once the training code is updated by the developer or the scientist then that gets uh we do a git commit push and that gets bundled in the training code gets bundled uh as a binary that gets bundled as an image and that gets pushed to the container registry so that happens uh this is the automatic flow we have here and then on the on the bottom row you can see the dev the devops we still have a pseudo automated flow for the devops to ensure there are whatever approvals we need uh to ensure the quality of the deployment that is uh supported and the the devops uh kicks off the deployment process and we have a rollback mechanism where if uh in the test environment we find any issues once the sanity tests are run we are able to very easily do a roll back and deploy the previous image and if there are no issues then obviously we our deployment cycle moves on to deploy the in the production environment cool um zooming in a bit into the uh into the deployment itself so we have a cloud native uh approach where we ensure our container the uh the container which is used for the deployments that is uh that we can very easily move uh to a different cloud for all you know and this is also uh very easily tested on a developer box right so it makes it very simple for us to test things locally deploy it and do it multiple times a day um some of the parameters we consume here are the hyperparents for the for controlling the training itself uh the data sets which which are used for training on and then some of the configs which are used for housekeeping in the container once this training is done the model is generated it is pushed onto a storage bucket which is uh which is which is a well well known bucket for the system and it is um it is hosted behind the model api for the serving system to pick up from cool and then let's talk about the challenges and the takeaways uh that is where somehow it's a summary of maybe not everything is captured here but it's still a decent list of uh things we we we have to take care uh in our while we why we build this system and also the we'll talk about the takeaways next so for us uh like uh we have to deal with uh non-standard data shapes right like it's not that the the incoming data is different for every customer as we i was hinting earlier and we have to keep the system flexible enough to take care of all corner scenarios and when i say custom data fields uh so salesforce lets the customer define their own fields and the data types as well so that makes it a bit tricky for us to um handle that in our training pipeline but uh obviously like we the corpus is limited and we have our checks and balances in the training pipeline to take care of this um then the model management this is also very specific to i think our enterprise setup and for us because uh we are not just training one model or a couple of models we are training uh hundreds of models and that is happening in an automated fashion and we also take care of um retraining them as data drifts so that's another key challenge and i think this might be common to many enterprise setup companies um then silo data sets we we also have the data sets we don't cross boundaries so that that's something which has its own downsides uh at the same time the the security aspect is also another uh thing tied to the silo data sets so which uh which i have covered here under data so we have to keep keep an eye on the compliances encrypted data sets um and then the quality of the data itself if it is too sparse as i was mentioning in the diagram earlier if it is too sparse then it might not even be ready for training uh so that's another problem uh freshness is something we we have to ensure that the data is hydrated continuously and not let uh training serving skew happen as a result of that um because the serving stack the serving side can very easily be having a very different data shape compared to what the model was trained on so that can very easily be a problem which can occur if one is not uh careful on retraining the models uh with the latest um yeah and and also like if the data is too dense right like uh it it for us we have to make sure that we are uh sitting within the sla bounds and also the cost bounce so that's that's another key key thing we have to take care of and then finally the cold start problem we talked about the training serving skew that can happen because of um like if the libraries are not shared across the serving in the training side uh if there's a feedback loop for example from the model to back to the algorithm or or like the data shape is drifting so that's these are some of the reasons why this cue can happen and then the goal start problem where um the data is the company doesn't have the data so we still have to uh have a way for them to get started and we solve that by providing a default model which uh which is not the best performing model for them but it will be something which they can bootstrap on so yeah finally to sum it up uh some of the takeaways we learned and i'm sharing here is that yeah it rate no surprises there build build small like we started with the poc so that's uh that's um that's an example of uh how one can it trade start with something which works and then keep adding complexity uh also like the model interpretability is super important especially in the beginning when you are also trying to understand um whether uh understand the data and understand the use case in depth so starting simple interpretable models like logistic iteration for example random forest those are great starters for uh some of the problems uh like recommendation system as well and then yeah prioritize the infrastructure because that the the ml tech debt paper kind of talks a lot about it and uh yeah if it can easily build up and and bite you by you if it's not taken care uh from the day one and first a few beta customers very important because they are like your partners and they are helping you improve the system um so that's very important we want to uh make sure that they are super successful and then the use cases they have are taken care of because majority of the use cases will be similar uh if not exactly the same so yeah that's this is super important as well and then uh scale the model uh to the size of the data like if your data set is small then um maybe deep learning is not the best solution to begin with so yeah something to highlight there and finally data privacy is super important like even if you are getting a minor improvement on quality uh i think there is no no reason to sacrifice the privacy of the data cool yeah i hope i didn't overshoot any questions [Music] you