Scale By The Bay 2021 : Alexander O'Connor, Transformers End to End Experience training...
thank you very much mike hi everyone and welcome to the session um uh it's a great pleasure to speak at this conference and this is an opportunity for me to update some of the experiences we've had with transformers which we've been using now for a couple of years with a great success so uh just to start off briefly to talk a little bit about who autodesk are so we're a 40 year old software company that is based in the san francisco bay area as our hq you may have heard some of our heard of some of our most uh um well-known products include things like revit autocad fusion 360 maya so you know across the whole domain of professional uh design and um indeed some hobbyist design development manufacturing and media and entertainment uh our software is the support that provides uh the bridge between the idea in the creative's head and the uh realization of that thing be it virtual digital or physical and very often you'll find that a large any number of projects that produce something often involve our software and that creates a very interesting situation from uh from uh from our team's perspective so the team i'm on are focused on uh the e-commerce and on the customer empowerment side which is to say that we look at how we can assist users in uh solving problems that they may have to resolve issues with the using the software or to which matters an awful lot because many of our users have spent much of their careers with our software and they know it extremely well so when they have a problem that they need to get solved it's pressing because it is their job and it could it could be a major project that's on the line and so it's important for us to be able to deploy the best possible software and the most advanced techniques to help get those solutions to the um to their resolution as quickly as possible similarly when it comes to making it easier for customers to uh execute transactions as they grow their company or expand their expertise we're you know being able to make uh better use of the whole digital infrastructural life cycle of our product ecosystem is a part of that range and so today i want to talk about that which is the experience that we've had using transformers as a way of facilitating the improvement in the experience for our customers all around our domain so i feel a little bit like this is probably a slide that uh when i first talked about transformers i had to introduce them in a very concrete way i think most people would not have heard of them and would not know what i was talking about when i spoke of them but nowadays i think it's very apparent that uh transformers are kind of a central part of the way people think about not just text machine learning but a wide variety of other applications in essence you can think about transformers as being applicable across a wide variety of tasks uh text classification being one of the most common ones language generation is probably the the highlight in terms of the most eye-catching one and then translation is also an area where transformers have had extremely high value and you know in all of those things you can see the same kind of components too which is that transformers provide increased flexibility they allow for a much wider array of success and they allow the models to take better account of what's going on in the uh in the in the language in a way that is more subtle than previous techniques uh have allowed for they are in fact you know as you think about it pretty much state-of-the-art in almost all nlp applications at this point they uh you know the underlying technology allows a wide variety of different uh approaches to training and to um you know greater flexibility in the type of data that you may have and the type of facility that you have so how does a very broadly a transformer work what's different about it well a transformer kind of thinks about uh language in a different way to previous models which were largely sequential or bag of words in their thoughts so either you looked at individual tokens one at a time or you looked at a sequence of characters or words in the order in which they appeared in the sentence and if you think about it humans don't probably do that it seems like intuitively at least or when i think about how i understand language i kind of encode the sentence so i listen to the sentence in a run-line way but when i get to the end of it i listen to the whole thing again and replay it you can think about all of the ways that puns work as being evidence that really you look forward and backward across all the different tokens before you come to a conclusion about the meaning of a sentence and compositionality is that sense that you know the meaning of the words beside each other all define the overall context and sentence and context and meaning in the sentence and transformers attempt to do something similar by looking at the position the uh uh the relative or absolute position of the token along with the um the uh um the uh neighborhood of the token when it comes to a particular meaning and so the other key thing about transformers that's important is that we don't come at this from scratch so you know transfer learning which has transformed is equally important in the in the change in the way we think about machine learning in general and coming from our colleagues in the vision domain has also transformed the way we think about how we do build these models so instead of starting from scratch one of the big challenges with these transformers is to get them to learn enough to do something effective they need a large amount of data and so thankfully uh you know this has come about in parallel this idea that you know instead of trying to just train everything from scratch and build your model from the from the from the ground up what you will do is someone will train a large language model on a general set of objectives on this multi-task learning and that can be uh token guessing it can be a wide variety of different uh objectives depending on what you're kind of building and then we transfer so we either take that model overall or we um we kind of fine-tune that model we we kind of make the model more expert on a specific area a domain a specific area before we use it so essentially the key the three key thing takeaways from transformers are one we look at multiple words at the same time or or windows of words together two we only use effectively that attention model as a way to to structure the model so we don't have our current structure and we don't have convolutions in there and three we take advantage of the fact that we have trained a large model and then repurposed it rather than trying to build it out in one go so the dominant kind of uh technological uh feature of um of the uh current world of transformers as well is the is the is the is the um use of the hugging face library and the hugging face team and i believe uh hugging face the ceo was speaking not long ago at this conference so i will uh hopefully not overlap too much on what he said or certainly not disagree with it but and the key point about this is that you know in transformers we've seen an unprecedented degree of sharing by large and small organizations in their ability to to concretely make models code and um architectures available for people's use throughout the uh the uh the community and so this has been one of the most wonderful experiences it's been the open source and open nature of the engagement the ability for for uh us to get from a research paper to a model to implement it in a in a business process has been extraordinarily fast and the amount of learning and value that's been going on has been incredible and so it's fair to say that transformers are reading the world much as software ate the world before and i think we will continue to see a value in the growth of these kinds of technologies as time goes on and we're only seeing them scale up and we're only seeing more creative and more fascinating uses for them and more techniques for making them available to everyone as we go forward and i think from my perspective this is a very exciting time because the technology has matured and has begun to take on another layer now a second leg where we're beginning to see it so we've been deploying transformers for nearly three years now and we would like you know we have some background on uh what we want to do so in my specific domain what would i like to be able to talk to you about today there are as i said lots of different ways to use transformers and ever growing every day and i think one of the ones that i want to highlight is the one that i think is often under um is often under emphasized in the uh in the kind of academic research area for very good reasons and also is probably one of the less exciting ones from the perspective of what you might do to get a lot of you know exciting social media likes this is not about generating um this is not about generating new text this is not about kind of image to multi-modal kind of work what we're really talking about here is a key thing that is very important to a lot of businesses and very industrially relevant which is i have some business process that i'm trying to automate or to accelerate and you know this can be in my in our own examples we've had great success with for example the question about how to root specific um domain specific questions about uh products to correct to the correct support team and that sounds like a kind of a flat or slightly boring or maybe even to some in some mind solved problem but you'd be surprised at the variety with which customers will phrase their problem and the number of ways in which you can have to deal with the overlapping and noisy data that you get from a real business process so it's important to think about the fact that while this isn't the most um futuristic or science fictiony kind of application area i think it's one that demonstrates the real value of um machine learning with especially with transformers and demonstrates the sort of practical experience that you get along the way and my goal with this talk is to talk to you about all the ways you can use transformers to solve parts of these problems and hopefully along the way also eliminate you with some of the techniques and methods that we used um when you talk about these sorts of things in an ideal world you would just say oh well look we'll just get everyone to tag all the data and it'll be we'll go back forward back and forth through the data and it'll be easy or we'll hire an external tagging company to do these things very often with these kinds of business processes there are two challenges there one is that the the process is designed to be efficient and so anything that involves additional work such as metadata tagging such as adding work to it often is costly and therefore will be noisy things move methodology shift organizations change and trying to do this with an external vendor it can be difficult because very often these are not intrinsic kind of differences like the reason you go to team a versus team b is different in one company to another and doesn't necessarily entirely depend on a logical kind of externally valid uh hierarchy of of conceptual differences and so there's this arbitrary nature to the way we're grouping these things that is very flexibly very flexibly uh accommodated by a a neural network which would kind of expand the manifold around multiple distinctive clusters but which does nonetheless have this challenge that em you need to generate the data for and lastly user text is always difficult because people type um with a lot of typos they type with all sorts of uh descriptive challenges and very often what you're trying to do is they don't have that kind of convenient thing of having the answer encoded in the question instead they genuinely don't know how to solve their problem and so we have to think about a little bit about how we're going to deal with that along the way as well so there are three stages that i want to talk about today the first is how to handle the data um looking at this thing about data cleaning labeling managing the noise and so on and then the second one is about training which is you know some of our experience with training models and some of the interesting things that have come out of training and best practices that have come there and finally i'll talk about our experience with inference where we had some uh we've tried a few different approaches and a few different techniques and some of the things that we thought about along the way and what i've tried to do is uh in this talk i have mostly reserve mostly tried to each give you an example or some code or some pointer that that i can give you in the direction of a methodology or an approach or some path along the way so if you yourself are faced with this challenge of developing a supervised learning environment or indeed something similar but not quite the same then hopefully in the process you'll be able to leverage some of the uh some of these tools uh uh on the process i should also say that while uh this is a particular example of these things we don't necessarily as a company endorse any particular approach these are just some of our experiences and mostly this is my personal view and my view as a as part of the team so data cleaning is why data scientists drink i think that's easiest point to think about along the way here is that you know we talk about as a rule of thumb spending four-fifths of our time trying to disentangle what's already been marked up we're trying to find out why a piece of data is in the format that it's in and trying to label out what's going on along the way and you know it's fascinating to look at how you look how you think about a business process you kind of see a very convenient and very nicely developed um flow diagram or a kind of set of documentation that describes a bit of behavior or even a process that seems very nice very tightly wound and when you look into the detail of it there are a couple of things that go on in practice one is for example that you can have the challenge that you know the misalignment of incentives is very strong so in the customer support domain one thing that you find very often is agents will go the extra mile to help their customers and that can be great but it also undermines your process because now for some reason one team is able to solve a problem unexpectedly vis-a-vis their um their ontology or their their taxonomy and the reason for that is that they don't want to transfer the customer because why would you if you can solve it for them and moreover that's a better experience for everyone so you know the challenges of kind of latently saying oh it's almost certainly the case that we will just uh track where things go and get all the data out of it never really works along the way so very often what you're presented with is a pile of data some historical record of some of the behaviors that have gone on along the way and you need to think about okay how do we extract from this uh the the uh the distinctive attributes and the clusters of distinctive behavior that we can use to then get us to something that we can build train a machine learning model from and begin to automate the decisions and begin to cover the three types of problem that you need to think about the first one is the model has to be right when the when the happy path is engaged in so when the customer goes to the right place and clicks the right thing the model needs to also guess that when the customer goes to the wrong thing we need to correct their uh behavior and thirdly when there's a mystery when there's some other exception to the the system we need to be able to understand what those are and prioritize whether or not we want to be able to to do that or be able to just simply pull things out along the way so given that open pile of data and that open pile of stuff what are some of the modern techniques that we can use to try and address this problem and some of these i wish i'd had three years ago because we did an awful lot of manual tagging and manual process engineering to do this and these days that's a much more it's much more of a tool-assisted approach now so the first one i want to talk about is probably one of the most exciting things that we've seen in the way that machine learning with large language models has moved forward and this is the concept of zero shot classification and in a zero shot domain what we say very simply is hey you know we have built this model it's several hundred million to several billion uh parameters wide or large however you want to describe it it has a huge amount of encoded information in it not just at a low level not just at the simple level of um you know understanding the um i use the word understanding loosely here but not in just in the sense of having a sense of the proximate order of tokens but actually those things emerge from those things emerge much more complex patterns that can be dealt with uh by the um can be used for the purposes of doing things so before we even get into training the model on our own specific way and um before we do that sort of thing why don't we just see what is already known what is already present within the model what it cut in for a long uh in the process and so what i really like about this is in this case uh we're talking about the zero shot classification using the um default uh transformer model and underlying this the way that this kind of works is that it takes advantage of the fact that the um models can score two sentences and decide whether or not they uh are similar enough or they they they entail with each other so for example i have my three sequences here are you looking for social media followers i'd like to purchase 10 units of your product and my product ordered on the june the 10th has not arrived yet can you help and i supply two things to the model i supply a set of sequences and i supply uh uh the um the the um the potential candidate labels that i'm interested in and at the top you can see here we have this hypothesis and what we do is we say to the model and we give it the sequence we give it the hypothesis with one of the labels in it and we ask the model how much do those based on everything that the model has seen in all of the thousands and hundreds of thousands of documents that it's seen before how likely is it that these encode with each other and so you can see that for example what you would have here is you could see you would say are you looking for social media followers and the model queries it says this is a sale this is a order support or this is a promotional um message and the model will then without being prompted in any specific training way give you back this score for these things and i've taken these examples and this is the hugging face code for this from largely from their website i've simply changed the examples and if you run this code you will get back correct labels for this in practice when we use this we find it to be a great way to start down the path of thinking about how we might detect clusters in the models uh excuse me one of the challenges or one of the biggest challenges here is you've kind of got yourself it's a bit like the kind of that old adage about you know when you start using regular expressions now you have two problems the same thing can happen with zero shot learning which is the choice of candidate labels is essential to how you uh begin to develop effective zero shot labeling and it's not at all obvious what types of labels and what types of hype and even if you want to change the hypothesis what types of hypothesis will give you the best kind of behavior and so um in my example here i wanted originally to look at spam and not spam as a kind of example and i couldn't find a good combination of examples that work for that so that's one challenge is that choosing the candidate labels appropriately can be difficult and can lead to some unexpected behavior the second challenge that you have here is that um these things need to have been seen and they need to be kind of intrinsic to what the model has seen in terms of categories so it works extremely well where you're trying to do very broad kind of categorizations of the of the kind of brown corpus type if i want to learn the difference between sports and news and technical data i might be able to do that pretty easily or science fiction it's very difficult for it to learn about things that are very fine-tuned differences or very subtle differences in high context because it just hasn't seen that stuff it hasn't learned in that way to distinguish those tokens so i think i'm very excited by how we've used zero shots classification as a first step towards these things but it's really just another signal and another piece of noisy data and so my recommendation is try it out but think about how you can use it laterally to the rest of the data that you have as a supplementary rather than a confounding value and that leads to the second thing that i think is interesting in data preparation which is the move towards using weak weak supervision so in this case we're going to accept the idea that that labels are uh difficult to use we're going to accept that we're never going to get a completely reliable set of data and in fact even if we do a lot of hand labeling by very committed expert users we're still going to have some level of error in the model and so what we can do is we can say let's combine a whole different set of rules of thumb if it mentions a particular token if it's if it has these keywords in it if it's of this length any number of rules that you can think of along with the output from different models previous information from the from the system as a way and then attempt to learn some sort of a surface some sort of a model that will then label that data for us and the the nice thing about this is this allows us to attack the problem of um getting really high quality data data centric uh training of the the models without having to uh without having to be perfect the other thing you can do at week supervision that i think is very powerful is that you can cheat so you can use post-hoc knowledge so for example you can look at what happened at the end of something use that that data to train backwards and use it as a as a label as well so because we can look at only for example cases that have closed or only look at sales that have completed and then use all of that data but the input is only going to be the piece that you get at inference time as well so it allows you to take out that post technology take advantage of it along the way and take advantage of all the active and interactive learning uh platforms that are out there so one example of this is the flying squid platform from hazy research who previously had built snorkel and uh i think that that's a very interesting thing to combine and now you have this opportunity to build things where your data set is now something that's flowing and you stop thinking about data sets as this diacronic slice that freezes in time and has spent a lot of money developing and instead it becomes a flow of data that's continuously being added so training is the next step that we want to think about when we come to training a model for supervision what we think about is this idea that what we want to do is uh we want to address the fact that um we want to take the model and make it very specific at autodesk we have this all the time because we deal with a huge amount of technical data and we use entity focused uh and we often use entity focused um uh type data so there's a lot of talking about specific functions specific uh words in a very specific context so the meaning of line the meaning of wall the meaning of those sorts of things can be very importantly different whether you're talking in a manufacturing or an architectural context so i'll refer to my colleague m c hammer who tweeted this a year ago now you know the difference between one-shot learning and zero shot learning and in fact all of the few shot concepts is very interesting because with these large language models it is a kind of an interesting open question as to understand how much are you changing that base model when you add the fine tuning on top of it and what how much is it dependent on the underlying model for it to be able to be able to recognize things and i think those are essential things to understand there's only so much changing you can do as you can kind of build up the model and how as you kind of build up the the the the training so i think very importantly when you're thinking about these sorts of things there's a lot of hyper hyper parameter tuning and stuff that can be thought of but very much the first step that i think is often to be thought about is actually just to think about the base model in a very in a very specific way and think about what the base model might contain that might help you move forward and so uh you know taking that into account there are tens of thousands of models available on uh the the hugging face repository all of them available to be downloaded them and it may be that one or more of them is the first place that you might want to address trying to change how your model behaves you know before you get into the very complicated hyper parameter tuning try it a few different fundamental models or base models and see if any of those improve the process improve the behavior along the way because they think about the relative position of the tokens better or they behave in a different fashion this is the simple example of the code again taken from aws this is how to train a model um on um on sagemaker and what i will say about this that's useful uh the key thing in here is that there's this trained.py file and one of the key things in there is to understand what loss function you choose and what um what um evaluation metric you choose and those sorts of things along with um the ability to choose the correct baseline model often have a much more productive effect than doing too much in the way of tuning and actually it's a nice relief that by and large for the kind of fine-tuning work that goes on there isn't a lot to be done in terms of the kind of tuning that used to be done or if you had to do stuff in svms or or or more shallow learning models very often what you find though is that people don't actually want to do direct transformer based modeling what they want to do is add the language of a of an input to a set of other things you want what you want there is you want a stable set of um embeddings from which to work from and the newer one of the nice things there is to have a look at things like the sentence transformer approach and what this does is this goes back to this idea of hey you know language is a feature maybe we concatenate that with some other operational attributes that we want to worry about and some other features that we want to think about and from there we're going to go rather than going into the kind of trying to get the machine trying to fine-tune the ml model entire our fine-tune the transformer model entirely to to fulfill our needs and i think that one of the things that we found along the way is being able to do this and to fine-tune these so one of the things you can do with these kinds of models is you can continue the training so you can say here's a bunch of um further examples that will allow us to customize the vocabulary or customize the um the uh similarity function that the embedding space by means of sort of showing more domain-specific language on top of the existing one and whether you do that with a classical single transformer or in this case with these sentence transformers which use twin uh typically use twin models where and are contrastively learned depends on what data you have available and how much time you want to spend but i think very often it's very useful to be aware that you have this opportunity to use over you use the encoding uh use some sort of a substantive method to get a stable embedding which is then just a feature to something else to some other feed forward network or xg booster or whatever model you want to use on top of that that would allow you to uh to build a larger model because you know the context there is that these things allow you to do these things in the past there have been experiments to try and use custom tokens to try and use other approaches to do that and i think you know uh your mileage may vary but certainly along the way we found that stable embeddings are a very powerful way of addressing a lot of problems and using things like fast a fast approximate nearest neighbor search along with an embedding can be a very powerful thing one warning i will uh give you here is that for all of these things we do find and there's support in the literature for this finding that um where they're entity focused where there's a lot of focus on the specific words rather than the surface meaning of the sentence you can run into a lot of problems because uh model similarities is kind of aggregate over the over the the whole sentence and so trying to learn and trying to get a model to learn the difference between how do i install revit and how do i install autocad being very different things compared to because they're both installation questions that kind of iconograph iconographic kind of question is very different difficult to uh to fine-tune so something that you need to have a very strong understanding of the behavior of the model you expect before you do there and that leads to my final point on training which is the importance of being able to do a a discipline to approach testing of the model in a discipline manner and i will draw your attention in this case to work by microsoft research on what they call the checklist process they define three types of tests that they think should be done and they use a transformer to help test these things so this is why this ties back to this talk so nicely and what happens is that the first one is the idea of a minimum functionality test which is equivalent to a unit test in programming these are some things that the model should just get right so if it's a sentiment model and i say i am happy it should always uh return a positive sentiment for that an invariance test uh is is is then beginning to sort of think about the fact that these models can learn spurious correlations and so it should be possible for us to um understand whether or not the model is beginning to behave spuriously so for example taking my sentiment analysis example it should not matter if i say it is monday i am happy or it is tuesday i am happy the model should be able to uh know that the day of the week is probably irrelevant to my statement about my sentiment however i may feel in a cure way about what whether whether i do or don't like mondays and finally the directional expectation test i am not happy versus i am happy those sorts of things where you expect a change in behavior from the model making sure all of those are happening and these checklist approach where it uses automatic augmentation to help you with this and they have a whole interface for it i think it's a very powerful way of scaling up uh the training and allowing you to build and test more models because you can automate the whole process or at least um program i programmatize the whole model in an effective way so finally inference and inference has been a kind of a an exciting topic i know that uh all of the major providers including hugging face are very intent on helping people to get their models into production this comes from the fact that two years ago it was considered difficult verging on impossible to scalably deploy um um transformer models they did and do require um gpu or other custom hardware uh certainly the cpu inference of models is expensive and difficult and so along the way there are other things to think about there two approaches in the that are kind of more common in the past one was to use distillation which is to say that we would train a simpler model from a larger one that's still popular but um and in fact training on distilled base mod based base model has proven to be a very effective approach and quantization which has taken a large leap forward now that there are the kind of dynamic and mixed mode quantizations that are highly effective and people are doing all sorts of fun things getting down to 8-bit and below and i think this is very exciting but uh very intensive and very expensive and so from my perspective where i have a limited budget of time i think i'm more interested in opportunities to scale things faster and more easily so one example of that is in custom hardware so we had an opportunity to work with aws sagemaker team on their infras inferentia custom inference chip and the nice thing about this is that this is all the code you need to take a pre-trained model and make it a a one to deploy and i think anything that is like this where there is a simple path from um from development to production is something to be encouraged and inferential is one of these there are several other examples out there and all of them have the value associated with them that what we can do is we can build a model take it to production and then speed it up and scale it out and we saw a 4.9 fold increase in throughput which for us was highly valuable and highly powerful so in conclusion what i want to say is that um this i think the best metaphor for a transformer in my mind is this uh swiss army knife it's a very powerful very extensive tool but it's not always the solution to the problem the thing about it is that they are they do impose some constraints on you they do have they do require a large amount of resources to train and to build the largest models right now that i've seen are in the 20 to 30 gigabytes size which is just and requires twice to three times that in terms of ram in terms of memory to load so that's a sizeable model to think about and you know thinking about all of the tools that are available to you can be a bit overwhelming so taking a disciplined approach to the to the understanding of it you know making sure that you have a a process that both involves a continuous data stream that continuously updates the value of things and that you have a checklist type approach to ensure that your model is um consistently behaving over different versions and finally looking at scalable inference and scaling up your your uh your um you know your your how you uh offer the service is a key point to controlling the costs along the way and you know maybe it is the case that this could all be done with a naive based classifier but most of the time the power of the transformer is in the flexibility it provides and in the ability to get something out quickly that then can go to the next step so thank you very much i will happy to invite any questions that may arise and hopefully i haven't run too far over time thank you to to everyone you