Bay.Area.AI: Lukas Biewald, Understanding the Landscape of the Latest Large Models
Recording: Bay.Area.AI: Lukas Biewald, Understanding the Landscape of the Latest Large Models
people are asking me if this is uh if this is our our kind of first Meetup post pandemic and I would say this is our first meet up as weight supplies I think ever or first in-person Meetup I'm pretty sure we did some classes before um but yeah this feels super fun um and I wanted to talk about um large models because I feel like that's such a like exciting um thing I was talking with a bunch of you about you know people are coming from really different levels of experience but I think it's really interesting all this stuff going on in the space I gave this talk um a few months ago and then I had to like really change so much of this you know just to make it up to date for for tonight which is a really cool um sign um you know the Lexi said uh I used to run a company called called Figure Eight that collected training data for machine learning maybe some of you remember it um before that was called crab flower and now I I run a company called um weights and biases and um you know I think many of you that came use weights and biases we started off as experiment tracking and then we've expanded into a whole bunch of um stuff to handle the the ml apps workflow this is not about weights and biases but if you do want to demo of weights and biases or a t-shirt send me Nema I'd be happy to to give you either at any time um I think it's it's kind of interesting to look at um how we kind of got to this place where we are with um with large models I mean it's a funny um adjustment for me right because I was running a company that did um data collection and so you know I'd always talk about this like unreasonableness effectiveness of data do you guys anybody remember that like Peter norvig you know maybe like 20 years ago now talking about how like data is really the thing that makes ml work in the applications where it works is where there's lots of data and then Google in 2017 they published this blog post where they talked about you know even at like bigger scales really it's the data um that that makes machine learning work but it's kind of an interesting moment that they did this because it was like just on the eve of you know coming up with Transformers and and maybe this sort of getting a little bit more complicated right and so then you know you have like attention mechanisms um which which came along I think originally to help with um uh lstms right because lcms had such a hard time like looking far back in time and it's so clear with trend translation that you know you don't want to look at kind of every single word to know how to translate a particular word and so yeah I mean so so attention mechanism remember that it was like so it was like exciting you get these cool pictures where you know it'd be like you know what what um words align with what for like a translation or you know kind of an image you know what parts of the image it's looking at such a kind of evocative method right and then um you know you that famous attention is all you need paper um came out that showed that you don't necessarily even need you know like gre's or lstms you could just kind of do it with um attention mechanisms um and um and then Transformers came along right and so um you know basically using this transform mechanism Pros in that paper um you see that at more and more and more parameters there's this um kind of consistent logarithmic Improvement in the the performance of the models and this kind of you know extends out as far as people have tried at this point right I think there's a debate right now you know how far this really goes but it does seem like Transformers that you know massive scales work um you know we're super super well um oh thanks thank you what um and so you know data size is um you know continue to grow and the bigger models kind of continue to take advantage of the larger and larger um data size you saw you know chinchilla came out um you know this year um training on like even kind of more data you know I think they're you know folks say there's like maybe another magnitude or maybe another two magnitudes of um you know like language left to to consume right but it's you know it's kind of remarkable how like you know they have like all of Wikipedia and that's like a tiny fraction of the data that um you know they trained on um and then um I don't know if you saw Andre karapati's mini GPT I'll just say like if you kind of want to get your hands on a lot of this is like about how you could get your hands dirty with this I thought this was such a cool project um that came out where he tries to um distill Transformers um and you know GPT which is the the open AI project down into like its Essence um and so you can actually use this project to sort of see how a few lines of code you really need to make a Transformer so I love to learn by doing and if you want to build a Transformer yourself it's funny I was like doing it and then I like saw this project and I was like oh actually Andre of course um you know made a you know made a what far better version of what I was trying to make as usual um so then uh gpd3 um you know you probably noticed uh you know it's kind of I think like maybe I don't know if you guys feel this way I feel like it's kind of the most sort of like impressive maybe like biggest train model that you can actually like access um you know right now um and you know GPT came out back in in 2018 um it was kind of like this this um breakthrough and actually like you know kind of worked well across tons of natural language tasks which was really interesting right like I think like you know not that long ago like when I was in grad school you felt like you know kind of each task required its own model and it's like own techniques sort of an interesting to watch how these models now like run across like so many different you know NLP tasks um at at once um and and again the you know the number of parameters really matter and then I think like gpt3 if you haven't used I'm surprised how many of my friends in machine learning haven't actually kind of like played um with these models but um it really is actually like an incredible thing that basically just is a text it's a it's a large language wall so it generates text but you know what you can do with it is lots of stuff so we've had you know at waste devices we've had tons and tons of people um play with it and write reports so if you go to I guess if you go to WB dot me slash dbd3 Doctor Who you could see you know kind of generating doctor who thinks it's kind of like a fun um example but then you can take these same techniques and solve all these classical problems right so you know for example you can do sentiment analysis just through text generation right where you're like showing a tweet and then you say like sentiment colon um you know positive or sediment colon negative um and um and then you can actually do translation just by setting this up amazing how many people actually don't know this this translation system is like pretty world-class you know like you literally just like hear some text in English and you're just like putting this into the gpd3 um interface and um you know I don't know if you guys anybody see this reasoning thing I thought this was super cool um where you know and this is sort of like the prompt engineering that's actually getting like really important to make these things work um that we see more and more of our customers doing right where you're basically you know you have this like zero shot learning case actually amazing this works at all right so you're just like feeding in kind of like a simple logic puzzle a simple like kind of math puzzle and then you're typing an a to answer you know is and then it like Auto completes off into the right answer right and this is like incredibly hard task right it's amazing that this model can kind of do it sometimes and then I guess some researchers found that if you ask if you add here's the answer and then you say let's think step by step it actually does much better which is such an evocative um you know result and I think cause people to to to try a lot more um you know stuff here and if you talk to the gp23 folks um they spend a lot of time with their um uh they spend a lot of time with their customers kind of helping them write this actually you know a good friend of mine um Anthony is a yes CEO of kaggle for a long time and I showed him a couple days ago how you could do this and then he was like really surprised that it worked so well and then it was funny his wife's actually a lawyer and his wife was a much better prompt engineer because it was like she was like really writing like specifically what the model should do and he got his accuracy from like already better than his like classical machine learning techniques that he was using too so it's better but then it got like even better when his his wife stepped in and did more like lawyer style um prompt engineering which I thought was really interesting and this is for um extraction which isn't like an obvious isn't super obvious that would that would work um and then um I just want to brag you know we have an integration with um with openai that we would love for you to use if you just call openai you know 1db sync then you can actually send your results um to weights and biases and we'd love if you use it I think like I've always been a huge fan of um fine-tuning and I think it's really cool to support cases where you know you're starting with this giant model that was trained on you know hundreds of millions of dollars to compute you can actually like get that tweak it for your task and then use it for kind of any problem or domain um and um and yeah you can use you know our tables to visualize the results and and um you know lineage tracking I think it's just like this graph is like so evocative to me right where like you sort of see this like Trend um of um you know the training compute and publication date and you sort of see this like inflection point um and I think this doesn't even include and this is because so I want to say this is because the model size like really um matters um let's see um and you know you can see this like you know the numbers you can see it in the the way the results look um but it's just I think like I guess I would say for those of you have been like watching the space recently you know what's about to happen right is this this curve is about to inflect a little bit more right because it's just like so super clear these Transformers work at these large scales people are finally like making enough um compute to be available so like I think right now like many companies are trying now to to train these models at like another order of magnitude of size so like the real winner is probably chip companies um there's also I don't know if you guys have seen the Jasper AI is like one of the fastest growing startups of all time that basically has just taken um gpt3 and and use it to kind of create marketing content right which is a little bit of like a sad use case maybe sorry just um but um but I think that I don't know like I think if I was like you know not kind of interested in this it's like this is this they went from one million dollars in Revenue to 70 million dollars in Revenue like in a year and so they this is like really like working on top of um uh just on top of gpd3 you know they did they just kind of like got it to to work well um and you know there's like a whole bunch of um these big models that have been made released in different ways right so um you know chinchilla I think uh at least I think meta recently said that they were gonna actually open up their their model they that they came out so maybe there'll be some competition for gp3 but GB3 is strong and it's the one that you can really just um easily use directly if you if you go to the website um you know I think and also like you know like people claim these chatbots are sentient but I actually think it's like At first I was kind of making fun of this myself and then I was like looking at the um the results they're pretty good and then and then we were talking with the folks at Google that like made these um chatbots and they were not so dismissive of this person as I thought so um you know I actually myself have not got a chance to play with the the internal chop up but I it seems to be a very very compelling um chat button of course you know we're about to train an unordered magnitude more data it'd be really interesting to see how how compelling that is um and so I don't know I'm always like excited about like sort of the the open source stuff and there's been a lot of Open Source projects around this kind of hard to do right because it requires so much training but there's been a lot of different you know models and and kind of Open Source um you know projects that I think are kind of exciting to get involved in none of these seem to be quite the level of gpt3 but um you know a whole bunch of kind of different places you can go to find this and then um uh yeah it's been it's been um it's been cool to watch um and I would say another thing that I mean I guess one thing I'm like passionate about with with advice is is to make this as public as possible so a lot of these people doing these open source projects have actually made kind of public reports about exactly how they're they're training these large-scale models so that hopefully makes an easier entry point if you want to kind of get involved in that um I don't know if you've tried cogeneration I think it's amazing I mean so you know it's it's I actually use it and a bunch of Engineers use it um but it's um it's not just good it's like actually like also getting better very very quickly um you know I think open AI is maybe a little a little bit better but not quite as like publicly available as the um you know the GitHub one but it's like exactly the same technique um and pretty incredible co-pilot and then replit built their own which I thought was like super cool that this organization could just build their own um you know co-generation system it's uh it's really I think this is like so exciting to see what happens next I've been shocked at how effective these systems are um and then I don't know every generation diffusion models have been um I mean like the the thing is like the last like you know month um they've been like incredible right and so you've got a slightly different um slightly different architecture I think like Lily and Wang always writes the best stuff but she did a great post um on diffusion mod as kind of an entry point if you want to you know understand you know more about the details of of how these work um you know when when Dolly came out I think it was impressive but dolly two has really been the one that I think um you know before again I'm amazed how many machine learning people actually haven't tried um these models um I when it's only two came out I actually built a thing with my daughter to do like her bedtime routine and it'll just like illustrate it um because my daughter's so small that like it's hard to make a contract with her you know to like make a routine because you can't read it you know but then we could illustrate it so that convinced her that we actually had agreed that she's gonna do these things before she goes to bed but now she's a little older and she's trying to use Dolly too to rewrite this plan that involves going to the park but I think a lot of people haven't actually tried these things and they are like spectacular like you take this and just like it just added this right like just you know extended it and you can literally you know and this is like two months old right like I you know I think the stability ones in some ways are um you know stronger um you know there's there's actually one thing I realized that I think you know you might not know is if you try these the first time you don't get very cool images at least the open the open air version um but then there's these books that have come out about how to like do prompt engineering which will really help you make more um you know compelling images so then there's like many variants of this image and from from Google and others there's there's crayon which I just want to call out I think this isn't the strongest model this is Dolly mini um and I think this isn't necessarily like um the the best mod there but you can use it for free um as much as you want and then um the this guy Boris actually he really built it in um public so he basically read um open as Dolly paper and then implemented himself and got a really good working um system and he really put everything in his Dolly journal on on weights and biases so if you want to see the like narrative Arc of somebody trying to get like you know system working um want to beat at me slash Dolly Dash Journal is super cool it's really fun to to watch him and then finally um you know stable diffusion is uh or um stable diffusion has really been um incredible they obviously like had a big launch um last night where they talked about how you know they do um video now and a lot of other things but the um the text image model is really what they're known for and it's it's really um pretty amazing and just to brag they also used weights and biases and you can look at some of the early work we had I think dance diffusion is really cool like maybe less well-known um but this is uh this is actually a model they built that does um also like I guess Harmony is sort of the umbrella but they built a model that um kind of generates audio samples and it's like actually really like apparently useful I'm not like a huge like EDM fan myself um but some people in the company are and um it's it's like it's really cool to see like you know open music generation that's actually like compelling versus like super boring you know like like I think there's been so a lot of Amazing Music generation where it's like technically amazing but you wouldn't want to listen to it but these guys are like really committed to making like you know kind of audio generation that you actually want to um listen to um and I was at the stable diffusion launch last night they had a lot of music I think was generated with the suit with the system and actually Justin here did it awesome or we did an awesome interview with the um with those guys and it's if you really want to nerd out on um uh if you really want to nerd out on music and machine learning I really recommend going to the WWE Harman AI um you know finally I think the folding stuff is just super cool also not quite the same stuff but kind of influenced by these these large models but um you know open fold it's really exciting to see right where they are kind of building an open source version of of Alpha fold to help with um with with research um and um I don't know if you guys have seen online I mean I guess I'm just showing you what I think is cool but like online task completion I like the the Adept AI demo um and they still haven't let me in their beta to verify this but I think this is like the coolest demo I've ever seen maybe where you are basically typing tasks into a box and then um through Transformers they're they're like turning it into um online research tasks um which I yeah uh this sort of seems like a real mind-blowing um they may have originally been doing coach and now I think they're doing um research and um yeah several people have been like I'll get you into their their beta and they they haven't let me in their beta some little suspicious of any I've just been doing this long enough that like you know an ml product where they don't let you into their like beta you don't know how it might not work as well as a demo but I I still think it's like I think it's intriguing actually another thing that that we saw is um the runway which makes like a like an ml video tool now has like you can like actually put in like text of what she wants it's like oh make the image like black and white and have it like fade in and stuff and like for me not really knowing how to use a video editing tool it really looked amazing that you could just sort of like type in you know just in words like the effects that you want and it sort of seemed like wow maybe that's like a new kind of interface that people would really um you know want to use um and then I think whisper came out which I feel like was underappreciated how amazing this was I actually kind of worked on um on NLP and speech to text generation and I feel like opening I just sort of like obliterated an entire field of research and I was like you know kind of like as an afterthought where it's just like you know this is trained with like most the optimization happening on the data set collection not on like you know not anything special about the model um and then they have like what to my eye is it better than human level um transcription system and I just thought it was like incredible thing like you know open source like you know if this was around a few years ago this would be like thought of as like multi-billion dollar like IP um and they just thought well you know this is like we're doing other things I guess you know so anyway um that's my like quick tour of um large sectors I hope it's fun and I hope that um you know don't be shy is if you come to our event you should definitely reach out and you know ask for a demo and ask for um a T-shirt and if you do use our product I would love to hear feedback um I'm Lucas with a k l u k s at wmb.com and please shoot me an email thanks [Applause]