SBTB 2023: Manasi Vartak, Can AI Write Its Own Story? Unveiling the Power of Self-Documenting AI.
all right great I think that is my cue um welcome hi everybody thanks for sticking around to 5 p.m. on a Wednesday um I've kept the talk light on technical details and more on pictures and memes so you can just chill and just absorb it all right wonderful um so my talk today is called navigating the Gen app development hype cycle I'll go into it gen app development isn't as much of a smooth ride as one might expect um as we learned firsthand and then repeatedly as we build gen apps so this is a bit of a reflection on that and then towards the end I'll talk about how we Consolidated these lessons into a really fun tool so that's just what I went over hype cycle lessons and then a reusable tool so a bit about me I'm the founder and CEO of Verda um technologist nerd atart um I started off in ml infrastructure during my PhD at MIT my PhD work essentially led to ml flow and experiment tracking so if you're familiar with that space that's where I got my start um and then did a lot of Hands-On data science infrastructure building um and now my company Verda sells Model Management and operation software to Enterprises uh Fortune 100s seeing all the cloud hybrid Edge configurations as you might imagine um and the reason why I'm doing this talk today is since the beginning of the year to now we've shipped a whole bunch of either AI enabled features within our product or we've built them for our partners and customers some of these have included AI enabl documentation for models text tosql um question answering type uh applications as well and so today I'll distill some of the lessons from having build those applications into things that hopefully folks can use and I generally enjoy building data and AI products that's what I've been doing for about a decade now so that's why I'm here all right so let's get into it um hopefully people here are familiar with the Gardner hype cycle they have this for um they release it every year different Technologies live in different places on the hype side ccle I find that this is a really great analogy for a lot of things in life um actually it starts off with there's a lot of excitement about a technology like GPT for instance you know there's a fever pitch as people try it out they realize how great it is and so on then people start actually implementing it realize that it is really really darn hard to get to do what you need it to do that's our trough of disillusionment and then you know you power it through and you're like all right I'm going to grind it out and I'm going to make something useful with it and that's where when Technologies reach a plateau of productivity that's like all right most people can adopt it and it's useful in the business the funny thing is as we built our AI assisted documentation text to SQL question answering you name it we realized that we were following a very similar process um as we were going from an idea to a application that we could deploy or hand over and so I'm going to map that over to the hype cycle CU that was the best analogy uh that we found and then we'll talk about lessons about how to get from this inflated expectations Peak to somewhere here you're probably not going to get all the way to you know uh productivity right away but how do you try to avoid this truff as much as possible all right so what does this look like um how many people here have built geni apps for your consumption for company consumption you name it kind of sort of tried GPT you know okay got it got it all good um I'll tell you what it looks like so this was me um when we were building the AI assisted documentation feature in our product so what that is is we offer a model catalog for models and a really tedious thing that uh risk people or data scientist or even software Engineers need to do is go and document the model and words do not come easily I have a technical background words just don't come easily and so that's where my idea was all right let's use GPT to actually help our customers write better documentation so right GPD generates text oo I can actually do that I hacked this um in a weekend evening that was great it actually worked I have a picture or a loom of that later and it's like all right I'm going to change the world because you know this thing can document no one will ever need to write documentation again we all know how much we enjoy writing documentation um so that's great I sent a slack message I got our data scientists actually working on it um and then we actually started testing it on realish looking data and it's like oh shoot this does not sound right this sounds like a high schooler wrote it or this is just this is hallucinations like these are pesky hallucinations I'll talk about them but it's making up information um then you're like it kind of sucks when you go outside oford you tested it on so what the heck are we going to do um at some point everyone is like why is AI so dumb like this is this is never going to work um because it you don't know how to instruct it quite right you don't know what examples to use you don't know what language to use all the fun things that come with prompting um go into that really truff of disillusionment you stay in there for a while and then you grit your teeth and you're like okay I just need to read everything about prompting there is I need to figure out what are the different ways to prompt I need to go find or label a data set I might need to F tune and then slowly and slowly you kind of you know keep getting better better is also an iffy term because this is not accuracy you don't know how accurate it is it's some sort of vibe based is this better or is this worse type of assessment that you're doing but hopefully you're getting better and most times it ends up with this isn't perfect but you know what this app is really useful so either I'm going to ship it um or this particular um firm is going to use it internally and so it's not facing real users this is usually what it looks like if you're going through this process don't be surprised that's just how it goes because we're still trying to learn how to best utilize this technology um and how how to best get the model to do our bidding essentially so this is a process that we went through over and over again um as we were building our applications and there were some key lessons that we learned that I'll go over next um that hopefully will help people get from you know idea to somewhere here a lot faster um and then there's a tool that I'll talk about towards the end okay 10 was a nice round number so I've evened out my lessons to B10 all right um there's a lot of talk about open source and I'm a big believer believer in open source we I've built Open Source before you don't need to use it for prototyping if you're trying to use build a application and not iterate on the model itself start with Open eii start with gbt you're going to know what the boundaries are of what is possible um and I put a little asteris there because there is an optimization you can do but that should come towards the end once you know what you're building and that it kind of works and this is just a benchmark there are lots of benchmarks that compare gbt to open source and so on just start with something that you know Works fairly well and then iterate from there so that's number one second one the flip side don't be lulled into how easy your initial prototype is or um how quickly you were able to build it that's my screenshot of the Loom that I sent the team and I'm realizing that is really really small for you guys to see imagine there's a loom of the app that I built um in streamlet it's great it took us about 3 months to get this production ready um and this was because we had to iterate on our models a lot on our prompts a lot the ux is really hard to get right um and I have one point about that towards the end the ux is really hard to get right because your users are likely not used to a tri fix try Loop and so your designers will need to think really hard about how do we let the user have control how do we tell the user what is automatic what are things that they've written how do you get consent from the user in order to use an AI generated thing it's really interesting and also hard um user interface problem and so if it's one evening just know that it's going to take you weeks um to actually ship it all right third lesson this one is really close to my heart cuz my PhD was in this versioning and management is super critical um if in ml you used to have notebooks that were not versioned in any reasonable way need to stop doing that now um because you're increasing the number of variables that you can give to your application so you have things like what is the model the model has parameters to it like what is the temperature now there's actually a reible seed so hooray for that but you're going to try different things there you're going to have different prompt variations you're going to have different data sets that you're going to test with and so records within the data set IO and so on very quickly you're going to try I don't know we tried about two or three dozen prompts and we tried them on different data sets with different models you want to be tracking how well these things are performing so that you can pick the best one for your application and so versioning and management make sure once you start doing this for real make sure you're doing uh versioning and management in some reasonable way all right related to that um build a good test set and this is really important we spent a lot of time building a test set even on AI assisted documentation so our goal was um let's write documentation that sounds like really good model cards and so you need go and find good model cards you need to then distill them to what might be the question the user would write that would create the model card and so creating the good data set turns out to be more challenging than one would think um it doesn't have to be massive which is a good news to get started you need five to 25 I would say like examples that a good representative set of what your users might provide um and this does not need to be a labeled data set so this could just be sample inputs that a user might give you so this is screenshot from the tool that I'm going to bring up in a little bit this was a summarization bot that we' built um and so the kind of test set we were providing it was this is the URL that you want to summarize the tone of the summary should be formal informal whatever it is and the length um you'll realize that GPT is really bad at things like math and it cannot count paragraphs so you'll also see the boundaries of what's doable and what's not but invest the time um because once you have a good test set it's like having good test cases you can iterate a lot faster all right um most people here I assume are software Engineers data Engineers data scientists and so on if you're a technical person you might be really bad at prompting um the one on the right uh the one on the left sorry is me writing prompt for a write a press release type bot the one on the right is our head of go to market and you can see how detailed and how refined and how precise his prompt is I could you not like ask your English major friends or hum's friends they're going to be able to use the right language because GPD is trained on language the better you are with language you're going to be able to prompt it better um and so don't don't do what I did get help from uh people who can write all right this is something I hope everyone knows but just reiterating we're likely not the user of the thing that we're building um we built a c uh customer service sort of email generation application not a customer service manager um our customer uh success manager said that's too sassy I would never said that to a customer I'm like oh that's that's good to know I would never have guessed that because I'm not sending those emails so test your stuff with actual users not yourself because their feedback is going is going to be the thing that steers your prompts and eventually a model to where you would like it to be next one um this one is a actual picture of our um testing that we did on the AI assisted docs um you can see a few of the prompts here that were you know some of our favorite ones listed and um I think there're different variations here with test cases and system prompts and so on it was it was me and my product manager going and trying to label these things it's tedious we didn't even have those many data points but it just spread sheets don't scale and so you need better ways for your subject matter experts whether they are PMS or customer success managers or whoever you're building for to provide you feedback and don't make them look at this they will hate you for doing that um big lesson learned um and we don't do that anymore so that's good all right you've probably heard of hallucinations um in the media or just broadly the thing is hallucinations are not just complete fabrication of information sometimes they're sneaky and they're just not appropriate for what you're trying to do so this is an example of um again the documentation example the input the data scientist had provided was I assume the demographic information is not missing and so we we were rewriting it as assumptions these two you know they make sense the third one is interesting if you can read it it says that demographic information is not missing this means that necessary attributes such as age education occupation marital status are available our data did not have those things um we had a couple of columns uh because for this particular application we couldn't include some of the columns and so that's where it's using its broader knowledge um but it's not appropriate for the use case is that demographic information yes it is demographics should this model be using it no and so you do need to look fairly closely to spot what might be hallucinations or just inappropriate information included for that use case um we're getting towards the tail end of the lessons this happened to us um we were working on a gen feature for I think it was again some sort of content creation we spent weeks user could not find the create button you could see them on you know the user interview testing thing they were trying to figure out like they wrote a thing and they're like how do I get it how do I get it to do you know to write ux is super super key um there've been some really good papers at Kai the ACI conference recently about how to build interfaces when you're doing iterative um stuff just make sure you're getting the right um because that's how your users will use your app um AI is not spart enough to be automated quite yet you do need a human in the loop all right and then the final one um I started off with saying start with gbt because it is the best gbt is expensive expensive if you're running workloads of any reasonable size um even GPD 4 Turbo that they announced last week and has a much bigger context window you can see the order of magnitude difference between these two it's still pretty darn expensive to be using these topnotch models and that's where I think there is a real opportunity for open source or even distilling these models which is something um we're doing a lot of work on is how small can you make the model if the only thing that you're trying to do is to go from say a text to SQL just how much smaller can you make it um and so after you have something working optimize your model itself to make it as small as it could be so those are the quick sort of 10 lessons um I'm happy to go deeper into these during Q&A I'm keeping it fairly broad given time of day as I mentioned all right so we learned this over and over again in being a coder I'm like all right I need to automate this I hate doing this I I don't want to do this um over and over again and so we distilled the process that that we went through on testing a bunch of prompts and testing different models to evaluations and then getting to a useful app um and then this app actually launched yesterday so I'm just going to reuse the screens from there sorry um that's the quickest so if you go on product hunt yesterday we launched this thing called verog nii uh workbench or LM Builder as we built it internally and our goal was I want to get to something reasonable in 10 minutes I don't want to spend weeks and weeks on really pointless U prompt engineering if someone if a automated tool can do that for me and I'll pull up the real screen so you know it's real I'll just go through what the steps are um and so what we did was we built an interface where the user could pick what they were trying to solve for is it text rewriting is it text creation are you trying to um we have rag coming soon too there's a bunch of use cases or translation you can pick uh proprietary or open source llms there um you then get some suggested prompts because it's really really hard to know what prompts work or not and here we're using the prom generation schemes that are well known and well studied we're applying that and then my favorite part is the leaderboard where there's a lot of iteration um if you do if you've done regular ml before you'll know that there's accuracy there's a and so on we don't have that in a generative uh setting and so instead we came up with a way which someone know was very Tinder resque is like all right swipe right if you like this um this answer and that's what we do to quickly compare the results of different models and different prompts to get you what works best for your particular needs and then you can do the same thing after you deploy your application um and you can your users can then provide the feedback that will be used to power the leaderboard essentially um and so as we're getting to time I'm going to quickly show you what that looks like um and you're it's open you're more than welcome to navigator app. verta and um check it out with me I'm going to quickly show that and then I'll open it up for questions all right so I've done this a whole bunch of times that's why have wow can you guys even see that kind of sort of okay um all right so I'm starting a project let me see if I can and if the internet decides to work it might do funky things to our UI but that's okay um so suppose you were trying to do something that's generation we did a lot of generation and so those are the use cases that we built out of the box you might be doing text creation you might be doing text rewriting summarization you might be doing Rag and so on so I'm going to pick something real simple that I need to do a lot um which is post on LinkedIn and I really dislike having to write things um and so I wrote this for myself when we started it's like all right I want to do LinkedIn great I want to test a bunch of models I'm going to pick just one cuz I want this to uh wrap up in a reasonable time let's say I'm picking just GPD turbo this is the screenshot that I picked earlier and this is so critical because this is the kind of input that you're going to test your app with so here I'm telling it I want to summarize these URLs and give me a LinkedIn uh appropriate sort of message tone paragraph and so on I'm going to use that and then prompts there are so many ways to prompt these things and each one of them gives different results and so so this is where we use um we use our previous experience and a little bit of AI to give you recommendations on which ones would make sense for you for this particular task so I'm just going to pick two of these I think this is teaser sure I'll just pick two so the two ones are a stepbystep type um prompting strategy and one is a really simple prom go do this useful thing and then it's going to go off it's going to run these examples and prompts against GPT and depending on how fast open AI is running today we'll get results back so I'm going to let that go for a second and these things will turn green when it's done um perfect okay so now we get to the evaluation part there isn't accuracy there isn't a numeric way for me to know which summary I like more so this is where it's picking using some algorithm like these two are the ones that I want to compare First Once you give it that information it's going to figure out what are the next ones you want to compare so quickly getting you to a useful ranking of your results so I'm just going to click through a few things that look reasonable you would look at it a lot more closely than I'm doing interesting Emoji situation going on okay um sure and let's pick this one all right great so now it has information about your preferences if you know rhf it's also doing a ranking it's trying to actually learn a model for ranking we're not learning a model because we don't have enough data points here we're using your preferences to do rankings and this includes Computing an ELO score for those of you are chess players from somewhere um so it's figuring out what is the rank order and you can go add your own metric to I'm doing the simplest bit right now so now I know that I preferred the standard prompt I would have thought that I like the second one but oh well um the in the standard prompt work best great and then you're going off and building an app so I but and our expectation is not that this app is what you ship to a customer um it's that this gives you a place from where you can iterate really really quickly so I'm going to let it run um for a minute let's see if it finishes in reasonable time and I'll show you what it does and then happy to take questions it's packaging it it's going to run it as um a Lambda and then we'll be able to play around with it all right I'm going to let that run for a minute I'm happy to take questions while uh oh there you go okay let's do this and then we'll cool um and I can pick I'm going to pick some random blog I wrote paragraphs sure um make it funny it's the end of the day what can you say um and then it's going to generate something that is you would do a lot more evaluations than I did but that's much more your tone hopefully than can GPT which sounds at least to me sounds super like marketing e this one also needs work if I'm actually going to push it but that's an example that's what we ended up building um to replicate and make the steps that we went through a lot easier it's available you guys are welcome to use it you get limited sort of open AI credit so at some point we'll uh will stop you from making a lot of calls but it's on app. vera. uh feel free to check it out and yeah that's all I had um I am more than happy to take questions yes you suggested using GPT for training and then sort of downshifting to either open source or some simpler version how easy is that and or does that how locked in you end up being especially shifting from GPT to an open source model got it um I like prototyping with GPT because it's going to tell you is it an issue with the model itself um or is it like fundamentally our work like the AI research is not at a stage where you can try to do the thing that you're doing and so that gives you a feasibility check once you're done with that then a big part of it's going to come down to how efficiently can you evaluate these things um because you can spin up the models that's what I was doing there to I picked GPT but you could pick like mistol or you could pick llama at the end of the day it's like how quickly can you um assess the relative quality of those applic of those two results and then pick one so it's not terrible I think it depends on how accurate you want to be if you're okay with saying like 60% accuracy then one of the open source models is going to be just fine and you can have GPT handle the more complicated ones that might be another way to go so the space of how can we optimize the model further is pretty large um but it's not too bad in our experience all right I think we're right on time uh feel free to find me after um happy to chat more thanks for staying guys