LLM Avalance: Anshul Ramachandran: The things we learned while building a successful LLM application
thank you hi everyone I'm anshul I'm from the team at codium and Cod ium and today I'm going to try to switch things up a little we've been talking a lot about a lot of the technical details and lots of fun training graphs and whatnot I'm going to kind of shift it a little bit more to the product of you and give a little bit of insight on things that we learned while building an LM application of Our Own so some little bit of context here what am I talking about about nine months ago my engineering friends and myself were all Engineers we all were trying GitHub copilot and we're like wow this is pretty awesome right this is going to be the future of software engineering AI helping to accelerate everyone but looking at our own kind of workflows you realize that just the you know autocomplete functionality was this kind of small piece of what we need to do as developers so we kind of thought well this is probably a pretty good place for us to try to build our own GitHub co-pilot plus plus right and that's kind of what the start of codium was and just to give some a bit of idea of where we're at right now we're kind of building this entire AI toolkit right the idea that autocompletes just one part we've also done IDE integrated chat natural language search a bunch of things that really address different parts of the software development life cycle today we have over 100 000 developers using our platform and I think probably more interestingly we've actually sold the software to Enterprises ranging from startups to Fortune 500 companies so pretty much the whole point of what I want to talk to you about today is what did we wish we actually knew to kind of get to this point right and everything I want to talk about is kind of our distilled look back in the nine months of things that you know how things kind of made sense we definitely didn't actively think about that in the time so hopefully this will add a little bit of structure to the idea of how do you actually go from a cool idea of using AI to an actual application that you can scale have users and sell right so I think this is probably the number one thing which is identifying a sustainable product and what I mean by this is kind of looking at the very end state right what does it look like three years four years down the line if everything goes well and we kind of distilled what a sustainable product really looks like into generally three bullet points the first thing is no matter what kind of application you're targeting like for us for software development you should be able eventually to address a complete set of subtasks otherwise it's just an incomplete product right otherwise people will have to pull in products from a bunch of different places to actually satisfy everything they need the second thing we realize is that if you're going to be is there a lot of echo if you're going to be kind of like the leader in any kind of space you need to actually understand the user and how can you make your product the best for that user for that task right a generic system will always be able to be assist it will always be beaten by A system that is more specific to a certain person and the third thing and something that we probably should have learned a long time ago is that they should be a clear win if you're going to kind of sell to a corporation or to an individual if you have them questioning like you know oh am I giving something up by using this you're going to be hitting a lot more kind of roadblocks along the way so what does this mean kind of practically for example the Codex uh the code situation like we are we realize that we actually need to accelerate every aspect of the software development life cycle right just stopping autocomplete wouldn't be the answer we needed to realize that we have chapter leverage individuals users data like for example a company's existing IP in order to create the best product for a company otherwise we'll get potentially being out by someone who does build a specific product for their company and at the same time if we're going to sell to a company we can't have any caveats on security legal cost pretty much anything that makes a company question whether or not this investment's worth it so some thoughts here uh probably heard the song before an API wrapper around something like open AI is is not a sustainable solution and there's a lot of reasons for this right that very first bullet point of you should be able to attract uh you know go for every single subtask this actually scales really quickly and if you just look at the costs right for any of these you know API calls something like you know an open AI I think GPT 3.5 is about 0.2 cents for every thousand tokens let's put this in a little bit of context a single call for something like GitHub co-pilot for code has a context length of about 2048 tokens it's about 2 000 tokens on The Daily you're creating an autocomplete suggestion for every keystroke you're doing that probably a couple of thousand times a day for every developer so what we're talking about is actually a few dollars per day just of open AI calls and that's not a cost that you'll be able to fully pass on to a developer right no one's going to pay maybe thousands of dollars a year out of their own pocket right even if the value is there that's that's a pretty large thing to to swallow and so you start adding like chat calls and search calls and your own Vector database and you're asking the user to actually front all these things really wrapping around this becomes an unsustainable thing from a purely Financial point of view so the second thought is understanding kind of what the actual differentiator that you could use that doesn't that actually could be a real moat for example we didn't go into codium thing that we're going to build you know the best foundation model and we're not going to drop you know tens of millions of dollars to train the next gpt4 but what we did realize is that any kind of generic model wouldn't have access to for example a company's internal code so now if we could create a solution that could leverage a company's internal code we could actually make use of a differentiator that any generic product doesn't have access to and so understanding where actually the the differentiators is I know there's been a lot of talk about what's the moat with llms I would probably say this is answer number one right leveraging data that no generic model will ever see the light of day so anyone building a wrapper product also won't have access to this right and the third thought kind of comes to that very last point of there shouldn't be any caveats and so if you're able to prioritize the control over your own models you can do a lot of really interesting things right and we'll go kind of more into details in in terms of code where that comes into play so the second kind of General overarching thing that we wish you kind of knew is figuring out all the different ways that you can actually differentiate your llm application the first one that I've already already alluded to is data right not only is data you know the user data not going to see the light of day this can actually become a real way to actually improve the quality of your product as I said a solution like a GitHub co-pilot might take about 2048 tokens of context just to put that into kind of like you know actual things that we can rationalize that's about 150 to 200 lines of code that's really not a lot right if you think about your own code base that you're coding in and you're typing well all of your internal libraries your syntax your semantics how can that all really fit into like 150 200 lines of code even if you knew the exact right best lines of code to put in there so if you have access to user data can we actually use that and actually training the model fine-tuning the model in order to improve the performance so this is one way that you can actually differentiate your model for a customer or a user from any kind of other product or generic generic model out there the second one is via ux right even if you are just an open AI wrapper this is probably what you're going to focus on most of the time right how can we create the user interface in a way that is not just more available to the user or more practical and streamlined but also makes the AI more powerful for the user right how can they actually discover how to best leverage an llm because to be honest all of us are still trying to figure out what all you know these llms are capable of doing and so you know here's one of the examples of our chat feature where you know you can see it's like a chat GPT window but how tightly integrated is Into The Ides with little code lens suggestions you know ways to actually copy in the code back into a diff back into the IDE you know a little bit of confetti for fun all these kind of things just to make that user experience a lot better this is another place to really focus on and the third part that we realized is actually there are some things on the model level and if you do control your model you can actually do model level the changes to improve the quality of the model for code you know there's a very interesting situation like unlike a chat GPT where this context in you produce context out for code both the context before and after the cursor really matters right so this is this task called fill in the middle right filling in kind of the tokens in between context and this is actually a training pre-task like it's a task during the training process and if you can control your model you're actually able to do things like this so code has some of these interesting things and I'm sure other applications have unique aspects of how the product could be best used which can actually trickle all the way down to the model level so the more you can actually vertically integrate your whole application the more you can do kind of fun things that will just improve the quality of of the of the product and the third overarching thing that you know we wish we kind of knew is addressing all of the needs that a user might have and these needs are often not actually the model so we spend a lot of time in you know these kind of sessions and these kind of venues talking about all the new coolest Tech to create all the best models but in reality if you actually want to sell a product but we realized that we learned is that people care about other things right people care about security right you hear this all the time I don't want my company's IP leaking out into the wild uh I mean there's been tons of really fun headlines over the last few months I think my favorite one was last week I think Google telling their own employees not to use Bard but you know there's real concerns around security and privacy and so you know one thing that we learned we had to do is that for any Enterprise we actually had to fully self-host our solution right make sure that companies can be very confident that none of their IP is leaving people have real legal concerns right uh you take something like a co-pilot and it's trained on GPL code and so regurgitates GPL code and people get scared because there's legal ramifications this is where owning your entire model end-to-end really helps we're able to kind of remove all of non-permissively licensed code from our trading training data again something you can't really do if you don't fully own kind of model not just the model architecture but the whole training process end to end so that's kind of pretty much the highlights of what I wanted to talk about right just to recap I think the main things that you know if we had gone back and we did everything linearly and this is all nicely formatted but we absolutely didn't end up here like that this is kind of what I wish someone had told us right really start with kind of that end goal what does it actually look like to have a successful sustainable end product you're not going to get there immediately we're still on our our way but the people who kind of like understand that hey user data is the mode on owning their model you already see them having success I think a lot of us here have heard about Harvey right their whole thing is being able to train on the legal data of all the people they partner with right it's these people that have figured out these aspects of the mode and really controlling that whole end to end to understand kind of how you eventually make money in a sustainable way then figuring out how to differentiate wish we knew that you know ux was a real lever that we could pull and we weren't just you know trying to rely on the ux that was well accepted from like you know chat GPT chat box or an autocomplete co-pilot and finally understand all the needs of your eventual customer right they're going to have concerns far past just the actual model and in your burning all of that will help you actually work on building the roadmap in Reverse right we started we realized okay we need to have a self-hosted Enterprise solution what does that actually mean well if it's self-hosted we don't get any feedback from the user so if we want to iterate our product that means we actually need to create an individual plan right so that we can actually have individual users use our product iterate on it give us feedback right if it's going to be self-hosted that means they're gonna have to pay for the actual infrastructure which means we need to invest a lot of time in optimizing that to work on cheap gpus okay and now if we combine those two we need an individual plan and we have cheap gpus why don't we just give this entire individual product for free right and that's actually what we've ended up doing so just in thanks for listening and my final blurb triocodium is free [Applause]