SBTB 2023: Beyang Liu, Open Source AI Coding Assistants: The Future of Software Development
[Music] all right so I'm here to talk about the future of AI for code uh specifically why uh we think that an open code Knowledge Graph is going to be important for making AI more useful in a lot of different situations uh so quick poll a show of hands how many here are using AI code completion today okay cool uh maybe like 60% what about AI chat for coding you're sort of like chat gbt oh more seems like uh people are a fan of the kind of like explicit instructions how many use nther okay a a a good showing um for everyone just shout them out what are your most common frustrations with uh code AI usually a little wrong at least right sometimes very wrong yeah exactly uh and it's really frustrating when you're like ah I can trust you and then you do stuff and then end up wasting a bunch of time uh following the wrong direction um so that's what the talk is going to be about like how can we solve the number one problem with with AI coding assistant using uh a knowledge graph for code um a bit about me my name is biang I'm the CTO and co-founder of a company called Source graph where developer tools company uh Source graph started uh its Journey as a tool for human developers to search and grock code across uh large code bases and comp complex code bases so we like to say we make working in large code bases tractable uh and part of how we do that is we built a knowledge graph for code and make that accessible to humans as you're searching and trying to understand uh existing source code so what does that mean um here's the search interface so you know let's say I'm trying to find stuff in my code base you know let's say you know uh I'm trying to find all the squirrels in my code base uh so I can go to the source craft search panel I can search for squirrel and it shows me all the symbols all the files all the kind of entities in code that relate to my query and then from there this is where the knowledge graph comes in it's not enough to just show me the the source result uh what I like to do as developer is I like to jump to definition find reference and basically walk the reference graph of code a lot to build up a working understanding uh and so here you can see you can find references to uh one of the methods on the squirrel service in my code base see kind of examples across all the code of how that's used and also pull history to see who's changed this particular method recently now what does that have to do with AI you know that's just a search interface search is old news right now we're living in the age of AI um well it turns out the code Knowledge Graph that we built uh actually is a really good context provider for large language models and other forms of AI and it actually helps generate much better things like completions uh question answering natural language code generation and eventually you know some semblance of of agentic actions in code so code AI today there's basically two interfaces that are incredibly popular lose the connection here okay so we got inline completions and then we got chat right those are the kind of like two most common interfaces uh that people use AI for coding today and let's go over each of them so here's an example of me using chat BT uh I want to know how to use the source graph API to fetch those references that I showed you earlier um and chat PT is more than happy to oblige to generate some beautiful graphql uh code that queries uh for the references and this looks very plausible um but there's one problem and that's there's a hallucination here uh it's lying to me about a particular part of this API it turns out uh there is no uh references node in our graph API that takes a query parameter this one weird trick will make this work a lot better and that's if I go and find an example of using the API that already exists it doesn't even have to be in the same language here it's you know use of the typescript API and I just say here CH CHT here's an example of me calling this API now go generate the response and then when you do that the generated response uh is much more precise because it's anchoring to existing usage examples so much like a human developer can take advantage of usage examples you know whether you're pulling them from stack Overflow or the fine references capability of source graphy or editor uh an AI coder benefits from from the same what about inline completions so here I'm using you know popular AI uh code completion tool and I'm trying to write a function I'm trying to complete uh a Constructor invocation that new interaction uh expression and this tool suggests a plausible completion here it's telling me hey you defined that prompt message variable uh earlier in the function so let's try to shove that in as the first argument but the issue is uh if you actually jump to definition uh on that Constructor you'll see that uh the first argument isn't uh doesn't have a type that matches uh The Prompt message variable prompt message is a string and uh the first argument takes a a structured object um and that's a really dumb mistake right like You' expect kind of your your entry level new new college grad programmer to to not make that sort of mistake again one weird trick will uh help make the output of the llm much better in this case and all you got to do is you got to find a usage example from somewhere in the code base paste it in right before your cursor and then trigger the autocomplete and then it'll do the right thing uh but obviously none of us here are going to sit down and just like copy and paste all these examples from all over the code code base that's boring and tedious um and we'll never do it and so the what we've done with Source graph sorry I gave away my punchline uh is uh we've tried to combine the magic of large language models with the kind of code graph context retrieval mechanisms that we built for you know code search and code navigation uh and use that to entally do the copy and pasting for you uh and orchestrate this process of finding the relevant information so that the answers uh you get from talking to llms the code generated uh from llms is much more accurate much more uh fits better within the context of your codebase and so there's a test that we've come to call the squirrel test which kind of hints at the importance of context so if I go to chat gbt oh got to log in again sorry okay so I'm in chbt and remember earlier when I said uh I was trying to find all the squirrels in my code base so squirrel is actually a component in the source graph backend if you go search through the source graph code you search squirrel and you'll find a bunch of references to squirrel it's it's component how we do uh uh code navigation actions like jump to definition find references now if I ask gbd4 you know what is squirrel it's obviously going to tell me something like it's a small Woodland creature or whatever right now obviously I would never say that that that's kind of like a dumb thing to ask what if I said uh you know in the source graph code base what is squirrel right our our code is publicly available so it should in theory be part of the the kind of index here um I think the gbd4 training set was was updated recently now it goes up to 2022 okay so now it's trying to like visit github.com and like do a bunch of agentic stuff uh let's see okay so in the context of source CRA code Bas squirrel appears to refer to a feature Branch related to adding more language support okay so I found a p request relevant but this doesn't really tell me the answer to my question uh it just tells me that like something related to scroll was merged in so that's not a great user experience what if I go to something like you know GitHub co-pilot uh they just announc a chat GA that's due out in a couple of weeks um with some you know workspace level awareness now if I go here and ask what is squirrel okay tell me squirrel is a high level imperative objectoriented programming language uh that's not also that's also not the answer I'm looking for now what if I ask it specifically hey take into account the code that I have open in the editor it's going to do bunch of fancy agent stuff try to figure that out sorry but I can only assist with programming Rel ated questions and the kind of references it found using you know its search capabilities uh there's nothing about squirrel in here even though for a human it's trivial I just go to Source graph and I search for squirrel and I get a bunch of references so now let's go to the source craft interface we've actually baked in some of the features of Cody into this interface and now if I ask what is squirrel it's going to do some code searches find references and actually gives me an accurate answer tells me it's a service that uses treesitter for fast and precise code intelligence and Source graph it's an HTP server and it tells me the main components and it tells me uh references to specific files so I can go and kind of fact check and audit what it's doing surfaces the read me which kind of describes at a high level what going on and also the backend component here here and then from there you know I can drill in and you know as a human kind of like do my thing with find references and walk in the code graph so that's why you think context matters like if you ask a question like what is squirrel it makes the whole world a difference uh how good your context quality is and there's going to be a lot of people that say they have great context um but to us it feels like a question of like Alta Vista versus Google right at a high level they're both search engines but uh one tends work much better than the other okay so that was the squirrel test okay what about uh like creating new applications um so how many people have here have heard of v.d it's kind of this like Buzzy new Thing verell released it it's for kind of like generating nextjs applications with just natural language prompting um and we thought that was cool but we thought you know what if we took that one step further and instead of just making it about xgs make it about you could use whatever libraries you want uh you know use natural language to generate uh a code for uh different sorts of applications that may or may not use a specific framework so in this example uh let's say I want to build like a simple you know stock tracking app um I have Library here called Alpha Vantage it's basically an API that provides stock tick data um and I want to make a react app and so I tag those into context for Cody and now I'm going to ask Cody uh create a react app that fetches stock tick data for I don't know I know what's a bank Wells Fargo using the alpha Vantage client then render that data in an HTML table okay so again with the benefit of context this kind of like open Knowledge Graph that we built we're doing a bunch of searches across the relevant repositories that we've tagged in and we're generating some code okay so this this code looks plausible let's see it's importing Alpha Vantage it's creating API client seems to be fetching the data and it seems to be rendering it in a table but does it actually run I'm very nervous because uh I think demos are typically uh a bit stochastic and generative AI demos are are even more so but pray to the demo gods and see if this works okay going to paste that in okay we get an error I expect that just got to tag in the dependency oh I exceeded the API rate limit but you saw it render the table right I was playing around this okay A little round of applause for me that's pretty cool um unfortunately I can't do the the next demo because I'm at the API rate limit um but maybe I can play this video uh which shows kind of like the next solution okay so you know HTML table is fine but what if I actually want to render stuff in a chart so essentially what I do next is I tag in a react charting Library called recharts um again this is part of like the open kind of index that Cody has built and now I'm writing a prompt that says like hey create a react app that fetches the stock price for Wells Fargo transforms it and then renders it using uh this charting Library I'm I'm really glad you all saw the HML table before it hit the the API error otherwise you would not have believed me um okay so I'm typing stuff out okay it's going to generate the code I'm G to try to fast forward a little bit so you don't have to watch it type out all the code okay then I'm going to go and paste that in I need to add the recharge dependency and boom now I have uh the Wells Fargo stock price over the past week so I think this is another illustrative example of uh what I view is like the power of context and why we're we're so excited um about combining llms with this kind of code Knowledge Graph that we built um well that's all well and good I hear you say like Beyond Great you know there schnazzy demo you answered a question you told me about what squirrel is you told me how to generate a new application but you know I'm not in the business of of writing apps from scratch every day you I work in a a legacy code base uh with many other contributors and a long history of code um what about that example so just real briefly you know Cody is also available in the editor uh vs code extension jet brains extension um we have these things called commands which kind of map to uh daily tasks uh that we find kind of toysome that we prefer not to do like generate unit test or write a commit message um let's say I wanted to add a new default command to Cody so uh the way I can do that is I'm in okay I'm going to create a new chat I'm going to ask uh where are the default Cod commands defined and like I could do some code searches here but it would take some effort I would have to like generate multiple queries multiple maybe like Rex's and string literals to search for um another option here is we do have this kind of like fuzzier search so if I search for something like default Cod commands uh that helps me get to the right file um but chat is cool because sometimes it just gets the right answer so it's like hey you should go to this prompts cod. Json file okay and oops forgot to delete this earlier uh okay so this is the file I need to modify and now once I'm at this file I can ask can you generate commands Json for a new command that summarizes a uh the current file so again it used context to find this file and point out this is the thing that I wanted to modify saved me a bunch of time of like poking around and probably a bunch of context switches uh and then now it gives me some Json I can paste in too so it's calling it DOC let's call this uh something different summarize then let's say I don't want the current directory as context I just want the current file um let me save that and I'll run it and this is obviously a toy example right like this is not a super useful command I just wanted to kind of demo the the the value that this can provide like I I want to add a new feature to a codebase that I've never add added before and then here's our summarize command close this oops and it's going to read the file that I currently have open uh okay two out of three ain't bad it's uh there's some bugs that we need to work through here uh but you get kind of the basic idea this points me in the right direction as a human I can complete the Remain the remainders of the of the steps um we also do things like you know inlight Auto completion if you're a fan of that so if I wanted a uh two string method for one of the the types I have um you know let's say I wanted function uh let's see completion response to string this is kind of context aware code completion so it's taking into account uh other stuff in the file but also in the broader code base again so we could generate higher quality Generations that actually work they don't have compiler errors and make sense with the within the context of your code base um so yeah that that's kind of like the quick demo uh hopefully this entices you to care about the technology behind this now let's actually talk about what's happening underneath the hood so underneath the hood it's it's pretty simple at a high level um if you're familiar with retrieval augmented generation this is basically that um both with a lot of care and attention applied to the details so the idea is you have whatever interface that you're interacting with code might be your editor might be Source graph uh and you're trying to do stuff you might be asking a question you might be trying to write code you might be trying to uh complete the code that you're already writing and essentially what we do is whatever your request um Cody is kind of this open core orchestrator of this process so the idea is that you're going to touch a lot of data sources in a lot of clients some of which might be open some of which might be closed but the fabric that weaves everything together we think is kind of like naturally open source it should be open because there's so many different integration points that you don't want to be constrained to kind of like a single vertically integrated uh provider and so what happens is when you do a request whether it's an inline completion or you ask a question what we do is we consult the code Knowledge Graph that we've built which is a combination of the reference graph that we've extracted from the code but also plugging in other signals like the G history and in the future we hope signals from other applications uh you know tools across the software development life cycle uh and maybe some tools even outside of it like you know chat uh a lot of the institutional knowledge in our company is just like someone sent a long chat message and that'd be great to surface as well the idea is we we we fetch the context from that using you know really good search we send that over to the large language model and basically have it anchor against those facts to generate the output whether it's generated code or an answer to the question I'm trying to answer we send it back to the intelligence graph to do some postprocessing you know catch common hallucinations do some factchecking maybe enforce uh open source license requirements you know if I'm using a model that was train on uh uh you know non- permissively licensed open source flagging that and either removing it or indicating that to the user they they need to attribute that properly and then we we send the response back to the user it's all very simple um but the details you know there's a lot of work in the details we think context matters a lot so if you compare Cody against your average AI coding assistant um you know the first thing that we think uh differentiates us is model choice we're not constrained to a single proprietary language model we can use the best model for the job and what we've discovered is in some cases the open source models are actually Best in Class for example for inline completion uh from our kind of early studies star coder seems to be the best at inline completions especially with the benefit of context uh if you look at completion acceptance rate which is kind of like an uh an objective is metric uh basically if you show the user a completion what's the likelihood they accept it uh the leading kind of proprietary coding assistant uh gets you to 30% acceptance rate after three or four months of use so at that point you've kind of learned what the common huc hallucinations are with Cody we're seeing 30% Acceptance in kind of like the first uh two weeks of usage um and this is even before we've integrated kind of like the full knowledge graph um completion triggers uh we are more language aware you know having built this code Knowledge Graph out we're we're kind of experienced in this process of like parsing language and figuring out what the the key entities are and that helps be much more intentful about what you're actually trying to do when you're typing out code we know if you know it's like an open open function Block in which case you might want to generate a complete function versus you're just trying to complete an expression where it might be better to go line by line um context scope so rather than just taking into account your recently open files or using kind of like naive Vector search uh we're trying to build this kind of like multi-index code search backend uh and you know one of the realizations that we've had uh working in this space is kind of like very anti-hype it's like the classical information retrieval techniques like bm25 and keyword search actually work very well especially when coupled with kind of a deep understanding of of the code at like a a parser level uh we're combining that with kind of embedding search and and other signals and we feel that that yields a much better result again everything comes down to context uh you know both both alavista and Google are search engines but the what matters is how well they are at fetching relevant data and presenting that in this case not to a human user but to kind of like a an AI coding assistant and then of course uh we think the most important thing here is that we have an open core model so again not everything about how the system works is open source I mean certainly we we still use closed Source models like gbd4 and Claude um but we think the fabric that weaves everything together that does the coordination ought to be open because as more and more integration points become available uh into this like AI automated software development life cycle uh we don't want this to be kind of a vehicle for um tying and constraining everyone to a single vertically integrated uh ecosystem and so you know the ultimate goal that we're working toward is we're trying to automate all the toil out of the sdlc uh you know when you think about how you spend your time uh day-to-day as as a developer uh I feel like often times like 80 90 maybe even 95% of my time is spent on like toysome tasks like I know this code it's not rocket science but I have to go in and kind of like read a bunch of files and try a bunch of things before understanding what I need to do um you know if I'm trying to solve like a security issue I have to go and do the tedious task of making the same simple change in like a bunch of different places uh that's what we're trying to do uh and we think the right approach is by combining the magic of large language models with the benefit of the code Knowledge Graph um and so that's what we think of as the future of of AI in code and uh we also think that future should be more open than closed uh so yeah that's the end of my talk uh any questions from the audience or do we are we taking questions or okay cool ah yes hello if you have a very large project with lots of files how do you limit the context if you have a very large project with lots of files how do you uh pick the context how do you limit it how do you limit the context yeah oh are you saying because like large language models have a limited context window you can't shove the entire project in there correct yes so the approach to that is basically to treat it as a generalized search problem uh so instead of just like saying here is the entire code base as like the input context and then answer this question which is like slow and expensive and also um we found like less good than first doing like a search so basically like as a human if you're trying to answer a question or generate some code uh you wouldn't read through the whole project You' probably do some search queries find things that you think are relevant and then page those relevant results into kind of like the working state of your brain and then do the task that you wanted to do and so like our approach kind of mirrors that it's like find these pinpoints in the code base that are relevant to answering the question or completing the task uh and then with that small but very precise set of context have the the model uh generates output hey thank you for the talk what are your thoughts on this closed loop uh with respect to debugging that's also a main point that debugger Engineers go through sorry what what are my thoughts on the close what are your thoughts on leveraging rag for debugging uh sorry it's very hard to hear with the echo in here can you just like shout it what are your thoughts on leveraging retrieval augmented generation for debugging leveraging something for debugging yeah rag for debugging oh leveraging rag for debugging oh yes so one thing we haven't integrated yet is debugger output as a context provider that's probably uh we should probably do that soon because like that's highly relevant when especially when you're in the Inner Loop um but yeah like that that's a great idea we integrating compiler output and uh debugger output into the code generation process something we definitely want to do uh the the core kind of uh Cody orchestration stuff is open source but we do have some context providers that are closed Source yeah I think for next questions um the speaker table outside table number two cool thank [Music] you