Can We Trust AI-Generated Code? | Baruch Sadogursky & Leonid Igolnik, AI By the Bay25

Can We Trust AI-Generated Code? | Baruch Sadogursky & Leonid Igolnik, AI By the Bay25

Recording: Can We Trust AI-Generated Code? | Baruch Sadogursky & Leonid Igolnik, AI By the Bay25

So, welcome to Back to the Future of Software. How to survive an AIP box >> with prompts, tests, and specs. >> That's right. >> That's the alternative, I guess, name of our presentation. >> That's all right. I think >> Okay, bad news. You've been replaced by air. >> I think all of us are here because this is happening right now, right? >> Yeah

All of you. And you giggle and you are in denial for your therapist will tell you why. Uh but uh it's it's actually happening. >> It is happening. >> And here's the news. >> One example like um two facts about Microsoft. Uh first they claim that 30% of their code is written in AI. >> Yeah

So does any uh AI vendor, >> but also they laid off 2,000 people recently. >> Not every AI vendor. >> So >> and then the music is absolutely [laughter] perfect. So thank you for that perfect timing. >> Yeah. >> Can we do this again? >> So uh question, do you think those two facts are related? >> Uh who knows? Uh I think uh AI is here but we're still struggling with it, right? and the industry is talking about AI like that. >> Yeah. Um well, that's um that's a common sentiment among software engineers right now

>> And hopefully you're here because you're in the same place both because it's going to take your jobs away and because sometimes it's still >> not great. >> It goes, I'm sorry I didn't follow your instructions. Let me try it again. >> Yes, you're absolutely right. >> Now, this thing is not new in in our industry. For those of you that have not seen this quote before, uh was not uttered for the first time in 2020s. >> It feels like very modern thing to think or say, but it is actually from >> the first time that the person that heard this quote in the 50s was no other than Grace Hopper. And for those of you that know the history of computer science know that Grace Hopper Ross the first uh compiler and a lot of old school assembly engineers said, "Nah, no, no, no, no

That I I I like to handr write. >> You cannot trust the compiler, right?" No. So, so this is not new in our industry. And if you look at the history of our industry, right, every big change in our industry starts with the panic. >> Yep. That it's not going to work. It's going to be too expensive. We're going to be replaced

But also, if you look at the history of our industry for the last 70 years, 60 years, we've been continuously going through those revolutions. We went through compilers, we invented object-oriented programming, we went to VMs. I'm old enough to have stacked physical service in the data center or waited for three months to get one provisioned. >> You had to enforce the floor and that's the interesting story. >> Oh god. Yes. There is a story for Q&A. If somebody wants to ask how physical realities of data center interfere with building software have a good one for you

So nothing is really new for the last several decades in this industry. We've been just doing one thing. We've been increasing the level of abstraction with which with which we build the software we built. Right? We went from assembly to high level interpreted languages and we have not died. The industry continues to thrive. >> I think that this one is different and this one is different because we know for a fact that we can trust compilers and we know for a fact that serverless doesn't mean that there is no server. But this time this thing really sucks. >> Much like every single slide in this presentation, this one was also generated by AI

It did its best to tell us you can trust it, but it kind of almost failed. >> And it's not just FUD. It's not just fear. There is a very practical reason why it does that. >> That's right. >> And this is because >> those systems are not deterministic by nature. >> Right. Every single LLM is a stocastic system

Otherwise, you will only get one result. Although pri GPT, you can only get three jokes out of that thing. >> Consistent consistent >> consistently non-deterministic. But by definition, the tools we're trying to apply to this problem are not deterministic. They're not designed to be they're designed to have noise in the system and they're designed to generate different results every time you give them the same prompt. And this is why we cannot trust them. That's right. This reminds me of the old saying about infinite monkeys theorem that goes like an infinite number of monkeys with an infinite number of typewriters and an infinite amount of time could eventually writes the works of Shakespeare

could >> could and they would because infinite gives all the possible possibilities and this is exactly the same with AI with slight difference. Uh we took those monkeys and we gave them GPUs and they are slightly faster and they're slightly more predictable actually much more predictable >> and they are actually skewed towards writing Shakespeare more than just random. >> We trained them. >> Yes, more than just random. But still there is a possibility that we will get Shakespeare. There is a possibility that we will get Shakespeareish >> and there is a possibility that we will get something that is not really Shakespeare. >> I'm sorry. You're absolutely right

Let me try it again. >> Exactly. Until we get Shakespeare or not. So yeah, uh my name is Baruk Sadagorski. I'm at Jay Bar on the internet's um head of developer relations at company called Taxare. Uh we do security patching for outofdate uh libraries and open source components. Um if you feel that you have some library which you are not ready to upgrade to the latest, talk to us. We'll make your life much easier >> because everybody loves upgrading libraries

Exactly. Because not um I call myself abstraction connoisseur and this is why I'm excited about AI and how can we manage those interesting abstractions a personal journey from development to devops to uh intent integrity chain that we're going to talk about today. Um Java champion wrote couple of books and this is my dear friend for the last what five years Leonik atnik on the internets. >> My name is Leonid. I'm a CTO at the company called Clary where we actually apply AI to the revenue and sales domain. We build tools to have sales teams uh work more effectively, more efficiently leveraging the data they all generate. I have been wrangling those abstractions for many many decades and I've seen some of the ones we've shown you in the previous slides. Started as an engineer uh used to be an architect once

Uh I still tell my team that every once in a while I'll I'll stick my finger where it doesn't belong. But I've been in engineering leadership for the last 20 years. >> Right? So the most important slide of this talk is this. If you go to show notes.taxcare.com, you will find there are the slides. You will find the video that we are recording right now. All the links for everything we spoke about. Um and a raffle for a prize to thank you for being here. We really appreciate it

It's conveniently on the bottom of every slide. So you don't need to take the picture of this particular slide. There is also a QR code that you can scan um if you want to. It's going to disappear in five 4 3 2 1. Next slide. >> Let's talk about nondeterminism. Is it real? >> It is real and this is one of the reasons why I generated code is not great. Uh there is a lot of research gathered by now about how not great it is

You can see here um 65 46 and 31% of correctness of the time. So on average, it's right about 50% of the time, which is >> as good as flipping a coin. >> Exactly. Um on top of that, it is actually dangerous. 40% of the AI generated code is vulnerable. >> So if you're looking for a next job, go work for a security company. >> That's a very good advice. Absolutely

Asking it to fix it as reliable as you would expect. Um it self-verification is a problem that we are going to talk about. New bugs are most frequently introduced than actually resolved. Um and it's uh it's not great and the industry has been looking into this and starting to know not notice those patterns. So recent survey shows that AI tools more often than not generate results we just don't trust. And uh well um people that are still on Stack Overflow obviously are not very big fans of AI as you might imagine. So this result might be skewed a little bit but um okay how many of you trust code generated by AI? >> Wow. >> Not a single

>> Okay. Last time we did it there was some hands. >> That's an AI conference. Those people know. >> Oh they know better. >> Those people know better. >> They see through the hype. >> When we go to like a general developer >> at least like third of the room will like yeah it's good

No, but those people know. Exactly. Exactly. Okay, we're going to change all that. By the end of the talk, we will do this again and you will all raise your hands. I promise. That's why we're here. >> Uh, so what we have is actually inverse Mr

Fusion. If you remember from Back to the Future, Mr. Fusion is um an engine that you put in garbage and get energy. AI is exactly the opposite. We put a lot of energy in and get garbage. That's I love this joke. But it's the reality more often than not. >> Um just a story that simple examples

>> Yeah. That you will obviously will um kind of um u will appreciate because we saw we saw the show of hands. Um if we ask AI to make um an endpoint an API endpoint faster what it's going to do >> it's going to do whatever it takes. >> Uh for example remove all the validation from the endpoint. >> Hey it made it faster. >> It did made it faster right. It did exactly we we we have tests. >> Yes, but tests with AI is as good as you might predict

Circle of verification is all the rage. It will adjust the test to the code instead of adjusting code to the test. We know that. We tried that. It's exactly how it works. >> Since I'm an engineering leadership, that reminds me of working with junior engineers. You give them a task and they do whatever it takes to succeed. Right

>> Well, this is how they became become senior engineers. The thing is it did exactly what we asked it to do, right? That's the scary thing. We asked it to make the endpoint faster and the endpoint is faster. >> We expressed our intent and it followed >> and we got the exact right result. Now, we obviously thought that it's obvious that we really shouldn't bypass verification of the endpoint. Um, the problem is we didn't say it [music] because we it's obvious to us. >> That's right. and not explaining something that is obvious is known as um a curse of knowledge from behavioral psychology

Um and we all suffer from that. Some people know how to um overcome it especially people who are in teaching and maybe give presentations. That's right. Uh but others don't even think about it. With AI, it becomes a really huge problem. >> It will do whatever it takes. And this is the problem we've been observing since we started playing with those tools probably for about a year now, maybe just shy from that. >> Wait a second

Wait a second. There is >> the the most important thing the the photographers here. >> Thank you. >> Sorry about that. >> Priorities. Yes. Uh what we seeing is we're pretty good at expressing intent. You know, wipe coding got us all democratization of ability to express your intent to a thing that can produce software

Pretty cool. The problem is between that prompt and the intent there's a giant chasm to the output. And since none of you have raised your hands, you clearly know that that issue exists. Well, we want to talk to you for the rest of the presentation and what we believe can be done and the industry is starting to notice uh that it needs to deliver in form of tools and how to bridge that chasm between the intent, the prompt and the output and do it consistently, repeatably and reliably. So one of the keys and this is this is it's it has been spoke about a lot is that mastery is not going away with AI and if you remember when the whole wipe coding um movement starts there was a lot of people who have nothing to do with software engineering that claimed that they successfully um managed to develop applications and the software engineering is not needed anymore. And if you remember the outcome of all those companies and apps, they didn't do very well. >> There's some great Twitter threads on that. >> Exactly

And this is because the mastery is not going anywhere. Why it is important. >> The mastery gives you an understanding of the art of the possible. You still through years of making the mistakes from the school of hard knocks and having scars on your back understand what's possible, what's not possible, when to use it, when not to use it, and you still have to go and master those. And again, if you look at the previous abstraction layers, assembly to compiler, compiler to object-oriented, interpreted cloud, you still have to master all of those for one simple reason, the law of leaky abstractions. No matter how much you abstract the layers between you and what happens in the machine, every once in a while, the obstruction leaks all the way through to the top layer. Every story that the developers become obsolete because of a next level of obstruction eventually didn't happen. >> Yeah

But with AI we it needs help and that help typically comes in the form of guardrails and if you look at the modern uh tool vendors they'll give you guardrail abilities and and they're starting to emerge in form of rules. You can start giving the machine rules of hey don't touch this code use those libraries let's not bring another language into the project. You can start expressing your guard rails to the AI but we don't think it's enough. So do we know what guardrails actually work for software development? >> As an industry, we've been putting those guardrails in front of nondeterministic variable feedback time machines known on humans for decades. >> And the most obvious of those guardrails are obviously tests >> of course right so uh one might think okay so [snorts] how do we wrangle those monkeys with tests? That would be kind of our our question. So I think much like junior developers uh AI needs structure to behave. >> Exactly. Exactly

And we can actually do that by expressing our intent of what we want to do in in tests. Now that might look familiar to some of you. Expressing intent of what software needs to be done in tests is called testdriven. >> We we may have done this as an industry once before. You know, we said you should be writing your tests first >> because this is how you expect >> make them fail, make them succeed, get that mythical 98% coverage. >> It's not about me. >> Uh but uh >> yeah. So has it worked? >> Uh how many of you practice TDD as of today? >> Uh what that 10% >> like real TDD? You write the test for last hand

Let's check real TDD. Like you write the test >> only [laughter] one hand. >> Yeah. Yeah. Yeah. Not like we have tests and coverage like no no we write the test first and then we implement. >> I have a theory of why um there is not all of us using TDD because it's obviously a good idea. Uh and my theory is that developers are biased for action when we think about a problem when we think about like hey we need to do something what are we thinking about as engineers >> that's right like I remember when I was 12 and I discovered basic I'm like wait I can make this machine do something I didn't care about tests and I still don't care about >> test yeah and >> well I do but >> yeah no now you have >> differently >> now you have and as engineers we think about the solution this is how we need to code this think in order for it to work

Now, putting it aside, starting thinking about um corner cases about how the test suit should structure takes a lot of discipline and it takes our dopamine away. We are developers because we want to build Writing tests is not that. So, this is one of the reasons why TDD is not ruling the world right now. And with AI, we have a solution. That's right. >> How about we can ask the monkeys to write the tests? That sounds a great idea. It's actually very nice because then we don't need to write tests. We can do what we love and get the doamine from and let the monkeys write the test

>> That's right. >> Seems simple. >> Yeah, absolutely. >> They're pretty good. Actually, uh the modern models are pretty good about reasoning what needs to be tested in a particular scenario. So that can be a very good start for our guardrails in order to wrangle the monkeys. That's nice. That's nice

>> There is one problem with that. >> The people who write who think about what we need to do are usually non-technical. It can be product managers. It can be business stakeholders. It can be some kind of managers uh that are claim that they are technical but um wrote production code last time 20 years ago >> 15 >> uh okay so yeah so they are definitely not going to write tests for their developers >> I I I thought we did something for this uh we we tried something there was this movement back in the day >> uh anyone has >> for product managers >> how can we do better >> PDD somebody said PDD >> maybe writing um requirements in language that even >> humans >> even business people can understand uh but also machines can understand. Uh this also looks familiar. This is BDD behavioral driven development is about writing specs that are human readable but also machine parsible. >> I don't know about your product managers

Mine loves structure. They just love structure. You say, "Hey, you shall express your requirements in this form." And they immediately comply. >> So I'll tell you a story. We had a conversation about it with a product manager and I'm like, why? Why you won't do this? How hard can it be? It's almost English. You know what they told me? You are as a developer take pride of your code and we don't tell you how to write your code. We write our documents. Don't tell us how to write our documents

>> Fair enough. So they are not going to do it as well because it requires discipline and syntax and our product managers are not exactly going to do that. >> So another great idea where the >> wait let's just verify who practices BDD uh yeah as expected. >> There's a half hand like a sheepish hand sheepish. Yeah. Yeah. So what we end up with the developers now need to write in order to practice BDD. >> That's right

they need to write the uh all the specs and again why bother we already know what the software has to do because we read the abstract documents that our product people gave us and we know the solution. We're going to go and write the code. So why are we still doing that? >> Because it's hard and because we have an alternative for the product managers >> now we don't even need that, >> right? They don't need any of this development stuff, reliability, the art of possible. They just wipe code. >> We can take the software definition document that our product people do and just shove it into LLM and uh get results. How great is that? What can possibly go wrong? >> What could possibly go wrong? >> Let's see. >> Now, spec is fascinating because spec is this thing that is readable by humans and executable by the machines. It gives us the ability to read to give the intent to people who need to read it

Make them read it and then let the monkeys act upon it because it's executable by the machine. So if we add behavioral driven development together with our testdriven development, we get what we call intent integrity check. something that ensures that from the intent that gets expressed in the document all the way to what you have in the working system there's that integrity that continues and continues through every single iteration of the working system. How does it work? Let's see how does it work. So humans write prompt that would be your business people, product people, developers get their input in and everybody agree on the prompt which is usually in the form of document of a spec or whatever you call right sorry document or product document. Yeah. Now monkeys generate spec which is human readable but already breaking down already broke down to the points >> complex ideas into small incrementable implementable >> executable >> executable chunks. >> Now of this spec we can generate tests and once we have tests we can let monkeys write code until the tests are passing

>> That's right. Those are the guardrails. They'll keep the monkeys in check. >> Exactly. So, this looks like a great idea and we could finish our talk here, >> but >> but never trust a monkey. Never ever trust a monkey. >> Right? We just talked about maintaining intent integrity and we've given a stoastic nondeterministic system a run of this whole thing. >> Bad idea

>> Bad idea. So, what do you do? >> There are pieces here that are definitely cannot be trusted to monkeys. For example, the spec that monkeys are generated from our talk from our prompt. Can we just blindly say, well, this spec is implementing our prompt because monkeys did it. >> I I remember when we first attached pet clinic to this thing and it hallucinated requirements that didn't exist. >> Exactly. >> It's great. >> Exactly

>> They're really creative monkeys. >> Now, if you remember, the tests are generated from specs by monkeys. Uh not a good idea either >> because obviously the tests might end up being very different tests from what it was in a spec and what we was in a prompt. >> So how do we solve it? >> We solve it by removing monkeys from the chain where it is possible. For example, if we have spec which is parsible by the machine, it means that we do not need to rely on monkeys >> or spend a lot of energy on those tokens. >> We can build a deterministic algorithmic solution that will take this spec and convert it to tests without any non-deterministic stoastic systems involved. So let's see how it looks >> and then we can actually tr then we can actually uh uh trust our tests because it's not monkeys anymore. The other problem is that if you remember uh the circular verification between test and code right if we say to monkeys hey make sure that this that this test passes what it's going to do >> it will remove the test or it will make it return true

It passes. >> Exactly. >> So, what do we have to do to prevent that? We have to take monkeys away from the tests. >> Just hit them on their little fingers and say, "Don't touch. Don't touch. Lock the tests." Right? >> So, you need a framework that allows you to log the tests while the LLM does its thing to make sure the tests do not change inadvertently. >> Now, Leon, you are a longtime Linux kind of expert. So you probably might have think about the idea of how can you make something that cannot be changed on the file system

>> Make it read only. >> How about that? >> How about that? >> It's as simple as that. >> As simple as that. >> Make the files read only. >> No guidelines, no rules, no rule files. Just make the files read only until you're ready to make them read. Right. And this is how you protect the tests and taking monkeys from another point that we cannot uh that we cannot trust

So now we actually have a system that is not trusting the monkeys in any point of time. Prompt is written by humans, read by humans, validated by humans. Spec is generated by monkeys. Yes, but since spec is human readable, everybody can read the spec and agree that what we captured in the spec is exactly what we wanted from the prompt. And then what we do is we let the machines do what machines do well deterministic work without spending bunch of tokens and energy on needless token generation and generate the tests, run the tests, validate the tests and >> and then let the monkeys run the write the code that fits the tests >> until it passes. Now this is a completely iterative process. We can go to the part when we nail down the prompt correctly and the spec correctly as many times as needed. We can add features

We can add features that we didn't think about. We can add more and more microservices and and like generally different requirements to the system. This is all iterative process until we actually do until we get what we what we wanted to get. Now the side benefit of all of that is you now have something that you have never I never had and I don't think anybody in this room had ever in your professional career. You have a fully documented codebase and a codebase that is documented with all of its intent in plain readable English. And by the way this also works and we run those experiments by applying those techniques to existing systems and extracting their current intent before you start modifying them. This is an amazing side effect. If you run the intent integrity chain on an existing codebase, the first thing you do is gaining understanding what the current system do

And you might think that you knew it beforehand, but more often than not, you will learn new things about your system just by applying the intended integrity chain on it. those little undocumented side effects that the system actually relies on get discoverable. >> Yep. >> Which is great. >> And um so the idea is that only what's in it spec get to the tests. >> Pretty simple, right? Spec to test deterministically and then only what passes the test stay in the code because this is what will make test pass. That's right. So let's review this one last time

The intent starts with the prompt translated into a spec by a machine >> by monkeys >> by monkeys. They are much better at this and they actually quite good at collaborating, brainstorming, thinking about various requirements, asking questions you may have forgotten, right? >> Verified. >> You still verify them by a human in plain English. It's easy to read. They're easy to understand. Those of you that old enough have done other older processes, spec review process, right? You go read the documents and by the way they have inversion control of course >> they work with the system those specs eventually get uh generated into the set of tests that again monkeys are pretty good at take this plain English and generate set of scenarios for it they're good at reasoning in this you verify those again with a human and then >> and then you let the monkeys you let the monkeys run until the tests which are protected are implemented and you got it. Now this is all the rage now. The specdriven development in AI is I think one of the hottest topics of the day

Um two most interesting tools in the space is uh Kira um IDE specdriven development IDE uh developed by Microsoft went to general availability uh sorry Amazon went uh general availability yesterday. Um and um another one is called SpecKit. Uh and that's a toolkit developed by GitHub that you can use with almost any tool and IDE. Uh both uh are linked right here. So if you go to show notes.taxcare.com you will find a page that has the links to do all to both of them. They do pretty much this. They might not call it intent integrity chain but they are very close. The vendors are starting to notice and others are starting to implement elements and some of the shops starting put to put together tool chains like that using other tools factory cursor wing surf etc etc all of this is doable in any of the tools you just need to have a lot of work and we believe that the next evolution of those AIdriven tools for engineers will be come with the system not just the tool >> and that's exactly what kuro and the spec kit are doing well uh something that we at least can get behind is the um you know what it's been almost 3 years by now since Andre said that the newest hottest programming language is English and we wholeheartedly agree >> we've been doing this as an industry forever we've been moving towards that >> up up the abstraction chain towards >> English >> towards a language and uh yeah it's not about pro prompting it is about prompting >> prompting is hard but that's not what makes a reliable software system at scale >> yeah it's It's about alignment

It's about how do we make prompt, spec, test, and code all >> coexist and produce a cohesive system >> towards the same goal. >> So, let's review one last time. >> Yep. >> Every abstracted leap started with a panic, a moment of confusion and led to an incredible uh leap in productivity in our industry. >> Yep. Shared understanding is now executable. If we all agree on the intent and we can verify it by spec, the rest down the road is just zeros and ones, we can get software to the exactly as we want as long we have sharing understanding that we can express and being able to express that intent and maintain the integrity from the idea to execution matters even more. giving the scalpel we just handled every hand it to every single engineer in the world >> and this is where mastery come in and understanding what you try to achieve is so important >> with that thank you >> thank you for listening [applause] >> Jay Bar I'm g intending integrity chain [applause] which we we're trying really hard to push this so help us with some hashtaging here show notes tax.com this is where you go um you can definitely still scan the QR code and see all the slides

Uh I will upload the video later today. Uh and all the links and the raffle for thank you for being here. Thank you very much. >> Rumor has it there is a 3hour video in internet that shows all these tools in action specifically. >> It is it is right there show notes that tax. >> So it is real. it it works in production in our use cases and uh we've been uh playing with this since January. >> Uh yeah so uh we doing this talk with Victor called no no Robocoders judgement day that show the the entire thing in action

Yes. Yes.