Building Tools with Tools: Modern AI-Driven Development | Edir Garcia Lazo, AI By the Bay 2025
Thank you. Um, hi everyone. Um, thank you for joining. Um, I am honestly very excited to be here at AI by the Bay. I love the bay. I'm visiting from like a rainy Seattle, so I love the beautiful weather that you guys have down here. Um, today I will be chatting about building tools with tools. uh modern AIdriven development
Um I want to briefly explain what exactly is uh this talk. This talk really is like a collection of learnings, tips, advice uh on you know like how things have changed. Now we're all just setting off agents. So these are some of the things that u me and some of my peers at Sentinel 1 have learned over the last maybe year, year and a half. Uh firstly and really quickly, let me just introduce myself. Um my name is Adir Garcia Lasso. I've been working in the industry for like 15 years. Over 12 of those have been in AI mostly uh uh on applied applying the AI and ML into cyber security
I've had like many names during all of my career like uh for role software engineer data scientist applied researcher you name it. Um on companies now now at Sentinel one um uh but pre previously as well as Microsoft Oracle etc. Um, I am a staff machine learning engineer and I work on the AI science team. So, just by a show of hands, who has heard about Sentinel One before? Okay, one person, two, three people. Okay, just really quickly, Sentinel One is the AI power cyber security company. Uh, they make the Singularity platform which is like an autonomous protection against cyber attacks. Um, uh, it works over endpoints, cloud workloads and identities and we have been doing AI for a very long time. We are the creators of this product called purple AI uh which is the industry's most advanced AI security analyst assistant
U think about it of like an integrated uh LLM assistant uh that you you can use to to you know all of your cyber security needs can translate natural language to structure queries. It can summarize event logs. It can thread up look up thread intelligence on the fly. And it's extensively used on the socks or security operation centers. and it really shines threat hunting and on incident response flows. Um this is not what I am going to talk about today though. This is just like a small ad. Uh I'm going to chat about other types of of assistance
Um most notably uh AI coding assistance. So depicted here in the screen uh some of the three that I personally use and some of the ones that probably you use as well and these are the CLI kind. Um, and I really want to just do like a small census just for my own personal information. So, just by a show of hands, who uses a coding assistant? Or rather, at this point, honestly, um, who doesn't, right? I'm actually just curious to see if somebody here does not use a coding assistant yet. See, so everyone uses one. Okay, great. This is this is at least good news. Um, as you can tell, there's like so many of them
Like, there's gazillion of them. like really it's just uh exponentially they have grown exponentially like I have here in my notes some of the names that uh that you have probably heard cursor replet codeex claud winds surf gemini loable cognition uh on this conference alone I just learned about google jewels about amazon uh about amp and yesterday uh but while we were already at this conference uh Google presented anti-gravity right so it's a lot of them it's so many of them how do pick the best one. Um, so I'm not going to go uh to kind of like get the senses on who uses what. Uh, but this talk basically will be focused more towards the agents mainly. Uh, not so much about the AI powered ids. So less of the windsurf replet cursor, more about like cloud code, uh, Gemini and OpenAI codeex. Um, okay. So who here is familiar with this? I bet that everyone uh actually Peter Norvik just showed the exact same slide this morning which feels very validating to me um you know to have such a great author be thinking about the same things
So this is the software development life cycle. Uh like the first concepts of this have been around since like the 50s and 60s. So you know it's pretty old like the version as it stands right now on your screen uh it's from the 70s and that's like 50 years ago. That's a long time. So software development and programming as we know it has changed fundamentally like we are not going back doing things like we were doing it even just a year ago. So what do we do about this like is this still a model that is valid? Uh I think it needs some tweaking if you really think about it carefully in in this diagram what happened was that you can go over it way more quickly. The bottleneck used to be like step number four implementation you know writing careful code it used to take some time it used to take like a lot of expertise um but now you can just do it really quickly so um we are basically proposing like a small modification to this uh and you know again I I really preaching to the choir in this conference I feel but like in the in the last year of my career I have never seen the change uh so quickly uh it's uh it's exciting but it's also a little bit scary so what to do Okay, so the proposal uh is really just this simple like an identic quote unquote SDLC, something that you are um like going through a new process that basically captures the same essence of the original SDLC. Uh but it's just uh modern and it's just uh with agents at the at the at the front uh at the front
Yeah. Um so you know the process goes something like this. you have an idea or you have like a an issue or a bug or whatever just like the concept of like what is it that you want to do then you raise a GitHub issue it could be like a bug a pull request it could be whatever like uh in the in the GitHub environment but that's the main entrance the communication point then the agent with a certain persona would look at it tackle it you review it obviously very important step and then you know eventually you make it to production so all of these things are really just like a collection of like three things or like the three foundational blocks are three uh for for me here I'm actually naming three particular things but it doesn't have to be that way for you obviously but generically they are a text editor a coding agent and a terminal to execute commands um so let me just expand upon those steps a little bit and this is how I have been thinking about this flow myself so originally in the ideation phase I am basically having a conversation with an AI like it could be just traditional chat GPT or deep research even better if you want know know about something you know you can just capture a lot of context really quickly by just kicking off a deep research on something that you're interested in or just asking the questions you know many of these things that we studied in school that you maybe no longer remember it's amazing to just go into chat and ask like hey how would I do this like you know this type of stuff um so the goal here is really that you're looking into more new ideas and faster and hence you have more ideas so you're literally just covering more ground so it might you might find new perspectives that you have not considered. Uh the second the second part is like the GitHub issues, you know, a GitHub issue can start the flow and this could be done either through like an GitHub MCP server or some of the tools can just actually run the command themselves on the CLI. Um you can raise an issue or whatever bug like it could start like it could be a refactoring, it could be a new feature, whatever you want to start it, right? Like it's just as as long as it's capturing an issue, agent can come and pick it up and and and go. Um on the agentic phase uh you know one of there's uh one of the things that has been really really good to adopt uh tent one and me personally has been uh the adoption of personas. Um many of the tools actually have that built in. I will talk about that in a second
Uh but the idea is that you have like uh a sui agent, a researcher agent, a tech writer, somebody that's always telling you no, that is always finding the the the flaws in your approach, security analyst, you know, because okay, if you don't actually have somebody that's taking into account uh uh looking at your security, you might uh I heard this statistic recently that about 40% of the VI coded uh uh projects have like some sort of like a vulnerability or or security issue with it. And that's big. you know, we definitely do not want to have that. Um, so you can actually have multiple agents work on the same task and then aggregate. That's also an approach that's been very successful for us. Um, and uh, additionally, you can also have like several kick off multiple agents in parallel to work on different features on the same codebase by using git work trees. This is a a really nifty trick. So you create a different git work tree and then you're like working on the same thing without having to do that back and forth because you're on the same branch
um and you agents doing parallel work uh at uh uh at at exactly the same time right uh review I will go into more detail later but uh effectively it's just like have an AI to look at your code before you check it in it's as easy as that but I will I will uh tackle about that more in a second and uh preferably being done by a different model that wrote it like uh I'll explain the reasons why so in order for to achieving this type of flow like um we are adopters of like the Python environment, you know, starting from the very very foundation like there was yesterday this debate on whether traditional enterprise tools would prevail over like newer uh AIcentric slower maybe tools and uh I feel like for me personally I'm very much on the side of of uh of Python. Um, it has an amazing environment of tools like from tools uh for linting and formatting like rough to static type checker checkers like like my all the way down to modern agent building infrastructure like fast MCP or or by AI that these are all partners and the people that you have heard here on the conference they have everything like the environment has everything okay so you've been promised tips and I have not really said anything yet [laughter] or uh about that specifically so I'm just going to go into like the actual learnings and stuff that uh I hope it resonates with you and honestly if you've been around here at the conference is stuff that you have probably already heard and the reason why that is is just because you know everybody just kind of like thinks about the same things um and it's nice for me to kind of like see oh yeah you know like this is something that I was going to say on my talk so it's nice so bear with me if it's something that you feel like it's too basic okay so uh the first number point is your CLI tool, your in my case cl code or codeex or Gemini, whatever you use is now your shell. So what does this really mean? This really means that you now tap your commands here. You no longer need to go into Google and search for the commands on how to do an interactive rebasing uh uh cherrypicking uh commit or rewriting your git history. You just tell the agent for you to do that. Like this has multiple benefits. um you know I for example don't do these things enough uh often enough to remember the syntax so this can save me massive amounts of time I just say like hey can you please rebase this and it will do that right like it's just kind of a great uh you obviously still need to understand the concept like so now at this point it's more about really knowing what a rebase is really knowing what cherry-picking a commit is rather than the execution itself so like the focus is now more on on like the concept um another thing is like uh several tools like cloud code like you can wrap a whole command and give give it to your agent so it'll execute that. So if it's something that you're constantly uh uh running often then you can just wrap it in a in a slash command and just give it to your agent and it'll just be part of your flow natural
So like you can make it become a first class citizen in your environment and that's fantastic. It can have parameters. It can actually be autogenerated by the tool itself. Like you can also tell code ape, please generate a slash command on this and it'll do it. And after that it can be directly invoked uh by your assistant. Another kind of like crazy thing that I've been doing recently is like you can actually add a slash command that would kick off another instance of claude. So effectively making it a sub agent. Um that is uh really really cool
So because claude can actually run in like headless mode with like the d-print uh flag and you can basically have like claude start off another agent and looked into like a completely different thing. So you can you know have like three different versions look at the same time and and then in the end just consolidating uh all of the results. Okay. Secondly, another uh tip is use all of the models you can. Um so this is uh I was comment on this regarding on like okay have one generate and have another one review. I think that is important because models are trained differently. Uh the same as platforms, models have different strengths and weaknesses. And make sure you explore them so you understand which where's the value in each one of these
Um uh I have a comment specific in this version of the talk about San Francisco itself because I know that in here there's a lot of people working on startups and obviously startups you want to run lean on the money, right? Um and you want to extend your runway as much as possible and I think that's perfectly fine. However, my suggestion is do not save here. So actually try to get like the highest quality of agentic tool that you can afford. Um uh because it's just gonna it's just going to make you go way faster and it's going to pay itself over over and and and just how quickly your your startup's going to move. Um I personally gather the requirements like I said mostly with chat GPT you be using GPT5 GP 5.1 uh kicking off deep researchers like that's really what I do. Uh after that I uh write like a spec uh markdowns and I use us planning like the opus planning mode in cloud code extensively. If you haven't heard about this like I really um encourage you to look into it and when it comes down to implementation and coding like I use for the most part uh claude uh son uh sonnet 4.5 like that's just my my favorite. Uh okay new tip new tip um enforce an AI review in CI
This is why I kind of like put the push pin on on the comment earlier. So, one of the things that we have done at at Sentinel 1 that is very successful for us is like we literally have like an agent looked at every single PR that goes into our systems. Uh this is really easily uh doable by like a GitHub action or or whatever build system you might have, whatever CI system you might have. And it's amazing. It really is amazing. It can catch drift on documentation. It can catch actual bugs. I've actually seen it capture actual bugs multiple times and it's great
You know, many of the many of the other companies I guess already have like a solution built this for this. If you are like on VSO online, Visual Studio, uh you already have this built in, right? But even if you don't, you can build it yourself. It's pretty easily and I highly recommend you do it that um and another cool thing is like cloud code for example uh uh integrates with GitHub actions. So you can cloud you can tag cloud uh in a GitHub issue and it will just like automatically start by just telling it hey at cloud can you please start looking into this enter that's it and it will just on the background continue and get you a PR as output and then you can decide whether you want to take it or not. Um, so yeah, it will always catch things that you didn't think about. It'll help. It helps me a lot, especially with documentation all the time. Things that I'm like, "Oh, this dock string was wrong and it's inconsistent." I'm like, "Oh, I never would have caught that." Like, now it's just, you know, I'm really glad I did
Okay. Secondly, do not accept agent misbehavior or hackery. Um, so these things are very powerful, but they must be treated like basically as every contributor, right? Like you would just not randomly accepted code in your code bases. you need to uh obviously have like a high security uh high quality bar. Uh so this is no different actually debatably you know uh it should be more right because it's not it's not authored by a human but you know like so maybe it really comes down uh to trust do you trust it yes or no right like there's also been this conversation of like how much you should trust the output of an LLM for coding uh so there's a uh a Russian saying and I I pretty much like it a lot that fits really well in this case which is trust but verify so yes okay good you're going to use that but like make sure it does what uh what it what what you want it to do. Uh so when I say here misbehavior or hackery, what really do I mean? You know, a very common thing is like if you allow your system to give it a goal of hey uh make sure that all of the tests pass. Oftent times it will just remove a test or it will just like literally just say like oh this test remove all of the contents, all of the actual uh and just make it pass, right? Like just do a a return true type of thing. Uh so you want to be really careful with those
Uh another thing uh that this thing u another example of of misbehavior or hackery is it will ignore type enforcers for example or conveniently insert non-types wherever it just deems it appropriate right um so this is for example one of the things that I personally have in my system prompt uh so we will never do this so it it reads you're not allowed to use any object cast type ignores or any other techniques to work around typing issues This is so I can use my pi and force that all of my code uh it's going to be type safe, right? Like the only time that you're allowed to do this is when you're like testing when you when you're like writing a test was like okay I need to actually feed something that is not the appropriate type is the only scenario when I will allow this. And yesterday also there was this conversation on whether uh Python can be type safe. It certainly can be. You just need to you know do these things and and make sure that you're running uh my pi constantly. It could be part of your flow as well. You can also put that here. Okay. Second, next tip
Context. Context. Context. Context matters so much. There have been whole talks about this uh on this conference and you have probably heard it at NISM at this point. But you know that doesn't make it not true. Uh accept it, embrace it. It matters a lot
And I actually have two concrete examples on uh what exactly this means because sometimes it gets thrown around a little bit like oh context matters. Yeah, but what exactly does it mean? So the LLM needs to know needs to know what it's working on, right? So 3.1 agents are great at writing documentation. Agents love up-to-date documentation. So they can create and maintain documentation and they actually thrive on the existence of it, right? There's like whole now schools of thought that are just basically saying that you write like a spec and it's just completely going from from the documentation is what kicks off the beginning of the agent. Um so you make a change and have the agent update the docs is as easy as that and then you know next time you run it will look through that documentation through those doc strings and it'll know better what to do. Uh you can also do automatic change tracking you can also do automatic release notes change logs all of those things automatically with an LLM. uh and you know maybe those are kind of like seemingly obvious things uh but it is really important to keep it up to date because in the end LLM's deal with text and documentation is text um so there is one thing though and I want to kind of see if somebody guesses it there is something better than documentation for a coding assistant do you guys want to try and guess what it is a trick question it's a trick question because it's code [laughter] So uh another thing that we have uh adopted internally at like Sentinel 1 and I've done it as well in my personal projects is give the agents real code. So creating dependencies as subm modules instead of like just having your piport like your UV package like you actually create a a a soft module on git and like grab all of the code
So I've done that for like fast MCP for like pantic. So whenever your your agent is like scarring through your project, it can actually look into the code. It doesn't only have access to like the signatures uh of the methods and things like this, but it can actually look at everything. So it can uh make smarter decisions. Um so like I always say is like having documentation is good, but having the code is better. Okay. So this is um some uh uh pitfall that I want to call out is like make sure that your pi project and dependencies do not read different versions like I have an example here on you know how to check this. It's just running a git of module and uh uh for each command like just the describing of the tags and making sure that that matches the version you have in your pipe project
This is very important because it's not if not if you don't do this you'll have like all kinds of weird things happen. Um, and you can actually also do do this like this validation on a GitHub action. Um, so make sure you do that if you're do if you're doing that. Next tip. Okay, so steer your agents. This one's actually really wide uh tip, but it's on purpose. So most agents will have like a a markdown that you can use for for general project memory or guidance. And make sure you use those usually called like agents.mmd or or claw.md things like this
So you want to add like important things that are valuable for you and your project, right? Like I said earlier, I have like every time you make a change, you're going to run my pie. I have that on my on my general rules. So there is nothing that I can nothing that the agent does doesn't ever comply with my pie or rough for example. Uh the other one of the other way of steering your agents I just mentioned briefly like have uh write your specs into markdowns like that is amazing because you have like done this idiation phase and you've done the the the the deep research you've done the conversation you have narrowed down all of the things that you wanted to do and then you just write a a markdown once you can manually edit that markdown like just being okay you're the human validation like no I don't care about any of this like why why did it do that you just delete that edit whatever want and uh and just get it get it running. Uh also use the different levels of effort in either GPT5 or for example claude has think keywords. I'm not sure if you guys are aware of those but those are fantastic. I think they are think think hard and ultra think. So it's just basically you're asking the model you know to look into it thinks harder
So you can help also steer your models by doing that. Um the other one is like uh write tests obviously make sure that the the like that you're testing the correct thing. I think that is essential. Um, I'll have here an screenshot of like some of the things that one of my colleagues, this is not particularly mine, it's one of my colleagues, but some of the general rules that we have in there. And um I for example have always one that says do not overengineer because sometimes I've bumped into many cases where like the solution that that the agent provided is actually really good but it's just like really difficult to read and it's just like using a bunch of uh uh design patterns that you know I'm like okay I just wanted something simple. I do not want any of this even though this might you know be more extensible down the road. Um so this is what I one of the premises that I put on it. Okay, so in all of this, where does MCP fit? So the reviewed version of all of this now has evils as well
And MCP servers are really the the the the glue of all of this. Like it puts all of this process together. Um so it'll enable your agents to have capabilities that are not necessarily built in already, right? like you get what to expand and also remember this is uh essential an AI coding agent can actually write you know uh its own MCP server and then use it. So you can expand the tools like the possibilities are really endless right. I as you notice here like evals are inserted and uh beyond human review. So here like after idea and after review right and those are inserted there on purpose uh because you need to have some sort of visibility to know whether the changes that you have made to your system are actually meaningful and right now in an in an LLM system because of the stoastic nature of it you will run the same thing multiple times and you'll get different results every time right so I think evals and repetition are essential like traditional methods like uh data science methods of like making sure that the change that you've made statistically significant actually still very much apply. Um and uh you know it's not only me who thinks this like no things uh uh the measurement of AI performance it's still a nent discipline and methodologies metrics validation standards are emerging but they're not fully yet established. So whomever tells you that that's the case, I would just not take that into account
And I actually copied this over also from our folks of uh uh friends at Biantic AI because this is really what they have on your on their documentation. And I completely wholeheartedly agree with this. So you need to think about your business case. You need to think about what's important for your business case. You need to measure it properly. But nobody can come and tell you like deterministically like, oh yeah, you should be doing this because it it'll vary. It'll vary from case to case. Um that was actually the last tip I have
Uh but I do have one more thing to say and that is um Sentinel One really loves the community uh forming events like this one and thank you for listening in. Oh there is one more thing actually uh that one last thing is uh so we are hiring and we if you want to push the boundaries of what is possible in cyber security uh make sure to scan that. We have some job some jobs uh openings and they're mostly remote. Thank you. [applause] >> Do you have a question? Yeah. >> Question about markdown. Do you put all of the description of all the features in a markdown? uh or like and you always keep it there like description of the whole project for any other additional uh features that you're working on. >> I love this question
I love this question. The question is like whether you put everything in just a single in a single markdown and we have it broken down in my personal projects and at Sentinel one we have it broken down. So you have one that will be high level architecture where you have like descriptions of what component it's what but then you will have for each subcomponent or sometimes for even for ex for for each feature you'll have a separate markdown and then obviously the the LLM can like smartly pivot and just look at the architecture knows where you know like the related the related files to this are and knows that for example if you're looking at the authentication service and I gave it an authentication bug it's not going to look into other places right it's not going to look in like the backend data libraries it's going to look in like the authentication part uh and uh but no it's only a big level architecture one and then many as many as you require that describe like particularly the features and yeah you really just have a bunch of those things uh literally sometimes checked in in your branches >> question yeah I I was uh curious what does it mean for um collaborating with the rest of the team so are you looking at the ai agent is something personal and uh it's committing code on your name. But what does it mean? Okay, you have five people working in a team and they all work on the spec together. >> What what what is what is the dynamic? How does the dynamics look like? You know, >> interesting. Um yes. So we definitely do check those specs like the process uh is you know you you come up with it but you definitely want to give it to someone else to read, right? like it's just kind of like the the I would say is the mo the modern PR is just like the spec review. This have I guess existed since the ancient times you know people would sit in a room and like read the spec but now it's more important than ever I think
So you definitely talk to people and you want to make sure that everything is written down clearly. Um because if there's anything missing you know whenever whenever you want to say create test for that to do like TDD or whatever whatever approach you want to have like it is very important that everything is explicitly stated like if A then B if C then D etc. So everything needs to be spelled out. Uh but yes we definitely do collaborate on on like the specs. Yeah. And just a follow-up question, it's like how do you iterate? Like, you know, you write the spec and it seems like it's doing everything all at once, right? >> Rewriting the whole code, but maybe you want to be more incremental. You write this then you follow up with the test. I don't know in Mars and is how is how does that workflow like? On the agentic level, one of the things that we do and I I I mentioned on this, this is what I mean like it's important to keep the documentation up to date because this specs and this markdowns that were created during the the kickoff of the agentic flow are are still descriptors of what the what coding change happened, right? So whenever you fix something, you also need to go and you can have it in your in your system prompt, but you know, you can also tell it to go and change it so it'll be updated
Does that make sense or does that not really answer your question?