SBTB 2023: Melissa McKay, Navigating a Mined World: Repositories, Registries and Artifact Storages.
welcome welcome so glad to see you all here um I am originally from Denver so I had a short plane ride here but I was really excited to come to this conference it's my first time speaking here so excited about that anyway this talk is actually combination of material that I feel is super important to get out there in order to increase awareness about security as a developer and that includes the development of AI and ml projects uh other Frameworks and tools as well and awareness about where you are getting and storing the software binaries that you use and build for your projects is in fact just one part to be concerned with um how many developers out there okay uh how many are either relatively new to software development or you do it but it isn't your primary job okay cool we've got a bunch of developers in here all right a lot is going to come at you fast I know it's the end of the day uh but these slides will be available for you and there's lots of resources in them you'll be able to refer to later or you can reach out to me on Twitter or LinkedIn okay most important thing on this slide is my contact info here um also I am a developer first and foremost I work for jrog currently as a developer Advocate but for over 20 years prior to that I was a developer all the way from your lowly intern to a principal engineer and so most of my talks are definitely from a developer perspective even though I'm working for a Dev Ops company I I hang on to that developer side pretty strongly uh the other item I'll mention here I am an author of the book devops tools for Java developers Java is my primary background and I co-authored that with um Baro sersi as one of the co-authors and he's going to be here tomorrow so just a quick plug for him um he's doing a session uh in I think around 12 at a little after 12 all right fairly recently I had the privilege of working on a little project with Damen Curry from enginex at F5 we were developing a series on YouTube around the application journey and Damian comes from an operations perspective I of course come from the developer side and we decided to have an informal conversation about what it actually takes to develop and deliver a cloud native application we had a number of episodes we tried to start from the very beginning from just planning a project all the way to um not only just deploying that project but observing that project and making sure that you get feedback all the way into the beginning of the development loop again and some of the topics in episode three are what I'm going to be focusing on today and if you can imagine some crazy things came up and one of the most most important things that I realized was that developers are not Security Experts we're optimistic folks we like to build things we like to build tools that solve some problem in the world or even just to make further development a little bit easier for other developers I believe I'm right in saying that securing our software isn't often at the Forefront of our minds especially early in our careers this was my very first intro to security how many of you recognize this page all right this is just a typical Apache um default homepage Administration and stuff and my first job was to go through and make sure this did not appear on any of our projects that we had out there in in uh production and this was security through obfuscation and I know never had even thought of it before but I I was told at the time and we can argue about this later whether this is a good uh way to do things or not but I was told at the time that even just revealing the version number of the web server that you are running could be potential attack Vector um if there are known vulnerabilities out there you're done for we can all agree that downtime for any reason is a problem it can affect your organization's credibility uh it can affect your customer personally it can certainly affect your pocketbook in the way of lost time and production lost customers and in worst cases lawsuits and fines and with that we'll whip through a couple of old but famous hacks and talk about some of the real consequences so this is log for shell and in a nutshell this is an exploit where one can trigger remote code execution by providing a string in a certain format that ends up being picked up by the log for J library that that turned out to open up the ability to initiate an ldap lookup fetch some compiled code and execute it and just about every Java developer out there uh cringe their way mitigating this particular vulnerability and according to Steven McGill vice president of production Innovation at sonatype approximately 70,000 open- Source projects use at log forj as a direct dependency and nearly 174,000 projects use it as a transitive depend so this is not a simple problem to solve it was widespread and a big deal the Equifax Equifax data breach happened uh through March um March through July of 2017 so time some time ago but enough time for us to now realize uh some of the costs associated with this this was due to the Apache struts vulnerability and unfortunately for Equifax it was a known vulnerability $1.4 billion in cleanup costs is estimated uh 1.38 billion in consumer claims estimated 147 million customers were affected the Equifax breach was an example of a data breach just one though and there's been a lot more since then so this is another scary stat about data breaches this number is huge 4.45 million in USD um the average cost of a data breach in in uh 2023 and that's a 15% increase over the last three years we hear about these a lot and this is a website that I just I found and it uh keeps a really good uh update Cadence on the new ones that are coming out there's at least a few every month some of these may be due to social engineering maybe uh fishing or employees are getting tricked into giving up a password or something but three breaches that were on this site uh that happened relatively recently stood out to me and all of them are from this year so we have June 1st July 20th and August 11th there were three data hacks that had to do with the move it transfer tool there was a a featured article on Tech crunch about this I have a QR code for that um Carly page described the move it hack as the biggest hack of the Year by the numbers and if you were to read that article it's it's pretty horrendous so why does this keep repeatedly happening we aren't helpless it can feel that way but uh we're going to dig in a little bit and figure out what uh vulnerabilities actually look like how we can better protect ourselves and I mentally divide attacks into these two buckets uh those that involve code artifact or package security and then everything else which involves maybe configuration system or network security I'll touch briefly on the everything else bucket at the end but I'll be focusing more on some of the ways that I relate to given my background as a developer that's going to be mostly around artifact and binary management and coding defensively so this brings me to our first step in troubleshooting this problem let's go to the very basics of a developer workflow this is intentionally simple and naive but this is actually what we do day in and day out would you agree we write code we build it and then we run it and then we repeat so where is it exactly that we get tripped up how many of you agree with this statement as a developer it is my responsibility to write code that is secure good this is reasonable I found this essay out on the oasp website um just a quick plug for OAS they are the open web application security project they are a nonprofit Foundation that works to improve at the security of software they were launched in 2001 incorporated as a nonprofit charity in 2004 so they've been around quite a while and are um you know there's a ton of experience behind its members and one of the things they do is provide security resources for developers and other practitioners so I went to check them out found this essay it's pretty funny it lists ways to write insecure code opposite of what you would expect so if you enjoy sarcasm and some irony this is definitely for you there's some pretty clever points in here and a few of them I wanted to pull out just because they made me chuckle I like the always used default deny so deny that your code can ever be broken deny vulnerabilities and there's unless there's a proven exploit and deny to your customers that there was any ever anything wrong and uh deny any responsibility for Flaws secure languages who all here writes in a secure languages only yeah you all are you all are Smite none of us none of us write in a secure language there's no such thing mix languages I've been on projects where we've had a number of different languages involved and for a good reason there were legitimate cases for this but um this is funny just because yes the more languages you add the more difficult it is to learn them all so the idea is to off escate vulnerabilities through using all kinds of different languages together in one project this one finally I put on a slide on its own I'll let you go read the rest by yourself but uh this one really made me laugh uh rely on Security checks done elsewhere I've heard this argument a lot especially for uh a development team um it's redundant to do Security checks twice so if someone else says they've done a check there's no point in doing it again all right how many of you were formally educated in software engineering or computer science and you took a security focused class before getting employed in the field okay there's three or four by security focused I mean a class that maybe had the word Security in the title how many of you the rest of you if you were not formally educated have you taken a security class good good this is getting better I will admit here and now I did not take a security class during my education this was way too long ago maybe before it mattered as much my personal experience was getting trained on the job it was not part of my formal education and fortunately for me my employer was very much about this making sure that we learned learned how to code defensively and safely so I learned all the things and I really thought I knew all there was to know about secure software I was done after I was done um clearly that's not the case there are a lot of other things that a developer needs to be concerned with we really need to consider developer maturity when we're making decisions about our security posture Junior developers obviously or anyone whose primary job hasn't been software are going to need more guidance and oversight than more experienced developers even if a developer has gotten training and the best practices to code and they're a master writing code without Securities there are other significant vulnerabilities a developer can still become prey to and I'll get to these in a moment but some of these you may not have control over so it's important to know you know where you can actually make a difference and where we need some help from other teams in our company or organization so let's review our very simple developer workflow and let's make it look a little bit more like reality and it turns out developers love to write tools to make life easier that's really all that software is and developers also detest Reinventing the wheel it turns out that organizations also don't enjoy paying for Reinventing the wheel so this whole idea of free and open- Source software that can be used and built upon by developers pleases many levels of the business business hierarchy so we continue we write code maybe we declare more dependencies as we learn more about our project and our requirements we build the code we run the code and then lastly if the code is not proprietary a developer May contribute their creation uh as free or open source for someone else to build on top of so let's accept this as a reality we need to talk more about dependencies and and I went to a session earlier today by Dean Wampler just before lunch did any of you attend that session no okay he touched on this topic a little bit um that there's a concern about imprudently bringing in packages and libraries and building off of them without considering carefully where these packages and libraries come from or vetting their compatibility their maintainability or even their security this is really easy to do package managers and build tools often make that easy on purpose and many that are creating don't stop to consider or they just might not be aware that this is a problem so when it comes to analyzing the projects that a developer typically works on the reality is there's a ton of codee that he or she did not write it's code that someone else has written and there's a lot of trust involved trust that can sometimes be misplaced an answer maybe we shouldn't bring in any independencies uh this is not the reality of today and in most cases it would be cost prohibitive to require a development team to write everything from scratch just for grins I went to a demo project that I've been working on I selected one of the components uh to analyze this one component is one of seven microservices involved in this project this is a spring project uh spring boot project and uh I did a little ENT Tree on it to figure out how many dependencies I had there were 114 direct and indirect dependencies as many as seven layers deep this is code I will never look at I've never seen before it's it's code that I build on top of so this is concerning in a way this is from the open source security and risk analysis report and it includes the cyber security Research Center findings for the year of 2022 they scanned 173 Commercial Code bases across 17 different Industries and I took these three graphs from the section on the use of Open Source I thought these industries these three industries were pretty interesting to look at and note that the lighter color represents the percentage of the code and code bases that was open source and also note the red line indicating the percentage of code bases containing high-risk vulnerabilities and notice that little uptake in 2020 this would be a fun discussion to have during our happy hour tonight I don't actually have an answer for this but I do have some theories and I suspect you all might have some too there are some risks that we inherently take when we bring in dependencies without vetting them we could bring in a dependency that's vulnerable already we could bring in an unintended version of a dependency something we weren't planning on but we ended up up with um this could be both internal or an external dependency so it it could even be one that uh was built in house we could bring in the wrong dependency altogether we'll talk about that in a little bit or we could fail to update dependencies as new vulnerabilities are discovered and then of course what about those dependencies if we can't Scorch the Earth and just get rid of all the dependencies we're using then what can we do realistically well the next step is to understand what kinds of vulnerabilities we're dealing with and put tools and processes in place to protect ourselves so this first one an unintentional vulnerability that could be a weakness in the code a developer perhaps wrote something that is now exploitable and it's out there in the wild the second one and this could be an intentional back door built by a developer either for legitimate purposes or malicious purposes but it's something that was done on purpose and these two once that's discovered will be registered with a CV ve a common vulnerability exposure ID to be shared publicly so that everyone can respond and mitigate appropriately the third one malicious components these don't get a cve these are ones we need to watch for ourselves and I'll talk about those more in a minute it's very difficult to manage dependencies both internal and external or to even respond to publicized cves if you don't know exactly what you're using and if you don't have a centralized location for them and this is where a repository manager or a binary artifact manager comes into play and despite the type of license that these packages have whether they're free or open source or their corporate licenses these packages are made available to developers often through public repos you all should be familiar with these um I'm talking about npm U Maven Central pii um Docker Hub the list goes on and I'm not saying that these public repos are bad um they absolutely have their place this is where you collect them to begin with right you go out you look and see what's available um these these are for developers to contribute to and also for consumption but if you choose to use a package that that originates from a public repo bring that package into your artifact management system at the get-go this way it's under your control having all of your packages you require for your builds organized within that repo will make some other tools work better for you uh you you'll be able to scan uh regularly scan them for known vulnerabilities for example all right let's get into malicious packages you probably are familiar with these um we're talking about the ones that involve dependency confusion dependency hijacking typo squatting and brand jacking um these are pretty serious and a lot of the public repos need some help with finding these um the we have a sec security research team at jrog a lot of organizations have this now with that continuously monitors popular repositories with our automated tooling and we report malicious packages discovered to the repository maintainers in The Wider Community this is something we all should be involved in and helping with now I want to pick on something other than python right now um this is a dependency confusion attack and it was described by Alex Veron in a blog on medium and it was a couple of years ago that he wrote about this most projects have a collection of dependencies that that need to be brought together they include open source or publicly available packages as well as internal packages and what I'm showing here is a typical package.json file for an npm project and what's highlighted are internal package names meaning the organization has built these packages they could be proprietary they're meant to be internal packages noted note that the version requirements for these packages are um either the version specified or the one with a or something greater that is available and that's indicated by the carrot in front of the version number and also these package names are not secret when you request make a request um npm those npm requests inform about these internal package names so they're easy to find so what that means is for example if a request is made made for a corporate lib that it's expected to come from an internal repository but it just so happens that there's a package with the same name with the greater version out there in a public repo and it's not yours and potentially malicious your package manager will pull that one instead and who knows what's in that package or if you'll recognize it right away so by the time you deploy it you've got trouble so one way to protect yourself if you have a repository manager or an artifact manager is to control these requests for internal packages instead of seeking out a public repo right away proxy through your uh artifact management system in order to um you you can specify that internal package requests can only be resolved with internal repositories and it's one way to protect yourself this is if you aware of the problem to begin with I don't know if you know do any of you um use pytorch no okay P torch is in open source machine learning framework and this is something that happened with them um for about 5 days at the end of last year there was exactly this scenario where if you took the nightly build of pie torch you would get a um compromised dependency coming from a public repo when it was supposed to come from their private repo so that that vulnerability has been around forever it was described in really good detail by Alex and um he got some pretty big bug bounties off of this one okay here's another fun one um this has more to do with managing open source packages and libraries themselves not really about a malicious package or a vulnerable package but this illustrates a trap we fall into when we rely solely on public repos uh the left pad incident so kick is a chat app and a developer had a package named kick and there was an a disagreement between the developer and the kick organization so they brought it to npm npm sided with the kick organization that they should own the packaged named kick the developer maybe I presume he was unhappy with this decision decided to unpublished not only his kick package but he also uninstalled 272 other packages and one of them was named left pad um any of you use react all right so react was affected by this this is a tiny little Library this is all it was and this is all that got uninstalled from npm and caused a lot of Havoc so this could happen uh what about a package that is compromised or replaced entirely by a malicious actor I mean this is one that just disappeared because the Creator decided it needed to uh this could happen if someone were to gain access to the maintainer account somehow all right this is a combination of problems here um a little bit contrived of course I'm not going to go through all of them but just line one um picking an untrusted parent image image a lot of these Docker files that I see especially examples online they're pulling directly from Docker Hub consider pulling in your parent image into your own artifact management system and pulling it from there instead um even if you indicate a tag a tag of an image can get replaced out from under you and this is a huge problem um the other issue here line seven uh dealing with something that you're pulling from a remote resource I see you know curling out for these things quite often if if this is an internal location that's fine but if it's not it could get moved out from under you updated out from under you uh it could be updated with something that's incompatible all kinds of problems also I feel bad about letting uh or talking about containers running as root sometimes this is something you should not do I'm sure everyone's heard of it already but even so uh cig pul um put out a report in 22 saying 76% of containers uh were still running as route all right so is there any hope um there are things that we can do there's things that have been done uh we've had some regulation come about uh we've had creations of the cve the national vulnerability database and CNAs um jrog is CNA these are the organizations that can actually create cves uh we've had a lot of development of new tools we need to talk about this more we need more education so more on education uh oasp has a fantastic set of cheat sheets if security is new to you if this is something you want to research more definitely a place to go there's also a trio of free courses by the op ssf I'm in the middle of these right now and they're pretty good um you can pay for them if you want to get a certificate but you can audit these courses for free don't rely solely on public repos we talked about that manage your dependencies but not just your dependencies think about everything that you bring in for your project all the tools and Frameworks that you use um do any of you recognize this landscape okay we got one this is a cncf landscape the uh Cloud native um Foundation L landscape everything that's involved there um I like jigsaw puzzles I thought this was great um it just gives you an idea of how much there is in the uh out out there in the ecosystem that you can use and just being a part of a foundation is good uh there's another Foundation I'm involved with called the CD Foundation it's a sister to that they're both under the Linux foundation and the reason I say that it's good for for a project to be in a foundation um there's different levels that they can have so you'll notice projects start as a Sandbox level possibly an entry level um when they get to graduated they have to go through um some pretty intense Security checks so this is you know some help um to indicate that these projects are well vetted that they're definitely maintained and U more trustworthy okay manage permissions we talked about what happened if you don't uh secure your access to things especially the accounts you use to deploy to remote repos or to public repos um keeping up with maintenance I would love to get in more discussions with you all later about this one um whether updating more frequently as in updating these packages more frequently causes more problems than it fixes um but it is best to know that a vulnerability will rear rear its head at some point point so it's easier to upgrade if you're on a later version than if you're on some super old version and then have to deal with that so another topic for our happy hour perhaps regularly scan your libraries and packages I'm going to go a little bit of on tools for this um we're getting better but there's a lot more work that needs to be done here um one thing I'll complain about right away is dealing with long lists of cves on a developer side we need to know when these things are applicable when they actually matter and when they should be prioritized to be fixed because it turns out that 78% of reported common cves even just on Docker Hub images are not really exploitable and there's some reasons for this it could be um the library that's being pulled in is not being used in an exploitable man manner U it could be a service configuration that's specific um it could be just the environment for example there's a a hop vulnerability that is only relevant if you're running on Microsoft Windows environment all right optimizing our cicd Pro processes um let me go back to that one are you rebuilding when you release this is something I've seen on on some pipelines that you should not be doing um you should be um taking taking a release it should be built once and then promoted through your quality Gates um before it gets to production um and finally are you pulling and building your release artifacts from a known and secure location like a privately managed repository I want to leave with who should all be invested in this process um pretty much everyone because it's not just developers that need to be involved in this uh we need to empower developers um don't just give them a list of cves they are not Security Experts and uh you need to have this culture in your organization to improve your security pressure all right any [Music] questions if you have any that you'd like to talk about later with me or if I've mentioned anything that you want more details on meet me up at the table one on uh the fourth floor and I'll be there ready to chat with you