Well Dressed: Scala with Pants

scala.bythebay.io: Stu Hood, Well Dressed: Scala with Pants

Recording: scala.bythebay.io: Stu Hood, Well Dressed: Scala with Pants

I work at Twitter and this is well-dressed Scala with pants pants is definitely the project name that keeps on giving so I was realizing earlier that we've never printed a pants shirt and that would be very confusing so maybe we shouldn't but there are the pants stickers that I mentioned I have more let's talk about what pants is so first of all let's talk about what the goal of a build tool is and particularly pants as goal let's talk about minor repose again we'll get to that and then I'll give an example of how to get started with pants since I have already convinced you in the first like 20% of the talk that you should and then why pants a little bit inverted because it has scholarly relevant features various other strengths and a very healthy community and I'll talk about some of the upcoming work with pants that some of you might be able to participate in so again if you have questions during this raise your hand and I'll definitely get to them so what is the goal the goal is to make it easy to build and share any size project so it's not just to build projects but to allow for sharing of the code you do write binaries you do write servers but what we want at Twitter and and probably a lot of other places is to share our code so we want to write libraries we want every person who writes a server or who writes you know an executable to also write libraries there's definitely reusable code in there make it trivial to to share also the size bit right it can be big if you've made it very very easy to share things are going to grow because people are going to share and you're gonna have this multiplicative effect and so our goal our sorry our solution to this goal was Monterey Pose so why am on a repo we want to make it ridiculously dangerously even too easy to share code we'll talk about that so all of these are kind of things about dependencies you want to introduce a dependency that's how you share code you don't want to have dependency diamonds when you're in you're making changes to code when you're sharing code you want to be able to if you need to depend on something that is some small portion of some other library you want to make it really easy to to mutate your graph right you want to make splits atomic so that I can split some library update everyone who calls it immediately and one go I don't have to make like for a core library I'm not going to change that and then have to change two hundred other repos it doesn't matter how much automation you have that's going to be a pain in the butt it's not what they're designed for so putting that all in one repo makes this a very easy process we'll get to that so top to bottom continuous integration testing it's just one repo how do you test one repo well you you run the test there in that repo which means you get it all the way when some core library changes all the way to the top without writing continuous integration infrastructure other than run all the tests obviously there is some small amount of infrastructure involved when you have many millions of tests but we can get to that linear change history so it's one git repo it's one you know your curio repo you have a linear change history you can get bisect to figure out exactly what broke you very commonly you bump from some major version to some other major version at least three things break you have really no idea how to figure out what that is without diving into that code as a consumer instead in a mono repo you just bisect you just bisect to exactly the commitment in that other library that introduced your problem so downside of that of course is that sometimes if your tests suck it ends up being the service that has to do that bisect and additionally there are no binary incompatibilities except the boundary we are down to two lines in our repo of 20 million lines of code just switch from one scala version to another swap those to run a sandbox see whether everything breaks it will but and really that's just an argument for source distributions like we have all the source that's great we call the repo of the source repo with a capital S and it's confusing as hell but that's what's in their source distributions are kind of great and it might be a direction that's called heads in okay so but yeah it's dangerously easy now to add dependencies I don't have to worry about diamonds I don't have to actually choose a version of a thing just BAM at it so you have to then manage your dependencies we have dependency and analysis tools and pants and regardless of the mono repo and probably regardless of the system you want to be able to analyze your dependencies pants has the step usage goal which implicitly is going to run DEP usage for our JVM targets we also have underused DEP warnings but both of these are about determining whether you're actually using something at all or if you are using it at all whether you're using it to a degree that justifies spending X amount more time building it right we have very good build caching but if you don't need a thing you probably shouldn't build it so those are existing rewrite tools I think are a growing and very very important component of how we do software development in particular we have massive amounts of source everything's in the open we can see we can see how much work is involved in making some library change you don't want to set that to scare the person from making escape scare them away from making the library change you want them to have some automated way to fix things and I say Auto as a question sure yeah yep sure yeah absolutely so the question was about how we accomplish that under under used depo warning and what it means to kind of justify a death so we use zinc and zinc keeps track of some file level dependencies class level dependencies and actually symbol level dependencies and together you can figure out which classes are using which other classes and so we dump that out in a either a JSON format or in a flat format that basically just says within this context that you just built you use this target at 0% therefore even if it's transitively something way down and you're in your depth graph you don't need it nobody needs it you can delete that def anywhere within your graph and using the output of that tool you can also rewrite your graph to say delete all unused dependencies which Twitter did recently and we drop to 30% of our depth graph away so that was wonderful what does it mean for something to be underused excuse me so one one metric of a healthy DEP is kind of what is the velocity of the code too high can actually be bad for you that's an expensive debt relative to something that's really stable if you're using just 1% of something particularly if it's a soar step you're imposing some build cost and you're also and imposing like startup time costs maybe depending on how the indexing ends up working there so yeah the decision is is an interesting one from what we can tell and I think this is this is kind of going a little bit off topic but from what we can tell people will continue to add depths as long as their system as they're like comfortable with their current performance and as soon as they get uncomfortable with their performance they'll drop them so really that means you need to be always pushing performance because you want people to share code so the last one here is build level refactoring tools this is a place where we're a little bit behind I'll get into how pants declares targets modules but in particular if you want to make it super easy to share code you want to make it super easy to refactor code to make it easier to share like lower the surface area of some library and it should be a one-liner to split some module some target did you start have your hand up or oh I'm sorry yes who raise your hand if you've heard of mana repose or no okay and then who knows is fairly confident they know what I'm onna rebo's okay oh shoot sorry why I said again because I this is kind of my thing and so I assumed apologies so a mono repo is essentially any time you have multiple modules in one repo and so a lot a lot of projects are Monterey Bo's at some level it's also sort of used to it's primarily used to describe it when it like really starts to go overboard and you have more than ten but really if you have two you begin to have a mono repo um yeah so the question was how do you not have merge conflicts unless you have hundreds of people working within the same directories or on the same depths you're you merge conflicts are very rare actually I mean it's kind of remarkable our very largest projects and in the Twitter mono repo touch I'm sorry so our very largest services touch 1.5 percent of the graph at all right and so then you but also they tend to write code within their own module so things will come in below them in their transitive graph but that doesn't actually cause them to fail to merge they have to be the tests have to be a rerun before you merge that but conflicts are surprisingly rare so yeah does anyone have any more questions about minor repos because now I'm realizing I might not have absolutely the question was do you run into version control performance issues and yes absolutely three or four years ago when Twitter moved to a mono repo we were using git we patched get quite a bit to make it performance enough for the code and it's really not too large of a repo it ended up after pruning ended up being something like 10 gigabytes the entire history of terse code obviously other people have larger ones but since then in the last four years a lot of the patches that we created have either been up streamed or had variants up streamed and also the get to get project itself has as matured a huge amount to the point where some of the things one of the things we have is uh excuse me a log base based fetch because get fetch in particular does this here all my refs which refs do you have and if that's something like 40,000 refs then you end up touching discs at least 40,000 times but a log base thing just pushes that out pushes out all changes as quickly as possible so but yeah there have been improvements to fetch upstream and we actually have to see whether it's now not even like out of the box it would just work so I think there are a lot of companies pushing on sem like this and so it's not unless you are one of those you're probably not going to hit the issues their existence proofs so yeah yeah good question so how do you open source some Project Romano repo get subtree is generally sufficient you can also use sub modules pants in particular though is just gaining support for merging multiple repos at a native level such that you could have a sub module and have it be using an independent pants build and I'll kind of get into what that looks like but as it stands a lot of those projects have their own separate build like they have an SPT build that's that's unfortunate we didn't want to actually be able to fix that and so either that that support pants native support for sub modules or there are a bunch of different tools Facebook and Google both have one and then I'm sure there are others related to this problem where you essentially whitelist a series of things that you need and so in particular within our repo you could whitelist essentially the transitive graph of that project somewhere so all the source steps are in that same repo and then whatever config you need out of this much larger repo so okay let's move on so example minimum viable pants now you're all convinced let's say that you wanted to install pants pantses is designed for Monterey pose good we have one of these kind of scary curl contraptions to get you in started but it's just basically putting this in a script that you will then execute put this at the root of your repo and all this is explained at that link none of this is technically necessary but we wanted to show an example you you can actually start pants with an empty pant syenite this is again at the root of the repo it's useful to pin the version that you're gonna get one - oh is the latest stable release if you don't pin that you'll just whatever the latest is and it will float and so you might not want floating I'm gonna pin that down and then this is just an example of another argument section with a fancy new feature enabled at all I'll show you make a directory to put some code in put some code in that directory create a build file now this is using a default target name default sources and it's a library with no no dependencies right so it's just this one liner of Scala library right it's a default target that means it actually is named example source Scala common example the libraries name is example because it shares the directories name and the default sources for Scala libraries are star dot Scala essentially then run it so echo print line to our rebel pants has that script that you downloaded and set executable it sits in the root of your repo there's no need to install pants on a user's machine at bootstraps itself the repple here is gonna take the standard in voila and we execute when this is actually running the first time it's gonna take something like a minute because it will actually download everything it needs to compile this compile this and then execute it which includes ink IV which it will bootstrap etc so first time you run will take a little bit longer second time it should be on the order of you know 500 milliseconds okay so that's it the reason I walk through this example is that pants has gotten orders of magnitude easier to use in the last year pants hit 100 in May and our goal for 100 is to essentially get to 2 to zero config as I said even though if and say and I was not strictly necessary but you probably do want app in your version and also pin your Scala version you saw here that we got to eleven eight just because that's the default that we have configured for the two eleven branch there's a one-liner to go from 211 to 212 in those options right so that's it like re-examining some of the stuff we did pants has lots and lots and lots of options we put some of those in the pants I&I pants options pipe to grep for that option we had okay true this is set to true came from config in pants ini these implicit sources is what allowed us to not specify what sources that target owned and I haven't really even explained what the target is apologies this is a target you put it in a build file it owns some sources though things can depend on it I didn't show any examples of dependencies but a Scala library target if it depended on some other target would literally just say dependencies equals source Scout Scala comm foo write that I depend on that library that target yeah yeah so that the question was about is this the recommended style yes this is definitely recommended style the the reason this became the recommended style is because of this property that you helpfully mentioned which is this one one one property it's actually super helpful to explain how we like things to go down we like a single directory to equal a single library or binary to equal a single package in the language right so ideally source Scala comte example contains only software for the com example package right I know where that package flips throughout the entire repo so I have this normalized namespace and introducing default target names and default sources is the idea is to push people toward this these best practices by making them very very easy so Twitter stripped out all of the names and as many of the source definitions as we could in our repo recently which was which was awesome any other questions about the examples okay well yeah example options there are a lot of options we use options for a lot of things now that that doesn't give you very much information about what this is actually doing right so let's get more CLI args the pants ini file and environment variables are all normalized into the same auction system so pants help advanced for target arguments which is the the kind of namespace this election was in target arguments and Blissett sources right it's the Nate it's the namespace concatenated with the actual option name defaults to true if true pants will infer the value of this sources argument so it it is optional to allow implicit sources which is what allowed me to have that one-liner Scala library where the default is just to glob everything right you can turn that off we don't roll things out breaking changes without having some option associated with them right so we move very quickly we have releases every every week and then we have stable releases every three months and part of that it's making sure that we don't break things so yeah but yeah this help this is a fairly this would have been something like 200 lines of help no the exception to this unified unified set of options and CLI args and environment variables is that when you define a target you give it arguments so these are there's this differentiation between things that you want users and in your repo to be able to configure locally versus conventions that you set globally within the repo and you can there are a few different ways to look at that we can get into it but as a very simple example like a resources target which defines a set of resources that some some other target might want to depend on has a few different options here and this whole page is significantly longer with all of those expanded so finding page works so yeah why else pants scholar relevant features we have all the things you'd expect right from a tool that builds scholar we have integration with IntelliJ we have demonized warm compilers and we have multi JVM platform builds in the sense that you can have a Java eight targeted Java library depending on a Java 7 targeted library that's totally legal and you'll get the right JVMs for each of those we have native support for maven and Ivy style publishing so pants publishes itself to Ivy central or to maven central excuse me along with a lot about their libraries Scala doc Scala format was added recently pumped I can't call it foam sorry Scala style Czech style findbugs scope dependencies which are similar to maven scope dependencies except rather than being on a dependency edge you define it on the target the idea being that in a mono repo you don't want to just change your own code you really do want to try you want to bias toward changing the code you depend on right because you're going to have multiple people depending on any particular library in fact in the right places is beneficial etc what we also have though is we have very tight integration with zinc zinc has a weird history because it zinc used to be a just a demonized wrapper for SBT as essentially how you could think about it but it's been that project died and it was rebranded the incremental compiler of SVT is now just called zinc so SBT uses ink it's the new way to think about it right so we also use ink via demonized wrapper so we have very tight zinc generate integration for Java and Scala it's important to have it for both Java and Scala because if you have a path from Scala to Java back to Scala and the Scala library behind the Java library changes you don't want to have to recompile all of the Java happens a lot and so you need analysis and this analysis is sub file again as you're pointed out sub file and sub at the class level information about which classes depend on which classes and which classes depend on which symbols basically additionally we have an incremental aware distributed build cache the incremental aware bit is very important essentially because you really can't trust us in KSP T if you use SBT you know periodically you have to clean all we've inherited some of that right I'm gonna not gonna lie we have inherited some of that do to our usage of zinc but we are aware of which compiles are incremental or not we never trusts the incremental compiles so only trustworthy compiles go in now with the optimizer where it is according to everyone with regard to 12 it's really seeming like this is a great time to use this incremental awareness to conditionally enable the optimizer wherever you can write you can't use the optimizer with the default as SPT incremental compilation and so we will be very careful to differentiate those artifacts cash and as I said we have the dependency analysis tools this will spit out either like one line per per transitive depth or an actual graph that you can then analyze also yeah so we're also the CIA and user friendly you can test all the targets that have changed in your branch for example because we are actually looking at the get diff and we can tell you which targets have changed so pants - - changed parent or you could do pants help changed and it will give you information about what options are available there but pants changed parent master test is basically going to choose the routes for your test command choose which targets to test based on what has changed between your branch and master or master in your branch you can also just report which files or transitively use bad target oh this is very important for caching and getting this right is critical to actually catching the right things and not having the wrong cache keys and so exposing that out this is this is frequently used again I said what do you do continuous integration testing wise in a mono repo well most systems because there is not a standardized distributed test runner are going to do something like figure out what targets have changed partition them some way use whatever system you have happened to have available I'll talk about how we want to improve that a little bit later so why pants again community so we are an inclusive nonprofit organization pants build and it's independent of any participating company so I've got this this Twitter shirt on we're at Twitter but this is a completely independent org all development has happened in in public on github for the past four years and we released we released 1.0 in May so again we do unstable releases weekly stable releases every three months and the big difference from the last time I talked about pants is that we are absolutely ready for wider usage now this one Oh milestone was was not an insignificant milestone in terms of what our focus was and as I showed it's down to very very low configuration for the common case also the stable releases began at 1l so we have this this good migration path for people there's a very safe 6-month deprecation cycle so two stable releases before we'll remove something you'll get warnings on options it's kind of amazing so strengths so as I showed our goal is to kind of try and minimize boilerplate we want to minimize the boilerplate required to define a library that we're down to one line that's pretty good for this for the this like default case that we want to encourage right we want to have a directory that represents a particular package and has at most one line hopefully plus dependencies to be defined by default pants targets are not publishable off to IV and maven you have to add some additional config there at least within twitter that's the right decision outside of toriel it may not be but it's easy enough to add a macro if all of your libraries within your repo should be publishable under some org you can just make that a required argument with by using a with a pants macro so by default if you try to publish something it'll say this isn't this isn't hasn't been declared publishable and the reason again is that you have twitter despite having a fair number of open source projects has a relatively low fraction of the graph that is open source so we can do better there but we don't want to introduce magic almost all of these behaviors are explicitly driven by options so consistency again we want to make it easy to enforce repo wide defaults local overrides in some systems are kind of the norm and so when you're looking at some repo you want to figure out what is the right way to do a thing in this repo you it depends which directory you go into you'll find like entirely different things going on in sub directories consistency is very important in growing in growing a code base as an example JVM platforms you can't just in some build build file to find a brand new platform we need to know which ones are supported by the repo it's not significantly harder so this option right this this is in the option syntax that will also go in pants.i and ayah that I'll also go in environment variables but generally you pop this in pants ini to say this repo supports this platform and then locally you have this relatively simple way to specify why I want to use the default or I want to override to a particular other platform as I said incremental compilation when even when zinc is not in play if a target has already been built we don't rebuild it and we support this distributed build cache and then additionally zinc another strength we had relative to other in fact even to other mono repo build tools is that we kind of support huge real depth graphs and I'll get to that so you have tens of thousands of source targets you have thousands of third-party depths and the third-party DEP is any depth that's coming in his binary right it's it's open source code it's closed source but it's a big blob of something and so we support less-than-perfect for third-party dependencies company a company like Google which is where this idea was really born probably at least pantses model was born there has never probably had enough third-party dependencies to need to have a messy def graph where they support multiple libraries for very long but the real world of open-source libraries generally means that within some repo you're you're immediately gonna have two versions of elasticsearch you're gonna have two versions of hadoop spark hasn't I don't think spark has screwed us yet there but yeah you will outside of the world of Monterey pose big version bumps screw everyone print so you you end up with these divergent is it similar yeah okay yeah you end up with these big Burton version bumps thick essentially bifurcate your your your mana repo a little bit I can't depend on you if you need to do two and I need to do one right and so we kind of recognize that we support that as best we can so extensibility yeah we have support for a few different languages we'd love more there is kind of fledgling Android support there is no I have a support although somebody was mentioning that recently but these are all things that we absolutely want Twitter in particular is going to be focusing on mobile support in pants over the next year along with a few other things but almost all deployments of pants have some custom plugins and macros because it's easy to do that in your repo without forking pants you can extend almost anything within pants without forking it which is another benefit oh and I don't I didn't really mention but we have another part of one oh was that almost all api's are marked either public for for usage and plugins or not and any public for use in a plug-in API has that same deprecation cycle so let's talk about upcoming work we have recently refactored the internals of pants post one Oh in a completely transparent to users fashion to implement a new native engine so the effect of this is that no op ie I didn't change anything I'm just re running and incremental I changed just a small amount of it and I don't want to rebuild everything executions get significantly significantly faster we already used Damon's as Inc was always a daemon so zinc is always warm JVM is kind of to be warm but since the rest of pants is in Python which I might not have mentioned yet it is now a Python and rust system and the daemon is primarily rust it's still optional it will become the default as soon as we shake out the last few bugs and but I think we the fundamentals are very good so we're excited about that it will likely not justify - well we're gonna save - oh for the next ux changes we make so some major version between now and then well we'll turn this on by default strict ups so this is the thing that begins to become an issue when you have depth graphs involving two thousand modules right when you have two thousand modules in one server right 2000 libraries targets whatever you want to call them invalidating those upstream targets every time one library downstream changes can get very expensive the compiler doesn't actually use the entire transitive ie recursive ie depth graph of some dependency unfortunately with Scala it's not entirely clear which it uses and so we have work to do there Java has had good rules up until eight where if you imported something and he'd stay on your class path if you didn't it almost certainly does not need to be on your class path and that allows for things like well I have this 2,000 module graph but this target that I'm about to compile only actually at compile time is going to need 10 or 20 of those right that's great for cache keys the library that I don't directly depend on has changed I don't care I don't even need to wake up to recompile this so a compile will walk only as far as it needs to and then stop so we'll be turning that on for Java it's already implemented and we hope to work with the skull community to do this we're very happy to have Eugene joining us to to help accomplish that it will certainly benefit they're build tools hopefully so and additionally we want to work on supporting remoting which is kind of transparently executing builds remotely it is at least partially blocked by strict depths the thing on the previous slide because shipping 2,000 things to some remote node is massively more expensive than shipping just 10 or 20 the strict depths so that's that's an important important change that we're gonna try to roll out before we focus too much on remoting and yeah right so the upcoming Twitter open source project you'll hear more about that fairly soon and I don't think I'm giving too much away to say it's it's called scoot and look out for it additionally we're interested in in attempting to infer depths this is probably the longest term of all of those when you have consistency throughout your repo and you have these namespaces where this particular package could only live in one directory and the entire repo if I import something from that package I know where it lives in the repo and if it doesn't live there it lives under a third party so you have these two namespaces of binary depths that are coming in from the outside world and sources that live in the repo so if I import it why do I need to repeat that in my build configuration so this consistency allows us to hopefully and consistency in normalization I say allows us to hopefully infer that based I imports so stretch goal but we're very excited about that we don't really believe in the kind of masochistic well I should have to I should have to repeat this should have to repeat myself to like banging it in my head that I have a depth we have tools for that okay that that's everything anyone have any questions yeah so the question was about what do you do when you want to start servers or have essentially different deployment stories is that yeah so pants supports all sorts of different plugins and what we have built in is not super extensive but we have things like rpm build and docker plugins actually currently out for review in terms of starting servers I think one thing you could certainly do it in a plugin but one thing is that that will interfere ever so slightly with the idea of remoting you want your tests to be sandboxed so our goal has been to sandbox tests to make them as reproducible as possible and and so that's its ever so slightly in conflict but totally possible to do in a plugin [Music] yeah absolutely yes so none of that goes in the build definition yeah I'm sorry so the question is do you if you wanted to start a server would you put that in the build definition so pants has this slightly alternative model not particularly though it's kind of similar to the Maven model in that you have goals and I have some minimum amount of configuration on the target but then I might have a goal to I would have a command line goal that I want to accomplish and if the goal is to start a server the plugin wakes up there if it needs any additional configuration on the target you can specify it on the target or you would have options on the goal that are kind of global options for the repo here's like here's what we here are the default port ranges that we we grab any other questions yes yes yeah yeah yeah so the question was about source steps and this is kind of this is fundamental to a mono repo so I apologize for not for not getting that but yes so nothing is versioned in the repo you don't have an organ a name so there is no way to even say that something had a version there's nothing to attach the version to in theory you could version a target but so internally that's the best way to do this is an example internally our CSL team shipped some change where they ship a bug what they say is Oh a bug a bug a bug was in master fur for this sha range right because we know it was in for this char range we know these services are affected by it because these services were deployed that sha and so Shahs kind of end up being your version but you never ever ever think about them except when you're dealing with third party so third parties we're all of the external binary version Depp's live and then otherwise you have soar steps and a source step is it's easy to add a sore step as and like a single line in the build file when you do that it's very very very very unlikely for you to have a DEP conflict because third party is this massively smaller you've got well-maintained set of versions that is self-consistent mostly with the exception of things I could have one intuitive so thank you everyone there'll be stickers over there and I'll be around [Applause]