Scale By The Bay 2019: Ahir Reddy & Li Haoyi, Speedy Scala Builds at Databricks

Scale By The Bay 2019: Ahir Reddy & Li Haoyi, Speedy Scala Builds at Databricks

Recording: Scale By The Bay 2019: Ahir Reddy & Li Haoyi, Speedy Scala Builds at Databricks

[Music] everyone so yeah till my name is a here I'll be presenting with how we were both from data bricks and we're super excited to talk to you about our speedy Scala builds on basil so to kick us off like who are we so we're data bricks we're unified analytics on the cloud we're kind of like the SPARC people where it came out of the amp lab in 2013 from the team that created SPARC and we love Scala back then and we continue to use it quite heavily today so who are we within data bricks where well we're on the developer tooling team and our goal really broadly is to make make sure every data bricks engineer is able to do their best work at data bricks we want them to be the most productive they've been in their careers so what we own we own the different language tool change the CI system docker github integration Auto formatting IntelliJ basically all the tools that you'll be dealing with on a day to day basis so like I mentioned before data bricks is a pretty heavy Scala user so SPARC itself is written in Scala and all of our back-end systems are also written in Scala we sort of are in the more Java Java plus plus usage of Scala so lots of cheap sort of standard Java libraries that you'll see in at other places and we actually also have a lot of custom Scala tooling which we'll get into we try to make compile test and build faster so we're a company of about 150 engineers so every incremental improvement we make is multiplied by 150 so that's sort of the premise of our team cool so today we'll go over sort of four main areas we'll talk about the process of basil fiying our Scala builds we'll talk about how we do dependency management and then I'll hand it over to Howie who's going to talk about how we do cross build to make it really easy to upgrade between Scala versions and finally come compiler performance and the investments we've made there so cool I'll start us off with the how we got to Basel so just sort of a timeline here so we're pretty early Basel users we started way back in fall of 2015 we did our initial prototype it was actually just one of our internal hackathons where we're like okay it it can't be that hard to get some subset of our set of our build running on Basel so we were able to pull it off in about a week we got things up and running and we sort of went from there so we sort of realized over the course of the next few months we kept tinkering with this project and by the second quarter of 2016 we're like okay we should just do this and fast forward to late 2016 we actually did the conversion we actually deleted SVT and then fast forward to 2019 we applied the same conversion the same techniques to our second mono repo so I'll get into more details as we go along so alright so summer of 2016 what what what was the state of the world before the migration well we had a we had a mono repo we had about 90 sub projects in our SBT build and about 15,000 lines of SBT build definition so the motivation was our build was super slow and why I say slow I mean no ops were 20 seconds pure SBT overhead it was super hard to maintain I think this was actually the thing that really pushed us over the edge there were two to three people who could review or really make any changes in the SBT build and sort of the last thing was it was really hard to make tests robust we were always making this trade-off of more parallelism versus stability so as we added more code more engineers more tests they just got slower and there really wasn't much we could do about it and but one thing we realized is sort of doing a whole cell migration saying we're gonna merge one pull request that's going to convert everything to basil was too large and really too risky to do in a single pass so we took this approach of maintaining parallel builds over the course of a three to four month win and we built some tooling like an SBT plug-in to generate some like our basal build so we could kind of maintain this with as little overhead as possible so yeah so by September of 2016 we had fully deleted the SBT build so we'd spent those six odd months building out the migration tooling actually doing the migration of each sub project onboarding new projects to Basel and by September of 2016 local developer laptops all of CI everything was running on Basel so fast fast forward to summer of 2019 we started hitting these similar problems in our second mono repo so but here the the the core was actually slightly different so the reason we didn't migrate this repo initially was that in terms of the number of projects the overhead of the build it was a lot a lot less impactful but what this repo had instead was a ton a ton of tests so as these tests grew our test times grew from an hour to three to four hours and as I mentioned before it was really hard to make tests both fast and stable we had tons of flakiness in CI just to just do two inner tests interference and we couldn't yeah we couldn't paralyze beyond 4x so basil as we had learned just runs things in sandboxes it's you have to try pretty hard to make a basil test interfere with another test so at the time at the beginning of the migration we had about 20 sub sub projects and an SBD build of like 1.5 K lines which an order of magnitude smaller than our original mono repo so just to go a little bit further into the motivation it's not that we had seen these problems were like no it's fine like whatever we'll just wait until it gets really bad we actually tried so in 2017 we got a lot of complaints hey my PRS take a long time be great if you could paralyze tests we tried hey testing like test don't run in parallel it's slow again it sucks having to wait three hours it's like okay let's let's do it for real this time let's let's go to the thing that we know works so you could actually see here kind of the test time and you'll see it sort of lines up at different points where we actually tried paralyzing things so you'll see around like June of 2018 between May and June we worked really hard and we paralyzed a bunch of things and then damn Engineers kept adding more tests and it kept going back up so we couldn't just throw this wasn't this isn't just a question of throwing more cores at it if we could just solve this problem with money and more cores we would have done it but we can't we're sort of bounded by these the sort of underlying infrastructure and how much parallelism we can eke out of the system because things interfere with each other so we started this migration in this summer of this year and after we went through the migration process somewhat similarly to what we did for our original mono repo we improved test times about 5 to 6 X and not only did we that we we're able to have more parallelism we also now this just let us throw whatever large ad of this instance comes out next we'll just throw it at CI and then we just get the parallelism for free so we still keep the SBT build around for local development for this mono repo and it's more usable than our than our basil build at the moment but ideally we're hoping to sort of switch all the entire development workflow to use basil cool so yeah here's actually the the the test times sort of over time so what we had done the way we'd we'd gone about this migration is that we started with the SBT build and we rolled out of basil build and initially the basil build didn't really do much of anything it's just compiled things so what we did was as we incrementally ported tests over made them stable make them work in the sandbox environment we would remove them from running in CI in the SBT build so that's why you see these two lines sort of converging on each other every time we added a test to basil we would remove it from SBT so as you can see we move we move me moved we got from an average test run time of 150 minutes to today I think about 25 minutes so we run the SBT build which is literally just compiling and running two or three tests that we have imported yet and you'll see that takes about the same time as the basil build that also compiles and runs the other fifteen hundred and ninety seven tests so the parallelism really helps and sort of having these the the primitives or the foundations to to be able to say hey I can run this test it's isolated I don't really need to think about it anymore is really powerful so what's the current state of basil eight data bricks we have about fourteen hundred Scala modules 1.7 K lines of Scala and to be clear this is across both of the Monterey Bo's tests run up to 96 X parallelism if we give this talk again it'll be whatever the largest AWS instance is without flakiness and some other things that we're not gonna spend too much time on but these other things that basil gives you distributed build cache when we roll that out in 2018 we cut down our CI compile times by 7x so that made people pretty happy and it was really easy to implement sort of new build related functionality across both of these mono repos I think one of the most impactful was selective test running so the core idea here being that when I make a pull request I should be able to do some pretty simple analysis to say oh okay these are the files that were changed what are the downstream tests and targets that I actually need to compile and run again so most pull requests or most changes really don't need to run all the tests in the repo most of them run a much smaller subset so this really cut down on pull requests times and really minimize the impact of flaky tests where they still around cool so moving along next we'll jump into how we do dependency management in basil so just to sort of set the context here you probably all familiar but dependency management something that's common to all programming languages at least ones that people use so languages tend to build their own set of tools for managing dependencies job and Scala use the maven ecosystem javascript has NPM Python pi PI etc basil at least at the time of our raishin didn't really have any built-in support for this so what we did was we're like okay well if we're gonna do dependency management let's sort of set some requirements set some goals to try to adhere to after having learned the lessons of sort of painful dependency management problems and SBT and maven and sort of our other package managers that we deal with so the first goal is reproducibility so what that means really concretely is that a commit must be fully self describing and reproducible this kind of sounds crazy when you when I like when I say it because it seems pretty obvious when I build a commit if I build it today or I build it tomorrow it should produce the same code and this is largely true in the Java ecosystem it's not necessarily true in NPM or really depending on how you set your IV config in SBT so the results of the bill should not change over time and dependency should be reflected kind of in an understandable way in the commit it should be really hard to introduce a dependency change that people didn't know happened so this sort of leads in the next when all dependencies must be locked so dependencies can't differ based on the environment in which things are executing we really don't want em to cache pollution NPM cache pollution we really don't want that to be the problem we never want to tell people hey this doesn't work on your machine just try it somewhere else so and then another really important one is speed and stability depends your resolution really shouldn't impact build for build performance some ecosystems have really slow dependency resolution and this sort of leads into try avoid pulling from external repositories at Build time I we've all probably had twiddle our thumbs when maven central was down or we've been owned by like NPM or sorry we've been we've had CI breaks because NPM took down a package repositories can disappear that's happened that's fun and sometimes repos just get owned I won't name the resolver but for awhile we were we weren't getting jars we were getting cat pictures from one of our third-party resolvers that was a really weird discovery so okay so what's our approach the approach is to resolve dependencies outside of the build system we want to remove dependency resolution from the critical path of development the the core idea here is you can only really do this if you if you believe that dependency changes are actually infrequent so if they're infrequent we should avoid redundant resolution in every build so what this becomes concretely is that we've materialized dependencies into basil build files these dependent these generated build vials are lock files that are just committed into the repository so it's really easy to see artifact versions it's really easy to understand what changed and finally when we resolve these we don't we don't resolve them and say ok I'm gonna go pull this from maven central at resolution time we simply mirror things into an internal s3 bucket a content addressable content addressable store so we don't rely on any external systems that can go down and we also avoid storing large artifacts and get soaked it doesn't tip over you know anytime I add a new dependency so the workflow is pretty straightforward we define or update dependencies and just a standard maven pom we run a little tool that reads in the pom and spits out build files this also will upload all of the files to s3 and then we generate we depend on the generated build files within our project so looking at this what we have at the top is just part of a maven pom this thing is fed into like our maven update script it will spit out a bunch of build files so here's actually the the bait the the result of the basil query that generates this like dependency graph so we've got the Jackson's Scala module and this guy cursive Li depends on like Parana mer and jackson core and data pined and Scala library itself so yeah okay so today get step back we've contrast this with SBT the depends your resolution is is fasted and really reliable and this is just mostly a function of we put stuff on s3 and we don't rely on maven central version upgrades are fully expanded in generated source so things are auditable when I make a change or dependency I understand just that change but anything recursively that might have happened so it makes it easy to ensure that like the mysterious versions don't get pulled in and we've set it up such that any module that shares a dependency with another module there's no divergence there's no notion of per module dependency overwriting so we don't really have these surprised binary compatibility issues in production anymore you should they'll happen locally in your tests instead where if they crop up cool so now I'll hand it over to Howie to cover cross building hey thanks I hear so I will cover cross building and the compilation performance he put in to make working with scale data bricks comfortable first cross building the basic problem across the building is what to do if I'm building a module or a library there needs to be used across multiple scalar versions and maybe SVG support sense built in but you think about it further this applies to much more than just color versions SPARC versions maybe I have different versions of jetty in different parts of the code base we want to test different JVMs in different parts of the code base because some team wants to use the new the new GGC but some other team wants to be on a more stable JVM even running for a long time so all of these are basically the same problem where I want to do the same module or the same test but with one facet which is different from what it was previously the way we do crossbills data breaks is via cross this helper called cross color library so this is a basil build definition which says I have a Scala library in this folder his name is author G sources as a scholar version and is slightly simplified but more or less discovers how you define a Scala library if you want to cross build a Scala library what you do is you write cross Cal Lib this is something that we implemented ourselves and you just give it multiple scholar versions and possible multiple SPARC versions or you know even multiple different axes JVM versions etc that you want this module be able to against and this will generate basically a matrix of modules they exist in the same folder all Suffolk's with how they are defined so you have a 3:11 version to 12 version us part to 3 versions part two version and it effectively just expands out it as a for loop or maybe two for loops to give you a whole bunch of separate modules so why do we do it like this versus a SBT where you have a global flag which sets us calibers and plus plus to twelve ten plus plus to thirteen one yeah so this is more that's what I just showed you earlier the big advantage of doing this is that we can cross build against any acts easy ones including multiple axes so a lot of our code cross builds against two axes Skylar versions and spark versions because the other bricks has to support many different spark versions and the different spark versions use different different dependency trees and a binary incompatible with each other and occasionally we have other cross builds for example we cross build against different jetty versions doing a big upgrade so we could build and test both at the same time and we may build against different JVM versions depending a JVM upgrade compared to SBT where is global in basil the way we set the top the different modules cross build against different versions of dependencies totally separate which means that it can be compiled in parallel they can be tested in parallel and basil knows that all this is one big graph that you can query and run all your normal commands over there's no magic that's cross building cross bills are just separate sets of dependents separate graphs of modules each of which with the different suffix underscore 211 underscore 212 and the normal basil craze that basil tests everything will really cross build and cross test against every single version in your repository the other thing is that the build tool basil is aware of which module support which versions by virtue of weather has underscore 211 212 underscores part 2 3 2 4 suffix that means it is impossible to select a version of X if you take a module it is impossible to select a version of a cross bill that is unsupported for example if I try to if a module is only supported for scarlet we leverage Scala to 12 I cannot build it again scale to 10 the build tool just tell you know this is not supported there is no 210 version for this module and these errors happen even before compilation starts does not I get a compile error which is weird or words you get a runtime error with like a method not found exception due to binary compatibility the build through that sells you there is no 210 version here and this even applies to dependencies so if I am Scala 210 211 and he is car 212 if I try to depend my module on him I'll again get an error saying that he does not support Scala 210 211 I will not even bother trying to compile it so this gives you quite a good user experience even when you're quite a messy graph matrix of cross builds as we do for spark scholar and other axes cross building also makes upgrading easier so apart from SVG many other build tools do not support building multiple versions of Scala in your code base or even multiple versions of anything in your code base the whole code needs must be on the same version which means if you have some like legacy dependencies that depend on some old version of spark they're stuck on Scala 210 your whole codebase is stuck on Scala to attend even your totally separate web services or API servers do not share a new class path with your legacy dependency at all because many of these build tools do not support cross building in contrast later bricks has code for skeletons car 211 scale 212 all built tested and worked on actively and concurrently by different parts of the company so the web services new version sparks 212 most of SPARC is to 11 we have some old version of spark that customers still need to use yarn 210 and y'all can live happily in one repository and there's no Big Bang upgrade you just add new cross builds to the work to the modules which need to be cross build against different versions and because of the gold leaf happening one repository upgrades are really easy so upgrading everything from Scala 210 to scale to 12 that we have upgraded so far that took like 1/4 worth of one person worth of work so it wasn't a lot of work to do a huge version upgrade for maybe of several hundred modules all into a new version of Scala with several incompatibilities so after cross building last thing I'll talk about is compilation performance so computation performance is a bit of a lingering problem for the Scala language it compiles all those a magnitude slower than Java then go even Kotlin is compiled somewhat slowly compares much faster in Java then Scala and it's a big problem in our code base it's always been a big problem so when you have a million lines of skull and seventh hours of source files it takes a while and we send out service to our developers twice a year and you get feedback like building takes too long building takes 10 or 15 minutes sometimes takes 30 seconds sometimes takes 90 seconds is always too slow and gets the way of people's productivity people do not want to sit there watching their code compiled so as a two example of modules I'll use for this section of the presentation we'll consider common which is a common module that many of our downstream dependencies use it has a lot lot of code itself but not very many transitive dependencies as well as application which is the web application that has not that much code itself but depends on a lot because this is the thing that actually deploying to production depends on the rest of our libraries common has been sea graph will look something like this if you do the Basel query and print it out so initially back in 2015 2016 the naive compilation of of app and common look something like this where app would take 15 plus minutes common would take three minutes to compile cleanly and if you make a single change like add a print line remove a logging statement app would take 30 seconds agree compile and come over take 40 seconds I mean this is somewhat expected given that relative numbers of source files and relative are transitive dependencies for each of these modules part of the reason this is slow is that at least we started initially the naive Bayes or scale integration with spin up a separate scale compiler for every module that you compile and this is inefficient because scale compiles are expensive they take a long time to warm up you spin one up and throw it away you're not getting any benefit from the JVM performance we're getting a lot of penalty due to the JVM set up overhead so basil does support worker processes which we had to kind of wire up ourselves where your multiple multiple modules which shared the same compiler process so compiler process lasts a long time and get properly warmed up and jitter then fast taking advantage of JVM optimizations and we took a step further and consolidated them all into a sink a process and the reason for that is simply that a compiled worker is quite a heavy piece of software it takes one or two gigabytes of memory in order to run and you might as well have one of them rather than three to gigabyte processes hogging memory on your laptop so welcome process is cut down compiled times by about three to four X or across the board just from having a hot JVM goes to the cold J via JVM what's next the next big feed that we got was upgrading Scala to twelve as I mentioned earlier we took about a 1/4 one-person upgrade and the big motivation upgrade was compiled times so this is the light Ben Scala teams compile time benchmark and previously we were on 210 because that's what was available when company started and when we did the upgrade to 12 was already a lot faster than 210 was at compiling things if you look at it today it's even faster to scar to help compile is literally twice as fast as a scar 210 compiler was back in the day so we did the upgrade we spent a bunch of time cross building things making sure everything work very dating testing and we also validated that this performance improvement in compilation also apply to our own code base not just the benchmark set like Ben was maintaining so if you look across many different modules or many different kinds we our computation performance almost doubled thing to half as long in compilers it didn't - things took half as long to compile into 12 as it did in 210 and the if you look at two benchmarks that we had earlier you can see that the killer speed up in compile times well visible the next we that we got was from remote caching so this is something that basil supports built in not just what scallop or anything you're gonna do is you're compiling golang modules you're bundling a JavaScript code creating docker tarballs basil allows you to share the output of compilation from one laptop on another laptop so when it compiles it gets sent to a remote cache server which is just some kind of s3 bucket with nginx in front of it and anyone else who compiled the same thing will also can just download from the bucket rather than having to recompile themselves so this does not affect Delta builds at all because if you are just adding a pin line the chances are no one else is added the exact same print line to their debugging they have environment but for clean bills it sped things up by maybe three to four X instead of compiling the world you just download you a bunch of jars from s3 and you're done so that was great and effects everything not just Scala so Dockers fast JavaScript bundles faster it's great the next thing we did was cloud their boxes so the idea here is that rather than having your laptop compiling everything on your machine and having to interact with all these remote services why not just send your code which is relatively small at kilobytes of code over to a ec2 machine which is co-located in the same data center as all these things that that machine needs to access not only is it collocated with all the other services it also is a lot faster than your machine like by default we give people 16 gigs of RAM sorry 16 cores and 64 gigs of ram on the box and you can ask for 96 cores and 384 gigs of ram if you really want to compile and run a lot of tests at the same time and you'll just give it to you it costs $5 an hour so if I just turn it on for two hours and turning it off it's perfectly fine and lastly the dev box doesn't have all this stuff that runs on a laptop hogging resources that's taking away resources that could be used for compilation like you have IntelliJ using 4 gigs of RAM you have Visual Studio code electron you have your antivirus software you have Gmail taking up one gig of ram and slack taking up two gigs of ram because they use a lot of memory and why should all this be fighting with the Scala compiler basil and docker for resources and the answer they don't really need to be you can run all these locally and you can run your build cooling in the dev box and you're both running perfectly happy your laptop does not get hot and your bill does not get slow and everyone is happy so the experience of using a dev box looks something like this I can compile something you see here I passed it on there again so the first time you compile it that it's passes I make a change you see it syncs basically instantly and you go back to your terminal you run the thing your changes there will fail you can fix it and it sinks again so basically the cloud air box has basically give you a local development experience but with a remote development performance which is basically the best of both worlds you don't you don't need to be thinking about I keep pushing it pulling but calling our sync on the correct folders or correct files just edit it syncs you compile you test everything works but just faster how much faster so for non cached things which are not presently remote cache we found that cloud their boxes basically double the performance of our Scala compiler so running on the ec2 machine is 16 cores and 64 gigs of ram is twice as fast as running on a laptop with 4 cores and 16 gigs of ram and slack and gmail and antivirus and all these other things so twice the sauce is impressive but even more impressive is if things are remote cached everything is already in s3 I just need to download it it literally takes one or two seconds sit down with everything on till our dev box and like why shouldn't it right because your dev box has 10 gigabit per second Ethernet connected to the next to your remote cache is probably the same datacenter and you're robbing 10 gigabits of jars are you just having a few hundred megabytes so it should take one or two seconds and once that's downloaded you can just immediately start working without having to wait any longer so 2 seconds down from 15 minutes was a pretty good speed up for clean builds and lastly we all this we were using here was not using a sink incremental compiler so that's recompiling the whole module from scratch every time you make a change and with zinc that speeds up encrypt the Delta builds from 5 seconds to recompile the whole module maybe a few dozen files to one second to recompile the file you care about and that helps feed the last model to speed up the Delta builds by just adding a print line interactively debugging so in conclusion we've covered these topics so basil Fang our scalability discovered that even though Basil's are large investments if you look at the timeline we saw earlier literally a 5-4 year of work with many periods of intense activity try and get us over the basil it was definitely worth it for our large code base we talked to OSS dependency management because basil doesn't support it and we to roll our own but in doing so I think our own was better than most of what is out there like we get locked files you get auditable dependencies it's fast you never you never failed you the maven central being down it's great cross building where we've found a good cross build against any axis you want not just Scylla versions and made lots of upgrades and other things much easier than if we couldn't do that and lastly combination performance where scale compiler performance is slow but with a bunch of work not just on a compiler side wall so a language side and cloud infrastructure side we've managed to bring these 30 seconds to 15-minute workflows down to one second to 30-second workflows and the Scala compiler performance says more that's dropped off the feedback that our engineers give us the other thing is complaining about but they don't complain about Scala compiler performance anymore and I have a few references even the Google these you send out the slides later the dog post and toxic being about this before I want to learn more thank you [Applause] [Music]