SF Scala: Ulf Adams, Bazel - a Brief Overview

SF Scala: Ulf Adams, Bazel - a Brief Overview

Recording: SF Scala: Ulf Adams, Bazel - a Brief Overview

[Music] thanks for inviting me I got an email a week ago saying that Oscar wanted to have a second speaker to talk about Basel and we don't actually have anyone in the Bay Area well we have one person in the Bay Area who works in Basel but not you know sort of um corners things of Basel not basil basil and but I'm I'm here this week I'm actually visiting from Germany and so they asked me and I said okay I'll do it and so as you can imagine I had a lot of time to prepare these slides and so this is going to go so smoothly you won't notice that I didn't practice all right so since Oscar already talked about Basel I'm now going to talk about machine learning instead No all right all right so so when I started on Basel nine years ago we didn't set out to to write a book system that is a functional transformation of your source files but somehow we arrived there and we arrived there step by step so as we were working on it Google was using this build system and Google was growing and we were trying to make it faster and better and more correct and at some point a couple of years ago we said well you know we have five different graph libraries in Basel this isn't working let's start over from scratch how would we design this if we had to start over and so we came up with this framework which I'm going to tell a little bit about today that actually does implement what Oscar said a functional transformation of your source files all right so what is the build system actually well who doesn't know what about system is excellent so I can skip looks like very good all right so what's basil specifically well it's a build system obviously it's a cross-platform build system one of the things that Google had to do early was to do cross-platform compilation our data centers for all 64 bits our workstations were 32 bits and so we had a cross compile and so basil naturally evolved cross-compilation support you know and I think it's pretty good its multi-language basil is used for pretty much all projects at Google with some major exceptions chrome doesn't use it Android doesn't use it yet we'll see about that it's it's extensible you know at the beginning everything was built into the build system but that could not possibly scale right with tens of thousands of people writing code and wanting to use functional languages and Fortran and lists we couldn't we couldn't put everything into the single boot system so at some point we came to a you know at some time we came to a point where we said all right this isn't working we really have to make this extensible and make it really easily extensible because everyone is kind of want to do that it's hermetic sort of what does Hermetic mean actually it means that we know what happens in a bird and by knowing what happens in a build we know if something changes whether I need to redo the bird and which part of the build every to redo and I'm going to talk about that for a couple of slides it's correct mostly I have you know two or three changes for the remaining correctness issues that I still haven't quite been able to submit but we're getting there and it's pretty scalable and well let's let's start with that let's not start with that sorry about that all right so so one of the things that we found was that correctness and performance are not opposed to each other for the most part by building a correct build system we were able to make it faster and faster and by what what happens is that when you build stuff what you really want to do you want to do as little work as possible and by having a correct build system we know exactly what goes on in each step and that allows us to do as little work as necessary in order to update you know whatever output files whatever book you want to do whatever tests you want to run what you have binary you want to build and a key part of that is that we need to make sure that the binary said we're running on your system like the compiler and you know whatever tools you have that they are correct but there aren't unfortunately you know if Oscar writes the tool that's just going to go to the disk and read files here and there and everywhere and so what we're doing is we're sandboxing every action individually so that the action can only read the files that we allow it to read and that's a critical part of correctness if we don't do that it's so easy to get it wrong so here's a story when we open source basil we didn't have sandboxing and then one of my colleagues implemented sandboxing about a year later and he found 20 bugs in our own rules because we didn't have sandbox and we didn't enforce correctness and so the rules sort of back slid and we introduced these errors so so at Google what kind of scale do we have if you look at Google's repository most of Google's code isn't a single repository all right what does that mean we have tens of thousands of engineers we have alright the numbers don't match up I'm sorry about that those aren't my slits we have two billion lines of code nine million source files and 150 million test cases per day but just just numbers what's the billion all right let's look at this this is this is you know a couple of software projects every dot is a million lines of code so you know the Linux kernel is 50 million lights all right then Open Office is a little bit bigger Windows Vista Facebook is a tiny little bit bigger and almost as big as deviant five all right so how big is Google's source code okay that's the first slide this is the second slap what - all right and three this is Google's source code it is that large and bays were had to work with that and this is really challenging you know because in this code base you get every corner case you can imagine everything right we found bugs in the Java 9 compiler where people make changes to the Java compiler and say excellent you know they fix bugs except that our code was relying on the bugs happening so that's sort of the scale that we're talking about all right so I want to talk a little bit about how basil works and now as I said we didn't set out to build a functional build system but we somehow ended up there and I want to give an example and I want to walk through that example I'm not going to give you functional programming code like Oscar and basil is not actually written in a functional language it's written in the imperative imperative language and that mismatch is sometimes causing friction I admit it but we make it work all right so so you know this is a simple Java file you know anyone could write that and this is this is a build file this is what how you tell basil what to do and it's sort of a high-level description of what you got your intent is so here you say I want a Java binary and I you know it has some Java files in it and it has a dependency and has a main class it has a name and then basically goes ahead and puts that so so what happens here now if you look at this built rule what it says it this is a you know this built file is a package and within a package there are rules and there are source files you know the source files here is the source file is implicit in the glob we actually basil goes and looks at the it what files are in your directory right and then there are some implicit things that are going on where you know every rule can declare things and can say I'm gonna build a deploy jar all right but there is nothing else in this package there no there's no magic we know all these things when we read the package we know all the dependencies and there are no other dependencies there's no hidden dependencies everything is explicit in the source file and so you know in terms of terminology we say this is a rule this is a Java binary rule it it declares some dependencies it declares dependency on a bar library it declares a dependency on on a java source file and there is one more thing which can happen which is that the rule can itself also declare dependencies in this case the rule says I depend on a Java compiler now if it you know if it was a scholar would say I depend on Scala compiler if it was a C++ rule it would say I depend on C++ compiler but there are no other dependencies so now when you say basil please build this what we do is we built this grass this dependency graph and we start with what you don't want to do you know that's a node over there and then we look at that and say well in order to build this I first have to figure out what's you know what's this label it what is it about what and to do that it needs to read this package foo and that requires reading the build file on disk and also evaluating this club to figure out you know what are the source files that are associated with that here I'm using this this sort of directory this is sort of the the note that says what are the files in this directory and so you know we read the package and and everything's good and then we say alright we want to build it well before we can build it we need to look at its dependencies so there is a depends on bars so we need to read bar we need to read the boat file for bar we built a package all right we got that now we want to build it we want to create the deploy jar all right let's do that in order to do that we figure out oh we have to do two compilations we have to compile bar over there and then main over here and we have to do it in this order so there is this edge here which says I can only did compile the main class after I have compiled bar and then we do that in exactly that order and we're done awesome all right now someone comes and says oh but I want to make a change so makes a change and then basil sees this file was modified how would you how do we know that well there are several options we can ask you know the operating system which files were modified since the last time or we can step the files and say well you know the step for this file has changed it's probably been modified and then again we start at the end and the first thing we do is we mark all the things that depend on the sink that was modified and then we do the redo the steps and we're done and we did the minimum amount of work necessary to update this binary we didn't recompile part we didn't have to we knew exactly that it wasn't affected all right let's say I had a fire now this is where is this node comes in this represents the list of files in this directory so since you added this note was changed and we recognize that and then we know that we need to redo that glob and these are all the things that we need to redo and again then we when we get to the compile we see oh there is now an additional file there's also an input to the compile so we need to declare that dependency and we have it all right so so that's pretty cool I think now how does how does extensibility work and when we started to work on this bless you when we started to work on this we were talking about you know which language should we use well how should we do this should we use a functional programming language that would be pretty cool because it really matches the model and it would be so powerful but unfortunately most of the developers at Google aren't functional developers they're imperative developers and so let's stick with an imperative language sorry so we decided to go with Python because the newly built files written pison and so we decided to take a subset of Python it's it's a proper subset it's not as powerful as Python but it's the same syntax and it's really easy to pick up you know you look at it you can write it almost immediately it's really easy it's very familiar you know most people have already written some Python in their life one of the things that we did intentionally was to leave out all the complicated features of pison so there is a lot of things you can't do you can't do monkey patching sorry but for for a build system it's not really necessary so that's nice it's also the way we designed it was we wanted this to be high performance and Python isn't necessarily known for high performance so we didn't go with the existing Python interpreter but we designed our own and we also designed a thread a thread safety system into this where we can very heavily parallelized loading packages and analyzing rules you know which creates actions which are then executed there is no global interpreter lock because it's you know heavily paralyzed it's deterministic it's deterministic and you may not realize how difficult it is to get a deterministic we intentionally chose not to have floating-point types in the language which people have complained about because they want to compute like a percentage in order to do a progress indicator over here but unfortunately floating-point numbers are not the standard is really nice and I recently wrote a paper about floating-point numbers but unfortunately there are a number of operations that are not fully specified and so you can't guarantee bit identical results across machines if nothing else and so we started writing rules for for for basel and the community also started to write rules for basel we have SCADA rules we have rust rules the net typescript and the list goes on if you want to see the full list you can go to our github organization which has a lot of rules that's pretty cool all right so because it's this functional transformation system and because we use sandboxing to make sure that actions don't access things that they are not allowed to access there is one there is magic that comes in which is for every action we know what it depends on we know if actions are independent and we can ship the action the entire action to a different machine and this is the main thing how we scale to a deeper the size of Google by running hundreds if not thousands of these in parallel so if you look at a typical software project at Google it requires running like 100,000 compilers compiler invocations right a hundred thousand even with the fast compiler this is gonna take a while and the one thing that we do to make this faster is remote execution distributing this across many machines and not just just reading the execution but also caching the execution if we get a second request that is identical bit identical to the previous request that we've seen we return the result from the previous run and we have gone to great lengths to make sure that you don't get time stamps in your bird and you don't get any other source of non-determinism in your bird so that you get really fast incremental birds and really fast clean birds really right you start at Google you set up your machine you do your first bit most of it's already done because the CI has already done it this morning there is an open source remote execution service it's called boot farm it's pretty cool google also has one that we are looking into making available publicly as a service yeah what's next for Google for basil we we want to really improve our route story we want to not just for Scala we don't just want to have for us we want basically want to every language Under the Sun and in order to do that we need to do support the people who write these roads and to give them the tools to write really good rules and the data they need to do their job you know Android and iOS obviously is a big topic for us coverage is still a little bit coverage is a difficult topic because basically every language invents their own coverage tools and coverage data formats I hear that Haskell has a coverage format that has a per character coverage information which is very cool but okay what we really what would be really nice what we'd really like to have would be a single discovered you know data format that is cross language and the reason why why that would be so cool because it's because you often have we have integration tests we have integration tests that run an iOS app against the Java server talking to another app that is you know in Britain and Android plus a web front-end so we're talking about five six seven different languages already and getting a single unified coverage report would be so powerful but we're not questioning quite there yet and of course we we want to do even better on the caching side on the remote execution side that code is working and it's working well but there are a couple of corner cases so in practice many people don't want to check in the two chains into their repositories which I completely can't understand I mean why wouldn't you want to check in it Scala compiler into your apposite Ori unfortunately many people don't want to do all right I understand why you don't want because it's you know it's pretty large dependency if you look at it all right if you look at the JDK which I'm familiar with sorry the JDK is about 150 megabytes do you want to have that in the github repository probably not but that means that we lose hermeticity if you don't do that we can't guarantee hermetic but we don't yes Oscar please so Oscar says at stripe they check in the the shower of the compiler so it is actually hermetic and so we're looking at ways to support that in the in basil itself across the board even if you don't want to check in the shower so for example I mean in order to really get hermetic you have to check in a shower but in order to get you know correct bills it's enough if we know which files were part of the build so we can actually track if you have a local C++ compiler we can try we know we can know which files are belong to the C++ compiler and we sort of check some all of them and then use that in order to make sure that if you do an incremental build say after you upgraded your operating system that you get correct results that's actually what happened to us at Google we have people at Google who use basil and we recently had an upgrade to a new version of Debian and people's both broke because the compiler is locally installed so that's something that we want to fix and then you know we'll see who's going to use it you know maybe angular wants to use it darker is pretty big maybe we can do better support for darker and maybe we can do something more for skylight Oscar wants us to we'll see about that all right another thing that we're doing is we're actually talking to Microsoft they have a book system that is sort of similar to Basel we're talking to Twitter they have a booth system that's sort of similar to Basil's we're talking to Facebook they have a boat system that sort of similar to basil and we were wondering if maybe we could get together and do something together that improves everyone's lives and that allows us to to sort of bring this technology to more people in a better way and not have this sort of divergence where everyone uses a slightly different birth system and you can't move your code around that you can't use its rules and whatnot so maybe we will be able to to agree to common api's we'll see about that and of course basil is still better we're still not at 1.0 we want to get there a lot of people don't want to use a build system until it has a one in the front or you know some integer number larger than zero i stripe was happy so why not weren't you all right and of course we want to be more open we want to make sure that we replied to pull requests in a timely manner and yeah he's been asking buts about that before before the talk and also that you know all the changes were making to basil goes through a public review process all right if you want to start you know try it out it's basil that but thank you very much [Applause] we have a little bit time for questions anyone else in the North car all right over there yes go for it the question is how much progress have made with the unified coverage format and is there going to be something open source internally we're doing everything without cost because the C++ compiler is still the biggest part of our you know C++ the biggest part of our code base I have been looking into open sourcing that but difficult and L cough doesn't work for every language right so maybe you will have to ignore the Haskell people yes how much caller support does basil have currently ask are you one answer that [Music] Christina your name was minimize blog post about to you'd like help programmable Skyler buzz for whatever reason she chose Scala as an example to make it into an example and I saw that about almost three years ago and puck that looked kind of interesting and so me and some other people have certain working on that give their base a little slash moose we also have to get her set up because a lot of these a lot like skater please join you can see but we have a library spell binary we have Jam each pitch mark we have squirt perspex test if support for Jaden test we support for Scala test tests we do not have support for Tut if you like your documentation compiled check the documentation we don't have a support for code coverage we don't have any tooling that can inspect the Scala code and write down the dependencies that and I have looked hear more about that president but if you saw the build files that all should there's a little bit of the duplication between the imports of maybe a Java file and these dad's third category similar but not exactly the same that especially properly facility security people so no tooling for that you've got to like kind of write that but please visit that repo which are used in production for to a ninety percent strength it works very well we just deployed caching works fair reasonably well our laptops because this hermeticity laptop certainly a challenge to integrate with our CI but it's where I grew up because our build times RC I like down to from up to 25 minutes very decent cash amounts is only like three minutes now it's it's what you should well so very very good sir thank you I have one that takes 45 minutes so I'll look at it well with the repo if you don't mind well these will build / . my phone so it's gonna be there for oh you'll see official scoundrels but they're used by many organizations now myself and gentlemen acquits are the main maintainers and we responded poleward of us very rapidly big believers in the reserves one of our technolog detective sorry master yes person over there [Music] so the question is how many build sites do you have and the question is what's the recommendation huh how many should you have my recommendation is one per package one per directory basically not everyone follows that and Google is pretty good at edit but if you if you especially if you migrate say from from maybe where you have a single maven you know pom.xml file many people then end up writing a single build file and it gets huge and that's not such a good experience because then you know you often end up with a single Java compiler and that it becomes really slow and we get paralyzed and so on and so on but I can tell you what Wixted the company they actually as part of their migration they wrote really fine-grained build files and they get super speed up on their books so it's really worth it to do fine-grained code files because you get generally get a much better experience now maybe there are languages where it's not a good fit I haven't seen one yet so all right rasca now you go and there before previously and it fits like that a model of modeling she and basil as she was kind of up you know when that bill whereas basil and so I'd be interested their take on that and getting comment on it and secondly this monadic nature the lack of the ability to see what's built-in visual hampers us in particular when we want to bring tools in as repository rules but so repository rules they just like the binary block you gotta pull them in it's very difficult to make a hermetic repository so paper has your thoughts changed at all about Internet of monadic kind of cycles in vivo or is it like solve this problem to sequence builds all right let me summarize that there has been a recent paper about those systems by a group of three people one of who is at Microsoft they analysed a couple of build systems among which is basil they actually managed to write like a ten line has the description of basil which is very cool and the question is whether that has changed my thoughts on something that I couldn't quite understand because I have no idea what about that is but I have been in contact with the author of the paper there were few things about the paper was I didn't quite get it right and I think what you're aiming for is what they call dynamic dependency discovery which is the idea that you might want to do some work first in order to figure out what work you need to do next and if you look at basil if you think of the slides I had about you know how these things fit together with these nodes and these dependencies that's basically exactly what basil does already except we don't currently expose that to rules we don't have an API for it I think we will have an API for it at some point in and I don't think I'm changing my mind here but I've always thought that this is a really powerful feature that we'd really like to expose but so far we haven't found a good way to do that there was a recent proposal on the mailing list from from Paul from I can't remember yeah first got to do something like you know to look at Scala source files in order to figure out what the dependencies are between libraries and it's a really cool proposal and we'd really like to do something in that in that area and we think it's going to be very powerful and create a bunch of so allow you to do a lot more a lot of interesting new things in the boat system now the question is do you want to do interesting things in your build system or do you want to do simple stupid things in your bird system and leave the interesting things to the source code but I think I think we're going to offer something in that area we don't currently have specific plans for that and there is the it's a little bit tricky because it will break a few invariants that we currently assume so for example we do large-scale static analysis over our action graphs to figure out for example to generate for example ID information and if we can't do that anymore if we have to start running things and reading files in order to generate the the information that potentially makes IDE updates much slower and we still don't have a good handle on that and how we're gonna do that all right viewers say that in fact the schools take advantage of these features that you're talking about as you're saying maybe we don't want to allow and so maybe one of the best things for me though would be all of the rules work if you say other people should have momentum maybe that would advance it more rapidly because in fact plus the even performance also uses this trick that it's allowed to reach in and break the rules of starbug and is that is that the pulse a mischaracterization all right you asked very long questions that's guy let me summarize Oscars saying that some of his colleagues at stripe who previously were at Google are saying that we are actually not following the rules ourselves because we have C++ rules that sort of circumvent what we provide in the skylark ap is and that allow us to do set sourcing I have recently been working on open sourcing exactly that part so that people can look at it it's not all that smart about it and there are a couple of problems with it that we haven't quite been able to solve yet but yes it's sort of doing something like that it we call it include scanning so so when we initially implemented basil at Google we previously had a previous build system that was called G config and in G config you also had built rules like here basically the same syntax but G config was generating a make file and then the make file would be run through make and make would run with C++ compiler on your local machine and the C++ compiler would then go out and look at source files and header files to figure out what files to include and so C++ developers at Google were used to not having to declare the dependencies and build files and so as we brought basil we thought well maybe we have to somehow get them into the new system first and then they will declare the dependence pretty please unfortunately they didn't but instead it turns out that includes canning is actually very it's a performance improvement because it allows us to subset the list of files that we sent to the remote machine right so we basically we implemented our own C preprocessor that goes through the source files and it uses our sort of incremental functional incremental execution system to do that so that it's correct but it reads through the source files to figure out which header files are included which header files and then we get a list of all the head apart and we said only those to the remote machine and so for some of the libraries at Google that's a very significant performance improvement because it reduces the number of files from thousands to a handful and so it's not quite you know fully I read files and then I declare dependencies based on those but it's more like a subsetting thing and so I want to make it open source so that people can look at it and see that we're not doing anything sinister there and all right alright so that's one part of my answer my answer is also very long apologize for that second part of my answer is we have just recently started on rewriting the remaining rules in Skylark so we want no rules to be in Basel we want basel to be sort of a minimal birth system and have all the rules be implemented in skylark as I said earlier skylock came into being after we had already written a couple of rules sort of built into the build system and now we have this extension mechanism and now we want to do everything with extension mechanisms there are a couple of reasons why we want to do that you know we want to decouple the rules from the core we want to disallow doing these sort of sinister things with include scanning or alternatively we want to find an API that allows us to do that properly and that allows other rules to also do that if they want to thank you so that's that's the second part of my answer if we're going to work on that over the course of this year probably next year as well the C++ rules are incredibly complicated because C++ is so complicated and we have to see how far we get and how quickly we get there certainly for some of the simpler rules like for pricing for example it's you know not quite a done deal but it's gonna happen this year and it's basically straightforward but for for Java and C++ it's going to take longer because those are the rules that we have put the most manpower into to make them fast and to make them fast we also introduced quite a bit of complexity and so migrating them is going to take a while but there we go so it's it's all happening I promise it's happening but in terms of timeline it'll take a while that's the best I can say any questions from someone else oh yeah so the question is there I talked about compiler versions but what about optimization flags and other requests for different optimizations like across a repository [Music] she is assuming that there is one standard flag for the entire repo that's not quite accurate we actually we have a fairly complicated C++ compiler configuration mechanism in Basel that allows you to do a lot of very interesting and very powerful things the reason how it came into being is that at Google we have a lot of C++ developers and we actually have a compiler team we have a team of people who are working on the compiler and they wanted to be able to do interesting and novel things with the compiler and so this is it's pretty complex if you are an expert in it then you can actually write a compiler configuration that allows the users of the republic' you know the other developers in the repository to apply compile it you know optimization flex to subsets of the repository there is not currently a mechanism for it for example to to apply certain flags to a subtree but right now you sort of have to go in and and you know add something to every rule but you can add it in a generic way such that if you see oh I want to add this additional flag you do it once in the configuration and then you don't have to add it you know - every - every rule individually so yeah it's pretty powerful but it's incredibly complicated and there's they're only like three people in the world to understand it there was a question over there yes so I didn't catch all of that so you said there was a slide that said we had 150 million test cases at Google and the question is so so basil has a test command and has a special representation for tests and we have a fairly complex testing protocol that allows you to do a couple of cool things let's see one of these things is charting so since we work very well with parallelism we can give you a way that to take an existing test which maybe consists of multiple test cases say in Java and then to deterministically you know compute subsets of the set of tests in your target and runs them as separate charts and then we can run all of them in parallel or even on different machines so that's pretty cool we have mechanisms to handle flaky tests so if you have almost everyone has flaky tests who has tests but we have a mechanism to automatically rerun flaky tests and that's very very useful for your pre submits in particular which we just recently enabled for our own prism it's and it's made our life so much better because now we don't we only get a failure if a test page three times in a row and three reruns are sort of the default behavior and there are it's you know if let's say you have a test that saves one and a hundred times you rerun it three times it only fades one in a hundred ^ 3 times and that's so much less likely for it to fail what else do we have we do have support for coverage it's not awesome but it's there and so we can tell the test you know we can do we can build a test with offline instrumentation or we can tell the test at runtime that it needs to collect coverage and then it can do runtime and thread instrumentation and we can collect the data and actually get it back from basil it sort of currently doesn't work in all the cases but it's there what else do we have for tests so we we generally encourage people to run the test in the sandbox and we actually have an for Linux we have a networking sandbox so you can say that my test isn't allowed to access the network or you can explicitly clear this test is allowed to access the network you can you can mark tests as you know they are accessing the network and then basil will rerun them on everyone which may or may not be what you want but yeah there are a couple of other cool things about tests that I forgotten but you know look at our testing protocol it's pretty cool all right [Applause]