SF Scala: Ulf Adams Interview
Recording: SF Scala: Ulf Adams Interview
[Music] hello everybody I'm Alexi crabber of the organizer of SF Scala and here on location at Twitter headquarters and we have this amazing meet up about functional programming and machine learning and it's about reproducibility and one of the topics were never covered before is basil build system we learn about it from Oscar Boykin who is a big champion in the community and he's gonna mention it and in his his man talk and so we thought we'll we'll get somebody to talk about basil specifically and we're very fortunate to have Adams who is the technical lead for basil who is actually based in Munich and he is visiting so it's all inside that beautifully welcome off we're really happy to have you so in our corner of the woods we use various aesthetic things to build things like we have our SBT which was supposed to be simple build tool and now like one of the most complex build tools in the world and we have some other Twitter users pans which is its own mono repo or a gigantic build system you know using Python and then we have some newcomers mill and CBT so we have programmers are very fond of build systems so we love this and so it's exciting to hear about basil but we don't know much about it so can you tell us a little bit briefly you know what is it how many people use it what is it good for but basil started out as Google Earth system ensign a couple of years ago we decided to open source it and you know you may know the name of Google system is blaze and basil is sort of a rearrangement of the letters we couldn't use the term place because of trademark reasons yeah and Google is using it for basically all of its entire projects not for everything you know if you know Chrome or Android those aren't using basil yet we're working on that so so it is basically a useful building very big for business that yes you could say that so I think you folks you know in the community probably build this kind of stuff right so so is it just us there is it is it kind of as complex as the code base you're building is it much more complex to make when I'm from one to ten right it's like if we make us three how complex's is basil this is certainly a lot more complex than make and if you're if you're dealing with such a big codebase everything becomes a challenge one of our biggest challenges is that naive implementations of you know dependency data structures and algorithms often run into a quadratic performance and memory consumption and at Google scale that's already a problem right we have to get it down from quadratic to linear and that requires very very careful coding and using all the right tricks and you know using all the tricks in in our repertoire to get it down to linear okay and so you see inherently parallel is it like basically a cluster by default so basil itself doesn't but it does it is able to run individual actions in parallel if they are independent so let's say you have you know it's called a program and it consists of two libraries and the libraries don't do anything with each other then it can compile them in parallel and it can do that on different machines so what we do at Google is we have a machine cluster with a couple thousand machines or so and then you can use as many of them as are currently free to to the Europe mm-hmm so what is the typical load you know how many machines are typically used for any kind of things I can't I don't know tens hundreds thousands well if you look if you look at basil you know basil will run about a couple of hundred threads and if there is is that much parallelism in your built then it will actually use a couple hundred machines to do that okay so it's so for you it's not a problem to harness two hundred machines and no process okay cool so and and how how long we should mastic is it right so I mean we're kind of in a weird is mostly scholar and this particular matter but if there are multilingual people I'm doing into itself is multilingual so you know how like language specific is it well it certainly has its roots in C++ and Java because those are the main languages at Google but we've been using it for basically anything there are people using it for Scala Kotlin Fortran Haskell you name it we have it I there I think there were even people who are using for Pascal mm-hmm I'm not sure if I remember how path code is written but you know it sort of diff to the past right but yeah it's all there it's all there so there so so is it is it hard to you know let's say rough people suddenly learn about you know like the like Python when they want to do it so how hard is it for for a new group to to use basil well it certainly requires some investment to learn how basil works you need to understand some of how basil works in order to be able to extend it because because of the scale issues we have developed very special data structures and algorithms to handle that scale and so if you extend it you also need to be involved in that that said the community has been able to write rust robots actually and Scala rules and cottonwoods with fairly little help from us mm-hmm so it doesn't seem too hard oh and I'm not a good judge for that okay okay so you mentioned that basically better kind of has its roots in Google's own build system right so so would you say that this is kind of it has almost like a decade the more of experience so so is it is it really so bicycles tested at all the scale right and so so it's it's really mature about systems it's open-source recently but it's actually very well - yes okay so yeah it's it's over ten years old now it was you know the first lines of code were written over that over ten years ago and it has grown with Google Google wasn't you know ten years ago Google wasn't actually as big as it is today mm-hmm the code base wasn't as big and so we had a lot of problems we discovered as we were going along mm-hm so what got you interested in in the space well I dunno I applied at Google and so I went to the office for the first time and the you know the sightly told me well there are two projects that you could work on how about you talk to the respective leads and then you know you can you can make a decision where you want to go and so I went to the first guy and he was sitting at his computer and I was like so what's your project about well you have to like it to work on it okay but what is it about all right so that wasn't the word that I said and then I went over to the other to the other guy and he said look let me start this built here and we'll let the plant run in the background while I explain how the architectures is you know how it works how its setup how its distributing builds on the data center yeah well like whoa that's what I want to work on this is so cool and then I stuck with it and so I've been working on basil for for nine years now wow that's awesome not quite so what did you do build systems before did you like make you know did you like to write micros not at all I just you know started to work on it it cool can I grew on me interesting and so you mentioned that the team is not distributed so if I folks in Europe pick up folks on this course but he said there's almost nobody in the area where Google started so what like was it an evolution I mean did they migrate well the you know the manager of the group at the time wanted to have thought it was a good idea to have people working on it in your neck or I don't know for what reasons and so he started the group in Munich and then the group in Munich was actually the main group working on the boat system for a while and then Google hired you know a director in New York to work on developer tools mm-hmm and you know a second group started to work on it from from New York interesting yeah I mean I know many folks Google but this is interesting to me like that heart or not like the build system is not a little developed in the vein which is interesting but maybe that has advantages right because when wake up you guys already did everything so so this is three you know we're really happy you know to to learn more about this so I look forward to your main talk and you know a portion to have you here so we know welcome back any time and tell us more about Basel you