SBTB 2023: Justin Kaeser, Taming the Beast - Better Bazel in IntelliJ
Recording: SBTB 2023: Justin Kaeser, Taming the Beast - Better Bazel in IntelliJ
[Music] I'm Justin and today I'll tell you something about taming the Beast basil uh to work better in intellig idea now I have a vision of of a world where editing and building your code is one integrated workflow a bright new future where you need not think about how to configure your system to make it compile it all a world in which code writes itself and we can finally T take the time to express ourselves in art and get an artist salary or strike that last part now actually I'm just taking baby steps in this direction I've been working on making build tools work better in the IDE in t for over seven years now it jet brains and well on the face of it it seems quite simple you know you just uh write a bit of code to make the IDE talk to the build tool like run a few commands get the output easy uh write a bit more code to get the tool to out put get the output from the tool let's say it used to be spt that I worked on uh easy well and then I found myself writing an spt plugin that inserts a hidden character into the spt shell so that intell can parse the beginning of an input prompt and uh then sideloading this plugin into the spt config through custom plug-in directory iies so yeah very easy now oh why why are we in this situation well the the build tool build tools typically are not designed with the IDE in mind nor was the IDE built with build Tools in mind historically at least so the build tool is maybe a bit like this itsy bitsy spider crawling along a direct a cyclic graph um eating some files running little programs on them to turn them into other files and then feeding them into the next node it's all nice and algorithmic uh whereas the IDE extensively also uses algorithms but it might seem like a bit more like this sorceress here trying to conjure up a piece of code a file from somewhere so basically Random Access across your code base navigating back and forth uh and you want to jump anywhere do things refactor the code without e even being able to build it so in a broken partially broken State um and all this this implies that the IDE and the build tool need a partially different project model uh and well we used to define the project model in the IDE directly in the IDE and we still have some nice goys graphical Windows where you can Define which uh which files Go in which module in intellig for instance um but I don't know not many people seem to touch these anymore at least I don't I just sometimes check out if check these windows out if there's any issue with what got imported from the build tool so yeah we call this import or refresh or reload or sync because we have to synchronize the project models uh and usually these days we want to use the build tool as a source of Truth uh so yeah ideally we want them to be the same they cannot always be the same because they're built against different constraints uh also parts of them in the build tool are often computed so some build tools try to be declarative but in the end they all end up uh having some kind of scripting either inside them or around them that modifies the project model or computes the project model because you don't really want to declare repetitive things over and over again so that means you have to run the build tool to actually know the whole project model so yeah keeping things in sync is hard uh so some years ago we came up we that's uh jet brains and the Scola center with this little protocol that I've spoken about here previously some years ago uh called bsp the build server protocol to solve or well at least mitigate uh this issue of talking to build tools so I wouldn't have to sideload plugins into spt or any other build tool ever again um and well those and and as a side effect maybe solve the N byn problem of integrating n build tools with M editors or idees well I only really care about one of those idees but may other contributors cared about others so we actually kind of agreed on this protocol as the name implies it kind of is inspired by and complements LSP the language server protocol if you know that uh but it aims to integrate build tools and the IDE uh and it does this by abstracting the project model into a few generic Concepts uh such as targets dependencies between targets sources and offers a few standard requests for you know building testing running and very importantly signaling errors and messages and progress that happen on the build tool side so this has been quite successful in the Scala world at least all the major build tools there support it it was also kind of inspired by in Scala back then every few months or so somebody came up with a new build tool which was kind of troublesome to integrate well meanwhile I've moved on a bit uh because recognize this new Challenger this beast called basil appearing on the horizon falling over the Bay Area and everywhere else uh consuming big monor monor repositories So What attracted this Beast here's the list large complex code Paces uh you know projects just always grow uh and these large code bases predictably lead to Long build times even if you have small changes you might complain over build times over half an hour hours days um and as your organization and project grows people bring in their favorite language so you build in multiple langu against multiple language ecosystems and well now if you try to reduce those build times one approach you might take is just splitting everything up into smaller repositories but then you land in dependency hell depicted here where what you have to if you change one important library then you have to update all dependencies or all dependence of that library to bump the version and probably be synchronous ideally and then update all their dependencies and so on and so on and and that and you spent a week just updating dependencies so the uh popular approach these days is to throw everything into mono repos just have one version of everything uh so no more dependency hell but well you have new problems so you need to be able to build all the languages in there and you ideally want to do it incrementally so making the deal with this devil what what do you get it promises to be fast uh achieves this by aggressive caching promises to be correct whatever that means uh it but generally it means that builds are reproducible and hermetic so independent of what what system it runs on so ideally if you run it locally or on the CI you'll get bit by bit the same result the cost of this is of course you have to pay if you make a deal with a Devils you have to fully specify all your inputs uh not just the source files and their exact versions but also uh time stamps environment variables I don't know what else uh basil is generally built to respect all these and well if you change one of them it might cause a complete rebuild so you have to watch out a lot uh and this also means you can't rely on system level tools or Library so stuff you have installed in your computer it has to be defined as part of the build but the real Crooks as I've heard has been the IDE experience so I've talked to a lot of customers over the past year and uh yeah over and over they say again their experience with basil is lacking so well we have this nice protocol bsp uh we have a client that was implemented in the Scala plug-in so just write a bsp server right so it should be easy wrong well we did write a bsp server it wasn't too hard it took like a few weeks um and it worked well enough for one project that was basil bsp um and the next project we tried it on uh well it broke of course and every other project additional project we try we find additional problems uh that we have to solve like one by one um so this has been quite challenging because there's always something breaking but after a lot of work uh well we built a new bsp plugin for intellig it's currently in uh open Hidden beta uh ask me about it anyway uh now most jvm projects Java cotland Scala uh work with it and some python projects too so yeah now yeah I I said jvm Works uh we still have the challenge that uh any larger user maybe inclined to write new basil rules so the the configuration is very flexible there are like standard rules sets of rules that or plugins you could call them for Basil to build individual languages but uh sometimes there are multiple such rules because somebody had a different idea of how to do it and we might need to integrate with them separately so for each set of rules we have to write a new integration both for getting it from basil and for uh importing it into intellig so for each language there are special considerations so that makes basil kind of uniquely challenging to tame and integrate I'll just tell you about some um of my favorite problems we encountered one of them we call the single file Source Roots so basil is nice and flexible and general so uh you can attach to a targets a bunch of sources you attach them either as uh individual files or as a set of files defined as Globs which then get expanded to a regular set of files if you talk to Basil now you will only get this set of files uh it expands them for you it doesn't even tell you the Globs unless you par the configuration files directly uh whereas in intellig historically at least it doesn't support individual files as sources and modules it always works on a directory level was just more comfortable more simpler to imp Implement I guess and worked fine with Maven with gradal to some extent with SBT uh with basil not so much so you have sometimes issues uh that yeah the mapping between sources and modules doesn't work correctly and uh packages so Java packages can be split between m multiple basil targets whereas in intell we assume at least historically a one to one mapping so we went on an odys to cure the symptoms at first as first we triy tried it doing it the right way you know like actually patching the platform to accept files because well we've been working on a new project model in intell for several years now in the background and it can cover some of the use cases like adding single files as sources to a module um but they were filtered out because none of the language plugins really supported this so it broke the language plugins so we tried to fix this uh but then we found the language plugins rely quite heavily on especially the Java plugin on on this imp implicit bidirectional map between package and package directory so oneto one mapping um now we tried removing that but it turns out that that has performance implications and the last thing you want in your IDE is performance problems so we found a new workaround classic hacks uh registering a dummy directory on the top of a package route um yeah anyway the result of this is if you use basil with intellig you probably want to just uh Define your sources put your sources in a very standard directory structure you're probably familiar with it just have each directory correspond to uh package segment so this is the standard Maven layout or standard Java layout really so even if basil allows you to circumvent this try not to it will just work much smoother okay another favorite problem of mine is a shared sources problem uh well in basil this is a common pattern uh or also inbt in Maven maybe not so much basically the IDE is you have targets or modules that uh compile the same sources in different contexts so for instance you want to compile uh Source against both JS jvm or native or against Scala version 212 213 30 um because you want to release your library to different Scala versions for instance um yeah this works fine inbt in Basel but intellig also has the yet another long-standing assumption that you have a onetoone mapping between modules and Source directories uh what this means in practice then you have you have uh yeah if you have multiple contexts in the same module uh you might get completions that are not supposed to be there or alternatively if you don't import them fully you might get uh red code where it shouldn't be um I can maybe show you an example soon uh if we have a few minutes now uh yeah what do we have if what do we do if we have a bunch of layers and problems mediating between these layers well we add another layer uh we call this the magic meta model this just a implementation detail of our bsp plug-in and basically it's a project model that's between bsp and intellig workspace mod model so now we have four models that we need to keep track of and well this one keeps is very fairly simple it just keeps track of which targets depend on which sources and allows you to switch on the Fly between sets of sources or sets of dependencies now these were my favorite problems uh they're not the only ones uh so day-to-day a big part of my work these days is just talking to customers understanding their requirements and blockers and talking to various interface points so the authors of Basil at Google Google or rules authors at various uh orgs so each one has their own interest uh languages they support if they're not supported by basil itself um the intellig platform teams and the individual language plug-in teams and so yeah this basically these technical problems become organizational and political and and that's just in a this small niche of build tools integration with ID idees uh yeah so we talk to contribute on all these fronts now so here the the Beast has maybe been tamed a little bit um any questions so far if not yes what you're talking about is different from plug for oh yes I I didn't even mention that one so yeah uh there's the basil plugin for intellig which was created by Google unfortunately yeah nobody was really happy with it uh but we did we are actually co- maintaining it until this new plugin is sufficiently mature yeah uh yes are people yeah pretty much yeah or that so if you want basil with intellig right now uh most of the time you want to use the Google plugin I think for jvm our plug-in will be ready quite soon in the next month or two at least for some use cases yeah uh anyone else yes uh not off the top of my head like time it takes to execute tests in basil yeah I don't uh have stats at hand uh but the main benefit is with basil yeah it will cache test outputs based on the inputs uh so you can avoid rerunning tests or building Parts in many cases with caveats uh like depending on how you configure your build uh sometimes the incremental build times may be longer than with spbt uh basil still depends a lot on manual configuration or configuration through additional tools that write the build files for you so that Define the build structure for you but when you have that you can get pretty fast incremental rebuild and test times yeah uh I don't have any in my mind right now well I think leh how you wrote something about that some years ago but uh I don't know right now any more okay how much time do we have another few minutes another few minutes okay so I'll just uh give a short very short intro into the new plugin uh let's see so I prepared an example project so that it wouldn't take forever to load and of course it jumps so I just opened this project very straightforwardly and well this is very uncomfortable to do on a double screen okay I do you even hear me I hope you hear me so I open this project uh you see a standard project structure uh you'll see is a SN with a bunch of steps and you'll have this little basil icon here on the right a list of targets that are defined so I only have a handful of targets but it can be thousands in many basil projects say double click this one to compile it must build ID or this you can also run it and it runs the main in that Target so it gets the information from this from from basil so basically the things you'd expect uh from a build tool integration uh what we have maybe s the here we go that's slightly better okay so this project uh also shows you the issue with shared sources so let's have a look U so here's what a basil build file looks like and uh basically this just defines it as a Java library with a glob of sources they're just the ones in the same directory as this build file and a dependency on lib B lib B1 uh so the specialty here is it defines two Targets within the same basil package which is a concept that is related but different from the Java package um and you have yeah lip A1 Target and lip A2 Target and they are both have the same sources in this case but depend on a different Library uh or different other targets so lib B1 lib B2 so if I open lib a um right depends on Li B I can n navigate to Li B and in principle when this is run it should output lip a that's what was happened here before uh now what you can see here is this little widget is Select it and you can load the other Target that is defined for this source file called lip A1 and well it marks it red because right now we don't automatically update the dependencies of the target so I can uh also load the dependency here from the unloaded Target and now it's fixed again uh yeah so this is maybe not the most obvious uh use of this but it's an example of how this uh functionality works so it should work maybe I'll make a example project that does this specifically for Scala sometime and the Scala cross compilation because uh yeah this bsp plugin we made now should in principle work very similarly not just for Basil but also for SBT or buck buck 2 if somebody at Facebook meta implements a bsp server or other bsp enabled build tools okay uh that's all from me any more comments questions okay yes what are yours us no it's embarrassing thank you