SBTB 2019: Justin Kaiser, Integrating Developer Experiences - Build Server Protocol and beyond
[Music] in ancient history software creating software involved a lot of manual labor you'd invoke your compilers by hand 'luntar on a bunch of compilation products in the command line and maybe if you're more sophisticated you'd create a makefile and sort of hand wire all your dependencies and compilation products and everything that goes into a completed piece of software by hand but yeah that's a lot of manual labor you know stone by stone so at some point along came the IDE the integrated development environment which would give you a lot of ways of editing formatting refactoring and structuring and organizing your code and then building running testing debugging packaging deploying all at the click of a button now I'm Justin I work on such an IDE the Scala plugin for IntelliJ just out of interest who of you uses this fine piece of software thank you thank you for thank you by the way who knows we're on github like our software's open source that's too so next time all of you raise your hands you can actually contribute to this with source code but also by just reporting bugs if that's more your kind of thing so what I want to talk about is integrating developer experiences namely how to integrate specifically built tools into IntelliJ more flexibly and in a better way than we've been doing so far and this is done with a build server protocol in short BSP and I'm gonna give you some specifics on how this protocol works and how its implemented in intelligent but first step back so we've had manual tools on the command line we've had an IDE which integrates everything but increasingly we've been working more again with one tool jobs like one job tools that you can run on the command line in a variety of contexts like on also from the IDE from the command line from a CI server tool because the way the IDE does it maybe not doesn't fit your workflow the way you want to run it on the CI so in the end you end up running a simple command-line tool when we are another but what about our general-purpose front-end we still want to use that right so classically we've tried to integrate this these tools manually one by one into the IDE but you see there's maybe a little interface mismatch going on here so you have a lot of friction getting things to work together there's idiosyncratic configuration api's you might have to context switch between command line and IDE you need to know two different interfaces in the and or more if you use web front-end for the CI but still we've we've done this for a bunch of tools like Scala tests specs for SBT specifically as a build tool in the Scala plugin for a bunch of frameworks like akka place cough FM t and other tools like bloop have you heard about bloop yeah that's what if you yeah bloop in the end integrates with build tools and it has exactly the same kind of challenge so build tools and Scala specifically are a fun story so many languages have one true build tool or maybe two really popular ones like maven and Gradle for Java and Scala nobody seems entirely happy with what there is is evidenced by the fact that we have a new coming out every month or a year or so so we have a lot of choices little coherence between them and in the end you still want to use your favorite editor front-end which is of course IntelliJ or maybe one of those others so yeah you have this these manual steps in the workflow that I mentioned switching between terminal and IDE because it's not properly integrated so you have this tangle of haphazard integrations that sort of keep parts together custom integrations for some build tools like SBT but it's still not very coherent so I've been trying to tie these loose ends of tooling together a bit to make it a little bit easier for you to create a more integrated developer experience but before I get to their build server protocol let's look at some prior art the language server protocol suspiciously similar named which has been created by Microsoft to address a similar problem in the editor versus compiler space so in IntelliJ what we do mostly is integrate compilers with the IDE one by one for each language and basically we implement large parts of the language in the IDE and most editor tooling support support there's a similar thing and the language server protocol defined defines a standard way for the editor to communicate with a language server which is basically a JSON RPC front and for the compiler and this has been a quite successful approach so successful that people keep asking us if we're gonna use it for Scala three and answer is like maybe but there if we were to do this in IntelliJ we'd lose a lot of the features that we can offer that the standard compiler doesn't really such as type aware highlighting or like giving you detailed type errors and comparing type tips and so on so you just get regular strings or whatever the compiler is able to give you and so that's one of the challenges another is like LSP doesn't really model the structure of a build of a code base so we have a bit more to contend with on the IDE side now I mentioned various build tools let's take for instance new build to emerging from the desert sands called fury which has been been worked on for quite a while now actually and maybe it will be released here or maybe sometime soon so right now it doesn't really have a big user base basically a few developers so if I wanted to support it I of course I want to support all the build tools but I can't really justify investing the time into this specific one most of the time and so another option is that the author of the bill tool John pretty contributes the support for IntelliJ but like he doesn't even use IntelliJ so he's not very motivated but maybe his users are but anyway I just want to support fury projects in IntelliJ so a what I can do is go here oh I just opened the Fury project in IntelliJ so how did I do this I implemented the build server protocol in IntelliJ in Fury and IntelliJ so fury is about server protocol server and I would show you around a bit but it's a bit impractical with the this set up so anyway yeah I was a bit inspired by a quote by Abraham Lincoln or maybe it's made up he said give me six hours to fell a tree and it will spend four hours sharpening the axe and I said yeah give me one month to support a build tool and I will spend the first year developing a protocol yeah there you go so the build server protocol yeah it sort of extends our compliments LSP it's also adjacent RPC protocol and it aims it integrating build tools and IDs we're also language servers so potentially you could have a setup where the IDE or editor talks with a language server by LSP and that language server talks with a build tool and that's actually what happens with metals if you're familiar with that so unlike LSP ESP has a project model that it can communicate to its clients and also a few generic concepts like build test run so to give a common language to talk about these very simple basic actions you can do on a build so these generic cop concepts dumped necessary and that perfectly to each tool but I think they're regular enough that you can use them in most places a client or server also doesn't need to support all of them so it's potentially enough to just support import or build you don't have to do testing and running you can still do that under command line if you're lazy implementer so like LSP it's a client-server JSON RPC protocol with a small specialty that it's sort of bi-directional while the client attaches to a server usually also it's responsible for starting the server either side can send requests to the other or us a notification so typically it's the client that sends requests and the server that sends notifications notifications here just mean informational bits that get sent one side and don't respect and expect an answer one thing we address that LSP doesn't is finding that there is server available that we can connect to so in LSP you need a server specific implementation in your client with PSP that's not strictly necessary because we say okay a server just drops a configuration file into your project or into your user space and the client can find that at specific locations and this file just tells you hey here's how to start a PSP server and how to talk to it and usually that's just standard in standard out protocol it's a very simple generic system and that allows you to write a BSP server and immediately import stuff into IntelliJ without we on intelligence I having to do any support for your build tool now one major part is the build structure that I mentioned so there's a bunch of requests that you can support namely build targets and a target is like abstraction for a bunch of information about sources or how to compile something so it has capabilities such as you can compile this target you can test this some language specific data such as this target should be compiled in Scala files as Scala and dependencies on other targets so basically with that you can model some directed acyclic graph and of course you can have sources associated with some target this is a separate request because the sources can be associated with multiple targets or likewise one target can be associated with multiple sources there's a bunch of actions you can do that I mentioned such as compile test and run so the way this typically happens is yeah you sit off a bill target compiled request with a bunch of targets that you want to compile the build server will hopefully compile it successfully and eventually deliver a compile result on the way there you can get a bunch of notifications from the build server it will send the notifications to the client and those notifications encompass progress so I compiled 120 over of 1500 files about 200 files and so on and diagnostics that's the word we use for informational and error messages so yeah errors warnings and connected to a specific code location LSP has exactly the same system so a language server can just push through Diagnostics from PSP to an LSP client the progress notifications we call them tasks also simple there will be a task start notification which is for example I started compiling this target and then you get a bunch of updates and eventually a task finish notification so I want to show you a bit how this works in IntelliJ scala plugin so I open fury over here now this is hard to do can someone hold my laptop please I'll just close it was kind of broken open akka come on yeah I should have done that right so it can trigger a build in akka and you'll see here a bunch of progress notifications and how long they took and a bunch of warnings or errors of stuff compile errors so thank you and what's useful about that is you see where potential bottlenecks are so you see this acha stream it's already taking 40 seconds acha actor tests tests is taking over 40 seconds to compile say so you can find which module which target actually is taking the bulk of your compile time easily and this is available already in 2019 - as an official feature next week or so we're releasing the 2019 3 version of IntelliJ and it will include some general improvements for the BSP support as well to make it more stable and usable so you can use it with a bunch of build tools already however currently it only really supports Scala targets well I'm talking about targets so we've got a few challenges mapping modules targets two modules so in IntelliJ you have the concept of modules which are kind of similar to targets and BSP they're also kind of similar to projects and SBT kind of similar to targets in basil and so on but they're all like subtly different so forth but as an example let's take SBT projects they are usually divided into several scopes compile test integration test and so on generally you might only want to be using compile and test because the other stuff makes the builds you too complicated so instead of different scopes he'll just use separate projects but let's take a SP t project with a compile and test scope now BSP doesn't have scopes so what we're gonna do is when mapping this to BSP for example when exporting to bloop you'll create a separate target for compile and for tests but they're linked laia their base directory for instance and then we'll have we have another challenge SPT and BSP support shared sources so sources that can be compiled under the context of different projects targets it's typically the case when you cross compile against Scala Jas Scala native and JVM or two different versions of Scala now IntelliJ modules don't support shared sources but they do support scopes but only compile and test scopes so what we do here is we they take the different compile and test scopes and map them onto the IntelliJ scopes and for shared sources we create a synthetic module which contains the shared sources and listed as a dependency of these different other modules that depend on the shared sources that's a bit problematic because depending on the dependencies of the shared sources again of the modules that share the sources you kind of have to just dump them into one scope or you need some way of switching it so that often might break your highlighting and IntelliJ if you've been wondering why cross-compiled projects don't work that well we're figuring out alternative ways of doing it but for now we have to kind of look with this workaround so sure look at the BSP ecosystem so as a client implementations we have intellij and metals metals is a language server that you can use from es code for instance and as servers we have loop as the primary implementation which in turn is an entire built tool but instead it tries to act as a compile server for different build tools but what you can do with different build tools such as Maidan Gradle SBT pants and mil you export their bills to bloop build and then you can import that bloop build into IntelliJ or into metals fury is a little special because it is both BSP client and BSP server so it uses bloop as its compilation server but adds a bunch of build tool things on top of that so towards bloop it's a client and towards IntelliJ for instance it's a server and then here in the lower right corner we have basil and pants which are currently also developing dedicated BSP support so you might already be able to use it if you heck around a bit let's have a quick look at the further development of BSP so protocol version 2 0 final is pretty much done we haven't changed muddin much in the past months as you saw we have IntelliJ bloop bloop URI middle Oh building on this version supporting to support BSP and what we're looking at currently is to support BSP with IntelliJ platform distribution so having it as a kind of a core plugin but that would entail of course supporting more languages besides Scala such as obviously Java Cortland I've also seen some Swift developers looking into supporting BSP server and maybe client side for better integrating their build tools and if we want to look a bit beyond bsp beyond what we have right now obviously yeah we want to think about integrating more tools with standard protocols maybe so have to think about what would it take to integrate various linters should they be integrated in a build tool or should they also support BSP is this some kind of in between part and yeah remote building is very interesting and and yeah maybe more fine-grained testing support though BSP is already pretty good at that but maybe we want to be able to support testing of individual tests so have some standard way of addressing individual tests the way this is done in SBT and IntelliJ kernel is it's pretty happy and very dependent on specific testing tools and if you have more ideas I'd be interested in hearing them ignore this slide apart that was for the last conference no yeah obviously one challenge with creating a protocol is yeah are the tools similar enough to warrant actual protocol all this coordination work on a protocol and but the big part is adoption say you know the old xkcd you have 14 competing standards add another one you have 15 competing standards so right now I'm just trying to drive the adoption a bit that's why I'm talking to you now if you're interested in working with PSP in some way or another please do talk to me so maybe we can get a cooperation going creating another making another bill to work better with IntelliJ for instance so if you look at the slides later you've got a bunch of links to the specification libraries test kit and blog post which basically talks about this same things I talked about today so I'm Justin I'm on Twitter is even Beth and Scala plugin team is on Twitter is IntelliJ Scala so you're welcome to follow and tweet at us and now please I'd like some questions or comments if you don't have any that's also fine or if you want to talk to me privately that's also great okay