SBTB 2019: Ólafur Geirsson, Metals - building rich IDE features beyond the Language Server Protocol
[Music] so hello everybody I'm Olaf I'm gonna present metals building rich ID features beyond the language server protocol today if you're not familiar with metals it's a language server that gives code completions and job to definition and I edit errs like vyas code boom so and if you're not familiar with the language server protocol don't worry I'll explain it just in just a bit so this is my first time presenting here in the US and it's also my first time presenting as a Twitter employee so I'm super excited to be here and I'm really excited to show you some of the amazing progress that has been made in the metals project for the last five months and when I say we it's actually been a large collaborative effort from from a lot of people so I'm here on stage sharing what what's really been done by a large team from different organizations so it includes Jorge who's here in the the room as well Gabriella Thomas Marek Chris and also 50 other contributors that have been working for the last year on this project so I want to talk about bridging the idea gap and it's the idea gap is not a thing on Wikipedia it's a thing you I'll explain in a bit so really what once upon a time before you know and it's not really a long time ago it's maybe five years ago if you were doing you know language tooling you you there was the situation where you'd build a custom if you were you know a compiler author of a new programming language and you wanted to have code completion working in IntelliJ and in vim and in sublime you you ended up doing a custom integration for each of these editors and then on comes the next language and repeat and as you can imagine there will be holes in this matrix of support you know some editors did not work with some languages etc and usually the situation was that Intel just had support for everything and that's pretty great so enter the language server protocol that came a few years ago I kind of solved this problem where if you're a compiler author you'd only have to know your own compiler stuff you didn't have to know any details about a what the editor provides and then if you're you know the author of an editor you could just do the stuff that the editor does best which is you know rendering completions and sending save notifications and manipulating buffers and and so the language of protocol kind of created a contract between these language servers and editors and what's amazing is that you'd only have to do one client for them once and then it will work for you know a list of end languages and vice versa if you're a language author you can just do one language server and you you know support multiple editors and in reality now metals I'd say this you know theory is working great it's in pria in in in reality you know you can see here that we support an incredible number of features and it works with you know we officially support six editors and considering the how small our team is and this project is less than a year old it's incredible that we can support these number of features in so many different editors there are tiny holes here and there usually by lack of support in some editor for specific feature but otherwise it's pretty amazing and some of the latest things that we've been adding just in the past two weeks in metals which I've been told some people are excited about we've also added renamed symbol and fine implementation which is where you can say find me all of the subclasses of this class or you can also find all of the methods that override a specific method so so this has just been merged into master but it's not an astable to release yet so the ide gap is that you know we grind through we implement all of these LSP features we fill out the protocol and people are excited and then I'm like why don't you switch to 2 vs code and I want to maybe do a small survey in the room how many people here use IntelliJ and yeah it's I'm not surprised I mean it's usually 90 plus percent especially in the Scala community and for a good reason I think IntelliJ is a fantastic product it's very rich with a bunch of great editing features so even if you do the work of fully implementing LSP you're not at feature parity with an IDE such as IntelliJ or or Xcode or eclipse or Visual Studio so what what I'm talking about in this talk here is kind of that gap it's like how do we reach that feature parity in in the language server protocol worlds so I hope that excites you I'm gonna say that there's four elements to this here that that maybe not going hundred percent but go a long way so in this talk I'm gonna be presenting debug adapters tree views worksheets and build servers so these are all concepts that just have no notion whatsoever in in the language server protocols back but I think are essential for anyone trying you know aspiring to build an IDE so let's begin with debug adapters there is a debug adapter protocol that originated in Microsoft and has currently implementations for dozens of languages so it's it's I'd say has widespread industry adoption it's less known than LSP and it has support in multiple editors I found out last night doing the slides that it also works in BIM which was mind-blowing and but the best part is really envious code so if you do a bit of legwork doing a language server you can add buttons to say run and the button to say run is actually displayed via LSP but what happens when you invoke that is kind of like an Annie to unit function you can do whatever you like and with a bit of customization on our in RVs code extension and a bit of customization on the metals server side and also on the build side which I'll explain in a bit we're able to provide this experience where essentially it starts a debugger and we treat running as debugging without breakpoints by doing that we get a lot of the nice benefits in vs code where you can see that the status bar becomes orange it opens up a debug console and on the left you see an empty kind of watch variable called stack part which which will use once we have support for debug breakpoint debugging so I think this is exciting running a main function is great if someone doesn't know anything about programming they can just write a main function run it in vs code and they don't even have to start a build tool they don't have to know how to run a terminal I think that's phenomenal it's making you know coding accessible to just so many more people and not only that we also have support now for running tests so if you write a test suite the button appears there currently it's very raw as you can see there's just a test text string it'd be nice to add a you know a button above each individual test case kind of the way it's displayed in IntelliJ and that's pretty much we're on track to go there this is the the the early earliest feature that we could ship and this is merged into master right now it'll be available in the next snap shot now in the next stable release what's coming very soon is actually breakpoint debugging where you're able to say stop here and as you can see on the the left side there is a variable you can look at the eval you can go to it you can step into methods from your library dependencies and you can also step into Java and this is enabled thanks to the fact that we run on the JVM and there is a Java debugger that already exists and we're able to just plug and play and this will be coming soon it's not merged but I'm super excited about it because I personally never debug I just print line debug but having just talked to a lot of people who code they this is a blocking issue essentially a lot of people work with debuggers and I think is especially it depends on if you're if you work with large dependencies there's something weird in your dependencies you just can't sprinkle the print line in a third party library then debugging is just a much better tool for that and I think it's also if you have an application that's just really slow to start it's useful that you can you know you don't have to add a print line save wait for compilation takes the time it takes to start the method you know the program and then figure out what's going on so the metals debug adapter which exists is was contributed by virtus lab they have been collaborating very closely with the Scala Center where I was working before and they implemented this in bloop which is the build server behind metals and I'll be talking about build service in just a bit but really metals doesn't do the debugging itself it's just a proxy for the debugging which happens in the build server and I think that was a conscious design choice to say that the language server we should not be running and evaluating code it's the build tool that knows how to execute code it's the build tool that as we've seen talks on basil on that it's it's complex machinery how to launch maybe it's running on a you know on a remote dev box you never know so so metals use delegates but what metals provides is saying that we know that the main class is on this line number and as you're editing we can update you know we can understand that like the syntax tree of the program this is a test suite we you know that there's gonna be a button over a test case and that's really beyond the knowledge of the built tool the build tool does not really work operate at the level of a method it's a test case etc so so in in practice really how this is implemented is that there's the editor speaking debug adapter protocol to metals which metals will just forward onto the build server there's a very rare case where you're navigating to third-party library dependencies metals knows where they are and metals knows how to produce those source files that we sometimes populate before passing on to the build server but metals does a really small job in this kind of picture so I'm super excited about debug adapters but I'm personally for myself even more excited about well three views but also the next one after that but I'll start with three views so three views are maybe less exciting at a conference presentation but I think they're very important to giving that good use X is we added a tree view protocol in metal so it's really just not an industry standard whatsoever there is a server implementation you know for Scala and but I'm very excited to say that there is a client of limitation for Emacs that you know we did not implement so that's a goal and an accomplishment and what they really are is just file explorers as you're used to you can expand the directory you can list the children's you can expand other parts of that thing and it sounds super boring like it's a file explorer what what's the deal but it's a huge deal because metals has it's just sitting on a wealth of knowledge about your codebase about your library dependencies and we can give you a tree view here for example that allows you to navigate your third-party library dependencies and as you can see we list you know methods inside of a class this is not the file structure on disk this is the structure of your code and what's really cool with the button that we press there at the top there is that we're able to synchronize the cursor position down to that method that you have in your library dependencies which and then you can click on a class and it will move the cursor to that position and this is a feature that just is not an LSP at all you take it for granted and IntelliJ and it's very helpful if you're just trying to kind of oh what does this package contain and so we released this in in July and I've heard just a lot of positive feedback so another cool thing you can do with tree views is this this part here where you have a large multi module build you're compiling 10 different projects in parallel and it's kind of difficult in an editor where you just have text to display that information which is so rich that you have progress on three different modules one of them is at 10% the other one has been running for two minutes and here we do this with the tree view where we can tick on every second and say these are the things that we're currently compiling and that's the progress we've made on these projects I think this is very cool and essentially metals is not a bill tool it's it's a language server so the actual information that we're displaying here is streamed from our build server which is bloop in this case which I'll explain in just a bit so that's very cool and I think this is probably the least impressive but I think if anything it's not the least impactful feature that we can do with previews is that how do you get help if a person you know a user who just started programming is trying to figure out like where do I ask a question there's nothing in LSP that allows the server to tell the user like hey you can report a bug here you can actually open up a chat room there you know read our Docs you can go to that website and follow us on Twitter and and this was such a small stupid thing like links to two websites but if you do a basic language server and you just follow the protocol there's no way for you to display that information to the user and I think yeah so it's probably it's not technically impressive at all but I'd say it's it's a major innovation in the LSP spec like on the LSP side like I think it's desperately missing in the protocol itself it's kind of sad that we had to invent this from scratch and I'd love to see something like this get picked up kind of more broadly and so this is why I'm super excited to see that there's someone who implemented support for the tree views in Emacs as well so if you're an MX user see some that was there's always one person in the audience like no there's more three four people okay cool so there you can see a very similar experience where you can navigate tree views expand and collapse and like the protocol itself is super simple it's actually like most editors have some wait to explore you know a file tree so if you turn that into an async operation you just want to list out the members of the class you can delegate that to a separate process metals for example so I thought that this was super cool and down the road you know this is I'm just mind blown when I use IntelliJ and I run the tests and I'm able to you know it shows me the tree view of like this was the test suite this test failed this test ran for so long if I press on that test case it shows me the standard output for just an individual test case this is super helpful if you're just trying to you know debug why something went wrong it's way more helpful than scrolling through you know amounts of console output and not everyone have the kind of you know dexterity to go into their terminal and figure out the 50 flags you need to run an individual test case which changes from every test suite now testing frameworks every other testing framework so I think this is just an amazing you you know user experience that I'd love to see in the LSP world and hopefully we can implement it with has views tree views in metals so really I'm just a call to action if you know anyone who's doing language service or editor plugins we have on our website a spec you know very well specified data structures what the product what the contract is between the server and the client and it's very generic it has nothing to do with Scala so I I think I encourage everyone or working in the space to kind of do these extensions because but also share them and document them and because I think standards is really how we get you know move the industry forward so this is the one I'm most excited about to share today which is worksheets so worksheets in metals got merged a week ago and they are files with dot worksheet SC file extension and they evaluate on file save the whole idea is you just write ass nipple of some bit of code and whenever you save it gets evaluated and the evaluation will display right next to the code as comments but you can't edit these comments they're uneditable if you try to select the code it won't select the code it won't select the comments and I think this is such a great replacement for the repple if you're doing you know just trying to learn some new API and what's amazing I think is that the lower end like if you have very small worksheets it's able to render them in roughly 200 milliseconds and it kind of when you start working with it and if you have ten expressions and you're trying to see the the relationship between different variables you can just save and I haven't had that experience with Scala ever before where you can just save and you get that feedback right away and you're it's it's almost like your thinking process is at the same speed as how you can kind of see it evaluated in the program and so if you're working with very large data structures uuuugh parse a large data structure and then you have it in memory the worksheets are using a library from how he was presenting earlier today called P prints which lazily kind of renders out very nicely line wrapped data structures so if you have something like here you can see it's gonna nicely align them so that it's it's still understandable like it's not just gonna display everything in a big long line which i think is gorgeous and maybe as you didn't notice from here as well the first line stream is an infinite list but it's just lazily printed out just to the point where we can display it in the editor so you can even work with infinite data structures and have them in the gravel and it's not gonna just run forever hanging I think that's cool so this part is super exciting to me as well is that you can have a worksheet that depends on code in your own library so you can just have in your own library like a file utility methods you can I would have complete them how to import them in your worksheets and what's amazing is you can go back into your main code file and change it so sorry so here we go into the example dot scala and we bump up the we change the method implementation and when we switch the focus back to the worksheet it has already picked up that compilation and it updates the worksheets to the latest state of your projects which is I think a phenomenal like if you've been in the if you work with the repple before you often have to like escape the repple and then do console again and then re-evaluate all the expressions you've had and here it's really just switch to tab in your editor and you don't have to even press save just the act of switching the focus will reevaluate the worksheet so it's it's super effortless and I'm really excited about this and I think if you're a library author it's just having that being able to try out your API right away as you're iterating on it you see how bad the API is you start tweaking it you say oh I want the worksheets look like that I'm yeah I think it's it's gonna be I'm real excited about it another cool thing with the worksheets is that I haven't seen this in another tool but if you if you have a runtime error it's actually gonna report diagnostic red squiggles and if you hover over that error the error message is actually gonna be the full stack trace so what's maybe not obvious is that this is a runtime exception that's appearing as a position error message in your editor which is super cool so it's like that runtime and compile time or just folded into a one experience I when I got used to it it's just super cool and yeah I was surprised when I even I implemented but I was still surprised when I ran it I was like I can't believe this so this is a bit crazier but if you like if you if you're trying to understand super hairy code maybe there are some threads and low-level API is and you're like I really don't understand you know you can actually still print line debug and and the standard output is going to be captured and it'll be attached to the actual expression where that thing got printed so here we we do a new thread we start the thread it prints out start it sleeps four five hundred when you do join it captures to stop output and I haven't seen something that visualizes that experience in the same way in one view so so this is pretty cool as well so this is called in metals there's on the website a decoration protocol but it's really nothing but just a single JSON RPC notification where metals just pushes to the editor saying these are the declarations you should display in the in the editor which is done during that LSP communication there's only one server implementation and there's only one client that supports it but it's really something that I would love to see in LSB because just everything that I've shown you right now is something that is just not possible within LSP you really have to do your own custom thing and you know imagine if that is available to all of these languages that are out there that would be just incredible so the last part is is but absolutely not the least build servers so so metals is I think a bit unusual compared to a lot of other language servers is that it has a very close integration with the build server which is loop and so you remember the compilation view where you would see that dynamic updates of what's being compiled in the background so so that is not happening in metals it's happening in a build server which is ploop in this case and we have a website built server protocol there's a organization on get up build server protocol that we collaborated together with jeff murray and so on so you can also use BSP to import projects in IntelliJ and it's just been one of these projects that I'm super proud of of having worked on because I think it really is solving a core part that exists in the whole ecosystem where what yeah it's pretty much the same problem as the language server protocol where you have a lot of build tools and a lot of language servers and currently you need to do like a custom integration with each of the build servers built tools for every language server and you end up with like a large matrix so instead with BSP you can just have a contract you do one implementation in IntelliJ one implementation in vs code and then you get support for all of the build tools so this is something that I would really love to see gain sort of wider industry adoption as well and really the the on the server side I'm the picture points to SBT Gradle male and maven but none of those tools actually implement BSP we're kind of lying that it sits bloop which is a build server which this talk is not about but it's you should really check it out on has ways to understand all of those build tools and and then we have one build server implementation for all of those so there was a talk earlier today please check it out and check out loop so in conclusion LSP is amazing don't get me wrong it's just enabled us to ship so many features in such a short time but it's really not enough if you're going to implement the full-blown ID so I encourage people to do small LSP extensions they really make you know make a big difference in closing that gap and please document them spec them advertise them go to conferences speak about it so debug adapter protocol for running and testing code I think is really the way to go tree view protocol I think it's working great for us people have commented inside it's nice it has support works in Emacs an obvious code that's a good endorsement a decoration protocol and I hope that the demos that I showed you are appealing enough to - for that to get adopted more widely and finally and yeah just give medals a try so thank you [Applause] I don't know if we have time for any questions oh yeah I think we have a couple minutes for a few questions anyone see a hands bendin here we go hello that was a fantastic talk I wanted to say one thing which is that the interactive kind of worksheets that you showed was extremely exciting I would say that to me that's one of the really biggest benefits of using Emacs and Emacs list is being able to not only they have your expressions in a poll but have them constantly update and then even be able to have a file with all of your results so that's extremely exciting to see especially when language like Scala which is much less I go to quote dynamic so thanks a lot for showing that I'm available in the hallway tract as well great great well let's give her a round of applause and thank our speaker thank you all you