Scale By The Bay 2018: Justin Kaeser, Fork It Harder Make It Better
Recording: Scale By The Bay 2018: Justin Kaeser, Fork It Harder Make It Better
let's see yeah actually I'm going to talk about eclipses but first yeah I I'm Justin as he said I work at that brands on the intelligence got a team and you can follow me on Twitter as an event you can like post questions if you have them during the talk or ask them later later during the unconference I'm just gonna be there for you to rant at me about all your problems with IntelliJ but for now this is for kit harder and make it better so you saw this this background picture who recognizes this yeah that's right it's it's called the Scala tuning echo Society so the purpose of this talk is to give you a bit of an overview of various tools in this gala ecosystem and outside of the Scala ecosystem and maybe I'll have a chance of inspiring somebody to work on any of these things or different things to improve the tooling ecosystem and bring it together better so because I think there's various tools that have the potential to improve your development experience all our development experiences and they might be underused because of this is how they are integrated or not that's integrated as they could be into the overall workflow so I'm sort of structuring this talk is a Hegelian dialectic with a thesis antithesis and synthesis as you see it's illustrated by Hieronymus Bosch and the lyrics are by Daft Punk so the the thesis because like this work it make it do it makes us harder better faster stronger what am I saying with this so I'm using the term forking quite liberally here as making anything new that is sort of related to the old but a bit different so that can be a literal source code fork or just a completely new tool and so this this for King might make the whole process of developing harder to learn but it might enable us better and faster ways of doing things and in the end make the ecosystem stronger so I'm going to assume some familiarity with common tools but overall that's not a requirement so let's just zoom in a moment on a few things we can find we can use tools for like learning experimenting editing code exploring understanding debugging building dependency management publishing versioning so since my talk got expanded to 40 minutes suddenly I can actually go into most of these topics briefly and the first one will be learning and experimenting it's I think a very important and undervalued part of the whole coding process because you don't usually start with a complete code complete way your application works in your head you have to experiment with the API is you code against the actual problem you solve this maybe not completely finished so there's some tools that can help with that and I'm slight dick that's unfortunate let me just switch I hope this one is the right one oops so the classical tool for this is the M&I triple and that the repple as such Scala comes with one and it's the one that is delivered with Scala is a bit limited so ammonite is an improvement on that which offer to various features such as history search syntax highlighting block input undo redo and many more artifact imports and others I don't even know about improvement on this is or not an improvement a different way of approaching this is the worksheet where you have a persistent state instead of going line by line and editing stuff you have editor or your intercept here's sky ste which is on the Scala website and and it allows you to import SBT artifacts which I'm doing here I'm just trying out the Giga horse HTTP library I'm gonna copy the example from the website edit it a bit to download the page this is a web-based thing that I'm actually working on right at this moment just look at me and mine coding and yeah you see it downloaded some stuff sorry noobs and this is practical for a quick reevaluation of self-contained pieces of code and and sharing working code examples and my favorite approach to the experimentation is actually notebooks which combine the iterative updating of your program state of ripples with a persistent editor like worksheets so here's a Jupiter I'm sorry it's a bit small but you get the idea you have cells that stick around you edit the cells you put in some code here and just loading creating a histogram of pictures of characters in a string and one feature many notebooks have is some graphical output and here like a printed histogram for which I just use a library I play around with it and it's all dandy there the notebook was actually pioneered by Mathematica some twenty thirty I don't know how many years ago and I think Mathematica is still one of the best in class in that regard so Mathematica doesn't just use it for adding in code but also for documentation here I'm looking at the nest function and what I can do is I there's some neat graphical examples I can just instantly change them and see how what's happening here a change X for you and the nest function and I got inspired to download this picture from the internet I'm gonna scale it down a bit and when you pixels yeah that's neat and put it here instead of the you and you've got a Penske triangle you're welcome say it invites playing around I love it I think it's the gold standard so I think this would be really great if we had it for a Scala I think we support something close to this in Kotlin nowadays so there's actually a web page where you can see code examples change them and run them so next thing editing and understanding so editing code is not just typing otherwise we would still be using editors like II D or VI but often you need to understand the code in front of you as well so another example form Mathematica so I think syntax isn't something that should be fixed it's the syntax is your interface so why don't can't you play around with syntax more so Mathematica is helpful in that it can pretty print and now I'm doing here I'm just pressing one button and it transforms the format of this mathematical formula and I can also do this as a function so here's the input form which is what you use normally and here's a full form which is close to the list s expressions and why not print this is a tree a syntax tree right in your notebook editor now down from the ivory tower back to the bare metals one thing I came across is the interactive disassembler so if you know assembly code it's just one long big list of commands and hard to see the structure in so why not print it as a graph word or you can follow the code structure more clearly that might not be entirely applicable to scala which is more structured language already but another approach that when around a while is this code bubbles IDE which is yeah promise Eclipse this is based on Eclipse unfortunately not available yet in IntelliJ but where you see here you can actually navigate from one part of the code to the next and and have the relevant parts of code next next to each other on your screen instead of jumping between windows and maybe arranging when split windows and moving things between them you just have them here neatly arranged next to each other I think this would be really great if we could develop it further the state is this IDE can't even import maven projects so there's a lack of integration again but maybe files is primary way of organizing code could be obsolete or at least go into the background so compiling so you have to get from from from your code to working software you have to pass the compiler at least in Scala so in this sense a compiler is yet another tool and here we have an actual fork type level Scala is a fork of the main Scala life and Scala compiler which tends to maintain binary compatibility but make new features available more quickly so if you have a cool idea of what would be great to have in Scala then you might try to get it into type level Scala first and make it available to those that use it before getting merged into mainline another approach eugen for marco talked about it yesterday day before I forget the reasonable Scala compiler so who watched that talk yeah many of you so just a kind of an experimental reimplementation but as he told us it's making progress with that and maybe making it possible to actually compile compile Scala more quickly now bloop is not a compiler in itself it's a compile server developed by scholar Center which runs on your system and keeps around a hub compiler to actually take advantage of the JIT compilation that the JVM offers and that can significantly improve actually compile speeds and it integrates with build tools such as SBT and others so that was sort of on the on the edge to building which is my next point so you don't to get working software you don't just compile it you put it together to create releasable artifacts and that involves many steps such as dependency management so let's have a look at build tools not everybody or developers in general don't seem very happy with their build tools is evidenced by the fact that there's one for every letter of the alphabet that ends an ache and these are just on one sending an ache so in in Scala I don't think we use many aches but we have aches like head aches yeah these guys are feeling it so so in in the right corner we have the old-school JV and Java build tools such as and maven and Gradle does anyone actually use ant for Scala and I didn't think so well actually I we recently as recently as last year we still had and in our code base in the left corner there the more Scala focused one such as SBT of course CBT and more recently Mill and here in the middle corner I placed the new-school cool tools such as basil pants and buck by face book get it Facebook which are more focused on mano repos so big code bases and Yury just snuck in there John did you do this job where it's done he's hiding so it's a basal developed by Google originally blaze is yeah as I said focused on building tools and a big unified mono repo codebase and it aims to still be able to provide fast and correct builds and houses done by providing so for each task each little part of the build who fully specify the inputs and that means this part is completely reproducible as long as your code is otherwise deterministic your build code and thus outputs may be cached the SBT no speaky contenders no and CVT so SBT used to a long time ago have pure Scala library configuration then they sort of have moved away from that and people were unhappy so they started making new tools which are configured in plain Scala they differ on some in some more or less minor ways mill tries to take a bit inspiration from basil by having targets which are by default cashed in CBT and caching is left to the user of the tool we had 40 minutes you just said I have two minutes left okay label use is nail gun as a for running in a background process so you have more responsive responsibility than with SPT typically okay so yeah we've got our build tool now we want to actually create a library and this is notoriously complicated and jvn land so looked around I looked at cargo the rest build tool and I'm going to see how hard that is to create a library I'm just generating a token just pasting that into my ripple and creating a new project and let's open this up see what cargo did for us okay just a simple configuration file a simple code piece of code I'm gonna check this into git commit and see if I can publish this and no that didn't quite work out it wants me to fill some fields one wants me to be explicit about it so I have to describe what I do here I fork it harder make it better give it a license I think the WTF PL is appropriate here and let's try publishing again yeah I take slow publish since working out uploading looks good looks good yeah okay I'm gonna check out the crates IO and there it is right on the first page yeah so that took like maybe 2 hours of 2 minutes of real time and in that time and I couldn't even create a ticket on sono type baby I also looked at Elm a language I like a lot the publishing process is similar slightly more involved but not much it involves pushing to github but what I liked here is the versioning so I have this package for kit harder and in version - OH - it just contains this one function harder in student I add another function called better and own package offers a tool package diff and it tells you if something is a minor or change or a patch change so here it's the compiler is basically already requiring that I document everything that I want to publish as a public update so I do a diff and it shows me this is a minor change because I added a function and [Music] you see I entered the bump command it automatically bumped my version up to two one and so I commit this and try a publish and oh I need to get tagged ok fine but it gives me everything it hand feeds me everything I need to do manually try publish and that seemed to have worked out and I have two-one-zero in the package repository now I decided okay this better function doesn't really make it better I remove it again update the docs appropriately and doing a diff it tells me it's a major change because the code is incompatible not backwards compatible so do the same thing again and publish yeah yeah looks good version 3 you get tagged okay yes Version three zeroes online nice so yeah I mentioned dependency management also known as scavenging code from the Internet in Java if you use SBT just downloading dependencies takes forever so carci is a nice tool I hope we're gonna move SBT to car CA soon other tools such as milk already use it in that by default and it just downloads faster and it's very reasonable thing to have maybe less dependency manager but package manager NYX is not Scala thing as such but there is a whole Linux distribution based on it it's kind of similar to basil and that the ideas you fully specify the input and the environment in which a package kit gets built which allows you to deterministically create exactly the software that you will install and also cache these artifacts so there's actually a SBT plugin which will allow you to create annex building project anyway back to Java so I mentioned publishing a sauna type is hard maybe you just want to get some code out quickly for someone to consume you can publish it on github and the user just adds the jetpack resolver and dip pack will build it for you I'm not quite clear on the limitations of this like maybe it might feel for complicated SBT builds but in general it will just work actually so that was a collection of random tools that I liked so now to the the antithesis more than ever hour after hour work is never over so I'm saying like creating ever more tools might yeah make everything harder because we have to learn more tools learn new stuff integrate these tools so why why shouldn't we work on that with the tools we have and just improve them incrementally bit by bit hour after hour and put our effort in there I mean sure that's that's happening so the Scala compiler is being continuously improved currently maintained by light Bend Scala 3 is of course around the corner in a year or two and yeah you've probably seen what's coming up like the collection simplification some compiler performance increases and so on SBT 2 has been around since quite a while 2009 I think it's also maintained by lifeand and myself I've been working on integrating SBT into the IntelliJ scala plug in making the integration smoother bit by bit and yeah you've so maybe or maybe not seen that what kind of improvements are coming so for me the server extensibility is important because it will allow me to integrate it better into IntelliJ because right now the way it's done is quite happy and of course I need to mention this little product and working on myself the IntelliJ Scala plugin has been around for quite a while we're not in December we'll be 10 people on the team so we're continually invested in improving it some of the things we've been working on are of course always and improving the error highlighting and making all these edge cases where red code that should that actually compiles is doesn't get marked as bad we've also had some work done a few other things to make it easier to get started like the one-click bundle so my colleague Pavo once went through this and found you need about 100 clicks if you don't have from a fresh system to have a working IntelliJ IDEA with Scala plug-in running so what he did he made this bundle and you see you just open it and you have a working scala project that you can run and test and play with recent feature from last release is the you know I implicit radar or implicit x-ray I don't know if you've seen it actually so so far we've only had tool tips but now you just press a button and you get the implicit parameters that you're passing or whether you're passing are implicit explicitly have you seen this feature yet oh well yeah there's you just press ctrl alt shift + and and you will get these nice yes sorry we ran out of free keys you know but yes it's there look look it up it might make your life a little easier if you're wondering why is this compiling or not compiling why what's this missing implicit which implicit is actually used okay finally synthesis fork it harder make it better do it faster makes us stronger more than ever hour after hour work is never over and my point being yeah we have lots of opportunities to make new tools and we have all these old tools that we actually want to keep using a lot of the time and creating new maintaining old minutes it's not necessarily a dichotomy we can integrate the old into the new so agreeing on styles and tools as a programming community makes us more productive cohesive and so from the point of view of tooling developers we've actually been coming closer together especially over the past year to work together on new things so I've worked a lot with the SPT team they've also cooperated with the Scala Center and as well as me working with the Scala Center on new ways of integrating tools John pretty has also initiated the LSP STP so Scala tooling protocol working groups so that we actually work together more you know Oh one way of bringing things together have you heard of the language server protocol that's yeah many not not all of you that so actually it's a protocol developed for Visual Studio code editor by Microsoft which allows you to get many IDE features into into your editor now maybe I shouldn't be mentioning this because it's our competitor in a way but the methyls language server is one is a tool to a language server based on scala meta which tries to give you some of these features via LSP and it's developed by scholar Center now likewise in Scala 3 I think Martin mentioned in his keynote on Thursday it comes out of the box with a LSP language server to also give basic idea features now for for the scala plugin this is not directly relevant because we of course have our own way of doing everything but integrating build tools has been historically a challenge like who's had problem with SBT and IntelliJ just some of you maybe I'm not doing that bad of a job but I've had lots of problems because there's always some edge cases to fix and so if I wanted to in support any more build tools then this would get this word just compound and I could support maybe one more build tool with my time so the alternative that we've been working on together with the scholar Center again is the deliver protocol which extends language server protocol and gives a standard way for Bill tools to communicate the project structure to IDE or language servers like methyls this is a protocol that supported by loop as a server currently and there is already implementation and the IntelliJ Scala plug-in Knightley's how much time do I have ten minutes yeah then I'll show you a quick demonstration of how this looks so I this doesn't look very different from SBT i opened the akka project i select PSP click finish oh I had it on the wrong window that's unfortunate oh yeah what you see here I imported the akka project it's not via SBT that's why liya BSP but you see the whole project structure right here I can click compile I get up again it would be a C build running and a nice output of all the errors which is nicer than what we have when SBT right now and this is important enabled by the protocol offering standard messages for compile errors and warnings and so on so back to this and well lastly I gotta mention various integrations that we already have or are working on so yeah we have already SBT the ho con configuration language recently added support for Emma night scripting and Scala format in the ultimate version you'll have place caller Jas akka and we're currently working on supporting spark and the build server protocol is already in the Knightley's as well as in the upcoming release you know but still as a hidden feature because it's kind of under development now I said I wanted to like inspire you to do something yourself like okay maybe you aren't paid full time to work on tooling like I am but you try to consider how much time you spend working around tooling and if it might not be worth it for you or your organization to invest some time in this and personally I'm always happy to hear what we're your challenges with tooling are so that we may support your use cases better I can't promise I can do anything right away but getting the feedback is very important and as I said like right at 3 p.m. after the fire alarm there I will be at the unconference just to hang out and listen to your troubles and for as long as we got now this is the question-and-answer pool just bear yourself jump in give me comments if you don't have any that's fine or it can come to me personally if you think it's an embarrassing question thank you [Applause] thank you