SF Scala: Anatolii Kmetiuk, James Thompson and Guillaume Martres

SF Scala: Anatolii Kmetiuk, James Thompson and Guillaume Martres

Recording: SF Scala: Anatolii Kmetiuk, James Thompson and Guillaume Martres

foreign my name is Jamie I'm from the Scala Center like I said and I want to talk to you about how you can compose your Scala projects um and I'm going to sort of uh explain a bit about um Scala CLI and some changes I've made to it by hacking on the weekends Etc so that I could build a full stack app with scholar.js and Scholar at the same time on jvm so yeah we're going to explain who we are at the scholar Center what we do why do we why did we introduce scholar CLI as the new scholar command um what are some sort of features that are lacking in this color command and why would you want to do what I did and spend you know four weeks without any vacation or weekends okay so scholar Center um uh yeah so we're a not-for-profit organization in lausanne Switzerland uh it's been uh founded in 2016 uh and well I will delegate to my colleague Anatoly now and so on um and then we have as a part of our resources dedicated to the work on the Scala itself The Scholar compilers finally we also do governance so governance is about balancing interests of different stakeholders of the language so different parties such as research industry communities the old one different things from the language and all of those interests needs to be taken into account and balanced and we are also concerned with Education and Training we have three Coursera courses that teach the new scholar programmers to to become proficient in the language as well as we are concerned with maintenance of the docscalaland.org which is the documentation of the Scala language um so yeah here are some examples of those educational materials so functional programmingists in Scala is uh Coursera course yeah that's the only one okay yeah but you can um you can also uh book with us one-on-one time if you um uh want to learn how to do programming in Scala and it's called the courses called effective programming in Scala and it's on the what is it extension the epfl extension school and yes you can pay to have one-on-one time with us to teach you yes yes so here is a little diagram that explains in uh in one picture what we do so here it is all about transmittance momentum and balancing interests between different stakeholder groups so you see several Cooks you see epfl which is a university where we originated and it represents research industry Partners which are also companies who make money doing skull then we have the community uh Community people those are people who do the open source libraries and they want all of those stakeholder groups want different sins and this was the main driver for the scholar Center to come into existence in 2016 to be a parties that balances all those interests uh here is our team we are as the team composition is very uh at different times but basically we are around 10 people at uh any given time around six engineers and the rest is Administrative personal we are all based in Switzerland in Los Angeles at epfl where Scala was originated so this was the introduction of the scholar Center and now I'll give the word back to Jamie to Showcase one of the projects that he was working on so thank you everyone [Applause] if you scan this QR code you should be directed hopefully is it frozen again no no okay so if you scan that QR code you should be directed to the lovely five-year impact report that we did so you can scroll through here and see every little thing that we kind of worked on over the last five years and I'm pretty happy and proud of this website because I worked a lot on it anyway um yeah scan the QR car code if you care or I can show you later the URL foreign [Music] CLI is it's like 50 percent um so if you how many people know that Scala CLI is Now the default Gala command okay less like a quarter okay or an eighth even right so I'm gonna now do a brief overview of Scala clo because it's the main focus of what I've been working on um it's a tool developed by vertislab to basically be a One-Stop shop for everything you need to compile sort of a small sized single project with Scala um so the Scala improvement process there was the the number 46 proposal basically approves that Scala CLI should be the new uh default way that someone gets started with Scala uh what does it let you do basically you can this is a very loud fridge or something um yeah you can run code you can test code you can run the rebel you can um load scripts with it uh it comes with support by default for uh the ID for an IDE with Visual Studio code and the Metals extension and the kind of key part of Scala CLI that sets it different from SBT or or Mill or any of the other sort of uh bigger build tools is that all of its configuration is just data so you have this thing called a using directive and I'll give a demonstration later it's basically just keys and values and it's in your code so you don't have to think about uh arbitrary things such as Turing complete configuration uh yeah the main goal is to be instantly productive at writing typical scalar code so uh yeah you can use a library just by declaring it in the code and you can mix and match Scala native or Scala JS with basically just one line of configuration uh so I mean I guess a lot of you haven't really seen everything you can do with it so I'll give a small demo um I have to quit that one and restore it okay so the main idea of what I have here when I make this bigger and probably make that bigger uh hmm okay yeah so the base the basic thing with this is just obviously uh I I opened uh something called a script files because I just have the um that was chat GPT doing that okay so yeah you can just run hello world really easily inside your text editor but also by declaring this configuration at the top with um using toolkit latest then now suddenly I'm able to for example um use stttp so I think I could do a quick request uh yeah sure thanks what does that do yeah so I just did a web request in like 10 seconds pretty cool um if you haven't seen this you can also basically just declare any Library dependency at the top like using lib and then uh well maybe that's a real Library I don't know okay it is oh wow now I get to demonstrate another cool feature Scarlet suite life which is you can do actional dot actionable Diagnostics so this is out of date so I just click this and you didn't see what happened but it it changed the version and then when I save and refresh then it's cool um this isn't the only thing that you can do with this though because basically you couldn't have you don't it doesn't just do Scripts um get many more files but uh so yeah like I I recommend that you do this as basically the basic way to start with Scala and in fact it will be pretty much very soon because when we go to I can't see this So currently with Scala the way you get started is uh you're you go on the home page and click the get started or you click the install button and then you just copy and paste this command in your terminal and then when I do this then I'm going to get the new scholar command like uh when that's finished then all I do is I could type like Scala version and you see them using yeah yeah yeah sorry absolutely a big font okay so it and that one bigger no uh Homebrew I don't know what's going on I shouldn't have done Homebrew never mind okay so ignore that uh basically yeah with scatter command you can uh let's go to one of my I'm really vibing right now what do I even have in here I have I already want to go to scripts okay maybe I'll save the demo for later but the yeah the idea is that you could uh run tests and I wanted to open a project that had tests so I could just show you that it worked uh whatever okay um yeah if you want to learn about Moro you use the scholar CLI then we just added to the Scala website The Scholar toolkit section so skull toolkit is basically the uh a collection of libraries that let you get started and do really easy things with Scala like running your tests uh interacting with the operating system uh picking a default Json library that has never been proposed before um and and connecting to http and running some requests so basically the demos didn't really fully explain uh but here we go then uh yeah if I copied and pasted this code into uh into the text editor then I can immediately just run that um but okay so the the thing I really want to talk about now that I've kind of badly shown you Scala CLI is that I want to show you uh this project I'm calling Scala compose but it's not really a real project so please don't think this is like an official announcement or anything um it's more like I what I hacked on the weekends with uh with lucasranski from uh from vertis Lab so uh yeah the problem with scholar CLI at the moment is it doesn't really have any support for Source generators or resource generators uh so yeah you can't really load your protobuf schema and then just generate a bunch of boilerplate that people want that uh I haven't supported that but I still want to do that in the future or someone can do it there's been proposals uh and we can also have a uh it only supports a single project so you can't say uh have two libraries that depend on each other that you are currently developing so what I want to do with this color compose thing is basically just say take two of these scholar CLI projects and smash them together so that you get the class path output of one into the input of the next one um and I'm still keeping the idea of config as data and so that basically allows you to declare just the way that you join the modules together and so now I have an actual demo that I have prepared okay so um has anyone heard of Cask by Howie okay one person two people three people uh okay so the idea with what I've done here is that yeah maybe I'll just show it running okay so what I have here is um a front end that's written in Scala Js and the back end that's running on jvm with with Cask um cake why not yeah uh but the the point is that this was not built with um spt or anything like that so you have to wonder how did I get the Scala JS to um uh front end like the web page to be bundled with my uh cast application because uh the way that works in the web server is that we have basically if you if you go to the index.html that has to load has to load this index.html and this references something called like the main.js from assets and that doesn't really exist because it was compiled with scholar.js so we have to bundle that somewhere in here which is just basically defined here but the main um magic that makes it work is I have this uh configuration file which I chose tomal as the syntax to use but yeah I can say that I have a web server here and I say it's an application with this main class but it also uh I don't know it needs to depend on repo which is the repository that saves my notes to the to the the file system so that is persistent and then this classic extension has just allowed me to have uh uh what's it called websockets I needed web sockets so for example just because I wanted to do something cool if you have two of them open at the same time then you can do yeah an instant update but that's done with the websockets anyway um so the kit the key point I have here is called like a resource generator and this is going to say yeah bundle the package the web page and put it in this main.js stuff and when it's all built it's actually in here somewhere and we're seeing the managed resources this is like the main.js but it's the packaging output of the web page bin and this is what scala.js looks like if you've never seen it when it compiles to the actual JS um but yeah I guess the the point is like how did I make this work so probably needs to explain a bit uh each one of these um so each one of these modules that I declare in the toml file is a single scalar CLI project so we have the web server here as so if I actually went into um it's probably probably model is better because it's at the very top of the food chain here so model is just defining my notes the data structure that backs one of these nodes and some uh communication with it web sockets but yeah each one of these itself is just a scholar project so if I went into the um if I just went into the model and then this Scala compile or whatever oh that's why I needed to do that yep I just overloaded that with my CS setup [Music] this is very bad on in store CS installed okay now this is great so um yeah so now I can actually show you like so if I did Scarlet test thing then it can run some tests and that test was uh written with munit but the point is that I just um ran scholar CLI on just this directory but when I use uh this Scala compose tool on this directory then it actually like brings all of these different Scholars see live projects together and I can also use it from the command line I think so if I did go back out and I did Scala compose file then they should do nothing yep that's right it did nothing because I already built it with the IDE so there's nothing to happen but if I did get clean and try and build it again uh then you can see it's compiling every single one of these uh modules in the kind of topological order and it does this thing where like here when it um in order to build the web server then we have to build the web page and then we package the web page into the main.js and then we copy it into the Resources directory um so so that's how it kind of Works in principle but to to explain what it actually is doing you need to kind of look at um skelei's internals um cool uh so has anyone heard of the tool called bloop uh okay so people at the back so bloop is basically like a persistent process that runs on the back of your computer and it uh just Waits and lessons for um commands to come through that are basically saying like compile this code run this code compile this code whatever the the reason is happening in uh as a sort of single process is that it reuses the exact same class path to to um basically jet compile keep a jit compiled uh scalar compiler in in memory um and then the job of Scala CLI is to take your source files and and any of the um any of the using directives and to remind you this is what using directives look like uh the point of scarlet CLI is to take all your source files take all those directives and turn them into something called a bloop fight Blue Project file and the way that looks is for example if you go into web server Json yeah so a Blue Project is basically uh just a big giant Json file that um says what's the name of the project where does it exist what source files it has what other Upstream dependencies you might have which are other bloop projects and then where to find all those libraries that you um require so really bloop does nothing except read data and compile it doesn't know anything about resolving libraries Etc so scholar CLI has to do the job of resolving the libraries writing this bloop file um then if I go back to my presentation you know [Music] um yeah so it's a basically uh scholar CLI runs in the pipeline first thing it does it scans all the source files scans all the using directives and then it creates something called an input which basically um yeah Aggregates all the settings then it has to transform That Into the Blue Project and then it writes the Blue Project to disk and then it runs that blue project and that's called a build what I have done in my Edition is basically um when you read the toml file you create a bunch of things which are called like a module I'm in the wrong um okay uh I have to scan it out manually um so we have the module and then we have Scala compose and then we have the old uh Builder that I have nope not that one foreign yeah okay you basically uh you pass the the config file into a bunch of modules which basically just says where is it on the file system what platforms it supports uh if it's a library or an application and then the other module so it depends on then in for example the compile command or maybe the Run command uh um once we've read that config then we get the modules um then what is that thing I'll do the compile command I think oh yeah that's why I moved into this thing um yeah so then you you get the modules and basically you uh just read the the root field of that module and then say okay tell Scarlet CLI to just make a project with that directory and then at some point you have to basically uh figure out the top topological sorting of all those different modules inside of build I think this is a bit yeah yeah here it is so in so we take all of those modules and then in parallel we're going to fetch um actually read the files and read those using directives and then we make a little graph of basically a pair of the name and the platform and that basically enumerates the the real projects that exist so each of those products is going to have a jvm version and a JS version for example for the for the model that has two versions because I wanted to have uh yeah the model has a jvm version and a JS version then what happens is we get to the point where uh we have to basically figure out um what is the real dependencies between both of these uh between all these projects because going back to this one uh basically we have two ways to get dependencies on a module we have uh depends on which is like read the class path of of web server repo and read the class about the class extensions and add it to my class path and then we also have a dependency on this resource generators thing which says I have a dependency on web page but I'm not using his class path I'm actually using its uh package output um and and the reason why I record both of these in the dependencies of of this module as a something called pre-build which is just saying it's the data structure before I build it um yeah the reason we need to record the the resource dependency is because we need to say that at the point that we are going to actually build the web server we need to have already built the the the web page then once we go to those pre-builds we're going to do a topological sorting and I don't know if anyone knows topological sorting did you have to do it for algorithm lecture class okay cool uh maybe we could just do it on here um yeah it's basically just like if you have a huge sea of nodes and a bunch of connections how do you uh know what should come before the other um why that's important is basically because when you build a module B that depends on module a then the file system better already have those class files written before you can use it so it's just saying uh how do we know what actually comes before and it's a bit more complicated than if you have a tree because maybe you have arbitrarily uh uh you know nodes going in multiple different places and even got looping back on themselves so it's not as simple as just sorting by alphabetical order or whatever um yeah then once you do that you just run the build exactly as if it was a standard Scarlet CLI project it's just there's more than one of them um but in order to prepare the build we still have to do a little bit of preparation so um what am I doing here so yeah yeah so for example um we're going to say uh how do we actually fetch the class path of the previous thing then the the reason um I do the top logical sorting is because we actually need to um fetch the class path of a specific platform and project name which is what we're doing here so we have to say um yeah when we're reading dependency we need to know what was the classes what's the class directory that will eventually be loaded at runtime yeah that's kind of how that works anyway um I don't know how long I've been running for yeah let's just say that uh I I'm kind of looking for uh what else we can explore in this space because there's also this other project that exists called bleep and bleep is like what I showed except to uses yaml files um so it's the fight of yeah yamo versus tomal who who's going to win but in seriousness uh I think if we want to get this project seriously taken then a couple of things we need to have probably a way to um rip up these uh these using directives and probably aggregate them all inside this module file and the other thing is really I had this resource generation thing but it's very basic it's just package this uh what would be nicer would be to have the resource generators as well and I think that would make it something like everything a typical person needs um if they're not doing anything complicated so yeah that was uh what I did and that took me like basically all of uh made to do like the first half of me I wanted to talk about it at scholar days but I didn't really find a way to weave into my talk so uh because I want what I wanted to show with this was how we could make maybe use it as a testing Bay for different compilation algorithms because theoretically it was less complicated than SBT or something like that anyway so thank you for listening in oh but I have one more thing to say which is um yeah where is it Bam Bam oh yeah no okay so uh scholar Center we are uh supported mostly through donations but we also get some money from when people buy our Coursera course but uh yeah we're basically looking for uh anyone who you can either do an individual donation but also we would like to partner with uh companies and if you are partnered with us then you can join The Advisory Board and at The Advisory Board basically you get quarterly reports from us and what we're doing uh in in and you get to actually be inside a meeting with us but also you get to formally make proposals that we have to uh listen to and maybe act on most the time I think 90 99 of the time we actually did uh act on these uh official recommendations so uh it's a really good benefit to you it's amazing to us because then we can hire more engineers and improve all those pillars that uh totally talked about earlier so if you really care about productivity of uh and improving the Ides uh getting more making it easier to train developers so it's even easier to hire more scale Engineers than uh really please help us thank you [Applause] questions yes um so to summarize I think the question is what are the biggest design design decisions I had to make uh in the The Scholar composed idea um it's a good question I think uh basically uh for me I think the hardest point was uh how do I sort of add this module bit on top of what's already there because you can either just like duct tape it together by if I um by just doing the minimal possible thing but then architecturally it looks a bit ugly um that's at the low level of implementation from a high level um I think there's definitely a lot of debate about should we put all configuration for the entire thing in one file because people don't really like to know well yeah people are often concerned that with scholar CLI that putting the config in basically any sort of file whatsoever could make it more complicated to understand because where does the config come from you have to go and search for it and it probably makes it harder if you're doing automated tools that try and recognize your project because if there's no clear config file then where do you go to look for it um in my opinion I thought it was okay if each module had their own uh config and then the the that tunnel file just said how to glue the pieces together but I think overall I think more people want these to aggregate everything in one file so that would be the the biggest design design decision I guess is just where does the config go I hope that helps okay oh so um when you ask so the question is uh what design decisions do we have to make about at the scholar Center um do you mean about the organization The Scholar Center itself or just in general across our project what makes us stay awake at night um yeah I think is it's really I guess you have to be quite passionate um just because you know also what's keeps me awaken out is maybe like just knowing that people are giving us money to work on free stuff and that seems like quite a a luxury and a privilege compared to what everyone else is doing we get to be like our own in in charge of Our Own Destiny and that seems pretty luxurious as well so I really just thank thank everyone for the situation I'm in maybe I can elaborate on the decisions that we are making the challenges that we are facing so the entire past year we were working on setting up the community structures that would uh well go to the direction of governance so concretely we started to the SIP which is a scholar improvement process which existed for scalatu and now it is for scholar 3 which directs how exactly the language is going to evolve in the future how exactly an idea becomes from idea to an implemented feature in the compiler so that it doesn't break anything and doesn't doesn't break anyone's code so that's One Direction so the development of the language is one vector another Vector is the development of the community itself so we are working on the governance projects which includes moderations moderation team moderation rules how to how to enforce those rules upon the community so that the community is stabilized and can be trusted by big companies who you look at the community as one of the factors to evaluate the stability of the language and the reliability of the language for the future projects so those kind of imposing structures structures that would serve the language in the long term was one of the big challenges and is one of the big challenges that we are currently facing uh yeah I think a big one of course is dark sky 2 versus carafree and how much verse cast Center or invest in either of them because of course like uh companies uh care a lot about the long-term support of square too and about the migration strategy and of course we want them to keep being happy using mascara and also to help them migrate for all the reason uh Holden says about how upgrading your code is good actually [Laughter] and so I think yeah it's supervisors we have to strike uh between Innovation and maintenance um I think we've doing uh an okay job uh I think scarafix is pretty good uh people can can tell us if they have things they want to be improved in that particular respect but there's been a lot of work for example over work on the IDE support uh like the center has been working on improving the debugger for metals and maybe this will also impact the intelligent because we've been talking within 3j um and but yeah we the debugger is being improved for both twins color free so you can write either Square two or Square free code in your breakpoints and have them integrated at runtime it's really nice um and we've also been having like recently we had a Meetup with um a lot of the scheduling community at epfl so we brought the virtual people the jet Branch people together and even greater and we try to like decide what we wanted to do in the future and uh how much do we invest in these different areas and how to collaborate more so that's your challenge is how to get everyone because Scala is like a big tent kind of things and we want everyone to collaborate as much as possible uh so I mentioned this guy debugger support which we are trying to integrate with uh into AJ another project with Oscar Center is working on is tasty query where Jamie did a lot of work which is a library for um basically doing uh inspecting the output of a square free compiler and doing uh interesting things with that like checking uh the kind of code that we compile words and doing compatibility checks for new versions of libraries and also any kind of uh semantic information you would want from your code you get from this library and the idea is instead of having this information I need to compiler but other tools can then start using that without having to replicate a lot of work done in your compiler thank you foreign thank you