SBTB 2015: Tobias Schlatter, the Scala.js Compilation Pipeline
Recording: SBTB 2015: Tobias Schlatter, the Scala.js Compilation Pipeline
alright so this dog is about the Scala jess compilation pipeline scholar jess compiles scala to javascript my name is tobias and i'm one of the core contributors this is my github handle before we have a look on how Scala compares to JavaScript let's first have a look and remind ourselves how does the scallop I'm work for Scott on the JVM right so let's assume you're writing an app you have your app scholar source file there you feed that into still out into the Scala compiler and the Scala compiler will create a class file that class file you feed it to Jay to the JVM and the JVM will run your application for you usually you package these class files into a jar but fundamentally doesn't really change anything the JVM will look into your jar and get those class files out there for you if you start using libraries which is I guess something many of us do something very similar happens on the side of the library author right the library author writes a source file feeds it into the compiler and creates a jar this jar is what he publishes right when you launch your application in the JVM you give both those charges you put them on the classpath and once you start using a class of the library so that's for example say your application uses a list class of the library when you use that class for the first time the JVM will look in the classpath and search for that list class and loaded so it's dynamically linking the class into the application if the class is not there you will get the class not found error which I guess all of us have seen in our very unpleasant I said you're using the list type in your application so actually you might see that there's an arrow missing here on this slide if you write lists in your app scholar file and the compiler doesn't know about your library it will tell you hey I don't know what you're talking about list here I can't verify that what you're doing here is all right so you actually have to put the library you're using also on the classpath of the compiler so the compiler knows about the other things you intend to use now if these two arrows get out of sync right the one that goes from the library to the JVM and the one who goes from the library to the Scala compiler then you even though your stuff compiles properly you will get classnotfoundexception when you try to run your application with Scala justice looks actually very similar right there's a couple of differences first of all we're using the Scala J's compiler plugin when we run the compiler instead of emitting class files we emitting scala GS intermediate representation it's a class it's a format similar to the class format but it's tailored to scala jess instead of just running these files right you want to produce a file a javascript file which is JavaScript coronated a single file so we need to run this through the linker the linker is a separate application the scholar Jess project provides and it takes all these iron files and links them together and produces a JavaScript file and then you have JavaScript and you run it with whatever javascript vm you wanna run it no GS in the browser Phantom cheers when you compared to the previous slide you can see that there's a file there's an arrow missing right this cross arrow here we actually don't have it yet in this graph so currently if we're using the list type in app the compiler wouldn't know about this time would complain right so the easiest solution is well just have to compile a read Scala gsi r and figure out that there's a let's type and go on from there right unfortunately the Scala compiler is not doesn't know about Scala GS all right it hasn't been built to do that so therefore what we do is we emit class files as well so thus colleges compiler actually emits class files and Scala gsi are the IR is used for linking and the class files are looking for symbol lookup when you compile all of this stuff comes back isn't it package it in the jar and this is what you publish so in the end it looks very similar there's really two key differences there one in the jars you have class files and scholar JSI our files and instead of running directly with a virtual machine you run stuff through the linker and then you compile to JavaScript okay so from here it is actually pretty obvious that we have two key components to that whole compile to that whole pipeline we have the scholar GS compiler and we have the scholar jess linker and i'm going to talk about these two components in order all right these are the phases of the scholar J's compiler we are going to go through these phases one by one and have a look at what they do individually and hope we're done in the half hour half hour i have just kidding oh this is hard to see only these two red guys here two guys prefix with jas are phases that are actually specific to the scholar J's compiler as I mentioned the scholar J's compiler is really just a compiler plug-in so on the black stuff is just the Scala compiler and the red faces are things we add you will have amex a better picture in the next slide and the red faces are things we add right so this is actually much more looks like that there's only these two guys are actually scala jazz specific everything else is just plain Scala compiler so what happens here again you have your source file that feeds into the compiler front on the front end is mainly the parser and the type so make sure that well do you have right syntax and it makes sure that what you're doing is statically sound so you have type safety the front end is also the part that would potentially use a class file of a library to make sure that you're using your list type in the right way front end is also responsible for implicit resolution and macro expansion all right so we got all that from there we feed into J Center up face Jet Center up face is mostly doing error checking of the scholar jazz specific things so if you're using facade types the wrong way your exports doesn't matter really if you don't know what that means its ecology has specific things that we checked there so we can give you nice error messages rather than just the later phase crashing and telling you hey I failed sorry let me feed into the back end at the back end is really the part that simplifies the complex scholar syntax to the simple representation that the JVM understands all right so it is all kinda just hyper Asia it does mix-ins it is all kinds of very fancy stuff that fortunately we don't have to take care of prepare the path for cheese right we have the JVM face that emits class files not going to talk about that that's just standard how the Scala compiler dozen and we have the GS called phase that will eventually omit this ecology as I our files so let's make let's look at an example right this is our piece of code she's going to company us through the rest of the presentation we're gonna feed that thing through the whole pipeline until at the end hopefully we have JavaScript that we can actually run so what do we have we have a multi elected class it allows us to alert the user multiple times very useful feature it it uses this message factory that allows it to create some hello world messages you can see that on the bottom and it has two multi other method right so he can give it a number hey I would like to alert my user ten times so I'm sure he actually understood why he just opens up an alert box which is busier message box and gives the message to right all right questions about this code good so when we feed this into the front end what happens first of all the type we will add all the types right so we're lazy there in some of the type annotations we didn't put them that the type is going to add all of them then this for loop is expanding so you can see that the in trapper that's an implicit conversion and the for syntax is to show get into a for each the body remains the same and down here the string interpolator expands to a call to the string context that's just how internally string interpret early Interpol ages are implemented all right again I'd like to mention this is just the Scala compiler we get all this for free there's no code in Scala Jess at all to do this all right so we got from the source file into the front end through the front end and now we're going to the J's Interop face first phase which is actually part of the Scala compiler I have the scholar jess combined excuse me so what does jay is enter up to it is two responsibilities one catch javascript interoperability errors there is no error in the code I present so we're not going to see anything there and exports exports are methods on Skylar classes that JavaScript can call right so we may be seeing this j/s expert annotation they're basically says hey I want this method to have a sensible name in JavaScript so I can call it otherwise methods will just get a very strange name you don't have it we don't give any guarantee so you won't be able to reliably call it is interoperability phase basically just does this it adds this additional method but this fancy name basic says hey this is jazz exported method what happens there is we're going to pass this method through the whole compile a compilation pipeline and in the end we're going to catch it and co there's an exported method they'll have to emit something special for that right the reason why we propagate it through the pipeline is that for example you could you can overload experts and when you overall experts and you might do something fancy of the type such that the erase types they're actually ambiguous and they end up being the same type and you want to prevent that now scholars he has a whole infrastructure to do that so we want to leverage that and when we use it so we just feed it into the whole pipeline and if that happens Scala see will just complain and we don't have to do anything about it we really get it for free that's the reason why we we add these guys here I'll show you later how we catch them and translate them into actual exports all right from there is this additional method not a lot happened we feed this red arrow here is actually hard to see how there's a red arrow here where we are hmm we feed it into the back end damn it's kind of ironic I said the back end simplifies things but this is how it looks after the back end so I don't expect you to understand all that this stage just a couple of things I'd like to note for example the constructors they have been lifted from the class body into a separate method there's this thing here and extension method pops up an extension method is a scholar sees way of implementing mixing so at this point we don't have trade methods anymore we have pure interfaces and classes and down here we see some very fancy stuff going on for varargs alright but that's again not our job scholars he takes care of it okay so we are already ready to translate all this into the IR unlike all the other phases j/s code is a translation phase right the other phases just take scholars into internal representation of code and transform it the giant the J's called phase takes his internal representation and translates it into the IR so to better understand what it's doing we'll have a look at the IR first how it is in some properties it has so we can better understand what transformation we actually have to do alright so the intermediate representation of scholar jess is in AST form and type tasty form what does that mean let's have this little example of cold up here in a yeasty form that basically means ok we have a block then we have a valid f the name is ax we have a literal string on the right hand side of the assignment we have we apply a function here where we select a member on this X variable here and we have a literal that we pass you can see that some things that you see in the source code are abstracted here so for example why do you have a semicolon here or this is a new line doesn't appear anymore in this abstract syntax tree form which is practical because we don't really care anymore right typed just means that each of these element has a type so itself so we know that the literal there is a string access string and so on and so forth this is the same representation that the Scala compiler uses internally so the translation is almost one-to-one it's really trivial so there's nothing special to be done there we allow complex expressions in the india st what does so for example a block is an expression and so the upper code there or you have a result and then inside you have a a variable right a helper and you just return this value from the block many languages like JavaScript for example they don't allow you to do that so you would have to write the lower code there and basically move the helper variable out again that does not matter to us right now because that's how Scala represents these internally so we can just translate them onto one next to having normal skull operations like you would expect we also support on any kind of value you have any ID say hey by the way I want you to call this JavaScript function here on that thing and just like javascript is if the functions there it's gonna work and if it's not there well it's just gonna fail miserably I'd run time just like javascript is um this is something we'll have to take care of because Scala did definitely not foresee having emitting JavaScript operations so we need to do something special about that so I'm I'll mark this what types do we have in the IR we don't have generics just like on the JVM that's all fine the Scala compiler took care of to care for us to resolve is we have primitive types primitive types int double and so on and so forth then we have class types right so this is all like literal one-to-one translation as far as classes and interfaces are concerned we have single class inheritance we have multi interface inheritance but interfaces are pure so this is really just like on the JVM we don't have overloading we use name mangling instead and as we've already seen a bit is every class can also have JavaScript methods on it right so these are definitely things which are different these three guys right the JavaScript operations the name magnet for overloading I'm not going into detail there because it's pretty trivial and the JavaScript methods so how do we do this translation calling javascript from Scala code let's look at our example again the multi alert method we have a for loop we all know how that works and then we have this finger Dom alert what what's down there right let's have let's have a look at the definition of Dom DOM is just to play an object but it extends this weird type JS global scope this type tells the compiler hey this is not actually an object so the compiler is actually not gonna make any code for that it's gonna say oh this is JavaScript world this is just a declaration I'm assuming this stuff is there and it says hi all the methods I have defined they're actually just global JavaScript methods so the compiler is gonna translate this to this guy right so here you actually see an example of how we use name Langley for overloading you see we append a couple of things they're based on the types in the arguments in the return type then we have the for loop I'm going to spare you that coat looks terrible because it's basically an implicit conversion on the interent and an extension method an integer and then the collection librarian for each stereo okay until you see that special thing your global alert this is how we represent calls to the JavaScript world and down here you see the call to hello so the interesting thing to observe here is that although we have two normal calls here we have Dom alert and messages hello these calls translate to something very different here right the Allard call translates to this guy whereas the hallow call translates to this guy this again is just named angling for overloading again and this is all based on types so this is how we can call javascript methods even though writing normal Scala code all right so how do we handle JavaScript methods on Schuyler classes I said Jay Center up generates this magic method here right if we encounter a method like that the ire of um it looks something like that first of all it's just the method itself and we omit this additional method here the export method again we use strings to represent JavaScript methods that basically checks the argument types you can see the argument of that method isn't any so anything is allowed there and in there we cast the argument so this will fail if we're given something which is not an it so we can then afterwards safely pass it to the exported method so we can still in there assume that all we actually have in it right this is also the part where we would resolve overloading of experts in case there were any overloading all right so we have I are nice so this is the first part right so we're basically now on that jar here we have compiled the application let's also assume you have compared to library so now let's have a look at the linker that's this box over here these are the phases of the linker all right so be feeding dir this now is the ire of both the library and the application this is all the eye or unique just like the classpath you will have a new JV yeah hey the linker assembles all the classes you need to run your application right so it figures out which are the classes you actually need and it reads those files from there optionally you can move it you can run it to the ir checker this is basically just a sanity check it's very useful for debugging that the I are actually satisfies the constraints you have otherwise the later phases might fail not because they're wrong but you're giving a buggy I are the optimizer reduces sighs and increases speed of the code the refiner removes classes that thanks to the optimizer are not needed anymore we will see an example of that and the emitter finally translates the intermediate representation into JavaScript code alright same game again you're going to take gir we're going to feed it through this pipeline and see what happens oh yeah by the way I forgot that the closure optimization thing down here is you can also feed that stuff into the Google closure compiler to do some more dad coda been eliminated elimination and renaming it will just produce a smaller javascript file which is a good for production the linker phase is the phase we used to call that code elimination now in time we have realized this is actually much more alive code inclusion than that code elimination so but it does it's it's looking at the entry points of this college as code so these are the exported things and it looks at everything that is reachable from there so from our original example we have an export here right that the multi allergic class is exported she's all okay the multi larger class is exported so it's reachable all right the class is reachable that means its constructor is reachable okay in the constructor we instantiate hallo factory so hallo factories reachable all right instead all know that's not me we also have this audio exporter alright this expert is a multi alerter and multi a lurcher is reachable therefore all its exported methods are reachable right if there is no instance of multi allergy it doesn't matter whether your method is exported or not because there's no instance so you can't call a method but here for instance we it's possible that you have an instance right so he's okay good this multi Allard method here is reachable right now the for loop here will reach all tons of classes in the scholar standard library not gonna go into that the dumb alert call is not really there anyways so that doesn't reach anything then we reached the hello method in Halifax alright that's it and you see automatically without even looking at it we have not included the hello debug method down there because no one uses it right just like concurrent hashmap for example we also don't use it so that's why a life co inclusion we have not eliminated concurring hashmap we have just not included it all right from there we feed them to the optimizer let's take that piece of code again night and I have a confession to make I actually lie to you I told you I'm gonna spare you how the for look look like this is how the for loop look what it looks like okay don't look at it it's ugly all right the optimizer will turn this into that guy right so instead of having this nasty little for loop which you say this is just a while just have a ver and volume right the nice thing there is well eight is this smaller be it's faster and see it also doesn't use the scholar collections library anymore ha nice so when you feed this into the refiner definer can actually now eliminate range and all these things we had in the scholar collection library because we don't need them anymore nice okay so we are ready to meet JavaScript code again we're at the translation step so now we're going to do the opposite we're going to look at the IR again and say okay which of these things do we have to change so that we have JavaScript again right which of these things are not okay for JavaScript trees are naasty form a type date is C form a you see easily translates back to source code so that's fine we don't care about types we just ignore them right javascript doesn't have types hmmm complex expressions I mentioned before JavaScript does not allow complex expressions like that so we'll have to do something about that javascript operations is a JavaScript tends to handle very well so we find her javascript is no generic types we find their primitive types which is we use the JavaScript primitive types number boolean and so on and so forth class types we can model using draw the type chaining I'm not gonna go into details there that's maybe heavy JavaScript done classes interfaces we have single class inheritance right so we're gonna model single class inheritance using prototype chaining multi interface inheritance while javascript is in no interfaces but since it's dynamically typed we can just actually erase interfaces because we statically know the methods are there so we just call it no overloading well lucky us we did that because javascript definitely does not support overloading and JavaScript methods and classes that's also fine alright so you're actually not very far here the example unfortunately breaks a bit and I have to take another piece of code to illustrate how we do sugar so we turn these complex trees into easier trees let's look at this a bit strange nor method it takes an integer in a string and it converts the string into an integer and dislike nested block there and then calculates the norm right of these two values in in pseudo JavaScript code we would immediately translate that looks somewhat like that you notice the pseudo up there because there's a strange stuff in here right we have a return all up here that returns the whole block and then we have this var which also is a block is assigned to it right out here that's definitely not allowed in JavaScript right so we'll have to push down we call it right a left hand side push down so we have to push the return inside this block oh this looks much saner already right but we still have the the problem of this bar here so we're just gonna do the same we're gonna push the bar down here alright some of you might object to that and say whoa hold on you change the scope of be too right b2 is out here so yeah you could use it down here right and now you pushing it into their this peter is not even defined here anymore what are you doing this is JavaScript in javascript in javascript for unfortunately for a someone who writes javascript fortunately for us VARs how our scope to the enclosing function block so as long as we're careful and we don't have names that collide and we only optimize on the function level which you do we can perfectly do that so this is actually valid JavaScript code well nice all right so um we just write that to a file oh yeah why not um yeah we probably it's probably to block that optimizes these away so if you create the outer block here in nutella hey you have these two statements it will look at each statement and look at the block and say oh I'm actually a nested blocks I'm gonna pop the inner block immediately and and if it doesn't it's the google closure compiler that takes care it is always our argument if people say hey you're not optimizing is for that Google closure compiler alright good so we write this to a file that's straightforward alright so we're done javascript code this is how we simplified version of how many omit the JavaScript code of the scholar called the exports are not on here but all the rest is all right so the constructor here of the multi a litter the for loop which is now a while loop and the hallow method here which yeah I didn't go into the details of the string context that there's a lot of instantiations of secret reason javascript aires and whatnot alright i'm going to open it up for questions before i go over my shame slide this is the shame slide because it contains all kinds of stuff I emitted but still is actually kind of important but questions first go ahead yep I was expecting that okay hmm there you go okay the question was reflection reflection which i retorted I was expecting that all right let's look at the linker let's look at the linker right um I'm going to enumerate the phases in the linker that don't like reflection okay the linker doesn't like reflection the optimizer doesn't light reflection they refine it isn't like reflection and as of now the emitter is ok with the reflection but in the future might not be okay with reflection I'm a bit going into detail by okay the linker as i mentioned assembles all the classes that are necessary to run your application uh-huh class for a name anyone it actually can't do that right actually it actually can't do that since the JVM links add runtime loading a class as you go through code is absolutely normal right the first time you see the list type in compile code zero cold let's let me load that it's not I works in Scala Jess so it is very hard right to have an ad hoc linking mechanisms like hey I want this class there and then somehow the JavaScript code should go back it's a okay let me grab this I are again but we're from you don't want to ship all the IR out to your client if you're in a web browser because this is I'm big talking megabytes there I mean the IR is if you compile everything that is reachable we're looking at 100 or 200 megs probably of just stuff you don't need the optimizer in theory could be made to support reflection but it wouldn't be able to optimize that well because it does things for example it recognizes monomorphic dispatch in polymorphic dispatch positions things like that so if there's only one subclass of a even if your call it a method in a it will Co actually this is in theory polymorphic dispatch but I know that can only be one subclass of this thing so this is actually monomorphic let me end like this BAM faster but if you can just load the classifier reflection that knowledge is actually doesn't hold any more you would have to read optimize the code the JVM can do that but we can't we can't change code on the fly the refinery similar considerations than the linker and the Demetria I'm not going into details it's we're having discussions about whether we can mark things to be reflection about which is essentially the same to say hey include this all the time right it's it's still in the early stages we realize that their potentially will be something that it has to be something there but the story is really not yet clear how who defines what exactly right if you have a type in your library would you as a library authors say hey I want to mark this reflect a bull or do I as user once again hey by the way that type in that library over there that was never intended to use by to be used by reflection I want this to be reflective all and how would you specify that mmhmm yeah methods the same we get rid of them if they are not used so short of including everything for the jvm and then you do both annotation base like up in as well as xml based off it so anyone can do it i want to so that's like what the status quo is yeah faiman so the question was if he referenced this in Scala code that you write in Scala Jess what's the semantics is the javascript is semantics or it's the scholar the semantics this always in Scala jazz always refers to the scholar dis if you need to refer to the JavaScript this there is this thing we call a dysfunction they're basically inside the function you're getting the dis as an additional argument right so with this function that takes an integer and returns the string inside would actually have two arguments it would have the disc and the integer and you will have to return the string all right so there is access to this but by default we're being very careful that we respect scala semantics so that this has Scala style the semantics other questions yeah I guess yeah thanks you want to shame slide alright she calls me out alright alright so I'm on the left here we have stuff which is there in general so i title it i are so duty stuff that carries over the whole pipeline we have things that are different in the compiler and if things that are different in a linker so the first really big thing is hijacked classes some of the classes the best example is java.lang.string right as far as java is concerned the string is just a class right we're not really happy with that because we kind of want to use javascript strings to represent strings so the whole pipeline knows that well if someone says java.lang.string what it actually means is javascript string but there is actually a type 4 string in dir right and that's kind of strange because string is actually a subtype of care sequence so if you see a care sequence on very suddenly this could be just some implementation of care sequence someone wrote or it could be string which is in terms of JavaScript not even an object ha so the whole pipeline kinda has to be aware of that and work around it there are others example other examples like that notably boxes java.lang.integer java long long and all these kind of things because we don't box them in Scala Jess which is past normal integers around which is nice there's more types in the IR there's the string type there's a couple of types that scholar has internally and we could just make types out of them and in on the JVM they use like dummy types with them there's a rate types which are actually pretty painful right the JVM has refi the right types so the array knows the type of it's in your element that's not the case in JavaScript and there's record types record types they don't live in the files their helper types when when we do class in lining we actually a four scalar in lining it some people call it if you have a class small topple somewhere and you can inline all the methods you call on that topple you can actually be inlined at the fields of the topple rather than creating an option actual topple class labeled blocks classifieds actually support go tues and the Scala compiler creates go to sleep pattern matches and terry tail-recursive methods so the thus colleges compiler back-end actually has to recognize these patterns and translate them back into a more higher level structure and sometimes when you have very crazy macro libraries the backhand crashes doesn't hey I don't know about the structure so you have to change your macro wait for a version until we fix it and modules or AK objects damn I mean as in when you write object something in Scala we actually have these ndir whereas scholar class files dog the compiler its special cases Scala numeration because it needs reflection to work which we don't support so all the code that uses generations wouldn't work so the most basic cases of enumerations we actually do some magic that you can recognize the names that you still have to refight runtime names of the fields reflective calls and we support them but we need kind of some magic to make them equally broken than their on the JVM function literals how do you write a javascript function there there's actually a lot of like boxing and unboxing and going back and forth and export overloading right i mentioned that that experts can actually be overloaded and we do this patch based on the runtime type of the arguments so it's not the static dispatch you would have on a JVM but you basically do type checking right the argument comes in like oh is this an int all yeah okay so let's call the nth method oh this is string oh let's call the intrusion at the string version right or like oh no this is not okay oh this is two arguments all right so it's just completely other or overload over there all right in the linker I completely enacted instance tests right because if I give you a value and sing hey is this an instance of this interface as I'd interfaces are erased well then you can't reply to that but a lot of scholar coat relies on that notably pattern matching which is kind of important so yeah we need to basically carry runtime type in runtime type information around to do that check s is fine Long's javascript is not do lungs so the link he replaces the primitive type Long's by a class of hours which has three fields high mid and low sure each of them an integer and well if you can post them there long if you want you to scala Jesco to be fast a word of advice don't use lungs I didn't touch in the whole inheritance JavaScript story right so um especially this slide here this is actually not the full picture there's each class as two constructors one is the actual constructor and one is the constructor you can inherit from which is common practice in JavaScript and we also have a init method that's just how on the JVM the constructor actually just allocates fields the actual constructor in the init method does what you as a programmer would consider to be the constructor right so this guy here would actually just assign null just to default value for an object to messages dollar one and then you would call each time you instantiate the multi auditor you would call new multi a leader dot underscore underscore no init underscore underscore underscore which is the magic name we have for constructors yeah so this is a bit more complicated in there and then semantics and output modes right you can the link your supports just emitting something to acma script six so you can just flip a switch letting all that we actually emit es6 classes the link here has annexed I mean both of these equus critics and strong loader experimental strong mode is a strict modern steroids for javascript people that v8 is coming up with two they're busy each is trying to limit javascript to even more language construct so that the vm can be even more efficient you can just flip the switch and you have strong low support which guarantees slower than acma skip 5 which is kind of funny but and yeah the semantics are basically you can tell the link you to get completely rid of as instance off speakers do to slow the jvm is good at optimizing his instance offs javascript is not because they not really a concept that exists for these guys so sometimes there's just four weeks in theirs and different ways to omit the same thing and basically all I actually erase all that I know this another flag is clicked floats if you have a flow I think that actually is kind of the same on the JVM you can say hey afloat might actually also be a double it might be more precise or not right or you can say i want strict floats and if you say strict floats well you'll get the float but it's potentially slower because if the JavaScript runtime is older it doesn't support the relevant float methods notably F round yeah and that potential reaches I mean try and round a double to a float in user space when you don't have access to the individual bits that's what's gonna happen so you might not want that to happen unless you really have to yeah more questions all right I guess that's it thanks a lot you