SF Scala: Martin Odersky, Binary Compatibility in Scala
Recording: SF Scala: Martin Odersky, Binary Compatibility in Scala
Mike seems to be working well uh thank you for the invitation I'm very happy to be back not well Intel ver now Verizon and uh present uh to this group and uh to the South Bay Area in general um I'm a bit jetlagged uh I just arrived last night so if I'm sort of kind of dizzy then please excuse that I'll uh you have to keep me awake let's put it that way good so I'm gonna what I'm going to talk about is a more specific thing uh about binary compatibility and I'm going to say more about that and how it relates to other things uh in my keynote that's galad days on Monday so here is sort of one deep plunge into a fairly technical subject and it's about binary compatibility and it's a subject that is sometimes quite painful to talk about so we definitely wear a little bit the hair shirt when it comes to Binary compatibility let me maybe ask you who has already had a problem upgrading to a new version because of binary compatibility problems well quite a few yeah um so uh we are often the Scala Community or type save or epfl gets accused of not taking the problem seriously enough that uh it's it's a matter of discipline we're just not disciplined enough people just change their code all the time and it breaks all the time so it must be somebody's fault right uh and uh what I try to do in in this work is to analyze this a little bit where where do these problems come from and then also maybe to point out the solution what we can do about it so um the problem in a nutshell has been that uh well it's been an issue with the Scala Community for a long time since it's beginning since it became popular and uh the um early problems were that things broke left and right and the community has learned to deal with it to a degree by becoming more conservative and uh becoming more conservative means changing less often and it also means I think having fewer dependencies you hear it a lot about this idea of zero dependencies and I think that by itself is is a problem it means that we can't reuse code of others as well as we could if we have if we had more dependencies or could dependencies use dependencies more freely but then right now we run into these binary compatibility problems so it's a bit of a problem the the the the dilemma here is well you can either innovate and break your client's bills or you free Freeze and stop uh improving and uh that's a sitting between a rock and a heart stone so the question is is there not a third way that we can do better so let's analyze first what binary incompatibility is because I think even though many people have been burned by it probably few people had the patience to really go down and say well what was the reason why this Library doesn't work in the newer version yet so to put it in a nutshell uh here's a problem where you have a source and binary incompatibility so you have a a client object and it uses a server and in the server you have a message and it's just a string and then you decide well message should really become an option because you don't want to use NS which is very reasonable uh but if you do that then of course uh that thing here won't compile anymore because you can't take length from an object and it won't be binary comp compatible because essentially the bits don't agree so we all agree that that's something nobody can do anything about except if you put some artificial intelligence into our sort of programming upgrading tools which is a fascinating idea and fascinating project but maybe not right uh there yet for tomorrow so here's a thing which is Source incompatible but it's actually binary compatible uh so we have a client object and it Imports two objects A and B and it has uh we do a a very clever thing we assign one to a string and that works because we have this beautiful implicit conversion which makes us pretend that we can do actually JavaScript like things great uh only now somebody comes and says well uh I have a great idea let's just do this just twice as good and have another implicit conversion uh that also goes from into Strings but this time it is just a fixed string ABC like this one here uh so if we do that then of course our program won't compile anymore because the implicit conversions are ambiguous there too now that imp imply but if you look at the bits at the at the bits in the class files they don't care because the implicit conversion has been resolved so these things will continue to run until you recompile in which case you will start to notice the problem okay so let's now go to the one that which is sort of the real problem some that Source compatible but binary incompatible so here we have an object call it Apple it extends edible it has a JS uh field or a method uh tra edible is a trait with the abstract method JS and now we say well actually it would be nice to uh also have a a calories method which is just JS by 4.184 and uh that's just a convenience method that we add to the trade if we do that then until uh for all current scalas uh you need to recompile why do you need to recompile well if you look at the implementation of this trait then that's what it looks like so a trade becomes essentially just an interface in Java I'm continuing to use Scala syntax but uh I now sort of restrict myself to the what I can do with Java interfaces so two abstract methods JS and calories and uh the calories method because I already have an implementation I write this in into a so-called implementation class which comes with the trait in the uh in the uh class file format for that uh Scala compiles to and now in the client so in the Apple object where I where I implement the edible interface I essentially have a stub for this method that forwards to my implementation class so in the implementing classes I fix out the methods that the traits Define okay so now the problem of course is that if I do that then initially uh I only had uh an interface that used JS like this and now when I add the calories method here I need to add it also to the implementation of the implementing class which is something that the compiler adds for me so if I don't recompile this thing then that method will be missing and I will get an error that essentially I have an abstract method error in the jvm which says the method J calories wasn't implemented by this by this object so that's a problem uh in principle that had to do with the way we implemented mixin and there's no other good way to do it and uh that couldn't be solved until Java 8 so Java 1 to7 essentially that's just something we had to live with with Java 8 it's slightly better so in this example one could say well eight has default methods in traits so now we could actually declare this thing a default method and we would actually be solved this this problem here and scalar 212 will do precisely that so it will use the default methods of java a to solve that but it's essentially a solution that is just one step so as soon as you have several methods that can Implement a thing so you have conflicting possibilities in traits that you inherit or as soon as you add vales or vs to your trades the the whole thing breaks down and you have to do the same thing again so it's sort of it in practice it will help you with some of the binary incompatibility problems but by far not with all of them uh and that's only talking as far as trades are involved and there of course a lot of other issues as well so here's another one uh let's say we uh want to say well our calories implementation is of course terribly expensive multiplications are Let's Pretend multiplications cost us millions of Cycles so we want to cach it in a lazy valow uh so if you have if you do this then uh the compiler has to work a little bit harder because lazy vs they have this evaluate one semantic so you evaluate the first time you call it and then you want to cach the results so the way it's done is that there's a hidden VAR that essentially stores the value of the calories and then essentially there's a flag there's a there's a bit in a bit set that tells you whether you have set whether have already computed uh the the value of of the calories and in which case you can simply re uh return it directly and otherwise you have to compute it and store it in the variable so that's the scheme of lazy values that we have in Scala um but it turns out that um it's actually lacking um so we spend one bit per lazy vals and it turns out that there are some situations where you can run into a deadlock uh just because of this lazy Val scheme basically what happens is uh in the uh in a multirail context uh essentially you um take the lazy Val uh you you find out you have to recompute the value you lock the object to say well while I recompute the value I lock the the object so that nobody else can uh implement this lazy well and essentially once the initializer has Runner can come back and the problem is that locking the object of course can interfere in bad ways with locking schemes that other people use somebody might might need to lock the same objects or worse you might have two lazy vales and you have two threads each one of them grabs a lazy Val and says I need to initialize that and then each lazy Val needs the other to be initialized then you have another problem here so it turns out that actually these problems can be overcome if we spend two bits per uh Las well where essentially the first bit says uh essentially two bits encode a little State machine that says well hasn't been touched yet or is currently about to be initialized or is fully initialized because then you you could say well I if I do that I only Mark the variable s is about to be initialized then another thread comes and says oops somebody's already working on this so let's just suspend and be notified by the other thread when the other thread is done and uh finally if the thread comes back and says well there's actually more than one thread that wanted the variable I notify everybody else so that scheme can be done can be made about as efficient as the current one it spends two bits so it's a clear that's not a big overhead so it's a clear win to do it this way uh the problem of course is if you do that then all your bit layout of all your data changes and that's another big problem with binary compatibility you have to recompile everything so it's a big uh disruption to do this we can't just roll this change in into a a build release into a patch release uh because uh well everybody would have to recompile and that's again just a thing to say well nobody is perfect so we cannot assume that the scalar compiler from 10 years ago would should do exactly the same algorithms and things as the scalar compiler now and the scalar compiler 10 years from now will again use different algorithms I think the world works like this you wouldn't expect a hotspot VM to have the same jip compilation as 10 years from now and 10 years in the future it will again be different so because Scala the the binary format was so much out in the open we have we had been restricted in ways which are really extremely severe because of this problem so if you look at what what goes on in a compiler then that's just a a little snapshot of a current compiler pipeline and you see there's actually a lot of things that the scalar compiler has to do until it can eliminate uh it can spit out Java bite codes so there's a front end which you see here up there there's a typer first a parser that essentially pars The Source produces syntax trees and there's a typer which essentially adds the types to the syntax trees then uh there's a phase that adds syn synthetic methods so like the equals and the hash code for a case class then there's a super accessor methods which is quite obscure what it does but important and then there are lots and lots of other faces so that one eliminates uh repeated parameters so varar arguments uh that one adds extension methods for Value classes that one uh makes records makes methods tail recursive uh that's a pattern matcher that one adds explicit outer pointers for inner classes that one is Lambda lift that puts all moves all methods to the top level and so on so each one of these does something that is interesting and quite substantial and they without having to go into details I just wanted to sort of convey to you the impression well that's a lot of stuff A lot can go wrong or a lot can actually be wanting to change in future versions so let's have a look at what how that affects binary compatibility so where break it's very simple you have a client a and uh it uses a library Class C and you have a binary incompatible Source change so then uh you get a new class and of course uh if you use the previous clients with the new class then because it's B incompatibly it will break so that's in a nut show that's that's what what people observe so why is that such a painful problem it seems to be a rather simple and trivia thing so here's the the one of the bad scenarios so you have your shiny new application and as things happen it relies on some dusty Legacy Library somebody it was hot at some point but then people have long moved on and this thing is forgotten but it's still in your build and you see you still rely on it and it works so there's no problem and this does the leacy library uh relies on an ancient version of Scala let's say 2.10 uh so so so now u in 2.10 uh we want to do it much better so we have a new version of let's say a class in 2.10 I just PE sequence uh and we have a binary Inc compatible Source change in 2.11 okay so no problem my application is still very much active I just recompile it right no because our Dusty Legacy library is too old we can't we build it nobody nobody even knows how to build this thing anymore we just have to jar somewhere right so that means that no you can't migrate your application because it relies on this thing and you don't you can't migrate this Dusty Legacy library because you don't know how to build it and that's sort of a problem that initially you probably can discount and it's it won't be important but in a big company in a big organization the bigger things become and the longer they live the more likely the scenario is okay and that's actually not just a problem for Scala it's a problem for essentially every system with components so we were talking here about the scalar library but nothing is specific for the scalar Library uh we could talk just as well about AKA or we could talk about shapeless or any of the other commonly used used scalar libraries if they uh have a new version which is binary incompatible then anybody relying on those versions will have the same problematic scenarios okay so what what's been done to deal with this problem so far uh it's not nothing in fact people have been working quite hard on that so the first thing that we did was we had this tool at least to say well we want to detect binary incompatibilities not only in deployment when you deploy this thing and it doesn't work but we want to detect it before that so we want to be able to say if we have a jar and we do some change to the jars and buck fixes is the new jar binding compatible or not previously the only answer was well try it out and see whether something breaks so now we have a tool that actually of since quite a while already we have a tool that detect this it's called Mima for migration manager and it uh can verify whether things are Bally compatible or not and then with that you can Implement a policy and the policy we currently have is that minor versions of Scala at the Scala Library must be binary compatible forwards and backwards binary compatible for major versions We are allowed to break them and but major versions are uh the the release cycle for M major versions is has already slowed down a bit and probably will slow slow down further a bit so it's currently at about 18 months so it gives people some time to actually adapt to uh not be overwhelmed by all these changes but the problem is that uh only affects applies so far to the scalar standard library but third party libraries need similar policies but often uh they don't enforce them yet and uh uh if they would and they will at some point probably then it's the same problem we already face now with the scolar standard library that Innovation is stifled simple fixes have a long time to get in and also what I noticed with our developers a lot of Cycles are spent on dealing with binary compatibility and I see when we get a patch let's say to the scolar standard library then uh it's one thing to say well we we do the fix and it's often takes at least the same amount of time if not more time to say well what are the ramifications for binary compatibility when exactly can we schedule this this this this this change and so on so a lot of effort is spent on this problem so that makes you ask well if we all spend such a lot of effort for overall not that great a result so what do others do let's let's have a look what what uh what the situation in other language ecosystems is so let's look first at our cousin Java so Java uh has the advantage that it's very close to jvm bite code so if things change they typically change at the jvm level uh so major changes reflected by changes on the jvm level take lambdas for for instance that use essentially new bite code instructions for invoked Dynamic U or take default methods that's again something that the jvm will simply support uh the other thing is that uh because the bite code format is kept very stable and and Innovation often happens underneath that is sort of mirrored by an attitude in the community to say we want to keep the bite the bite code files the jite code files very very stable even if it means keeping completely non-working and outdated things like Java util alive Java Java util data life for a very long time everybody knows it's broken but nobody gets rid of it because people just value binary compatibility very very highly and I think that's quite a rational attitude here and uh the language Java itself is quite restricted in terms of extensibility so uh until defa methods for instance you needed uh to have for each new version of an interface in Eclipse you made a new interface and then the client had to deal with all this things so the language sort of gives you a fairly rigid uh uh structure in which you can live and it sacrifices to some degree the extensibility of the of of of of the language and the the architecture for the sake of binary compatibility and that makes it work quite well um another way to go in the on the Java side would be osgi so osgi is actually quite quite interesting take on binary compatibility so if this whole thing were running in osgi you would have a different uh possibility so you would still have my your dusty Le Legacy Library you would still upgrade the scolar library but now you could just rebuild the application and keep this thing in its own uh class loader and that class loader for Dusty Legacy Library could have a version of scalar Library 2.10 whereas your application would have migrated two 2.11 and the two could coexist so that's the theory in practice it's quite fragile because you have two versions of everything then here and they at the interfaces they have to sort of talk to each other and that can become quite a hassle so in practice very few Frameworks have actually bought into that concept the best known is Eclipse but it hasn't spread that far uh further than than Eclipse so what about cc++ well it relies on a Linker for more flexibility of interfaces but I think generally it's not that great a story either so the name d hell doesn't come from from nothing uh so uh what about closure uh while closure builds from Source I don't have a problem there right so no binary compatibility because you ship stuff in source and you compile from source what about JavaScript JavaScript buils from source as well there is no binary format it's just JavaScript right well python python builds from source as well Ruby builds from Source go builds from Source yeah so it seems like a lot of languages built from source and of course then if you build from Source you don't have the a problem of binary compatibility you might have other problems but definitely not binary compatibility because it's all just source so why can't Scala build from Source isn't that something we should consider well we have a couple of pro problems here the first one is uh the in the other languages you typically have a standard build tool or deployment tool in Scala we don't have that uh so uh we could uh we would have to pick a winner here among spt or gr or Maven Ivy and and it's just too fragmented you will never be able to pick a St standard build tool and say well everybody uses that build tool even then the build tools are very very flexible and builds are very very re reproducible builds are very very rare so everybody has their own concoction of how to actually build this stuff and it's a it's it's quite a rocket science or magic to actually build this stuff so it's very very hard to reproduce and to some degree it's a chicken and neck problem because everybody is used to Binary builds nobody invests in making builds repr reducible because that's not the problem you ship the binaries right whereas in let's say if I pick go is an example where you build from Source well you probably wouldn't do some exotic uh acrobatics for your build because you said well I can never reproduce that but because the scalar Community has grown up essentially with this idea to say well our build system is built around Maven so everything is is is binary the problem never was looked at and now it's too late of course now it's it's not possible to get rep producible builds again but nobody invested making builds repr producible is actually not quite true because typ safe has invested a lot in the community build and we now can build more than 1 million lines of community projects with essentially every night uh with every release of of of of Scala we actually check that all these projects built so uh but that's essentially just what one company does and the total code code base of Scala exceeds this thing by several orders of magnitude so this is just one effort and it was quite a big effort to have this working so what we need would be then uh and what I'm going to propose is to have an interchange format instead of bite code that should capture the essence of scalar dependencies so that can't be the jvm bite code format what it is now because we've seen that causes us all these problems with biner compatibility and it can't be sourced because we can't get reproducible source so we need something different something independent of uh what we have seen so far also independent of javaby code so the idea there is then to say well what could be a good format for that and uh what has come out what has uh sort of emerged there that is that we should be able to use trees the abstract syntax trees together with the types of those trees so use type trees as The Interchange form so I'm going to show you that this is much more robust than source and so in particular it should be easy to get reproducible builds with that and it's much more stable than jvm bite code so it will essentially take care of uh all the binary compatibility problems or at least the major classes of binary incompatibility problems that we see there and I'm also going to argue that it's efficient so the new compiler idea pipeline would be to say well let's snip off the front end which is essentially parer tyer and then maybe some some number of faces that do don't do much so essentially immediately after tyer we snip it off and we produce these typed trees these typed asps and we can then feed the type SDS to the rest of the compiler and the the standard compiler would just go from one to the other and the type trees would essentially be produced on the side you could say that's nothing new because that's what the scalar compiler does anyway uh so at this point essentially it produces it dumps the symbol table the it dumps all the symbol definitions in a file into an annotation that it ships with the class files and it needs that to actually figure out what's another scalar file so that he can do separate compilation so that's what we we we do already anyway so the only change here would be that instead of dumping the simple table we type we we dump the whole a the whole tree with symbols well with the definitions of symbols with them but also with the bodies of methods with everything um and the normal comp compilation pipeline would then immediately go on go through the back end and produced by code but the idea would be that you could actually come back and take these type trees and run the back end again without running the front end so that's the the advantage of this separation so how would that affect our builds so the idea now is that if I have this uh application and then let's say that's the dusty Legacy library and that's my Class C then in together with the class files I would ship the tree the full syntax tree and I also would ship essentially a hash that tells me what is in the class file and the tree uh so I uh recompile that with a source change I get uh a new class with a new tree and a new hash and then I say I want to rebuild my uh a uh a do uh and I would then have to look at the dependencies and say okay it relies on the the b class and the b class has in its hash essentially all its dependencies so the the hash of C is included in the hash of B and now we say well if we take the new things we have is it still the same hash it wouldn't be because that hash has changed so we would know that we have to rebuild the class B and that would be done from the Tree of B so we don't have to have the source of B we have the tree of B and we can immediately run the essentially the back end of the pipeline to produce new class files and once we've done that we are in a position to do to apply the same thing to a and everything is over uh on the on the new side and it's all done automatic without having having to look at build scripts or manual recompiles okay so how is this better than going from Source uh it looks like if I would just take the sources I could do the same thing and that's true but uh like I said a lot of sources actually quite a subtle thing so uh the if you look at what happens in this tyer thing uh the so the front end then a lot of tricky things happen so the tyer has to resolve names it has to say in particular it has to say well I need to look at a class a name that represents a class I have to look at my class path and find out well what's the concrete definition of that class if the class path on your client is different where you ship it then you will have a problem here uh it has to handle Imports it has to establish implicit Scopes if in your new project you have let's say you add an implicit par definition or conversion somewhere then you probably have have realized that this can have a huge ripple effect on essentially everything in your build it can cause ambiguous implicits it can give you give you a different resolution of implicit or so on so implicits are very very subtle so if implicits change and you just chip this thing with different implicits and say well recompile from Source can but you can keep your fingers crossed that that the new build will still work but you can't give 100% Assurance I don't think you will be able to do that okay and there are lots of other things apply conversions infer type parameters assign type to trees so a lot of these things can be very subtle to do overall if I look at the current modern uh compiler the new compiler we look at uh which is much smaller than scalar C this whole thing is about 5 and a half thousand lines and well 5 and a half thousand lines can can be quite a quite a large source of Errors so whereas if we look at the back end if you take the type trees and actually the type trees uh I have to reproduce the the the types of the Interior nodes because otherwise the types would become very very big but this is design types thing has just 311 lines so essentially to produce from the serialization format the full tree ones I've deserialized with the types I need 311 lines much much better it's also more resilient than bite codes because with this new idea I can add fields and methods to trades I can add lazy vales wherever I want I can change the implementation uh technology of lazy WS in any way I de necessary because all this happens after I have the tree so after I have the tree I can just say well I I roll in an bug fix in a compiler it uses three bits per lazy valve whatever uh then I can say well that just will affect the hashes and everything will be recompiled so I just have to do this I can also add and remove implicits I can add methods anywhere I can change Imports again without affecting the format and all of these would be source and compatible so it seems like with the type trees we have we have this perfect spot in between where we say all the sort of things that can affect Source One way either Source or binary things one way or another at this sort of focus point they are largely eliminated then the question is well is it efficient uh the first thing it would be well this these trees on can't they become very very big can we still deal with them because now we're talking about let's say rebuilding a system with a million lines of code in it uh automatically when you build and you could have Ripple Ripple changes of incompatibilities with these hashes that force you to do that so size of trees is an issue on dis and in memory and also transformation time so size of trees um initially it looks like like quite a challenge uh so here's a simple expression xs. filter greater or equal zero uh and if you look at what the compiler makes of this then that's roughly the tree that it produces so each of these boxes is a node uh so they overall 16 nodes and I haven't even counted the types yet typically the types make it make it much much bigger so you say Okay 16 nodes take an average size of a note let's say 32 bytes 52 12 bytes total double that to include Ty type info so that would mean 16 bytes of source give you maybe 1 kiloby three so Factor 64 blow up for a 1 million line uh system you would have 30 megab Source maybe and two two 2 GB trees so that looks like uh a bit daunting but in fact if you look at it in practice you can compactify it a lot so here's the format that we we currently have this tasty uh so that format uh represents the tree essentially in a nested form and I won't go into the details of it um so essentially it it takes the the tree and uh lies it out like this one so it's overall it's an apply method and apply node and uh there's a select node so where we say well we call this uh filter method uh that uh takes a signature so it takes a function it produces a list and all these things overall so these are bytes so we end up at uh 36 36 bytes for this thing which is twice the uh source that we started with um if we uh do some bigger programs where essentially the savings are much bigger we we get to about 50% to 75% of source size which translates to less than a quarter of the class files we generate anyway so less than a quarter of the class files that we have we put into this thing and that's uh would replace something that we do already now namely the simple table serialization so it's essentially quite manageable to do it that way speed so uh the uh transformation and B code generation uh amounts to about 60% of total compilation time so definitely significant um we can speed this up by essentially refusing phases reducing the fast type assigner and building different files in parallel once we are at the at essentially at the backend level we have way more Liberties to do that and we can use incremental compilation so I I believe that we can make this fast enough to work well in practice speed is something that we haven't really tested yet whereas uh the the serialization format uh so the gains in compactness we we actually have a working thing that that achieves that okay so that thing seems to work and it can address the binary compatibility Challenge and it actually turns out that it has lots of other benefits as well so once you have type trees you have a great format for optimization inter procedural analysis you have precisely what you need to do things like aggressive inlining that code elimination whole program analysis um uh you uh can you have also precisely what you need to uh do sort of linter tools like uh abide or um uh Scala Centric versions of f bugs or instrumentation so if you let's say want to find out code coverage and things like that all these streets give you a standard format where you can suck in things and interpret them on a very high level much higher than bite code and much easier to consume that bite code and also it gives you precisely what you need for portability because these type trees now allow you to decouple essentially the fact of publishing a library to the question for which platform it will be published so you can publish a library once and have it then automatically deployed on both jvm and JavaScript for instance and also you will be able to have it deployed on new versions of jvm because the jvm also after a long period of essentially uh stability uh where nothing much changed for 10 years now the jvm is actually moving again there was a big jum jump forward this Java 8 and more more such jumps are planned I mean Java 9 adds modules Java 10 might add specialization and all these things are essentially again jumps very much like Java 8 so that it will become effectively a different platform to compile to and that means that if you essentially can have a layer that is high level and stable then you're not dependent anymore to what version of the jvm you compile to okay so what I see is how we're going to migrate to this is so the whole thing is essentially part of this new compiler framework dot do c uh which we are working on uh and uh we now have the serialization and deserialization but we're also working on getting the same front end for current scalar C so the idea would be that we would at first have that as an alternative mid-end and back end that we could go into this tast tasty format and the the the the the compilation pipeline that I shown you both from scalar C and from Doty and then use again the bite code emitter would be shared between the two uh and that over time we could actually snip this thing off and have that as essentially the standard way scalar code is published and once that is the case then encourage essentially the whole industry to move over to the new format and hopefully uh fix this binary compatibility problem for good that way so in summary B competi ability is a very is a highly naughty problem to solve it I think we we need to move to something that resembles much more a platform uh so the platform would of course build on top of hosts such as jvm JS and be as interoperable as it as it is now but Scala Scala interoperation would essentially be formulated at a higher level at this level of type trees and I've argued that this has lots of other benefits as well so to find out more uh there will be I will have a talk that expands on that a little bit at Scala Days and of course I would also be happy to see many of you and discuss many of you at scad days we have have three days next week for doing that and for now I thank you for coming thank [Applause] you wow everyone thank you so right now is going to be question time so anybody have any questions so is this new distribution approach how do you see the deployment going to happen will will be always built one assembly jar or will will be somehow partition build this you know distributed T into separate components uh it's it's largely in it's largely orthogonal to that so so the idea would be we will generate class so on the jvm we will generate class files as before and the class files will have instead of of the so-called Scala signature annotation which essentially gives you the symol table it would have the full tree so for for a user initially there's nothing to see so the the tree is just handles the separation the the separate compilation aspects like the signature annotation does now but you can do much more and more with it that means when you put this into a build tool like s spt they would be aware of that and they would say well I get a version from Maven which is incompatible with what I have I can detect that with the hashes and essentially locally change your um your u local copy in the caches to be compatible by recompiling so that would be one scenario to do it another scenario might be to say well I could have a service and I could have essentially the right versions for all platforms for all combinations on Maven uh there would be a thing that compiles these things as they come along uh I think can imagine one or the other and I'm not sure yet what will will win I personally think that the local caches idea is better because it doesn't lie on sort of a critical rely on a critical failure point but uh other people say well maybe it's we integrated with with essentially the maven Maven Central or something like that but whether you build a single jar or just leave class files or use let's say Java 9 modules is completely orthogonal to that that's sort of the output format yes yeah might there be a way to thank might there be a way to external to export these typ trees to for instance use them in DSL development uh yeah absolutely so uh I actually have U I sent to uh uh Doty internals uh the the link there's a Google doc that essentially is a specification of this tasty format and the idea would be to really make that available as as a platform like bite code format to say well if you want to like consume uh Scala Scala modules or generate Scala modules then that's how you would do and there could be tools that generate them uh which are different from the compiler uh so for instance I'm not sure whe whether dsls would go directly to it but one one idea would for instance be staging so staging is this essenti program specialization depending on one input where right now we have the problem that if we do that and I mean there's a there's a at epfl and Stanford we had a big staging problem uh project that that does this or companies like who are why who are doing this now uh the problem is what when you do the staging you need to uh generate again a program and right now you would generate scalar Source but of course that's fickle again because you have to be sure you have to write compilation options and things like that so I think staging tools would generate these type trees and just serialize them and be done with it and all sorts of program Generations let's say um code coverage you want to instrument your code you wouldn't generate Source you would generate this one yeah yeah there was a question over here oh take take the take the one in the back first yeah yeah so you first likey long time back now so like how much of a f Ty is it versus like how like real do he actually be uh well we have a essentially we have a uh we are we So the plan is to have something that we can release an alpha version so full of bugs no doubt and things like that by next scolar days so the Amsterdam scolar days which is in June so that's the current current plan here yeah uh that we can can can do something it is open source it is on on on GitHub it has quite a lot of activities uh so we have we're very close to have a working back end so essentially this shared back end which is the same as the Scala C back end uh is taking life it will be Java 8 only initially uh there's initial work on a JavaScript back end uh and uh the last thing I was doing was precisely the serialization format so that we we have that now and that will give us essentially interoperability and separate compilation and that's sort of the big things we needed to get in place to to get there what sorry how deep is it well all the way to the leaves so uh so so everything there there was this issue to say well maybe you don't but because of course the tree is Source you can get the source back so maybe some somebody doesn't want to ship the source is that should that be possible and I think it yes it will be possible so you can say I will stop generating code at the method bodies because you don't need that for in for for separate compilation you say you will ship the the Cally the the headers of the methods which give you the types you need that for being able to call them but the bodies you will you will essentially hide and that will make the trees more Compact and of course you won't be able to see them well you can always decompile it javaby code I guess but they are up for skaters and things yeah yeah so for so essentially uh for Java nothing changes so that means Scala can still call Java and Java can call Scala as before because we generate the same bite code as before but it's true that the Java client cannot profit from the essentially the automatic binary compatibility upgrades so essentially if you do a binary incompatible change to a to a scalar library then you will have to manually recompile the Java clients there's nothing we can do there I'm I'm afraid jav depends oness to Cod can iile it on the synx G and us and recile my Java with this new yes that that's actually a thing so if if you if you have a build no I have to correct myself if you have a build tool that essentially uh yeah can can can do the same thing of course yeah yeah yeah but if right yeah no no you're right yeah I was where do macros fit in this picture and if I change a macro will I need the source to pick that up or can I get away without that now you'll you'll hear more about that also it's galad um so uh so essentially that this has big ramifications for macros as well so um the idea is that macros would use the same trees uh for the expansion and so macros is essentially code that expands code right so the first thing is well how how can you essentially expand code that's in other compilation modules and compilation uh units and it would use the trees for doing that the other thing is that the macro to work right now for the macro to execute we have to use Java Reflection so essentially a macro is some scalar code that modifies code and it's executed using Java Reflection so that means there's an invoke uh method to a reflective call to the micro code and then the micro code will run that has several problems so the first problem is you know because that is so the macro must exist in a project that is already compiled before you can use it because if it's in the same project as yours then the the bite code doesn't exist yet you can't execute it the second problem is a problem of security because you you run arbitrary Java bite codes reflectively uh that macro could take forever it could do could do very bad things to your code but it could also just out of like bugs or things like that take forever or use infinite amount of memories and cause your compiler to crash so so one one example which I saw there which really scared me was that the play team had this great idea to say well we have a macro that uh when you when you compile uh a query then it will immediately go out to the schema and validate the query against the schema and give you immediate feedback whether your query is valid according to the database schema you have great idea right but only the problem is that this was done on micro expansion time and you should know that in an ID um the compiler is run and macros are expanded on every keystroke well those guys Ed use sublime or something like that not not eclipse and so it meant that if you use that thing that macro in an IDE then it would be like molases on every key it was a database request they would sort of come back with these things so it shows that uh sort of having un unlimited code running in macros is actually quite dangerous whereas what we want to do now is for these trees we want to actually have an interpret that would be The Interpreter that interprets macros and then we can bound the times we can do all sorts of things we can give it limited memory you can you you can have it run essentially in a fixed amount of time and stop it if it takes too long and so on so that's that's what what what uh the what's in store for macros here uh I think it they will be overall then much safer to use and much more flexible to use as well hi um is this the first step towards a Scala VM where you divorce completely from java and rifi generics um I'm not a fan of rifi generics um just to say I I think RI generics in a VM is a mistake uh and I think the the net people found that out um uh the problem is that in a VM if you have a VM you you have to to keep things very stable and a VM also in a VM you want to be able to be a backend for many languages so the problem with RI generics is always you have to burn the type system into your your VM and as long as there's no universally agreed on uh concept of what this common type system should be that that's just premature and I I don't believe we will arrive at that over the next 20 years so I think maybe in 20 years it will it will will be be time time to do that but for a VM well I think I'm I'm I I I think that's I don't really see the point of it because they're perfectly good VMS out there you will get in effect something like a platform so an so an interoperability format an interchange format and it will use essentially other VMS as the runtime but it will be sort of one level above above the your typic notion of VM I don't really see the necessity to take this and go go down to to Hardware there is some some work very initial stages on on having an llvm back end and that then you could maybe call a Scala VM that you say well you have a lower level VM that you map to and and and do that but it's way too early to to tell whether that that will be successful or uh yeah or to bet on that so did your treat format actually require a special syntax to do with macros or was that just something to figure out from the basic preformat Oh you mean to execute macros yeah uh no that that sort of falls out because the the tree format simply says where we want to have a complete way to express what Scala is and macros are nothing but scalar code right so of course that that means that what we need if if you have an interpreter for the Stree format then by definition we have an interpreter for for for for macros because macros are just essentially Scala code that we have in that tree format reason forces that we talked about s compatibility binary compatibility and really Fe tree compatibility next oh yeah yeah there there's no silver bullets that's right wherever you are you you will have a you will have a some some tradeoff this compatibility I have some hopes that the situation will be relatively much more stable than than in the other extremes you probably would also be able to uh optimize it'll probably enable optimization when you compile let's say to a single jar you would be able to strip unused code and maybe in line even put from other libraries yes optimize a lot more than you can indeed indeed yeah yeah yeah yeah that's right a lot of optimiz stations like this are actually already being done on the Scala JS uh which relies a lot of them both because you absolutely have to minimize code you can't ship multi megabyte JavaScript files and secondly because actually turns out that the JavaScript VMS optimize quite a bit less than hotspot so essentially you have to do more in the front end but a lot of these things have been pioneered and have been shown to work very well well on the JavaScript side so just to give you a thing that in in the JavaScript benchmarks there actually quite a few benchmarks in the standard Ben Benchmark Suites where Scala JS is faster than native JavaScript because because it can do these optimizations thanks for the great talk actually I was going to follow up on the new's question so uh you mentioned J two you mentioned one kind of benefit schol has Will derve from this are there any other implications for scholar s or scall on other platforms rather than G so what what other things could do what other platforms can do with these trees what other platforms well it's it's sort of well one thing you can do for instance is to say let's say you want to do a scalar specific code coverage Tool uh what you can do is you can write this tool once have it work on the trees and it will work automatically on JS as well as Java so that that that's a big way just to give you one example you can you can have more and more things that both in the libraries of course you can publish for and be platform agnostic uh but you can also have tool tooling that is platform agnostic and that way should that that that should make it more interesting how much of the compilation time is currently in the front end versus the back end uh the it's currently about 40 60 40% front end 60% backend uh but that is uh so in in the current Scala C compiler um what Happ so as you've seen uh the back end has a lot of faces that uh transform these trees so essentially there's a lot of tree transformation going on and the trees are have types until we go into bite code Java B code or JavaScript uh so there is this big issue to say well how do we maintain the types in the trees very very early on in Java at the one point x thing we we have said well every transformation has essentially to compute the correct types in the trees that it generates it turned out that this was completely unmanageable there were way too many things things could go wrong so in the Scala 2x series what we did is essentially we ran the type Checker over the generated trees so the so you generate a tree and the type Checker runs over it and it w it won't retype check the the trees that you got from the previous phase but everything new that you generate the type Checker will run over it so that gives you sort of this uh weird uh relationship where you say most of the code is most of the time it's actually spent in the type Checker but it's also in the back end because it's the type Checker running in the back end essentially recompiling these transformed trees and the trees become bigger and bigger so in current scalar C the trees become very big because every closure becomes an inner class and a class is much bigger than a function so that will be fixed in 212 when we go into Lambda so we don't generate these classes anymore but there are also other things that essentially everything you do just makes the code bigger so the code the Tre become bigger and so the compiler has more to do the type Checker has more to do to type check them uh so in in Doty what we going to do is we have this very simple type assigner which essentially can recompute the type without running a full type Checker because it turns out that a lot of the things that that you would that you do for the type checking you don't need to do anymore so it would be unreasonable let's say that you compute trees that re rely on overloading resolution or implicit search you could say well you should be specific in the tree that you generate here so you don't need to do that so that means that hopefully you will spend a lot less time type checking because you will have a lot less work to do but I can't give you the the numbers yet because the do compile is full with instrumentation code and we haven't really done any any serious performance measurements yet we have to rip out the instrumentation first good any more oh okay yeah so since you've changed the I mean you separated the the front end compilation you know from the the back comp generate from the scolar tree could you basically have I mean you can have a pluggable surface sytax for scholar you could yeah yeah that definitely would make it easier uh but we would sort of run into the same situation that let's say people have on the net platform so if you have a very rich interchange format with the types and everything then essentially what you can do is really having a different surface syntax but the types would all be essentially the same so it's sort of just different different skins of the same on net people say well essentially every language is essentially a CP in Disguise because that's what the VM demands and uh and I think we would have we would have a lot of maybe scalar dialects that could use the same thing but uh uh there would be limits to what you can do I think but it I I think it would be interesting maybe for experimentation to say well that could actually become a for sort of scalar like language but you want to experiment uh that could become a good platform in the future no more questions well thank you all for coming