Adriaan Moors, The Future of Scala, SF Scala @Twitter 20150217

Adriaan Moors, The Future of Scala, SF Scala @Twitter 20150217

Recording: Adriaan Moors, The Future of Scala, SF Scala @Twitter 20150217

so before I jump into my talk just wanted to say we're all really hoping to see you at fort mason or the busan backlog in amsterdam and as Alexei mentioned there's a nice promo code up there that you might want to copy down for a nice discount there's also a blog that we did on kind of an overview of what you might want to see if this is your first scala days and at the end of the talk will do a raffle for one free ticket to scala days so there's a google thingy up there that you might want to type in or see if you can scan that QR code with your iphone or phone in general or iPad or I don't know if you have your webcam point in the right direction but I dare you all to scan that thing or two to go there and fill out your name and your email so I know you know if your email is worth more to you than Scala days ticket don't you might get an email from from from our team in exchange for a chance of winning one of those tickets so I'm going to close down on that form after my talk and we'll pick a live winner or a pick a winner live sorry english is not my native language does the last time i'll use that excuse of bronze so I work for typesafe so a little bit of history the rest of the talk will be about the future I promise so I got started with Scala in 2007 or at least I started working on this Scala compiler around 2007 I was using it a little bit before that so I worked on type constructor polymorphism that was my my thesis and then I worked on the pattern matcher into 10 and 2 11 I worked on her and build and I refactor at the XML that was you know moving up in the world and so recently been working on our CI so I guess that's what a team lead does so the slides are up there if you want to there's everything that's kind of in a different color is a link so if you want to add me a Twitter for example that would be a link up there I had to do that of course but anyway the slides are up there's lots of links through to talk in case you wanted to do some research on our future because I won't have time to go into all the details also feel free to interrupt me at any time if there's anything that's not clear you want to talk a little bit more about so to me this is is the most important slide of the talk and you know what I want most fiercely for scholars future and what's already happening for the future as you can see here so its growth in the community and these are two slot this this slide shows you the numbers for the contributions that we get from our community and extrapolate from there for usage of course and so of course the most exciting multiplier there is a times three that we got in the commit pie hurts the size of the slice of the commit pie that was taken by the community grew you know three-fold increase so that's that's very exciting to see and we can definitely you know feel that on our day to day when we're reviewing pull requests see that come in so epfl has with with type save taken over more of the Scala maintenance has had time to go back to research as it should be so they're working on dotty and you know actually writing papers and didn't do a whole lot of that when i was at epfl anyway so those guys and girls are back to our research and that's that explains why why the overall contributions have gone down from them although the core team which is about four people at epfl and for a type-safe is pretty stable across time and one more thing to point out in his list is that you know there's roughly half as many commits into 11 astute n so that's that's a big part of the stabilization I mean maybe they're just twice as big every committee I don't know but I didn't do that math but the key idea is that we do want to stabilize scholar releases in every upgrade even you know after a lot of investment in your and your build infrastructure even without that you should be able to upgrade much more easily from to 10 2 to 11 and even more so for future releases so I'll first kind of give you a sketch of the outline of the talk and then I'll talk about the three major releases that I'll cover and in his talk so up next obviously is to 12 and so that release will focus on the compiler we've decided to alternate releases that focus on compiler changes and library changes also with the idea that would be easier to upgrade for everybody if not both of those change lots of a moving target and 212 is very easily summarized as Java 8 i'll talk a little bit more about that and the other really exciting thing is that Miguel Garcia's optimizer is finally landing it's already landing and currently as an experimental feature in 211 and it'll be to default back-end into 12 and that's Lucas on the on a type-safe team that's that's working on you know making that production ready so to 12 will be out early next year sorry wrong key I had to try the latest JavaScript presenting framework of course someone wanted PDFs of this does anybody know how to make pdfs of JavaScript anyway so the next one is code named Ida because we don't want to commit to version numbers because then the rest of the year we'll talk about what it's going to be you know to 13 or 14 or to 12 1 we've done all of that in the past we decided just to abstract from them and we'll just call it Ida so as I as I already implied it's going to be a library release and we're going to try and do what we can to make stew happy although we're still going to focus on removing stuff from the standard library but that doesn't mean we don't want to add more modules and that goes back to the community growth and contribution thing so in 211 we did we spend a lot of I'm modularizing a part of the standard library kind of where we saw the low-hanging fruit for potential contributions and those all have now those one will have standard SBT builds and have maintained errs or multiple maintained errs per project actually so that's that's very healthy except maybe for Scylla XML sorry but who knows so we want more of those and you know validation would be a great example the main thing that'll keep a lot of people busy I I imagine for in that time frame is going to be rewriting their macros I'm not going to be talking too much about that tonight but I have a rule that every talk today give about Mac about Scala I have to say I have to tell people not to use macros so please don't use them their experimental and if you do use them you possibly greater end user agreement that you will be for factoring your macros as we change them because that's what experimental means they're going to change and we mean it I know and as easy as it's been to upgrade from to 10 to 11 for the macros it's going to be a lot harder for the next generation you want to be using Kwazii quotes and you don't want to be casting down to like the internals of the compiler where we're factoring that thing can't want to have both be have the compiler be faster and stable macro a p.i so we chose faster compiler sorry I hope you'll forgive us but it's experimental so we told you so so we're roughly on an 18-month release cycle and I kind of count 12 months between the dot 3 or got for release so we're at dot five now because of little snafu earlier in the two to 11 cycle but so in about a year you'll see 212 land and then a year and at 18 months for that how you doing so I predict but we're already well into the future now I predict that Don Giovanni will have to act and the first one will rework the back end and a lot of interesting ideas are being tried out and proven already as far as the back end goes in indy dotty incubator at EPFL so there's going to be a lot of interesting things coming out of that both a faster back end so I back end I mean everything after type checking which in the end is you know still a significant part but the type checker is really where most of the work is done ya know i think thats that's just how it is there is a i know that they have a staging repository but to be honest it's been awhile since i was last animals a huntin summer so that's when I last really like talk to them face-to-face I know that back then they're working in a staging repository that might be under the same the same project under the same project user but in the end like that's still there official one and the poor requests have slowed down because you know they're also getting to a point where it's a little bit more mature for you know as far as that counts and first year of something so but anyway one of them one of the two of the ideas I wanted to highlight and the introduction here is that face fusion is something that we think is going to help really well with with the currents calico power there's you know two dozen phases that the compiler does to boil it down to stuff that the via JVM can understand and they're all you know spinning out new trees every time and the GC doesn't like that there's rumors have it that scala c is also being used to stress test the JIT because it's one of the projects that take the longest ever before it starts warming up and actually like you know emitting fast coat anyway so we definitely we're paying attention to that and I mean for from a user's perspective or from library developers perspective some recent ideas around binary compatibility they're already kind of being salat solidified in the doggie codebase are going to are going to lend I think predict in this timeframe I'll talk a little bit more about that as well of course and then the second act and the whole idea here is that we really really need to manage this change carefully because even though we have pearl and we have said and we have a WK and I'm sure by then we'll have you know abide fix it or something like that it's still we we don't take that ability lightly so we we are excited about adding new features new modules cleaning up mistakes in the past I mean even Java is moving faster these days so we definitely want don't want to get behind their pace and we won't but at the same time we want to be very careful when we get to changing the language and what it means so most of all it will be focused on removing stuff that is deprecated already now or that will be deprecated by then and just simplifying the internal foundations of the theory of the type system and very little of that will actually percolate to the level of where you write your code and we're looking to put Andrew and Norman out of business with their Scala puzzler series because we just can't have that stuff anyway nothing personal so some more details about about 212 so there was a blog post we do try to you know be more open about these things it's not like we're sitting at you know the office down the street and wondering about what we're going to do and then not tell anybody and then release it so we welcome your feedback and this has been out there for a while and you know this is basically to summary although I guess I can tell you a little bit more about it so I already said it's going to be out in early 2016 we debated it for a really long time whether we're going to you know even do Java 8 and require Java 8 because for the longest time we haven't we haven't revved the Java they were required java version I think our decision in hindsight was a no-brainer now that you see how Java 8 is being adopted but that by no means was clear you know year and a half ago or so when we're at least not to us when we were doing this and to hedge your bets and to help everybody in upgrading 211 and 212 both internally to compile our code base in the library code base will be very closely aligned which is why to 12 is the compiler release as well and so everything that will be in 212 and you may have noticed there hasn't actually been a milestone released for 212 that's because it's kind of secretly living it in the closet of the to 11 x branch like there's flags that will enable the functionality that's going to become the fault in 212 so we also have pads we call it D build and we build about a million lines of community code and we have it out running on 211 so we want to be able to easily test what's going to happen in 212 and that's much easier if you just have to flip it flip a couple of flags kind of going back to you know lessons learned from Twitter apparently so yeah so anyway I think it's pretty important to emphasize that you know the differences between major versions of scholar are trying to be smaller every time so one of one of the key features of 212 and this will line soon into 11 proudly to 11 7 is that we omit the same by code as Java Sea does for lambdas and that's that's basically always been our strategy right you know there's a bunch of jet hackers at Oracle and Twitter and you know lots of other places as long as not us we don't do vm hacking we don't do any of that and we're very happy staying I'm i don't i rarely dare to venture into the back end i stay the type checker that's a that's us down dirty as I again anyway so Jason Zog one of my colleagues is working on having our compiler basically look like Java as far as the jet is concerned and we just we'll just you know have oracle pay for that the fault methods are something that that's very exciting at first sight for you know getting rid of the expensive trade compilation scheme both expensive because of the interactions through static calls to the forwarders and the binary compatibility story that's a bit hampered why can't i add a new concrete method to a trait well sorry it's actually an abstract method in interface so we don't know yet for sure how far will be able to push this because traits are quite a bit more powerful than Java interfaces Java 8 interfaces with the vault methods I mean but we're definitely going to do it for all the function end classes and most likely we will be too lazy to write the bytecode or by hand or and when only not going to port that to Java so the Scala compiler is going to have some support for emitting traits or interfaces with default methods I don't think it's going to happen for all traits there's probably going to be opted in this is something that I worked on and a type checker and that's been in 211 since 211 for I think I bumped it a little bit 4 to 11 5 so this allows you in addition to pretending to be Java will also give Scala developers an easier time to call into Java 8 functional api's so of course java couldn't be bothered to reuse our function and traits they have to come up with their own solution which I love by the way I think it's a great solution but we'll have just have to go with the flow there and so the type checker will just synthesize the right classes just like Java Sea does so our interrupt with Java 8 goes both ways as I guess what I'm saying it is really long winded way wildcards make everything hard and I hate them but I'll figure it out eventually how to do type inference in our setting to to to make that work in general horrible anyway I'm not going to go too much through to code but that's why I put up the link i just want to do show you real quick what i was talking about before so I don't know if I can easily zoom in here is this somewhat readable from the back or not at all yeah that's ok ok so what you're seeing here is you'll just have to take my word for it because I don't have a fancy rapala embedded in this talk but you can look at the gist or the code of the talk so when you run Scala to 11-under X experimental it'll it'll enable this particular feature that will be on and default by default in 212 and so you can just call in to stream and that is java.util stream not our own stream or java.util strange stream matta factory so you can just you know make an array from a list commented out here is the Java code that looks just a little bit nicer to my Scala IDE and in Java but it's basically the same the same stuff right and so basically the idea is you know we just want to make sure that you interrupt alright cleanly with Java 8 this is where the wild cards come in this is something that you won't have to do eventually but type inference sometimes gets into trouble because it didn't furs wild cards for everything because Java doesn't have definition side variants but most of the code just carries over straightforwardly and I'm not I don't intend to go into the details good it's just the idea that you can write Scala code that uses java api is in the same way as a Java 8 programmer does and so the way that works if you're curious behind the scenes if you write the thing up top what the Scala compiler actually does is the same thing as a Java compiler it tries to figure out what is that interface that they're using in this in dismay that call to represent a function let me create an anonymous subclass of it and implement the single abstract method in there with that body that's lifted out so i think it's straightforward to see what the correspondence is but behind the scenes a lot of type inference and overloading resolution and wildcard capture and all that stuff needs to happen but we're just shooting for a par with java when it comes to calling their methods so to kind of illustrate the pain and and and you know I so we love the Java 8 provides better support for functional programming roles are confident that Scala has a better approach and many levels to it and I think this is one of the examples I mean this is not the true signature of map in the collection libraries I know there's a can build from there but imagine tacking that on to the signature that job I uses and my the main bein there is that since you don't have which probably shouldn't be pointing at the screen only I can see so up here you know in case your Java is rusty good for you these are wild cards you know and that's a function type and this is exactly the same thing semantically in Scala and I'll show you why so I'm just going to spoil the surprise I hope you won't mind so we define the tea as contravariant and the result as covariant and that means that this is what the library developer did once in 2004 or something and this is what the users do all the time they just write the function type and this is actually the same as argh arrow raz because you can write types and fix you can write your own types like function that you can in fix that way and if I was going to write the the Java equivalent in Scala syntax I wouldn't write any job invariance annotations here all interfaces are assumed invariant but then the price paid for that is every single use site has to treat this very very carefully and say well this is actually meant to be a contravariant one and this was meant to be a covariant one so you push down the pain to all your users you know I never think about variants I just write pluses and minuses until it compiles I'm serious I want its word a thesis about variance polymorphism and then just something short-circuited or something and Suzanne I just don't want to deal with variance anymore figure it out once a new compiler and then just let the thing let that do its thing you know computers are much better at fiddling those math thingies than we are so that's the point that I'm trying to make is library designers don't even have to understand this don't feel bad if you don't if you feel if you understand it that's great I mean that that helps you but the point is even if you don't you'll get it right eventually pretty quick pretty quickly if you you don't the great thing is in Java you don't have to think about it at all but then everyone else is going to think about it order types won't be compatible or they won't be subtypes you won't be able to pass in something that actually relaxes the requirements on the argument and Java will say but wait what this is unrelated so I wrote down some more stuff but I'm not going to go through it so you know the subtyping rule reads really naturally when variances at play I tried to read the papers it's been so long I tried to read the papers on on what actually existential subtyping is with bounded polymorphism I just had to write a talk so I just couldn't anyway this is equivalent so when you push this down you know in the end you get the same type subtyping rules on your function types if you're careful and have all your users right all the wildcards everywhere we just don't want to do that to you all right so this is getting farther into the future which is signified by the upper names we also blogged about this on scalawag so I could encourage you to read that but most of all it's a simplification and I think one of the lessons that we've learned with the collection library well we learned many lessons one of the lesson that we learned was that inheritance isn't always the best solution for for extension and I would have to sit down for quite a long time to make sure that I you know implement it traversable like correctly for my for my own collection and that's just that's not how it should be and we intend to fix that in Aida mostly by making everything final so you can't extend it and then providing cleaner hooks I'm exaggerating a little bit but there will be some nice clean up stir and and the moat the main examples will be parallel collections which give rise to this exponential blow-up in the parent the parent traits for all those collections with jency against Juan those will be moved out and parallel collections we're very rarely you want to you know you just it it's kind of going back to what was said earlier I think was to about you know overseas FP and extension methods versus overriding and dynamic binding and for parallel collections you don't really want dynamic binding you just want to have your splitter a turd like Java did it and then have and enrich or pip on whatever you prefer some parallel goodness on top of that in the same for views and the way that we that we plugged it in overriding all the right hook methods makes it really hard to to inherit things and just blows up the the subtyping diagrams so caveat inherit or things are going to change in your parent types but aim is that most users won't notice like there is really no need for you to be extending collection classes at least not you know in the middle there at the top and so we'll first go through a deprecated final deprecated non final cycle and so on so if you if you give heed to deprecation warnings you won't notice too much of these changes and in fact we did a lot of that already in 211 ok so call for help which I didn't adequately convey the beginning of my talk so we're really thrilled with all these contributions and we want more of them both on the you know submitting pull requests and review info request or coming up with new modules and and maintaining them and seeing where that takes you and if it proves out you know we'll be happy to include it in the distribution but hopefully that'll be as easy as just adding a dependency to Scala library hall one of the ideas with the 211 modularization was that we did a couple of modules we made an svt plugin to do them and so now we help other people will start developing modules like that and they'll fit right in and we'll add them into the blast selection of modules so what we're planning on doing is is lazy collections probably something with Java streams or maybe something with transducers or full transformers or whatever you like to call them are they cata morphisms who knows anyway so that's one of them that's going to move out parallel collections there's those are great scallop blitz project by some vfl grad students and actually Java 8 pretty good job we've been comparing the performance of just you know implementing a splitter ater for all the Scala collections and then benchmarking power versus dot splitter ater and then using the the Java 8 api's and it works pretty well so we're probably that's another thing we're going to make sure to play well play nice with Java 8 and validation is something that was the Scala team at type save twice the size we would have long done but we haven't gotten around to it yet so that's one of the things that is also an open invitation to find the right mix of scholars eNOS and well the opposite of scalzi I just want to re-emphasize that you know as I said before macros are going to change so Scott lomita I mean there will likely be a talk at one of the Scala days near you about it it's going to be a big rewrite its I think it's a nicer approach than what we have right now but it's going to be big change you know don't tell anybody I'm just really trying to take this case strongly so no one will be disappointed at the end so um yeah I think I'm kind of confused by these hip JavaScript frameworks yeah okay so that that was that for I've never done a two dimensional presentation before there's feel like I'm in a video game I have three options I think I'm gonna go down okay so i hope no i don't know how scary is waiting there so now enjoy body where I mean it's really cool to see if i'll do more research again i was there when we're really crunching to keep our big industrial users happy even though we might not always have succeeded doing that so now they're back to doing what they what they what they love and what they do well and so there's a lot of good ideas i'm dottie debt will use this inspiration to a factor the Scala compiler yeah so but like I said before we're focused on on some migration strategy in we're very well aligned with EPFL there so we want to drop procedure syntax very heated debates about that because its syntax but you know I really like being able to say it and in Scala all definitions are key award named signature and then equals body or nothing you know I really like that regularity it helps with code formatting you know actually doing it refactoring like this with said could bite you quite a bit I don't know what exactly right i don't want to see the size of the reg ex you come up with two to migrate from from procedure syntax to equal something but anyway there are some cool corner cases you might run into xml literals one's the single biggest selling point for scala are probably not going to be there they probably won't make it into the next decade anyone ever use early definitions i'm just wondering early definitions going once all right well you just sealed their fate they're gone so instead you'll get trade parameters which is a nice generalization and something we get asked about a lot who I don't traits have them well I just we had early definitions why did the thieves oats anyway please don't use them they're going to go away so simpler semantics I mean that's that's exciting further research and maintain errs I think personally if I was trying to sell the skylab grade I wouldn't lead with that I don't think pragmatically that's the most important thing I mean the most important thing is that your compiler will be run faster your documentation will be better you're tooling will be more stable and so on and then maybe somewhere deep down there you know we have a really nice theory but you know theory held r is for those guys across street so anyway and dotty there will be no more type parameters and all that crazy stuff will just have tied members members all the way down so we're really going to be object-oriented you know none of that FP Universal quantification blah it's all going to be members and that was actually I mean just to give you the background um I seem to be you know down all this this is what i wrote my thesis about oh just so we're clear there since then ideas have refined and evolved and you know crystallized and all that good stuff so it's coming too it's coming to a Scala compiler near you and it's already in a dotty compiler I won't go into the technical details but it's pretty neat and I'm sure you've all seen stack overflows into Scala compiler or ginormous types as a result of you know matches that were just a little bit too precise precisely typed I mean so we're basically going to delay those blow ups and introduce union types due to model them and there will be support for something like what you're asking for a type-safe records the compiler will will just say well we'll integrate a mini mile seven or something and so Martin recently gave a talk about binary compatibility he was standing right here he autographed pictures afterwards I don't expect that's going to happen today but that's okay I'm happy to stand in his shadow anyway so binary compatibility the idea is there they basically every class file every jar will ship with the source not exactly the source though the source after type checking so the source that is insensitive to changing your dependencies or insensitive to the exact nature of implicit search which we don't dare touch during minor releases because it might change your types so everything will be elaborated everything will be horrible you won't want to write that kind of source but it will be so specific that you can just rerun the back end really quickly and recompile everything if you're lazy val encoding changed let's say so that should speed up things like pants or D build and it should also speed up just in general evolving libraries because you will be able to add you know change vowels into lazy vowels or add stuff to trades as long as its source compatible it'll be binary compatible by definition or not by definition but construction and in Java that's I think one of the main confusions that are you know people are so used to thinking well its source compatible so it's binary compatible right and that's one of the pain points in Scala because all of all the trickier that we need to go through to make all that stuff compiles to buy code is that there's a huge gap between what what you can do source compatibly in where you can do binary compatibility and so this will resolve that with the smallest investment possible okay well thank you happy to take more questions comments give you a chance to fill out that form and which I'm going to shut down in a couple minutes and then we'll pick a winner but yeah no more questions no one trying to delay the okay attention to questions alright cool so one is when Martin Amis that nitro recently he got past the vault you know when and of regular more to endorse it electro complicit and she said that you know this as any feature it's going to be infra noise later yeah she says in da team loses secondary blaze bowler hats and everything is going to be used with the blizzards and the Blues are composable so what's your take on the future bliss is my second pitcher is when the date of Don Giovanni cathartic falls into hell and Greg's don't join with him so what's it going to be in the money okay let me answer the second question first there's a shorter answer I haven't seen the movie or read the book or I don't know what you're talking about I know I'm just kidding I haven't seen that one either but yeah so anyway I'll give you a longer answer to your second question i'll go back to the first and a mean Tom well I think about the second question so the first question was about implicit san mo Nats and I'm trying to reverse engineer de question because I think it must have been about how you deal with effects or how you do like an effect tracking system and whether you want to do that with frame 0 nets and you can composing them and so on or you know you want to pass around tokens that say you can do this or you have done this now you must carry on the taint and that's kind of what implicit SAR right there like a lightweight mechanism to either give you permission to do something you can't call this method unless you have this magic implicit that there's no other way of creating or you know when you do some side of side effect from then on you you will have to actually you will have to consume that token I guess what I'm saying so you know that's very much in the research stage I've talked to martin a couple of times about this but i don't think it's more concrete than thinking about how how you might model you know this idea of having capabilities passing them on and consuming them which is very close to affect tracking and kind of what mo nets do in a way but you can they can they do anything to can make them do anything right so and for the second question could you repeat it every on oven okay yeah yep my question is why I didn't mention SPD at all mm-hmm so so as big as a horrible so even though sometimes the jar files are already in the idea be possible to keep evolving in time so you and the inner top you also mentioned about your using pen sir I don't know so any time this week a convoy mm-hmm oh yeah thanks for your question so the question was about SBT and slowness of resolving dependencies so recently Eugene I'm not sure which is the T version at landed in did some work on cashing the graph for the dependency graph and cutting off like 30 seconds for resolution because it IV had to do lots of crazy stuff to build that graph over and over again even though we could just cash it and then just quickly walk it but one of the reasons that haven't talked much about SBT is that that's a different team and we don't talk to each other no I'm just kidding I'm kidding no SBT I think I haven't talked about it is because it's just pretty stable like we we have a where where we like where we'd like it to be it's going to go 10 pretty close in a pretty quick near future sorry and I think it'll just it'll just mature I mean I think there is not a lot of exciting stuff in spt's future except usage and happy user happier users better better documentation better performance but that's about it i would imagine yep question the back so the questions about SB T&D build integration so d build is is is to SBT what pants is too i guess aunt or something so d bill does is kind of an svt plugin that does a lot of really nasty rewiring to mimic google blaze or pants or something we're using SBT build so basically you have a bunch of links to repositories that all have SBT builds d bill forks all the repositories over i like injects its plug-in into those SBT builds and then rewires all their dependencies to go to be recompiled from source instead of fetched from from maven or or whatever yeah i mean d build is is is great for us and it's also open source and i think it's kind of from the same time than mine says BTW like we don't foresee any major new features for that but it's definitely something that we want to support other questions all right thank you Oh