SF Scala Guillaume Martres 7 26 16

SF Scala Guillaume Martres 7 26 16

Recording: SF Scala Guillaume Martres 7 26 16

[Music] let's see okay hi everyone thank you all for coming here uh this is a very tightening kind of torque very intimate very nice uh my name is gilm I'm a PhD student at UPF uh and I work on do and today I'm going to talk to you about the and say a bit about how it works and what the sort of things we're planning and and then dive into the internals of the compiler and hopefully get you excited about trying it out or even contributing to the compiler all right so what's the so the is this new experimental compiler for Scala uh that's been uh in development for a few years at first just by U martini and now we are a small team and development has picked up pace in the last year also uh the main goal of the T i' say is to have completely redesigned internals of the type system but uh the language itself feels very familiar uh from like a user point of view um and incidentally because we are kind of free writing everything from scratch uh we get to redesign components that have been existing for ages and that uh kind of work but were not extremely well working for example uh we had to completely rewise the integration with SBT to get incremental compilation working which is a thing that has existed for a long time but hadn't got much love and uh by doing that we are able to get SBT to uh be stop forgetting to recompile things that it should forget and hopefully uh work in more cases uh another example is that we have a new pattern matching exhaustiveness Checker so it's the thing that warns you when you have a case class in a TR and you forget some case in a pattern match and the new one is much simpler and much faster and hopefully it works better but we only have tested it like on a bunch of test test cases from the old compiler so and the interesting thing is that some of this stuff might be reported back to Scala 2 yet because Scala 2 is still in active development and these things are uh completely new design but they're not specific to the new compiler um so one of the thing you're probably wondering about is whe can you actually do things with doy right now or is it just a dump comp and we so as I said we have SBT integration including compilation so that's a big plus and we have very basic intellig support uh done by the intellig FKS uh if you try to use the features your like the type Checker of iner isn't going to help you because it doesn't know about Doty features but it can compile code with doy and we have a a guide which explains how to do that and we have a Repel so you can actually try it out easily and get feedback uh we don't have a presentation compiler the presentation compiler is the thing that uh an ID like eclipse or enzyme communicates with to type check your code and gives you information and errors that kind of things and uh that's a pretty big piece of work that we don't have and until we have that like ID integration is going to be very limited except maybe on intell side where they do their own thing which is completely independent of our compiler but uh that remains to be seen by the way you can interrupt me at any time because I think we have a bunch of time oh I just had a question about the you know SBT integration incremental recompilation are you um also trying to get that to work with other build Frameworks like Gradle um we are not but the BT Forks Have Done Works in uh splitting the incremental compilation part of SBT which is now called zinc from SBT itself so this already exists and it's used by CBT among others and it's getting a lot more work now uh in SBT 1.0 so hopefully I have to do nothing and it will just work at some point uh until then uh you have to put up with SBT or maybe use CBT so I don't think anyone has tried to compare stuff with CBT and but uh feel free to try and see what happens uh so I'm going to try to talk about the future which is always always difficult so Scala 2 is like here to stay for a long time uh it's like there uh sizable team behind scy 2 at lightband working on the compiler and uh they're doing great work and uh ver like 22 is about to come out with jav support and a bunch of other nice things so like shouldn't worry too much about that uh and D itself probably won't be production ready for at least a couple of years assuming that like it's successful I I still think of it as an experiment um so migration migration is complicated but uh basically we like the two compilers to converge as much as possible so so far we haven't tried to compile that much code that much SC 2 code with the we have compiled most of the standard Library already so we support a lot of things and to have to do that we have a compatibility mod where we enable some things that we want to remove from the long AG but but that we still want for compatibility and on the other side Scala 2 is going to implement some features that have been prototyped in inty elect parameters that I'm going to talk about a bit later um macros so we're not going to get macros as they are in scale 2 because macros in scal 2 are way way too tight to how the compiler Works internally they're basically the internal representation of the comp so there's no way that we can get that to work to on D however um the like basically the longterm plan is to have micros based on scal meta and if you're interested in more details on that there's no prototype of that on GitHub by Eugene bco and he has a torque at scad days where he explains that in more details so if you're interested in maos you should watch that uh until then uh we think that if it makes sense uh probably the best thing we can do is to take macros that people actually use like common Pro very popular macros and put them in the compiler itself if it makes sense which like has to be determined on a Case by case basis um we actually don't have a very good idea of which micros are very popular and that kind of thing so that remains to be uh researched and the other thing that could be done is compil plugins which uh are less involved than macros but uh we don't have any compar pluging yet and no one has expressed any interest in working writing them but we welcome people trying that out uh I think that may be an interesting way to do things until we have macros um back to compatibility so we have a rewriting tool or a very basic version of a rewriting tool basically when you compile your codes with the in the SC two compatibility mode you can pass a dash rewrite option and that is going to rewrite your code for you to it's like very small changes it's going to do locally to get some things to type checks that will type check in the and we'd like to get this to cover much more cases to like cover like potentially every difference we can in type in frence that kind of thing so um the Scala Center is going to have one person working full-time on uh this rewriting tool starting very soon to make Transition as smooth as possible yes um on the last slide just before we get away from theiler ums oops um does doy support um uh as the the same level of uh comp compilation levels as schol C or is there completely are they completely disconnected what do you mean by compilation levels like there's like like 20 or like 15 or 20 different stages of compilation in scaly so are they mapped one to one or are there just you mean the compilation phases like typer ER that kind of things I'm wondering about this the the compiler plugins in particular so would we add us another so um please question okay so the question was about uh compiler phases the compiler is split into a bunch of different phases that Tech has input a tree that represent the program and transform it and so scalac has its own set of phases and the has its on set of phases too and I'm going to talk in more details about that in a minute uh so but roughly uh we have similar things in that at the beginning you have tape checking in the middle you have arure at the hand you have the back end which generates bite code in the middle you have a bunch of stuff uh this stuff is different between SC and the t uh I think I've heard like 50 phases yeah there's a I have a nice diagram like don't be scared but I have nice diagram I show you um what's your take on implementing popular micros as a language feature I'm bit afraid that you know it might well I'm not afraid but aren't you afraid to kind of like grow language to B yeah which is why I said if it makes sense so that's that's a very big if I ask another question who decides it makes sense to implement features in D yeah so we don't have a formal process to implement things in the like scal C has the Sip process uh we might if if it's needed get something similar for doy for Doty specific things for now we put issues on GitHub and people discuss and based on that we do whatever we should do U so it's very informal because we haven't had to worry too much about it if more and more people start wanting to contribute and that kind of things you might need to make the process a bit more formal um for now I think that like we discuss things and R out is okay yeah okay cool how many contributors are there to D how many contributors to to d uh if you look on GitHub I think it says 23 uh that includes people who like fix the typo or that kind of things so but if you're speaking of active contributors I think we have uh five like less than 10 people working on it regularly uh sometimes I joke that we should pretend to be a Jon library because that seems to be the thing that gets the most contributors um are you going to talk about the calculus here nope I'm not going to talk about type Theory all right uh just practical stuff is uh scalar JS how does that fit into so uh scalar JS has a very experimental um backend for the which basically can compile Halo world and so that that we can get it to work and uh for now like like not many people work on SC basically two or three people and they're focused on making it work as best as they can on scaly and they're doing great work on that so uh eventually we we'll have scjs on the but probably not soon until people start contributing because they're interested in that kind of things uh there's probably lots of U low hanging fruits here uh but I couldn't tell you because I've never actually looked very much into the internals of Scala JS and uh also will you have like all the major features at least in some stage of completion maybe bux but implemented your question is about which major features we have now in when we you have like all of the major fees for like the first Milestone implemented so um the first milone was that we got the compiler to compile itself so the compiler is written using like it compiles with the current Scala compiler but it also compiles itself it Bo stps so that means that we have already implemented bunch of features stock on effect system there oh so like like crazy ideas like effect systems or every other things that we have proposed we have no Milestone atory on that yeah but currently scheduled features or there you're just like figureing out how to make it work nice so we are still very much focused on uh making it stable and making it work on more and more code so at this point we have tested it on very little code that is not the compiler itself mostly the standard library and I think that's going to uh occupy us for sometimes and then we uh we'll be able to like do some sort of alpha release or something and uh we focus on like more uh crazy features afterwards yeah because basically the question is if I will have some piece of code and I want want to try to compile it against d uh I want to be sure that it's actually like to learn what what sort of things changed uh I want to be sure that it's all of them for like the first first uh stage so the things that we change when when probably your guess like when we have this Alpha like alha year a year um just very I level like couple of months couple of months maybe hopefully like if we being optimistic more realistic it's very hard to say cuz uh depends on uh what we end up discovering as we try more things uh but like if you try dotin now the things that it's going to like reject the thing that it's not going to allow you to do uh I don't think this this is likely to get bigger like we're not going to break more Cod that we're breaking right now but weely to to try something already uh yes um like taking in mind that it's very alha quality and very early product yes like if you like you have some code that compiles now with it's likely to continue compiling in the future in the future like we compile with do and we hope to continue doing that so that's code well it's a compiler it's it's about 50,000 lines of cut so it's not a humongous project but it's pretty big it doesn't use very advanced typ level features I mean it's not the tri well then I hope you you contribute cuz if if it doesn't use like complicated things should be very easy to contribute to but um yes I don't know we'll see uh rewriting tool yes I think I have talked about that draft features uh basically there's not that much there's procedural syntax and early definitions and I'll like explain what we rep definition with a bit later and existential types written using for some so as an example in Scala 2 you can write R for some type t uh we don't support this syntax anymore because it's all you to do things which are more General than what we want to support but you can still write AR way of bracket underscore is a sub type of f so I don't think that many people care very much about that like who has used Forum here do you think you can who has used the for suome in types in okay do you think you can reite it using white curs or do you use Advan for polymorphism not totally sure I would have liked to write it in that sentence that's a lot less few characters but I gave me type errors when I tried so maybe I didn't try hard enough or maybe there was a BG in the comp but yeah if you can dig up an example that would be interesting but we think that like most of the time you should be able to rewrite it using white card so uh it shouldn't be too much of a problem for any sort of migration uh implemented features that's the cool stuff uh you can try right now because uh it's actually implemented and not just a planed or just uh an idea so intersection types and un Union types are a NE thing so basically you have two kind of things you have a and b which is a type it's the super type of all subtypes of both A and B so if you have a class which extends A and B then it's a subtype of A and B um if you have tra or whatever Tak s and be the same uh it's similar to the existing type which was a with b and we have a or b which is the typee of things which are r a or things yes I once had the case but I want to but match against thejunction and to use in the match I'm going to talk some fields which are common in the intersection I'm going to talk about that again because Bel was fine with that but not color compiler so uh what's the difference between intersection types and compound types okay so I are a somewhat involved example where you have an a a b something which has an a something which has a b and you make a subtype of has a and has B so it needs the get method which is both A and B and if you look at the type you get in Scala 2 the type that it in first it's has a with has B bracket and get with a with B why does it add the bracket with the get a with with v because has a with has B um is not not commutative and like the type of get in has a with has B so the first that thing is just the type of get in B so it's less precise that what you want and like if you switch things around if you do has B with has a it's V inverse so in it just has a with and has B there's a typo here it's has a and has B Not A and B and has a and has B has a member get and the type of this member get is A and B which is basically the thing you'd expect like when you do an intersection of two class the numbers the typed are the intersections of the members in each class uh Union types so Union types are cool like you can have an if which returns an INT or string and it type is in or string and then you can pattern match on it to get a thing out of it yes does this use reflection NOP okay one sec I have a different question yes uh is this like a intended to be used to implement like uh algebraic data types um still like SE trade in case classes algeb data types are still still tra in case classes uh you can use that when you have something which is kind of like an algebraic data type but uh it's not a seal trait which happens sometimes and we might uh I didn't talk that but we might Implement an alternative syntax or something for algebraic data types but there's no concrete proposal yet for that sure is this meant to implement JavaScript uh it's actually probably useful yeah typescript as Union types they kind of pioner that one and they do nice things with them so it's probably useful for JavaScript uh and what is interesting is that this is safe like if you do your match and you forget to another case you're going to get a warning because it's like a Sil trate you know all the cases that's very cool uh question yes I'm just wondering the type of the if if expression I don't know how it previously thre an error no so previously you got the least upper bound of in and string so that's the type which is the super type of both of them and in this case it's an which means that you can't do much with it except compare it with for referential equality um but the other cool thing you can do on a union type is you can select it so if you have two completely different class A and B with a member size with some type and you have an X which has Type A or B then you can call X do size and this does not use reflection we just do an instance of check this is just equivalent to that code so if X is an instance of a then we Cast X to a and call size on it otherwise we Cast X to B and we call size on it so you pay the price of a branch and in is pretty fast as far as I know on the jvm it's just checking like a flag on the header of the class so um don't be afraid to use that uh but yeah we don't infer Union types by default so that means that uh in my previous example if I didn't put the type ascription in string X will get type any uh we don't do that for basically two reason like unit types can be too precise and if you like put some have some value with a union type and try to use it later you might get some very complicated weird type error because uh things just don't match the way you expect them to match um this doesn't happen often but but when it happens it can be confusing so we turn that off by default the other is that if you have many Alternatives like imagine you have some uh pattern matching things where you return a different kind of subtype every time and you have 40 of them then you get a huge type and then the compiler has to do subtyping check so if it wants to know if A or B or C or D is a subtype of something it might have to do 40 different sub typing checks then uh that can that can be come wild so we we don't do that by default however for now if you want to play with it there's a a language import that basically changes the the compiler to keep the union types instead of replacing them by their upper bounds uh yes have you considered removing the any type uh and replacing it by what nothing yes because HC does not have subtyping [Music] uh so why do you hate the any type uh just results in some like bad errors sometimes mhm like poor inferencing because things get inferenced to any where in hosital you will get an error at the time of like where the error actually was in Scala often you get the error like in a in a different the question is uh have you considered removing the union types because the any type uh because sometimes uh you like have an error in your code that results in getting any when you driver have a compiler tell you that you're doing something wrong now instead of uh 10 lines later uh no we're not going to remove the any type because it's still very useful uh but uh I don't see why we couldn't have some uh linting features or something you can turn on to warn you when you infer any type um and maybe maybe we can infer unions in some places where it makes sense and have like less Weir messages in that case uh it remains to be seen like again if you have like interesting examples of things that break in weird ways uh can post an issue and we can see if we can think of a better way to do things great solution to that like saying like hey like I heard any here and I got a weird error but here's where the error probably was so getting the the typ Checker to give you more meaningful error messages is a very interesting problem but also very hard problem uh I don't know if we'll get there but uh hopefully yes like and does very interesting things with error messages uh where it tries to give you as much context as possible I think we can do more we can do similar things in in doy uh other feature trade parameter so they replace ear definitions and if you don't know what ear definition is that's okay you can forget about them and never care but basically uh when you have some tra with like um side effects when you create the trait like this print call and you have a class which extends this tra if you try to uh call newb uh at the moment when the print is going to be called X is still null because the construction of the trade is called before the Constructor of the class so all definitions are this weird syntax that allows you to bypass that by initializing things before uh calling the trade super Constructor uh we don't support the syntax in the and then intend to support it instead we have trade parameters which are basically like class parameters so in a class you can the primary Constructor of a class is just the parameters you pass you put after of the class uh and the name of a class and in the you can do the same for traits there's a few restrictions because uh of the way traits work you can look at the uh sip 25 document for more informations but basically um it does what you want most of the time well it does what you want or it's going to give you an error because what you want is not possible uh yes so should we use tress every time except uh basically if something is abstract us a trait if something is concrete us a class uh abstract class is only useful for Scala for Java compatibility that's the way you should think about it uh this is especially true uh now that we are based on Java 8 where tra have a much ler uh impact on performance and uh bite code size than before when we had to do a lot more work when scal had to do a lot more work to make traits work on the jvm now it's much simpler uh literal based Singleton types are the fun thing so uh the type was of one is int but one has a more precise type which is just one uh in the same way that if you have a value x uh you can x. type means uh the type where the only value is X and that doesn't seem very useful like that but you can do fun thing with it like combine it with Union types and other things to make a list where you can have a string or one and if you go look at the SI sip 23 you have examples of more useful things it come ups for example in in JavaScript when you interact interact with um JavaScript ipis uh which uh take string as argents and there's a new fixed set of strings that we take but that kind of things so this is compile time checked uh yes uh all of that is compile time so one so for example if you say list on the right hand side you say list of two yes uh at what time would you have an error you'll get an error at compile time because the type of two is not a subtype of the type of one the type of one is a is a subtype of the type of int so you don't have to care that the most precise type of two is two don't those tyes either uh yes so singl ton types are also never inferred uh because uh if you do that then terrible things are going to happen because you're never going to be able to for example mutate anything because everything will be uh have it's a type which is equal to exactly what they are so for string L yes yes we have string ler types too any object like case classes type precise type for this case class uh more precise than what soe like case class for the field which value is one uh no that's an interesting thing uh we don't have that uh but uh I'm going to is it just for primitive types or yes it's for uh literal so are integers strings floating points symbols uh that's it and also for every Val so if you have but that's that was already in Scala two like if you have an X it's most precise type is x. type um yes if you have the keep imported and not the type will it sorry if you have the keeps import and not The annotation type will it infer this type uh so we never infer single tone types even if we have keep unions we're not going to uh not wi and singl ton types uh we could have a keep single ton but it's like not going to do what you want to do like there's no way so uh I think you'll have to live with um type type uh annotations or like there are other ways like for example I think shapeless has a construct which is called narrow so if you do narrow of some value you get the singl ton type in thir that you have to put narrow in in your Cod around your value so we might do something like that or let that be a Library Thing uh so propos features so I'm going to talk about uh things that have been uh uh proposed but have not been been implemented and might never be implemented or we know we don't know but this is the kind of things we're interested in in doing but haven't invested much time in doing yet so the biggest one is probably having an effect system so being able to have a type for purely functional functions and uh have possibility to propagate effects have polymorphic effects the same way you have polymorphic types you can say that you map over something and you keep the effects you had before the map after the map um so this is probably like the main idea is to implement that using implicits so when you have an implicit in scope that means that you can use this effect and the implicit propagates through your program and So currently when we uh use implicits we have to pass values and this values have to to be present at run time but as part of this we like to have a way to have implicit values uh which are only there at compile time to prevent you from doing things or all you to do things but have no representation at runtime so that you have no no runtime cost from using that uh so there's actually a a pool request done recently by Nicholas which starts implementing that these sort of phantom values uh generic programming so if you're familiar with shape CL how many people in this room have used shap okay cool uh so shap allows you to uh abstract over arties abstract over case classes have U uh lists generate like heterogenous list that kind of things so we're interested in having similar capabilities like uh we like to be able to make functions that work on every case class uh that has a certain uh set of types as elements that kind of things uh this is again something that we haven't implemented at all and uh but we think that it makes sense and J Cas shows that it's something that people really want and is actually useful and non nullable types so I'm going to explain that in a bit more detail so null has type null in Scala and null is a subtype of every nonv value class so value class is uh every primitive class like int uh flot and every class that extends anyal so you have a class that extends an eval you can't assign n to it and what we want to do is make it so that you can't assign null to anything by default so if you try to write Val X string you will get a type error and you can combine that with Union type to make the type of something which is either a string or n and we probably want to have a shter syntax like a question mark to make this something that people actually want to use uh do you dislike the question mark or do you dis it's just like making one moment more special than the others and you need a spaceship for lists and and I don't know like elephant for for futures or something cuz well none isal because it's part of um the implementation and it's part of the base language and it's not a library thing it's really a a language thing uh and going for language Java like the jvm but also uh JavaScript also uh everything which uses pointers has a n pointer um so we we want to probably treatability as an effect so you don't want to always use this question marks everywhere of the Union n syntax when you're interacting with Java code or just because you decide that you love n and you you want to use it everywhere so if you want to do that you should be able to do that and have some way to turn on and off the nonability uh and we don't want Java uh become annoying to write so you need some way to uh interact with that easily but again that's uh not something that we have implemented at all yet um the so in Java there's an at not null annotation and we could take advantage of that to avoid having to think that things from java are un label I don't know how many much people use that because that must be horrible having to write at not everywhere standard library in particular doesn't use it very much not a very fun J having to add the annotations to everything um think should be opt in not opt out like that uh okay we're done with the first part and now I'm going to actually talk about how things are implemented so before that maybe if you have any question um maybe some off topic uhhuh but I think about it and are there any kind of like uh systems that don't have n reference nowadays our system without nules yes uh hasc doesn't have nules okay as far as I know there's nothing comparable R does uh so yeah rust has option types and it's I think it's implemented when possible using uh n pointers but it's transparent to the user uh so we could what's Sub in sorry like R I think you have to opt into your type including oh uh option gives an option type you have to write uh so my what I thought was reference is more like if you if you use an option of something in Rust it has no runtime cost because everything is a pointer and you can just represent the non case using a no pointer so option is just uh or and we could we can't exactly do the same thing because like an option of option uh can't be represented using a type which is n or not null uh but we could have like a nice API around the nullability thing so you don't have to actually write null and still get the performance benefit yes um so the compatibility with Java with this opt in or opt out and all like how at the B code level can you tell do compiled by code from java compiled by code and you um enable nullability for all Java functions that don't have that type description or that that annotation not know so the question is can we tell from the B Cod level if something comes from doy or Java so that we can decide what to do about NS and the answer is yes in fact if you just have a b code you like doesn't know what to do with uh the code CU it looks like Java uh when we compile the T code both the and scy do the same thing we add extra information to the class Files about uh the stuff that scy cares about but that is not representable on the jvm so we do know where something comes from so libraries written in in would know that their type uh can be nullable or can can be not nullable uh yes potentially okay okay uh another scary picture of the compiler pipeline so this is all the phases in the compiler uh every uh it doesn't really matter what the names are but every uh line is a compiler phase something that basically takes takes a tree uh which represents code and transform it and as we go through the the pipeline uh the the trees gets more and more simple and more closer to what we can represent on the jvm or another backend until we get to the back end which is the part that emits uh jvm by code uh so you start from the prer which takes your source code and gives you an un type threee and then the typer is going to get you a type threee and everything else works with type three uh the thing in the middle which is the most important is erer rure is the thing that uh is going to type check again your whole program but after replacing every type by a type that is representable on the jvm for example if you have a list of int uh the jvm doesn't know about a type parameter so you replace that by a list and uh you compute the types of everything thing again and so things which are after that beave in different ways of that things which are before that but it's all still BAS basally Scala code just Scala code with some extra constraints and so compared so to answer an earlier question compared to uh scaly uh we still have a typer we still have a Pickler of thing which is going to uh take uh bite code and figure out uh not I mean to take the Scala code and put it in by code like all the annotations all the things that let us know that it comes from Scala um and we still have of course a back end actually the backend we use is the code is shared with scaly it's mostly shared with scaly so we benefit from what they do and in the middle arure is also something that basically everyone on the jvm has to do Java C also does a rure because it has also type parameters which cannot be represented on the jvm yes so regarding the back I heard that do is going to have like a Linker some concept like that okay so the doy Linker is a completely separate thing the principle of a Doty Linker is it's a sort of whole program Optimizer for Doty is going to look at uh every Library use everything and if everything is compiled with the has a lot of information that allows it to do a lot of basically preoptimization to make the code easier to optimize for the jvm uh mostly it's doing speci ization because the jvm has a hard time with um very polymorphic function calls and it doesn't do uh a lot of function specialization so the Doty Linker is another experimental project on top of dty which is an experimental project there's some nice talks about it if you want to learn more it's not part of the itself currently there's a separate Fork of the with the Linker so would that mean that some information that Dr Linko would use would be encoded in j the way uh yeah so the way the Linker works is basically adds various phases in this giant pipeline uh to collect information about the to uh to be able to optimize things and it also uh uses information from uh tasty so tasty is the thing that we put in the uh bite code uh to get basically it's the whole like typee check scalar tree that we put in uh the bite code uh so that we can put it out when we uh like when we use a library like when you use a library that comes from that was compiled by D you basically have the whole source code Type checked in your uh class files which means that you can do uh all the optimizations that you want to do on it how is the f the SC I like why is faster why is the faster we don't know we actually haven't done much uh performance work at all the scy people actually are is it actually faster yeah I haven't actually done benchmarks myself so I'd rather not uh say anything that could be wrong on some code it's definitely faster but uh we have to do a lot more uh benchmarks to actually have a good idea yeah uh but we're planning on at some point spending time on like getting it to be faster uh one thing that Doty does that scy does not do is uh Fusion of phases and I'll talk about that later um because I was talking with Martini few back in a Scala exchange and and he promised me that the SC is going to be faster we yeah so definitely definitely one of the goal of D is to make SC faster to compare I think everyone wants that uh uh also if you're interested in in performance like probably Scala 2.14 so Scala 212 is not released but will be released very soon but Scala 2.14 will the developers will try to spend time to improve performance uh so you might get performance before the uh we'll see how any of that goes like uh Performance Tuning it's sort of of a black art and it so comparers are pretty hard to tune to get because they do so much stuff everywhere yeah you need to be code based to do that actually that too yes so uh if you're on jvm current code Bas isting is there anything else sorry for kind of like mean talking uh like what's the code base right now the Doty code base is uh 50,000 lines of code more or less but do you have kind like libraries or anything else uh we compile part most of the standard library of scaly yes uh most of the collections actually we haven't tried over stuff because the collection is where all the crazy stuff with uh F bound types recursive types and complicated class hierarchies are and I think we comp faster than scy so far um okay so let's get into some more details how do we represent things in the compil so basically uh we take code and we represent that using trees then we attach types to those trees for type safety and a type might refer to uh something an identifier that comes from a source or classier or something and to that we corresponds a symbol a symbol comes from a symbol table which depends on the current scope and basically each phase of the compiler as I said Will T your trees and your types and simplify them until uh you can express them using Java code or JavaScript turns out uh like you it's similar kind of things and all of this is immutable or looks immutable because actually we do a bunch of catching of things because you don't want to recompute things uh so paror is the part that takes your source file and gets you an an type three so if you have some code like for example ob. fund5 so the paror is going to look at that and see that you're referring to some identifier OB so the tree that corresponds to that is just an ident tree and then you're uh selecting something on it it doesn't care it's if it's a method or a value or whatever so you get a select tree which takes your object and select your method on it or your value and then there's an application and in the application you have five which is a literal and that's uh what you get uh once you have your this un type Tre uh you have to actually type check it so briefly and very simplified the way that works is that your going to go recursively down your tree and type check your sub trees remembering the expected type so if you have something like ob. fun 5 where you don't know what the expected type is uh the first thing you're going to do is to try to type so typing type checking is we use this words interchangeability we're going to type op. fun but at this point we know that uh this should be a function something that takes something and returns something else um and and to do that you first need to type OB and at this point you know that uh OB has to have a a method fun that takes something and returns something so that constraints what you can do and cannot do uh so once you're done you go back up so you assign a type to ob. fun based on like you're going to look up fun inside Ops you have type Che op then you type five and at this point for things to type check and to not get an error uh you check that the type of five so int is a subtype of whatever fun text as a parameter type and finally you assign a type to op. fun five based on what's the type of op

fund and based on five uh so what's the expected type business for basically if you have ever tried to compare anything with Scala and it told you U uh like given type something expected type something and an error that's uh what was going on uh we care about expected type because we need to do uh type in front very aggressively so I have actually a whole talk on type in front so if you're uh Super Fan of type in France definitely look into that uh and if the actual type does not match the expected type uh we do a bunch of things we do AA expansion we do implicit conversions so to be able to do these things at the correct moment we need to track what's what are we expecting and when that fails we just uh save that we have an error replace the type by an error and go on uh symbol so uh I talked about symbol but really explain what symbol is so symbol is a definition in a scope so if we look at the code here uh we have uh the symbols we have are a f and X cuz those are all things which are defined and that you might refer to later so uh like in the body of the method F uh we call X and to be able to do that we need to know where that X comes from so you need a symbol for X in the scope of the method so uh what's the type of the symbol of x uh we can just uh assign t as a the type to X in the compiler uh because everything is immutable symbols and types are immutable but as we go through the compiler we are transforming things and after Erasure we don't have t anymore because T is a type parameter and type parameters are don't exist on the jvm so we replace T by object so we can just assign uh T to x uh and we want things to be a mutable so we don't want to have to change the assignment um so what's underlying this and one of the most important concept of a compiler we need a way to represent time uh because each phase of a compiler is an instant of time and we have this class which is a period which represent an interval of time so for example you might have something a Type R Tre which is valid between after type checking but until aure so that's an interval of time uh um so you need a way to say that something is valid for a certain interval of time and when you go past that interval of time you need to recompute it um so a symbol does not have a type instead it has something which is uh very specific to Doty which is called a denot ation uh the notations are circular link list of lazily computed types this is uh as brief as I can explain them but maybe not very enlightening like that basically uh your uh Den notations uh represents a type and at some point in time so if we go back uh to this example X at some point in time like after type checking but before Erasure as type T and f as type T2 t uh so the denotation of X is uh something which has Type T and which is has a period like front end to aure typ to rasure uh so if you go past that in the compil uh the denotation is going to check the validity uh field like the period field and if it Sayes that uh it's not valid anymore is going to call uh the corresponding function in the phase to do uh the type computation and so this is done lazily why lazily because not all the notations are needed at whole period so your type checking might need to go look into some complicated class file somewhere in your Java class path and then after rer you're never going to look into this again so you don't have to do any transformation based on this thing so this link list is lazy uh it's also circular so I should have added some arrow that goes like this it's circular because sometimes you need to go back in time uh which sounds very cool and also is very useful because basically when you're at the end of your compilation like you're in the back end and you're going to generate your code you might actually be interested in what are where my type parameters back uh when I first type check my code because that's the thing that you can uh annotate in your uh jvm by code so you need some way to say what was the type of this thing back in time and so the implementation you just go through the lazy list and go go through the link list and go back to the first thing sorry no it's implemented using Point like just values and it's a it's a handroll implementation of a link list yeah cuz there is stream which is Las it's not implemented using a stream it's it's all done by hand I we don't actually use much of the standard library in d uh we use list we use sequences we use maps but we don't use uh any of the complex types we don't we may use streams for some things I don't actually know but not for that for that we do it by hand it sounds just like this yeah it's actually uh pretty similar to you could maybe do it with streams uh except streams uh can we loop I think so because it's just a function which called when you need yeah you can get streams to Loop but uh you can do it efficiently I think because you have to recompute values every time so so this is like the most fundamental structure of the compiler it has to be very very very fast yes uh yeah uh with these denotation uh Rings be attached to each term in the a or are they sort of a separate data structure from the sytax tree so the tree to each tree corresponds a type this is a mutable and to each symbol there is a denotation and some types also have a denotation but three themselves don't have denotations um I go in more details in just a second so there is one last concept about the internar of of the way we represent types that is fundamental to the it's references so suppose you have just some class A with a type parameter T and some LM which has a type list of t uh what's the type of x. LM it's list of int uh so the uh how does the compiler know that it can just uh look at the symbol so the symbol is LM but is list of T but you want list of in not list of T so you can just uh use the type of a symbol you want the type of LM as seen from the prefix X so you want list the of x.t where x.t is just in and uh so this Asin from thing is a pretty fundamental thing to how Scala is defined and every time you have some prefix and some uh thing inside you need to look at the thing inside as SE from that prefix uh so this is why we have references uh there's two kind of reference like a reference to a term and a reference to a type like you can have x. LM but you can also have x.t and those are types so the type of X dolm is just a term ref of term ref without a prefix uh you select X and then you select LM uh so this looks a bit weird because like that doesn't really tell you anything about the actual type of the think uh so every reference has a denotation whoops and the denotation is the thing that has the underlying type so in the same way that symbols have a Type Hello uh and that's it for the uh front end part uh I don't know how much time I still have you have a full hour if you want okay well I don't know if people want to hear me for 4 hour but I have less than a full hour of material uh uh I'm going to speak briefly about the middle end so basically we have the front end and the back end and the finger in the middle I decided to call the middle end because I didn't have a good name for it middle end so the middle end is the part where you have your shiny uh type check tree and you have all your your uh crazy Scala types and you're going to simplify all of that done until you get to uh basically Java but not really Java but something that you can represent on the jvm um so here's an example of a transformation if you have if you write T star in your code at some point we're going to transform that into a sec of T and when you have so it's t star is a repeated parameter and if you have some method through that takes repeated parameters like one two uh and in the same transformation this time on three and not on types you're going to replace that by a call to uh the SE Constructor on Fu um so how do you how do phases work uh so the traditional way phase work is that each phase is a full traversal of um the tree that represents your code so here's a pretty diagram of a tree so you could imagine that uh the top is something like plus and the numbers are one and two or whatever and you're uh after the end of the first phase and now you're going to do a phase that is going to transform this thing so the way it works usually is that you do things recursively so you start from the top and you're going to uh recursively apply the transformation to the children so going to transform uh your first uh leaf and your second leaf and once you're done you transform the parent uh and you do that again for uh every transform so this is fine but it means that you need to fully Traverse the full tree for every phase and it's not the actual computation which is a problem here but traversing a tree is costly in terms of memory access because a tree is not a single block of memory it's a thing that points to things which can be anywhere in memory and every time you Traverse the tree you're going to blow your catch because your catch is smaller than the size of your tree so you're going to uh have to reload things into catch every time uh so the the minase design that uh D uses is based on the observation that you don't actually need to fully uh transform a tree to start an an overall transformation so suppose that I'm at the leaf and I transform my leaf from phase one to phase two then I'm going to also transform it to phase three and I'm going to do the same on the other leaf and now uh the parent has to work under the assumptions that the children have been transform by the transform itself but also by over transform in the same group so you can't take everything in the compiler and do that because you're going like things just stop making sense at some point but if you take small groups of phases that do different things which are mostly orthogonal then they can operate in that way and just do their transformation uh without carrying but that over phases have done over transformation on children uh okay so that's it and as the last thing that I can do if I still have time is um actually show you how we work on the compiler by trying to fix a bug in real time I found a bug yesterday and I think I know how to fix it I can show you a bit how like how we look at things and how we debug things uh but uh before that we can probably uh answer questions any question yes you haven't T anything about Ty instructor and sorry you haven't told anything about her kind of types in new synx for type Constructor maybe you so higher kind some examples higher kind of types in do uh work pretty much the same way that they do in skyc uh we have implemented uh some of the stuff that is coming to Scala 2.2 so partial unification of uh higher candate types so if you have uh some type parameter and bracket underscore and you have um some actual uh value of type for example in in uh it can match in in with mcore by setting m equals to uh int X where X is the unknown so partial higher order unifications which some people are really excited about in scand and um the way it works in is pretty similar we have also implemented um type lambdas so in in Scala you can make type Lambda so a type Lambda is a thing that takes a type and return a type uh for example list takes a type and return return a type so list bracket into the type and you can make your own uh type lambas in Scala but the syntax is a bit cumbersome you have to use uh refinement types and projections so in the we added the syntax which is kind of similar to what exist in the macro which is called uh kind projector uh using so I guess I can show an example uh if I can remember how to do things those changes are for what 2 x or only the the the higher can type syntax the same yeah so that's only uh that might be something that yeah that might be something that will be implemented in scalac at some point uh I don't have a good example here but I can show you basically how working on the compil f i I'm going to try to do a short demo of of that uh one question about um so type lambas not type lambas um Cur types was not a part of as I 272 but our our type currying is that a potential goal of Dy as well so uh by type curing do you mean exactly being able to specify uh um being able to specify partial uh application of type Constructors uh without kind projection um so uh I guess like defining a type Lambda with a type projection or with a uh kind projection inside of it so you're kind of you're allowing multiple um type in fact that was fixed like two days ago or something uh do you have actually a use case for it uh I did and I can't come up with it but I I used it like two weeks ago uh so yeah we we do support that now uh so go do exciting things with it I guess uh all right so uh let's see if I can do some live hacking or fail at doing iing uh if I can get uh my screen to synchronize uh and you can ask questions and stuff so have a question um will support linear types uh linear types interesting interesting question uh I don't think any one is working on anything like that but uh I don't think it's impossible but you have to have someone actually work on that and that's like not a simple thing in our types uh that's a whole PhD maybe uh I think there were some uh prototype plugins to do things like ler types uh I've seen like the unique one right but but I think it It suffers from some serious bugs I have no idea there was a paper recently by uh Philip hler uh who he implemented um something something like ler types I don't actually I haven't actually read the paper but like there's definitely research being done in this area so something like lineer types might might come actually this was uh D was based on this that uh calculus types and I just wanted to wonder what are the use cases of usually dependent dependent type languages will be possible to implement with d for instance typ providers for Generate so uh uh like so dot the things that we s like the type theory behind is also the type Theory between behind Scala itself just uh simplify it as much as possible and uh the fact that we use represent things using dependent types is because it's a very unifying concept but doesn't mean that in your code you have to use dependent types for stuff do mean that would be useful to have type providers or uh what do you mean by type providers exactly if you have schema and you just compile a program and the compile time it generates type which represents the record table okay without C generation it's just a type which is yes generated by to databas schema or yeah uh type providers are definitely an interesting thing but I think that whatever your the theory behind your language doesn't actually influence much uh what you can and cannot do here like I think F sharp are something like typ providers yeah uh like if someone I'm well I don't think anyone is doing any research in that area currently okay uh and also about termination are you moving to towards this toal functional programming guante termination of the functions uh so we are adding effects and in a way you could have an effect that dis allows things like recursive CES or yeah so it's not impossible to have some form of total function but uh again can guarantee sorry how compiler can guarantee that certain function ter like uh again a very interesting and very deep area of research so yeah I just know that some language is guaranteed that and there are algorithms like f is yes and so the way you you can more typ things yeah we haven't spent much time thinking about that uh I can't get my unified output so in might be time to carry it stuff uh so thanks everyone and uh yeah so if you're interested in uh trying D or hacking on I I sent a message to the Meetup group with like instructions on how you can try things out uh it's getting a bit late so you might not want to do that now but feel free to do that anytime and we have a gter channel for uh chatting with people so uh definitely we welcome contributors and ideas and stuff so U if you have a complicated idea for programming language feature and if you can describe it in some details feel free to pen an issue on our B tracker and tell us your idea and we can discuss that thank you you want to show how you devolve uh it's getting late I think do people want to stay a bit and show see stuff okay I I going to need a chair uh so this is my awesome SBT prompt uh that is just a normal uh SBT project you compile code using like you compile code using the compiler that compiles the compiler so SC see in that case using compile and you run the compiler using run and so if you just do run it's going to tell you uh it's going to do stuff and then it's going to tell you that you need to pass arguments so anyway the bug uh fun recently is a very somewhat obscure thing but uh let's open a scal to prom so in scy uh you can uh your x. type is a type that uh represents that like the value of that type is X so if you try to do uh y x oops C x. type equals two oh I screw up so you should do Y X Type equals 2 you're going to get an error because x. type is not the same as uh the type of way anyway so if you have two value uh actually let's choose a string so if you have two different values and use e instance of to check if x has the same type as y you could also do pattern matching it's the same thing so X is an instance of Y type uh oh yeah this true because X and Y actually have the same value here but if I SC something like bar then it returns false so the E instance of check is more complicated than the jvm instance of check because on the jvm uh the things have the same type they're both string but for scalar they're different and is instance of uh we'll add uh this a more comp licated checks and I was surprised when I saw that uh the T which is building things right now I like hisb starting time okay here's awesome red po or you have cols yeah but when you do the same thing in D is X and instance of Y type you get true and then a warning but like ignore the warning so my question was like why are we returning true here uh so the way we see what the compar is doing is that we have this thing called X print which allows us to print uh a tree of a compiler so if you do xprint uh front end on yeah I'm going to have to write a f now because you is a string and that is this yeah but isn't S of should not just look at the type it should actually for single turn types like that it should do an equality check or it's the way it's work in scy 2 and I think that's okay and I think I already wrote an example yeah so I wrote a program that does that I have many things like that like most of the scalar code I write is just examples of things more than actual code just to test the compiler so any if I do that with xprint Fronton I get the code as it was uh after the type Checker so the type Checker is going to add some types it's going to transform objects into class with modu Dev but we don't care about any of that we can see that we have this e instance of check which is the thing we want uh so what we want to figure out is why at some point this becomes true what is going going on so if we do xprint all it's going to print every phas which is horrible to read Because when you have this bunch of things uh recently uh n implemented this cool thing which is called xprint diff okay I have to actually sh for it because I explain diff D which just shows you the difference between every phase yes that's pretty nice so we can see that after uh this thing so the why is is this whole thing why is this a huge name because we have Min phases which are fused in one phase uh so when you're debugging that's somewhat uh annoying because you don't know which one of this mini phase actually did your transformation so when you're debugging you can go into compare the Scala and change the mini phases to be uh separate phases but here uh we see that our instance of check has been removed and replaced by true and if I look here I see something called is instance of evaluator so it's probably related so all the the transform in tools. say transform but even if you don't know that you're going to do a grap or whatever I do grap because I'm old school and you see that it's defined here and that file doesn't like being that zoomed up so this thing does partial evaluation of is instance of so it's going to try to optimize your code to replace instance of which are not needed uh which is fine but it will not get rid of our uh his instance of check and replace it by true that's incorrect so what what's going on um so I looked at that earlier and what I saw is yes to check so the scrutiny is uh the thing you do is inance of F and the selector is the thing on the right hand side and uh uh this code the first thing it does is call Erasure on your both of those things which is wrong in your case because x. type is eras to string and Y do type is eras to string so they're the same after Erasure but we don't want itself to return through so uh something is wrong here so to check if this is actually the program I can take uh this function so it's functions that transform all the type applications and then check if they are is in sense of and it takes a Tre as input and returns to transform tree so I'm going to disable like this transformation just return the threee without transforming it okay let's uh try this again now we're compiling code again using the compiler that compiles the compiler and then running the compiler and and we still have true so what's going on okay so our test like test

X in instance of Y has been replaced by true but we are in a different phase now arure so probably something is wrong in arure too so arure is one of the most complex phase of a compiler because it has to retype check everything but you don't have to worry too much about that um the part that arise type is called type Erasure so let's look at what it does for uh oops so is instanceof is a type application so let's look how we arise type application uh there a bunch of thing going on uh intercept type apply let's go there I don't have uh jumping working right now in my ax so be with me uh it's defined in uh type test whatever uh yes in type Test cast Okay so this thing normalizes type test and type cast uh which also repace type test with Singleton argument type with reference equality check which is what we want we want uh basically we want X do instance of Y do type to be by XX Y and this is what the command says it does but this isn't actually what it does so something is wrong uh so let's look at intercept type apply what does intercept type apply do uh it has a transform as instance of thing and transform as instance of is called on the erased arguments uh which is exactly what we don't want cu Y type again Y type is a raise to string but we want as inance oft to know about the fact that it's a Y type so um if we just change that to be uh erased to be the non arised argument oops uh what's going to happen uh so if we have uh single t uh oh wait okay I know what I'm doing wrong we are looking at is instance of not as instance of so I'm in the wrong never mind the second one is as instance of yes yes I was in the wrong one we care about is instance of so let's not give to is and let's look at isense of so if we look at isense of we see that it has a case for single ton types like why. type so it should do the correct thing right it just didn't do the correct thing because the argument we gave it was already erased so it was not a Singleton type anymore so if we just have do this single simple change and I haven't tried that yet in the end we get yes the thing we wanted test. X equ test. Y so if we want we can brch repple and check that this does what we think it does uh yes that's correct f is not equal to of course well we to actually fix correctly we need to do a bit more work because for example if you have a list of in which is and you want to check if it's a sub type well if you have something which is a list of in and you want to check if it's a sub type of list of string the arrays types are going to be different uh but you still want to be able to do the optimization while you WR it to true because uh we don't do special casing for that kind of things uh so to do that properly I need to like change the code around a bit uh but that takes actually thinking about it so I think for now that's good and the coverage problems yes so the way uh well it's not actually unit tests in a way we have a bunch of test cases so we have three kinds of three kinds of test cases we have post test neck test and run test so post test are things that will compile neck tests are things that should don't compile and give certain error messages and run test are things that will compile and run without like and give a specific output so here if I want to write a test I'd write a run test because I have to check the run time so uh so the way it works is that we need a test with a main method and this is going to be run and we can uh have a check file that checks the output or we can just like we get an error if an assertion file so we can just do assert who is instance of uh why type uh and add uh a test of course to do V but proper you need to test over cases and tricky things and stuff but that's the basic idea and then you can use part test and it's going to run all tests and using as much of your CPU as it can and burning your computer by running everything in parallel so it's not going to run everything in P it's going to look at how many CPUs you have to things uh and once you're done you make a commit and you send a pre request and we say looks good to me and it gets merged and that's how we work on the comp [Applause] [Music] thanks