funconf 2013, Paul Phillips: Whose Program Is It, Anyways?

Recording: funconf 2013, Paul Phillips: Whose Program Is It, Anyways?

all right i take it by uh the turnout that you guys sensibly uh well everybody else sensibly figured out that you know i don't really have anything specific prepared here but i'm glad there's a very small number of people here so i will i will literally try anything here i i have nothing i don't know what i'm going to do so you you guys have to be like the primary drivers of content here but uh this i think the smart thing to do though would be to not like like try to get me thinking real hard everyone like unless you enjoy watching a guy like try to think his way out of it like a non-compiling program for a while just think like some guy who knew scala really really really really well but wasn't otherwise very bright but put him to put him to work like see what he could do because that's that's approximately the guy that's present um yeah i so i have it all set up for uh scala 210 so don't throw like 211 things in there but um it might be entertaining like or even educational to like if you want to see like how some features might be used together or just how you can crash the compiler oh yeah the fringes of value classes like i can't keep straight in my head what's only working in my my pattern matcher repository and what's in the real thing yeah well shoot i mean i don't have that thing built though i think i better try to stick to mainstream no no no yeah the general right well i don't i don't know it's kind of fringe for me i like to anyway um i'm going to start this thing and then i'm just going to sit down and uh we can just sit here quietly for an hour if you want but i'll i'll assume we'll do something better than that what's that so i'm really curious about combining interpreted scala with compiled stella and the reason i ask is because when i use clay chamber i'm often stymied by compilation times well there is no interpreted scala unless you have a interpreter i don't know about um there's only a rebel that pretends to be an interpreter and is actually compiling the code and then running it um it's it's the very poor man's interpreter now an interpreter is an interesting thing that has been discussed and continues to be discussed because it would change the situation for macros if you could walk in ast and actually like run it a little bit uh but that's at best plausible for a subset of scala um and i think it would be quite a project even though the question to compile times and play and the clips all right then let's get it out let's get ready i'm used to it so it's compiling reasonably well incrementally on the play and then when when i save the file blake has to redo it there's got to be a way that that they could be deduplicated that work wouldn't be done twice well so yeah we're we're working on on that actually what you're seeing is bugs play is especially sensitive to that because they do code generation creative spots and so on and so greg has pictures we're specifically optimizing for play right no that would be like that's one test case players are like right playing incrementally by itself understand i'm curious if the two things couldn't be oh so you're saying why can't they share yes they can play we have to do generating some code and then eclipse let's model let's model all this maybe we can figure out something um yeah i think in general you can share class files but it depends i mean since i don't know the mechanics behind it are there any other situations all right let's come back to incremental com compilers there there's a bunch of scholar features nobody ever uses uh it's but maybe you've heard of them maybe you could uh if you've heard of something and never seen a cause for it or never seen like it used in a way that seems plausibly useful maybe you could say it and i could if it in fact has a plausible use which it may not uh i can i can put that to work go for like the fringes here if you can because it's like this is your like i've thought about nothing else for such a long time that it's you know squeeze something out of me we could all be surprised together something with macros something with macros so the problem with macros is just that it is a bit laborious because of the uh you can't have the compile the macro and use the macro in the same compilation run so uh i have some like stuff set up for the like because you need basically two spt projects the one with the macros and then the one you depend on that uses the macros and if i had all that stuff in place then maybe i could if i was ready for a macro thing it would be good but i i'm sure you guys don't want to watch me going like ah stupid spt for no you know i mean that's as fun as that is for me i probably doesn't generalize so i think it'd be better to stick to mainstream language like ideally something that is not particularly like not something that involves like using a thing that isn't scala but something that involves like putting scala itself to work in a way that only sucks on its built-in capabilities all right now we're talking animals now we're talking okay so path dependent types so yeah so we've got something like you know stuff and inside of that we have you know yeah feeding animals right but let's do something better than feeding animals i don't know um feeding dinosaurs what do we what do we care about around here um my stuff is is my stuff uh all right all right well let's just say type universe value universe and let's give universe some stuff in it like i don't know uh planet alien all right and then and then we're going to import everything that's in the universe and then we're gonna write a method that takes an alien and a planet and i don't know populates the planet i'm gonna go give planet like a populate method it seems plausible what's going on over there not found value alien oh al i called him all right i'll give you good i got to keep those shortcuts going all right parameter type and structural refinement may not refer to an abstract type to find outside that refinement who's seen that message before yeah it's a sad state of affairs it's not even a particularly effective restriction um so yeah that's right so i'm gonna make that like trait planet instead of trying to be all abstract and then uh oh and then i guess i'll make a trait universe can i still have type alien please yes okay all right so clearly universe universe is just a ver a specialization something you find inside of stuff that is probably maybe not that bad a model all right and can you just explain for one second how that weird message led you to decide to do it oh all right sure um so basically when you have that when you're using abstract types um and trying to write methods in particular uh the compiler gets upset if it cannot pin that abstract type down to something concrete enough in order to actually generate the method because it's this is where it's kind of its feet have been put to the fire uh it has to actually generate something into the byte code and the jvm is quite unforgiving of any amount of flexibility here um but uh if as abstractly as i have defined things it has no inkling whatsoever what that thing is um in particular reaching for like an abstract thing that is defined outside of where you are means it can't rely on being able to know what that is at the time that it's actually called so it says get lost so by moving everything into because i said basically universe was anything like arbitrary thing and that it would just have these things defined inside it somehow but now i've made universe an actual trait which makes it concrete there's a real thing you know there's a trait called stuff dollar universe now um and it's now no longer going to be sweating that so much the exact circumstances of when this arises are a little fuzzy but that's the gist uh so we can have two different stuff places each with their own universe object my stuff extend stuff and for this to be interesting i guess um we need to take advantage of f from the concrete one um oh yeah so you is the guy we need here um let's see how that goes what that compiles already i feel like that was too too easy well it does compile already um and i don't know what's really going to uh trying to think uh because when you start the the trick with path dependent types is that it's generally unwise to use them in non-trivial ways um and so it's like the things that like because my my performance you can't turn off the stupid performance engines all the time so there's all these parts of scholar that like are kind of neat in theory and that i just kind of turn them off because for instance like let's you know there's a very appealing programming model very very very appealing um where like let's say i was writing a compiler well i would like to write it like the very top level a very a very abstract model of it all right and so like you know very you know here's the thing you give me like a like there's going to be a type source file and i'll be completely non-specific about it right and then there's going to be some kind of compilation product that's like byte code in our sort of standard case and then there's going to be something that just says compile and takes its source file and turns out as i don't know a set of compilation products and then you could take you know you can take this quite a distance you can refine you start with something very abstract like that pinning down only like you know and ideally you do this with your concrete models and minds you know going back to my earlier talk so that you don't like make a bunch of pointless painful abstractions for yourself but you know you're thinking about net and jvm and who knows whatever your like greatest abstract ambition is you define it there and then you go down a series of steps and you refine it this is great it works it is also imposes a terrible penalty on you if you have any kind of implementation anywhere before the end um i was i was showing adrian yesterday what happens when uh like the exact same code written in terms of scala.reflect.api versus scala.reflect.internal a method is 15 bytes in internal and 71 bytes in api they look exactly the same to you but the difference is that the types you're working with in api are abstract and so when you do this pattern match it doesn't get to go instance of at bytecode level it has to say okay here comes the machinery and so it whips up this whole implicit materialization business in comes the type tag with the unapply that it can call it runtime to find out if that abstract type is matched by the actual thing that showed up and it does it doesn't have the luxury that internal does of knowing that it's going to be an instance of food.bar.bass so that's the really adds up as like at one point after they did the sort of partitioning into reflecting compiler they uh we had like a 40 slow down on the compiler everybody's like ah running around with their heads cut off and i mean like that was basically it right i mean that turned that was definitely the problem so anyway uh these you know i really like abs you know this what i'm doing right here this abstract type business and especially i like and this also this is this is brutally pessimized right here you know you like you know put a method in here work it in terms of its you know even its own things because this is essentially saying uh this will be some unknown thing that has these methods but i'm not even going to give you like any kind of concrete thing to hang your hat on at all so it don't that's going to be invoked via reflection um unless you're lucky right but if you if we try to write anything that uses any of this at this level it's just going to be bad news in fact we should look at that let's do that but just one thing like how come you could create a universe didn't you have that abstract type there so that's a good question turns out that uh scala for reasons i um probably shouldn't speculate on um i mean i sort of know and then i'll wait anyway abstract types are not treated the way the abstract methods are like you you your intuition is you never implemented the abstract type compilation failure that's not how it's done instead it just makes it any um there again there's a plausible sort of implementation driven reason for that um i do not think there is a good like language semantics reason for that uh and i don't think the plausible reason is a good reason either i don't i think it should just be an error but it isn't what do you think adrian yes yes he says that means i agree with paul is generally what that means is trying to think back when it was changed it was a long time ago i mean it's been doing that since at least well since i had showed up so five years plus and you're right and that's i at some point i researched it and uh the re the right there was a reason and it was something to do with like separate compilation or blah blah blah right i mean basically it was like it was too hard like it was an implementation problem um i don't think it was a good theoretical reason um well i mean if you're being performance sensitive not much but uh if you're not then they're they're fantastic for just modeling things at like the layer you want to expose in an ideal universe you'd use them much the same way you use abstract methods which is to say you you have an implementation with a bunch of stuff in it but that's not what you want to expose you want to expose a like idealized lesser version of that the types of things are not like the concrete classes you make all those private and abstract types like you can give people nice like solid traits and stuff as their thing but if you if performance were in on consideration you'd much rather use abstract types because that gives them complete freedom of implementation like in other words like the abstract type that might turn out to be string or something right something that they have no ability to modify but if it's designed such that a type like that actually meets the test then it's good enough but implementation issues arise right i mean like if you write methods in terms of an abstract thing and then it turns out to be string then it has to do it via reflection there's no real option there because it couldn't you know issue the actual byte code that you needed to call it directly in string although that's not completely true you can there's other ways of like spinning stuff up and but the point is it's not easy for it and so you will pay um but i mean if let's let's say like take something like a tree right now now we just get this kind of thing all the time you've got a leaf you've got a node right and then now you can write some methods that operate on leaves and nodes and do all the beautiful tree things that we all know and love and we've said nothing about what leaf and node are and they can be anything and this is completely reusable like generic in the truest sense code this is the ideal it's just that you pay for it it gives you type parameters you don't have to specify when you're writing so i mean since list to write list is sometimes yes that's right in case you don't have to specify those so type parameters um are just enormous pain in the ass because like if you change your type parameter situation every call site must be touched it's a that is a debacle if you want to add a type or something you have this like combinatorial explosion of madness when you have like this thing with type parameters extends that thing with type parameters is terrible abstract types in principle can do everything that type parameters do but without these big problems but that is very much in principle they are like you know they're just not up they are not equal they are second-class citizens in terms of their robustness and just like general usage and that's too bad because in the ideal sort of unified there would not be both type parameters and type members type member type parameters would be just a special case of type members that's like the idea for some future version of scala that it would de-sugar like you'd write a you know list takes a t and it would de-sugar that to like private this type member t right and then when you instantiate it when you write new list of int then it would de-sugar that to new list type t equals int and you're done and then there's other neat things that come out of that like you don't have to specify the types like you know because there could be just whatever it's got some bounds going on at a certain point now you just say give me a list or you know give me a map i haven't even told you what the key and the value are but it's it's learned from whatever context as much as it and now it's just going to like settle the bounds and give you you know it's a foo and a bar because that's what i know at this point or you could be more specific by giving the type parameters point is you'd be able to write map not like map bracket underscore comma underscore like i never want to type like class of again because i got a class of foo underscore underscore what you're just going to throw it away don't i mean give me some syntax man just let me say the type constructor without the stupid you know meaningless applied underscores these little things you know become a big things after the billionth time personally um yeah so anyway that's the advantage of abstract types and then you can so here's like the refined version right it's like say and then like maybe file system tree or something and and then we can remain very abstract but maybe like throw some new capabilities in here oops look at that i'm so extracted i'm doing that like in principle that would be a good syntax in fact that that leaked in by accident at some point um from some change i made because yeah anyway miles was like is that the new syntax that's great uh oh i had to pull it um so yeah here so maybe you have like a vowel root you know which is a which is a node um anyway uh point being that like you can remain abstract start bringing in your buddies the values and they can be abstract in terms of what it is that they are and you defer everything to the guy at the end it's make it a his problem um but you you write in terms of the abstractions that make sense at your level and the one of the like most productivity killers i see all the time is the inability to like keep your levels straight so you have you know like stuff that's like pure logic operating on in sort of a pure domain of logic and it's just interspersed met with these mad compiler details of you know like this flag and that crazy bit and all this extremely implementation specific stuff this is death uh there's you should be as pure as you for any given thing as pure as you can be be that pure and then access it from the crazy implementation stuff and so you but you must be like you know adept at building these layers and composing and not composing the stuff so yeah i see this is just turning into me like ranting about programming rather than getting involved so i hope that's not too disappointing but i i i you'll get more information out of me by me just ranting than me like you know trying to type no i'm totally willing but anyway uh are there other questions i can answer with you know paul's crazy opinions about programming i'll check my book can you do any cool stuff with infinite sequences i don't know if it's any cool oh well i can show you one one trick i like because nobody ever realizes that it can be done without laziness let's see val x stream and equals okay so i can't remember what the operators on stream are i never use stream stream's not really my thing it's like this is like no it's got to be that right all right and then some reference to x here we'll do it wait let's just see what that does yeah um so that should be an infinite stream of ones i guess uh it's not the fibonacci i was going for which will need a little like some throw another one in there or like i don't know uh i don't know anyway point is though the fact that that didn't already like go bad is makes the point which is to say uh you can make an infinite sequence that's like self-referential that actually works without laziness which is surprising in like because if you try that in other circumstances that's going to be a big fat null but let's see here x take 10 to this no it's always been like that it's uh it's like you're sort of your internal model about how things work causes you to expect that like npe there um and well let's see i mean like if it's right if it isn't stream you do get it but the thing is that because this is called by name right the creation of the cell in stream is called my name then by the time it gets around actually going for x it's there right whereas with list it's not um so yeah i don't know how cool that is so when i do stuff that's infinite i tend to use iterator not stream because iterator has the wonderful property of actually forgetting things which is usually what you want or usually what i want um you know if i'm like going for the 8 billionth element of some infinite stream i don't really want to be pulling the other 7.99 billion elements behind me mostly i want to just get there and get the one because i don't have that much memory um what would what would it represent a cool thing i mean it's possible i can do cool things that i might be setting the bar for cool too high um like let's see so i'll give you some like iterator stuff that's fun i mean so there's a there's a dysfunction continually right so like you know that's an infinite series of ones although we could do something like util.random.net that would be a little more exciting so if we do that and then take 50 then show them then we have so that's a really quick and easy way to fire up like a whole bunch of something um and take i just took 50 but there were plenty more that went on indefinitely um you know or some i just added 50 random numbers that's a powerful useful thing right there maybe i got a non-random number out by adding 50 you don't know ah it looks random um let's see what else can we do with uh that well i mean i don't know does zipped work on pairs of iterators i don't know let's just like to stream it right so now i've got like pairs of random numbers going for whatever that's worth i you know i don't know that we're getting any cooler here now i'm adding two random numbers you know pairwise and then now and then summing that and clearly we're in totally new territory here as we just keep over flowing the end um let's see here's uh like here's something i do from time to time all right so there's a handy way to get like your one bit guy in each spot oh i know let me do something good with value classes um so value classes are super useful here's something i would love to introduce to the standard library which i don't know if i get to use do okay so we're going to give it a private constructor it's called index i bet you can guess what it is it's an in but it is much better than an inch because i'm going to first of all guarantee by construction that it's always non-negative if x is less than zero throw an exception the easy way else new index x i forgot to valid um but it's an int you don't pay any costs watch so here's like def f you know you've got a list of something and you want to know the index of something this will be index of in fact index of whatever t looking for some particular t index equals index and then i'll use the real index of but the point of this one is to look at the byte code of that see that second second thing in that public int you get to return an int that's just like the real index of does but it's better much better uh you know the mini the the the cavalier usage of int for just whatever comes along it's madness it is only from like being hypnotized by earlier generations that we allow this in a language with types it is nuts that like any given thing you think of for a number oh let's just throw that at the same time it's our catch-all number like things so my meters might be inches might be number of planets in some other solar system they're all ants that's crazy we have types there's no way you should be able to mix these things up so index is just like should be the first of a long line of things where you see any basically if you return in the api why don't you just return any and give it up just say like here comes a thing eat it figure out what it is right and at least with a thing you can like type match your way and find out oh well it's a this or it's a that you can't distinguish ants they're all the same of course that's also true of the value classes but the point is not to distinguish them after the fact the point is to retain the static type of what they are so that you never lose sight of what they are and then it's not a problem and you pay no penalty for that now and so the the argument against it is gone it just becomes like i like bugs at this point well uh the creation of uh there's an extension method call in here to into something right yeah it's this is the constructor where i wrote that so notice it's intent again we're not paying any price here this is all about you know hotspot's great at these like tiny in int methods you don't need to sweat that but we allocate nothing so it's going to call this exception otherwise non-negative you can tell because i gave it a private constructor the only thing that can touch that is the in companion object and the incompanion object it's got a nice one-liner that you can verify for correctness if it's less than zero we're going to throw it um and so this is just the beginning of like useful things now there's another uh one that i have and that is an interval and so the wonderful thing about an interval is that you can have two ins and not box because you're going to take your two inch and you're going to put them into a value class where the underlying thing is along and then you will inside the value class you will magically do the decomposition into the first thing and the second thing but you will still have no boxing because what's going to be returned at the bytecode level is a long so that would look something like this i'm probably going to blow any like bit manipulation here on the fly but um and then in interval i would have methods like low and high which take it back out of the long so they say okay and and that's like a really like obvious use to take two instances to come in along but you can do much better so i'm doing range positions in the compiler and i need more than two i need like i need start i need like the offset to where the point is and then the end i need three uh plus i want to encode some other information like whether it's a transparent position or not but that's okay so i've got i'm gonna go 20 bits 20 bits 20 bits and then i've got four bits for signaling right all of this you package this all up into a nice like you know it's low level stuff but you do it right low level stuff where you can believe it and let everybody else use the high level thing but they pay no price it's all they're just just primitives flying around it's it is really it's money um it's definitely the way to go and so the the real payoff though comes when you take this stuff and put it with the work i'm doing on the pattern matcher right now which is turning that into an allocation free experience in extractors so at that point you can take something like one of like index for instance uh you can write it something like that that's an extractor you give it an is-empty method for failure and you give it and so then you can just pick any inch you want that you want to be failure let's say minus one and then everything else that's up is success then you can return that as your extractor and so you've got the whole semantics of pattern matching but again it's just an int you just pay no price there's no so it'll come for me personally it's completely changing the way i program already which is why i'm so determined to get it into uh trunk because the the main killer on extractors is performance and it's legit i tried at various points to make some changes in the compiler and it was not tolerable but that argument's gone and so you get this just tremendous flexibility for what you want to expose and you can you basically you can take of any view you want of some piece of data and the cost of doing that is extremely low and you get these just you could your pattern representations of things tend to be far superior in terms of like believability than you know manipulating a bunch of text example of cool thing that i did a string so i'm doing this work on class path you've got like a string and that's fed to you by like zip file or something so it's a path and there's a bunch of slashes in it and i need to know like what all the substrings are the individual path components so i did that with no substrings instead i have a thing that goes and like finds the indices at which those exist and then packs them all into a value class along these lines and then then uses that as the basis of a sequence where it pretends to be these things but is actually just going to index into them as necessary and show you those cares so you there's no allocations in other words you're using the original string and a single int or long depending on how ambitious you are to like allow for really long paths but you can fit an awful lot of slashes into 32 bits given that you know it's they're mostly coming like right after each other right so you only get a very small amount uh point being it's you know the opportunities are are numerous like it's really shocking how much uh cpu is wasted on basically just string processing in our case for sure i think in most people's case um so yeah these are like neat opportunities it's a series of offsets so you've got a string and you want to know where the slashes are and so you ask and it walks the string and says okay on the fourth care there was one and then like five later there's one and so it gets all those and encodes them you know it's like whatever however many bits it needs for that less than 32 is the case in in practice um and then the value class is happy to just turn back around the value class has a method that says if you give me the string that this applies to then i'll tell you i'll give you a sequence which has uh those substrings as its elements but it never allocates the substrings it never has to it knows where they are so as you ask then you get them so you're saying the next slash would have to come within some small amount well yeah we're talking about paths we're talking about paths here right right right you're limited by right i mean it's gonna be within a 64 is the next thing you you can pick a bunch of different encodings here and they're all going to run out of space if you have like a gigantic string but fortunately it's not an issue like no nobody's got a jar with like a 10 000 character path um so like frankly i don't wish to support them in that i'd like to actively undermine them so what did you use in practice though what did you find was like a good trade-off for them well i found that 32 bits was more than like it never came up in practice it was never a problem to easily fit like the actual currents in 32 bits but if it was i don't know i don't know think about i don't know i think there's probably a few ways to compress it but it's uh i'm not i wasn't overly concerned with it because it's this it's the spirit of the solution that is like relevant in uh well for me there and for me here so yeah and there are lots of these things it's like any time you're returning like option in somewhere this is brutal you box twice not just for option but for ant because it's got to be go into an option which is not specialized so you are returning a sum around an integer box around an int both the boxes fall away with the capability i just described in the pattern matcher and so you just return what looks like option in but it's like it's a valid imagine value class option call it opt you return opt-int and then at bytecode level that's just int where some particular int is it represents failure this is the future so earlier in a different talk you were used to kind of shy away from value yeah well so all the basically anywhere where you you these are all like very sort of direct specialized uses um if you let them sort of infiltrate your general purpose models it's the problem is compositional and these are implementation concerns like value classes in in principle like nothing should stop you value classes are pure optimization it's all good um there's uh but in practice uh there are a bunch of bugs at the intersection basically if you're trying for too much generality so that's why you know these cases are like specific things that would benefit from being more typeful but where we have historically not wanted to do that for performance reasons there's another good one like ordering i mean it's crazy you what is that like if you look at you know comparable or whatever what is that method return that did when you want to compare two things what's like the the thing what is the type of the that it returns and you so you've taken the four billion possible values and you're asking a question that has three answers and you have stuff that into a four billion value space and act like it's normal it's nuts like it's crazy so we don't have to do that well we should never have had four billion but now we can get down to three which is a lot better in every way right so you can have um basically pick your values minus one zero and one that's it that's all that can be created and you return a value class type that wraps you know maybe a byte although it's hard to win like by shrinking in these spots because everything gets widened to an end in jvm anyway so i mean maybe you're still an in i don't know if anybody's packing a lot of like the only place you might win is storage and i don't know who's storing a lot of like comparison results so but the point being like we can at least limit the size of the type it's crazy what else what else this is this is what i should have done and it's like just your programming question because i do feel like i do nothing else but this so i've had time really to think of a lot of stuff like i'm i'm very i'm so i have a problem but you may as well benefit from it i have a question which will probably drive you up the wall but in general like i had a binary method right and i tried to have a value which was you know a function basically the method with the underscore and the binary method had a default argument value and i was just wondering why can't i call the function with just a single argument why does that require like two arguments i'm gonna see that did not make sense i mean maybe it did probably make sense in in your mind but not in a way that actually made it to me being able to parse it um by binary method you mean like two arguments where where one of them is not this we're talking about like just some method off in the universe it takes two things and one of them is a function one of the arguments so it has two two parameters and one of them has a default value okay and then i say val f equals whatever the method name is underscore oh okay so you're talking either way but you know i can't obviously i'm just wondering like well you're just you're it's just implementation stuff so that trailing underscore martin's tried to kill that trailing underscore a number of times it hasn't succeeded i'm sort of glad sort of not um it's like because the trailing underscore is sort of a weird like buggy expansion um yeah so what it comes like this i'm not even going to try to answer that question it's the implementation um when you start saying like well there's default arguments and i'm doing at expansion and there's a value it's at this point it's the implementation there's a million things that like one could step back and say well you know this ought to work forget it as soon as you intersect two or three things written by three different people across six years in like harry and harry are parts of the compiler then forget it it's just it doesn't work because it doesn't work default arguments in particular were a source of some real potential ambiguity and they still have some super annoying aspects like you know having to put the parens on to call anything that takes any default arguments even if you have no arguments passed and this was because of ambiguity with like something i forget but uh it's yeah it's uh that's right and in principle if there's like you could allow for like a sub like common cases where you would be allowed to overload and have fall documents but again it's basically like it's too complicated and you know we just don't have like this giant stable of people going give me a super complicated ill-specified thing to handle um so sometimes you know you just say discretion's a better part of valor and outlaw stuff that's what happened there is it just because you come up with a name for the class file to refer to the value that could be the default and that doesn't take into account the signature um well it's because overloading resolution is ridiculously complicated already and to have it to combine defaults where so that leaves you like even more in the dark about which methods are applicable in the first place it's too much go look at the implementation sometime you will quickly be disabused of any notions like that it's good idea to complicate this stuff um yeah it's just a bad scene there it's like and it's it's inherently like the language spec it's one of the most complicated things in language spec it's it's like it's famously cited is it 6 23 6 or something it's like it then it's it comes up enough that like the number is you know and i still don't understand it i definitely do not understand it but i don't understand most of the specs so i'm not the guy to say that mark's mark mark herrera claims to understand it i i don't know he's like the only guy i've seen like i understand it you know just like willing to stand on that but that's mark but you could kind of overload it by hand right like i mean you could say like i have the three oh yeah absolutely you can essentially reproduce the the but but you can't i mean you can and you can't you can't get the kind of combinations the default arguments give you right um like try to make give a method like eight default arguments and now try to overload that by hand and you will find yourself very distressed right you have two to the eight situations right whereas you can you get all those at once with the faults but you can't exploit that power when you're overloading um i don't know because just because i don't have a completely clear view of what we're talking about here and the tiniest details all matter um but it's the long and short of it is that trailing underscore mechanism of turning a method into a value is just going to have like sharp edges when it encounters things like default arguments that's just the way it is yeah i would note that because it's really handy and i guess before i since i no longer have the energy to like try to because i've tried before to change the signature of stuff like index of and uh and compare back before we had value classes i still thought it was more important to be correct than you know performant now but i don't have the energy to get this stuff in anymore so it's up to adrian blame him is there sort of a why slow for a scholarship like like there's a lot of tips and tricks about how to avoid certain things that'll slow down your combination in an automated fashion find out yeah we have something floating out there that we never really finished with the jesus aspect jade and weave timing into the type checker and we'll like bring yeah i i can tell you i mean for a typical code base it's almost always the same story people um use type inference in places where it's completely inappropriate and then are just suffering forever after uh like a classic sort of expression that you should just never write is like val x equals foo barbaz where all of those are these unknown things for which it needs to infer type so it's going to go all right guys get your boots on here we're really going to get we got our work cut out for us today we need to find out what all those things are so we can love this all right so let's not jump to any conclusions let's just start a bunch of type inference things going off polymorphism god knows let it all come back do a little analysis come up with weird refinement types by intersecting parents and dreaming up crazy stuff let's do that every single time they compile all you had to do was either like this because even if it turned out to be this that's not like it wasn't that obvious to this guy um or you know this right anything but uh yeah even even without the int there you're in reasonably good shape um just because of like the nature of this ends up calling the apply method with you know three things of the same type but when you when you use call and colon like that each of those is a separate call with a separately inferred type which are chaining on each other and then being loved i i mean i saw i i changed one line in one project and it was taking like 80 seconds to compile and then it was four no joke because like the guy just lucked into like you know that was a day i was like felt like looking at somebody's random spt project he's like it's really slow yeah it's really slow it's almost always type inference one way or another the the other big ones that'll get you potentially are like a just a rampant use of implicits um i i find that most people like are burned one way or another badly enough before it becomes a giant performance problem um that they you know don't paint these terrible situations for themselves mix-in composition if like is definitely a problem if you're too nutty about it like we are in the collections so you pay like if you got 80 parents spread out all over the place you know and a bunch of like basically you there's a you linearly with the number of parents that a class has you pay in a bunch of places so you know you have that many more base types it's going to work have to work that much harder every time a type is getting inferred or types are getting loved it's going to be looking through all these base types and like every time you throw in another parent with new type parameters then it just you know those things start doubling up ending up the o-n follows you around so fewer parents is better um if you can do it it's as usual it's like that's less appealing from an elegant standpoint and that's you know it's not yet you won't hear a lot of stuff like well here's how things get better and it's a more appealing thing to do well of course it's not that's why you didn't do it in the first place it seems like for a lot of not so spicy just especially beginner scholar programmers they get frustrated they're like why is it so slow and there's there's no automated way to really discover that um yeah i mean there's there's like 60 different obscure command line options you can give to like get out different little windows into the compiler that you can easily understand after a year of constant study so i don't really know what those guys are complaining about yeah i did do a typer timings thing that i shipped shipped all the way to github you know my private thing but long time ago when somebody wanted it that gives pretty precise timings but it's very difficult to measure timings in the compiler because of the because of the way the compiler is architected but there's a tremendous amount of like latent slowness just hanging there waiting for somebody to roll through some particular code path they're not necessarily linked at all so you'll find like if you're measuring things you know you were typing some method and like oh god that took 20 000 seconds that's the guy but that's may very well not be the guy it's just that that was the unlucky guy that caused some particular symbols lazy info to start unrolling at that point um he had nothing to do with it anybody might have done it but he's the guy that did it um and so you have that you it doesn't take much of that problem before your timings are like well now we know exactly how long each thing takes and the plus or minus one billion because you know who knows how much that really was attributable to that so it's you must have a sort of more sophisticated uh model for like how your laziness unrolls than we do um in order to accurately measure things and i i don't know if anybody's got a plan for that one other than like don't don't use that compiler but i don't so somebody had a i saw like a hand going somewhere this direction no fade yeah well i'm sorry i i that was the one let me just let that go i'll come back to you who was just starting to speak there just a moment ago well yeah so and it does there's a big old map of computed lubs it tries a bunch of different things to um ease the pain and uh i don't know from what i like from my occasional attempts to try things like throwing out all the caches like it's a coin flip whether it does any good at all caching stuff is unfortunately like in just very difficult to win on in the compiler i've tried a million things it's is quite often just better to recompute it so uh the the improvements to be had here are algorithmic much more than they are like memoization based i'm sorry calculation and the idea like everyone was saying like you know basically that's a better idea because you already know how long you do for everything else you had one more line and you do like and then it's lion's turn to be slow very deceptive though right i mean what you're really measuring at this point is more likely hotspot and whether you've been jitted than anything to do with the compiler itself right it's like it is very difficult to separate these things but i you you you cannot assume much from like i made change x and i saw thing y you must have like more controlled circumstances so yeah i mean it's possible but don't count on it i don't know it's just really hard to measure stuff and more than that it's really time consuming just tremendously time consuming to try to squeeze accurate numbers out and it's just like and then you know things change and your numbers are invalid right so it's just it's really hard to modular like the the noise again on jvm to really test the compiler like you need to actually we're moving towards unity we need to move down we're trying to like you just you you think like well we'll just measure the whole thing compiling a thing it's just there's no way to get info out of that we and because it's like not modular it's getting there right but it's still a great distance to go it's very difficult to actually like say like let's test the performance of this bit of it but that's that's one reason it's so important to achieve modularity for correctness and for performance yeah it's um i think before we get there we can do some interesting things with just like on your code and say maybe this is it or you know maybe that's it right well guarantees the type of timings thing was designed to say like i just had to spend a lot of time typing a particular tree like basically you would give a threshold for a number of milliseconds spent typing some particular tree and if it was over that it would give you a warning hey man i just spent a long time on this one maybe you should think about annotating the type and so it would be a low confidence sort of thing but it'd be a place to look which is better than most people have right now uh yeah these are all pretty sad answers i wish i could say like well it's slow because i you know i missed something on line 72 it's fixed in trunk now can you talk a little bit about separate compilation for those of us uh which part of it because that was mandated these these are these are in the realm of um of assumed constraints uh outside the realm of discussion not uh because i don't know no no no it's mandated by the powers-that-be um one could say we don't have separate compilation anymore we have got to do you know see whatever it is see whatever so and it's freaking tempting to me but it's you know definitely thinking about an optimizer right so it will work a more likely scenario is not to abandon the requirement but to have an opt-in thing that's like assume whole world analysis available and uh then let it go but the problem is that you still you know now you can't interoperate with the other kind of byte code um so it's well you did you see in my talk about all the little crazy methods going on in there right i mean so anything that's built with like expecting to find b dollar dollar m underscore setter underscore is not going to work with anything that's not built to expect that so either you need to provide those things anyway for compatibility with that or you've just carved off a separate universe for yourself now maybe you can do that and so on that aren't strictly necessary in the current code base but might be if you're linking against it and so if it's not there it's not there right so now we're back at byte code rewriting magic stuff maybe like you know maybe that's the answer uh but there's you know there's no simple answer unfortunately and with it doesn't involve punting on that completely but i think we should punt on it completely you can consider this as like a more general problem like just give me a bunch of class files and it's not really related to scholar you can imagine just doing this analysis at the bytecode level kind of like proguard where it's focused on getting some chatting class files that you don't need or click shutting by coding you'll need you could right optimizer right right right post-process it all at the end point yeah i mean sure that's available to do now it's just but you're you really are it's this needs to be your application right like we're shipping this bundle of byte code that we've made it work with each other but that's as far as it goes yeah but i'm saying i don't think there's anything i mean like well if there's you could do differently but so not really but there is in the sense that uh the more sort of mechanisms you provide within your language to partition things then the more pain you're going to suffer at the level of jvm if you're partitioning requires it to like also mimic those partitions so in you know in this example like i want to divide my class into 50 traits and put them all in different files well that's you know it doesn't stop there now the jvm has to have 50 things that are all able to communicate with each other and it has no channel for that except these ridiculous things that we do so that that is a scholar specific problem in the sense that you one does not have to have those capabilities in the language um or one could implement them somewhat differently i don't know but if you declared like those traits steel nothing nothing helps you here there's no there's no nothing will be the least bit helpful um remember that the example i gave had private this val m like it was literally that's as sealed as it gets nothing nothing should be able to see that except m itself no nothing could but nothing should be able to see that at all the reason that it is even visible anywhere is so that the value can be populated in the instance of x which nothing can see but the the the requirement that the trait is one set of byte code and the class is another set of byte code leads to that necessity because the trait has no state itself but it has the code necessary to populate the state i never pay attention to clocks so i'll just stand here until you guys tell me to stop well technically we are at the end but i'll still stand here till they tell me to stop all right well i won't be offended if you want to leave but if you want to ask more stuff or anybody does questions yeah so you're announced right there moving on any idea where like uh i'm not going to work for anybody else i'm just going to go and sit down and write code certainly initially whether whether that remains the case remains to be seen but yeah i got an awful lot of mindshare invested in scala so that doesn't change very fast have you tried to go it's quite an interesting game i'd like to write an ai for it so have i tried it well let's just say that i've seen enough like that they did one thing that i admire so much that i love them for that but it's like beyond that it looks like a language from the 70s to me like with a slightly less maybe early 80s right i mean you know i maybe maybe they just have like see-through time better than me but i feel like we've learned some stuff in 30 years you don't just ignore all that stuff that's my general feeling about go but you only asked if i tried it so the answer is no but the one thing that they did that i wanted to point out that i think is so wonderful and it's such a stark contrast to my own last five years is that they have two compilers they're like so that you know it's like these things have to be in agreement and if they're not one of them is broken right so it's like this is this is a team that's serious about like there being a specification and that it and the implementation matches specification that's like that's what you do if that's really what you want and that you get somewhere doing that right and like my you know my number one frustration by far has been the essentially abandonment of the specification of scala uh and so like wow two wow two compilers that's great well that's not enough to carry it what does that mean well it means that they have two two implementations of the language it means that uh it means that one does because if you have one implementation language that defines the spec doesn't matter what spec sense the the implementation is the spec that's how life is scala compiler as impossible as it is to like deduce what it's going to do that's the spec not the not the written spec that's life but if you have two now you got something right now you don't have an implementation that suspect you have two things that need to agree what they agree on that's the spec that's a much more powerful thing a much more believable thing than one implementation no matter how much like you may believe it to be specified three why not three that would be great because resources are limited uh but absolutely more more is better 100 percent but two is the first i ever heard of somebody starting with two like it's engineering right they also have a really cool tool called go fix it yeah that they use for language evolution so whatever they make a breaking change can you imagine can you imagine if we could just ship new versions and like just like the scaffolding to make the old version continue to work is comes with it oh my god we'd be out of jobs we wouldn't have anything to do what do you think of the built-in formatting as well yeah yeah i i guess i'm less like i i like the idea of uh of there only being one way to format code but from what i've seen nobody else has enough taste for me to even allow them to begin to tell me how i need to format my code if i it's again like if i'm the dictator if i can pick the formatting then i think that's a great idea uh but there's a like i'm very sensitive to the aesthetics of code exactly how it looks i think it's very very important for like the comprehensibility of it and in my experience most people are like uh philistines yeah you know but i think people are much more flexible than anything else in the world so you you you were given that you'd be fine in the year well again as long as i was choosing or let's say everybody else was choosing if it was consistent and you couldn't have a choice well you might need a gun to keep me programming or my editor my editor can reformat it it's just that like from what i've seen anybody else's set of choices is just not going to do it for me but you're right i mean it's i i do think like far too much time is spent on like twiddling with formatting but i would rather solve it with like you know scholar form on the way into the repository always consistentizing stuff than i would any kind of like mandated language thing i mean i i think this is a solvable problem with remotely decent tooling and the fact that it's not really well solved in general it just means that people don't care that much the choice is great yeah right but it isn't though right i mean like in many cases choice is not great like you think it is but it's in fact it's worse than no choice like but if we all grew up with a single format we wouldn't know if a choice existed probably well i think a lot of like horrifying atrocities have been justified on similar grounds right really it's like it's you know you could say like well nobody knew about there'd be alternatives right i mean yeah it's it's tough uh let's just say it's not an easy one which like oh i don't know really i literally what i know about the go language is that i assessed the feature list and thought it beneath uh my interest um because my time is pretty limited and a lot of languages come by and i could just like i forget with the particular things that they punted on were maybe it was like no exceptions or something like check all your return values no polymorph yeah no polymorphism forget it i'm not interested i mean that's that's ludicrous that's crazy uh you like without generics and you know the implementation of generics in both java and scala is like loaded with pain the idea of programming without generics is not on the radar for me at all no way um i'm saying that generics are spectacular and i am that uh because the implementation is like you know especially from java but i mean because the many constraints imposed by like jvm and etc it means it's not the like completely satisfying programming experience it ought to be like say it is in haskell uh but even just yeah despite that i would never give it up uh even under these conditions whereas i you know i can imagine much better conditions but if i'm going to move away from the jvm the last thing i want to do is move towards something that you know looks in every way to me to be worse yeah that's again it's like those guys i'm sure that they're doing useful stuff if like your target is systems programming and your mind is you know basically thinks unix is like the pinnacle of design but that i don't share those feelings and so that's yeah uh i guess you know about power mode got no power mode uh there's there is some hidden stuff i think well i guess hold out for a little while because i guess one of the first things i'm going to end up doing is writing the ripple i always wanted from outside of the thing that's because i've already done that bunch of times and i've just like never quite gotten you know shipped it but i've done so much replica work it's sick so if i can just pull it together anyway i so hold out for that and i'll try to make that a reality because i would really like it because it's very frustrating to me that it's still really terrible compared to what i know is possible and i've just never like as time has gone on i've had less and less time that i can really reasonably allocate to it being able to connect and resume oh god yes so i was just playing with that the other day with sbt as well right like i really want these things so i got nail gun doing that for me so that i could just like connect to an spt thing and have it all hot but nail gun and j-line don't seem to behave with each other if i can if i can solve that then then this is the way to go i think you ought to be able to just like you know connect to a previous existing uh session just it should be instantaneous and it was it's like this works in principle but you can't give up like j-line stuff but on the other hand i have a half-done-ish complete rewrite of j-line in scala sitting around from like three years ago the last time i went on this route sline of course um and uh yeah so this had some great stuff right do you know what the magic space is in bash so like if you say bang bang space then like within your command line it expands to the previous thing so i had that going in the rebel so you had the whole history with magic space so that you could go do do bang and like group and it expands that is awesome um but you know j-line yeah j-line is like the source of most of the struggle it's getting the formatting right and it's like the tiniest thing wrong with you know being off by like one character means that you think you're on the left you're over on the right and it's completely maddening and the i never had the patience to work out those little details of like j random terminal and especially windows gigantic like kill windows held so much back in the rebel you don't even know because it's like it doesn't work on windows and after a day of in like a vm sitting there banging my head it's like all right i just saw just live without it but that's not a problem anymore because i can just go write whatever i want and it doesn't have to work on windows um i heard pearl 5 but i didn't hear what the question that went with pro 5 was speaking of languages that have the definition of exactly oh yeah well i bet it's more forgivable it's more forgivable and pearl to some extent just because we knew less than right i mean like a lot a lot of things i will i will give a lot of latitude to the guys of the past uh not only did we know less but it was like it was less important as well we know now though you know what's like what matters in terms of you know how a thing makes its mark on the world and i think it's irresponsible not to take things like you know the specification of things so that other people can work with it without wanting to kill themselves those things take seriously so i don't know pearl mostly like i think as an although i would never choose to use it at this point as i i have i have warm feelings about its overall existence um and rust is uh is that brendan's language at mozilla is that the one so rust has some interesting stuff going on for sure but then like then i was reading like the they're another one of these guys that say like well you're not supposed to have covariant uh refinement of your parameters it's unsound but it's what people want i guess i guess that wasn't their exact words but it was something along the lines of like well we have these optional types but they're really optional right it's like you know we're not even particularly concerned with whether they're accurate they're doing some interesting stuff but not interesting enough like i i'm fully convinced to the core of my being that the answer to our problems maybe maybe this is one of those like types or like xml or sex if it's not solving your problem you're not you know you're not doing enough of it i but i but i really think that that's like if types aren't solving your problem it's because you don't have enough of them or because they're not sophisticated enough um and like any language that's moving away from more powerful types is not it's not my thing so and in that sense scholar for all its issues i have with it is definitely like you know at least trying you know like to provide a really rich type system but i think there's a long way to go what else do you want to work on besides slime well i mean i just want to write the compiler the way i want to write the compiler to be honest i mean so i mean that's probably what i'm going to do is just sit around and like do stuff the way i want to do it best case i can i do that and it's really useful and they they use it in the real compiler but most likely it's just me writing code i think looks neat and nobody else ever cares or sees it and that's fine with me because i will still be pleased to do that at this point that has started to sound real good all right well that looks uh you guys look pretty done it's late enough i gotta hide before anybody tries to get me to buy them dinner for for my last talk thank you very much for your attendance sorry if it wasn't the exactly what was in the brochure you