Scale By The Bay 2021 : Bill Venners, Advanced Features of Scala 3

Scale By The Bay 2021 : Bill Venners, Advanced Features of Scala 3

Recording: Scale By The Bay 2021 : Bill Venners, Advanced Features of Scala 3

hi everybody um i am uh so many buttons on my screen i couldn't figure out where the play button was but i am uh going to talk tonight about uh advanced features of scala 3 and there's not too much i can say in 30 minutes uh so i just wanted to kind of give you uh a few in if a few uh views of various features that that are probably not something you'll run into that often they're the kind of things that people use when they're maybe you know when they're designing a library or framework but you don't tend to to use them uh every day except through libraries and frameworks um so anyway uh i'll um actually just it's kind of a grab bag of things uh so the thing i want to talk about was opaque types which um isn't really advanced i think but but i want to start with it to warm you up um this is something new in scholar 3 that is trying to address an efficiency deficiency of any vowels and any vowels were trying to address an efficiency deficiency of any ref so any vowels look like this and this this works in scala two or scallop three this still works in scholar three and um basically you just make a class looks like a regular class we say extends any vowel except it has some special rules that you have to follow and one of them is that this class has to wrap one other class so here class text wraps a string it has to have one and only one class parameter there you have other implicit parameters if you want but the efficiency that this will do is that at run time when possible the compiler will actually just pass around strings instead of texts and any methods you add to this class will be optimized into like an extension method essentially where you pass the string in first and then it has the other parameters so it's it's an efficiency thing the um feature of this guy though is it will box when need be like whenever a primitive type would box this would box which in boxing means going using means going from the stack to the heap but in this case like since the string this is the jvm strings are always on the heap already so it'll make another object on the heap called text instance of class text that points to the string so if you like for example make a list of text it'll make you know if there's a thousand texts objects in that list it will have a thousand text instances on the heap but if you just pass a text to a method it will actually just pass a string so so anyway um it has another feature that for example you can implement an interface and so whenever you use it through at that interface it boxes it so there's all kinds of times when it boxes um so what uh was desired for a long time was a way to not ever box and that's called an opic type and so this works in scholar three only um an open type is sort of built on type aliases which existed already um a type alias is just you just say type uh fred equals ginger and now the name fred means ginger means the same exact thing and um if you put opaque in front of it in scholar three what that means is that inside the template that you're in so i'm in object example that's the template samples either an object a trait or a class um it's just the same thing it's just a regular old type alias they just are two names for the same exact thing the legroom just means double but outside um there it's a logarithm is opaque in the sense of you can't see through it to double you don't know it's a double and um and so that's basically kind of like if you look back here text at runtime is really a string most the time but you don't know it's a string so it's that kind of concept logarithm is really a double at run time but you don't know it's a double so because um there are two different things you're going to need to have offer some ways when you make one of these for your users to you know given a double give you get back a logarithm so so i've got two factory methods here um and what you can notice because this is inside the template like this first apply factory method it it actually the right hand side the body math.log d that's going to produce a double and i just say it's an algorithm because it's a type alias here it's just a regular type alias um because i'm inside the template object example so this is my factory methods and then the other thing i've been back up here to my my my uh any vowel is i can add methods right there like a regular class and you can invoke those methods on instance of text and it's optimized the way they don't box if they don't need to and so i can't do that here because there is no class so i'll never get an instance of class logarithm because there isn't one uh it's just a type name of a type so what i need to do is use extension methods so in these are scholarly extension methods and so this would be that this is basically the pattern of making an opaque type by i say some inside sub template i say a pic type something equals something else then i'm going to have some factory method so i can go from the the underlying type to the opaque type and then i'll probably have some extensive methods to define the public api of it so um so that's you know one use case for these for both of these things both any valves and opaque types is uh tiny types so if you have like a bunch of strings but they kind of mean different things like once an address once a street one's a uh a city wants a state then you could actually make these little types like class state that has an underlying string extends any vowel and and uh it sort of gives you type checking for not much cost at run time because there's a there is a cost to every class that's instantiated on the heap um so that's the cost we're trying to avoid with these two features um so that's one one um use case another one that's really nice is you make a little hierarchies of these things so for example it's galactic any vowels we have this package called any vowels we have a bunch of any vowels for different kinds of um uh well usually numeric but they can add other things but they're narrow things like right here text will take any kind of string but we have an anti-vowel called posint that has an underlying int but it promises that it will always be a positive integer and then we have one called posient that it takes underlying it promises that it will always be positive or zero and so if you think about it um if i have a if i need a positive or zero end and i have a positive in hand that should work it's really a subtype because both of them are underlying our ants and uh any any positive is a valid posey end right so that's like a subtype and the way we implemented that uh in scott in scholactic was we just have implicit conversions in scala too from pause and to pause the end it makes it kind of a subtype and if you think about like there's one way to think of the i can pass a subtype to a super type where a super type is expected there's like an implicit conversion from subtype to super type when i do that so we hard code all those but they're not transitive i have to have every single implicit conversion explicitly done in opaque types you can actually put a little less than colon something so i could say opaque type posit less than colon posey end equals int if i wanted to do that so that that's a nice way to make little hierarchies of these things so that's um that's basically opaque types um the only thing i think i wanted to mention is this is still around it does have other features so i mean different feature set maybe it's useful to stand on its own but it's kind of almost two different ways to do the same thing um there is a project installed in java called valhalla which is to put value types into the jvm and so maybe this will get sort of shoehorned into that use case if that if and when that comes into the jvm but anyway so that's that's the status of that so that's one uh and i just said this kind of a grab bag so i'm going to kind of go through these quickly uh but we can ask you can ask questions after so type lambdas um we're out of the scholar three because they were wanted by the community for a really long time and the community one is so bad that they came up with very uh well a very ugly way to do it in in scala which turned out to be unsound uh from the type system point of view so that got ejected so they can't do that anymore so they kind of needed this to replace it but also um a lot of people use the compiler plugin to get this thing uh but anyway um it looks like this and i made it purple for two reasons but it's a that's the color i call it type so i'm syntax following the others is a little less scary looking but it you can just think of this as another lambda but it's a lambda at the type level so um and it looks like you know these are all types k and v are types but this is really like a oh god i don't want to say it that way that makes it sound worse but it's just a lambda like like in lambda calculus like we have these uh function literals in scala source code that are terms uh this is like that but at the type level and so its parameters are in square brackets because those are types not terms if they're in parentheses that means there are values or terms that you have to pass in here i have to pass in type so that means this thing's going to be evaluated at compile time and so if i pass in like string for key for k and int for v to evaluate that it just substitutes and what i get back is map string comma hint so that's all it is and it's used to for example to uh like if uh you know you want to make a functor of an either uh well that actually got solved a different way but anyway it was used to um uh sort of in type level when you're when you're trying to change the number of uh usually it's used to change the number of holes in a type like this is a you can think of this as a type that has two holes in it i got to plug in two types and i get back one type so anyway that's called the type lambda and um the one thing it's a little interesting design choice that i wanted to point out is that in scala um type constructors and you kind of think of this as a a way to construct a type i pass two types in i get one back have variance right because you can have subtypes so the v over here in map if you go look at how map is defined uh the v has got a plus sign next to it that means it's covariant and in these type is in scholar three you cannot put a plus sign next to the v over on the left hand side you can't actually do that um it will always be inferred it says because v is equivalent on the in the body then v is going to be covariant in the parameter even though it's not you know there's no plus sign there so anyway that that was um that's how that's done in scholar three and the other thing i was going to say about this is you know people wanted pure functional programming like in scala you can do anything anywhere so our lambdas can have side effects this one is pure so it actually is like lampa calculus like it was meant to be it doesn't have any side effects when you evaluate these things okay so that's that so something looks a little bit similar is polymorphic function types which i think will be used a lot more rarely but it's two things it was you know there's use cases for these and people ask for it but it also reduces removes something that is different between methods and and function values um i think there's a kind of simplicity you can get when you when you don't have special cases and if a method you could like you should be able to think of that as a uh as a um [Music] like a vowel instead of a death it's a vowel that refers to a sorry i'm at my friend's house and this is his phone answer machine so anyway uh a vowel that points that sort of refers to a a lambda i mean that's another way to think of it i'm going to say that again so a def which is a method in scala is really conceptually no different than a vowel that is referring to a lambda it's it's you know you apply the lambda you pass parameters into it and you get the result right that's what a def does it's just like syntax sugar for that except it has extra features there's things you can do with that so you can't do with vowels pointing to lambs for example you can't do var args um when you ate expand a a death that takes vargs it actually gives you a seek right there's no way to write that by hand there's actually a weird way to get of our lambda but uh that's also not regular um so you know you can't actually write a literal with firearms as a lambda you can't um do default parameter values either you can do that in a method but you can't do it in lambda and the thing in scholar 2 another thing you can't do in scholar 2 is you can't have a polymorphic function value but you can't have a polymorphic method so this rev method here takes a type parameter a and then then the parameter that it takes the imprins that's the value parameter is a list of whatever that type a is so if you say a is string like a pass string in there then i have to pass a list of string and i'll get back a list of string if you pass in there then i have to pass in less event i'll get back a list of n right that's just called parametric polymorphism and uh we've been able to do that for methods for a long time but one if we were to a to expand that you can aid expand such a method and spell it two which means turn it into a rapid and a function value it will pick it will infer and pick a type for a so i'll get a function value in scholar two from list of into list event or one from list of string to list of string depending upon what the context is and that's the exact same default in scholar three but what's added as column three is i can actually write that same kind of thing and it looks like this the second line there it's called fun it is got in square brackets an a that means that you're first gonna have to pass in to this lambda a type sort of you think of it that way and then it will produce you know the thing to the right of that arrow uh which is a function value from list of a list of a whatever that a was uh so that's the fault that the purple thing after bowel fund colon is the polymorphic function type and so it sort of means the same thing as def rev up there given an a then it's going to take a list of a and give you back a list of a that's that's the type that means that and then the the literal the function lateral or the lambda is the the part on the next line here square brackets a rocket and then you have a functional regular function level that uses that a um so you always have to write that out explicitly uh otherwise it will assume it's going to like early decide what a is so that's a polymorphic method and i think what's going to be sorry function type and what's a little bit confusing is we got a lot of different kinds of arrows now are not arrows but um like this square bracket a regular rocket that if you see that that's going to mean that either to the right hand side is a type which is a function type that's the purple one on top or it's going to be a a lambda which is the value and and if you compare that to this guy what's different is um he's got a double arrowhead so the double arrowhead means type lambda you know i've got some types in square brackets and then a double ahead i've got a type lambda what's over the right hand side doesn't have to be a function type and down here i have the single arrowhead and what's to the right of it's got to be a function type because that's a polymorphic function type and value okay so that's one so there may be questions about that um but we can talk about that so i want to talk next about match types which also got added in scholar three and this was because um people wanted this what they want to be able to be able to compute a type from a type essentially and there is a way to do that in scholar two which is to use implicits to abuse implicit lookup and so everybody did it including me because that's how you did it in scholar 2 and so what scholar 3 added is this thing it's called a match type and so there's that type keyword again and i always thought it was kind of funny that sometimes type keyword that the type here would have square brackets like that because this looks more like a method actually um and uh it is i think a good way to think about this like the difference between a type of lambda and a type match type sorry is that a lambda is like the the type lambda is a lambda and a type a match type is the method uh it's you know the name of this method is it's all runs at compile time you know it's a type level method its name is elem it takes one parameter x which is a type and it returns a type and you know basically when you call lmx you just replace that or when you mention it and as a type it will just replace it with one of these result types here so it's just works like a match at the value level except i'm matching on a type so i say x match we're here x is a type at compile time if it was a string at compile time then the result of this match type is care if it was an array then it's going to you know the t is the element type of that array it's going to pop out and be available on the right hand side of that rocket and and you get uh that is your element type so if there's an array event then the result of this is going to be end otherwise if it's an interval again the t is going to pop out so if it's integral string then your result will be string and if it's not one of those then that's a match here right but this is a compile time this is always run at compile time so it gives you a compiler error so that's a match type and so i think just one way to think about this is i you think about this is like the method the type level method to the type lamp is type level lambda that's really what they do the same thing given types being passed in you get um a type as a result and it just substitutes okay so that's uh that and i'm just going right along i'll give you another one dependent function types um so i just as an aside one of the sort of the original uh tasks for scholar three was to simplify the type system by replacing three pieces of it with exact replicas that were defined in terms of other existing pieces so an example of that is in the scholar 2 language spec there's their description of type parameters and a description of type members a definition of them they just define them specify them and they're two different things so in scholar 3 there's you know the description of type members but then type parameters are now defined in terms of they're like syntax sugar for type members so that you you define this in terms of here's how you rewrite that's the type numbers so really you just have one feature so that actually was a i thought a big accomplishment it had to do with like the soundness proof follows the soundness proof just talks about type numbers um but then a whole bunch of other things got added like you've just seen several of them right so typical is another one and this one again there was reasons for wanting it i think a big one was just consistency or regularity uh that you know it's another case where in scholar two a method can have some feature but a lambda cannot so in scholar three the lambda can now have this feature and the feature is that a result a method's result type can depend on its parameter a type member of its parameter type so here i've got a trait entry and that has one type key that's an abstract type so to have an instance of entry i have to have a class that extends entry and fills in some type of key so let's say type key equals string class string entry type key equal string or class and entry type key equals n so i have some like concrete subclass and what this extract key method the dependent method does is that every call site there's some subclass that's being passed in and i need to know the specific subclass so that i can tell at that call site what e dot key means so if i'm you know passing in a string entry somewhere then at that call site edaki means string and that type checks so if i'm passing an int entry somewhere then edaki at that call site means int so that's how that works that's not used that often but it's actually quite useful um it's used in libraries uh dependent you know path dependent types um but before uh you know i don't know if you got an error when you try to aid expand something like this but you couldn't actually make a lambda like that but now you can so that looks like the the second line there val extractor and then the purple thing is the type i make types purple and uh except forgot to make that one type purple but anyway this type is purple and it is a a lambda type or a function type that takes an entry and returns the key type inside that entry there's a path spin type there so that's now possible in scholarly also it's called a that dependent function types all right okay one too many types of types i think uh this is not a scholar three marketing this is all like these things are in there you'll rarely see them but um uh it's kind of good to know that they're there because you might see them now and then so i wanted to finish with talking about meta programming because um i think that's a really big thing that got matured and and finalized in skull three it's something that uh was experimental in scholar two and um everybody used it because it was so useful uh so we all had to rewrite our libraries uh to use this stuff and what it really i think is about is that i think i've whenev ever since i've been programming i i often would feel the tug to automate writing source code so uh i had one summer job where i was given all summer to to finish this guy's work where uh he had written by hand all these really long lines of ones and zeros to test some hardware and that was going to be my summer job and i thought well this is very tedious and you can automate it so i wrote a c program to write that code so it's a program that writes a program and so basically after a week i said okay what's next so that that was one uh and it just is like we're programmers and a lot of what we do is repetitive so we tend to want to write programs to do that and we had like at one point i decided that none of the java web frameworks is a big mistake one of the java web frameworks were good enough um so i wrote my own and as we're still using it we're still stuck with it and it uses a lot of code generation because there's a lot of repetition in these web frameworks and that's what i thought could be done better and it works really nicely actually but what it does is generate java source code and then we we use that java source code and you know it generates a class and we extend it that kind of thing we do things by hand but there's a lot of downsides to it like um uh like when you go to your id and look at the by hand written classic since the generator one well it hasn't been generated yet so it doesn't see it right so the ide is it's just it doesn't it has clunky id support and there's a bunch of other issues um so one time i remember mentioning that i like to do this code gen thing on the early days of the scala mailing list and i got a huge backlash that like you know complaining about that all the problems with with code generation which are real um but what one person said is that if if you're doing if you need feeling they need to do code generation then there's something missing from your language and i think what all this meta programming stuff is about is trying to bring that kind of thing into the language where i can write programs that write programs but it's all part of the language so that you know you'll get good ide support and all those other issues are not there right so um so there's a whole bunch of things to it i'm not going to show you everything but what what uh what i'm going to start with is inline inline is is just it's a new keyword uh there's an atm line in scholar two which is different that's like a hint um but this is like inline uh and inline um without an ad sign means definitely inline and inline just means you know if i have an inline def like here i have an inline def log instead of when i invoke log pushing a stack frame you know creating a new stack frame on the stack and pushing you know the parameters there uh and then jumping to the log message method and it executes using the stack frame and it uses the first local variables and it puts its result there and then it you know returns it takes the code of the body of log and it just sticks it right it just replaces where you call log it just replaces it right there uh and that's um that's what inline means it just do it in line it's the same thing the jit compiler does to make your program faster if it's if you're calling a method over and over really fast and and you don't want to have the it decides it would be faster to just take the body of that method and stick it right there so that's why that like the uh you know the hot spot of your code with the you know hotspot uh jit compiler you know a lot of stuff just gets flat there's no method calls because it inlines everything that is needed to be in line for performance so it's a space time trade-off so anyway what this is going to do when you call log is it's going to in because it's in line is it's going to actually put that body there except there's a little bit more to this story because there's logging up there at the top is inline and we have a little if you know and if then else in the inline def that it switches on that logging value so because it's an inline it will actually decide it will either do the then side or the l side every time it's used it won't do both of them it won't do this if at run time it will do the if at compile time so for example um if i i have this factorial method where i say hey log that i'm calling factorial and here's the body of factorial that's in the curly braces that's the op by the way if you look up here opt that is a by name to some type t and then it returns t um so if if logging is false at compile time then you're just going to get the up you see let's go hop down to the else it just gives you whatever the op was but what it's going to do is it's going to take the code that you pass in these curly braces and just stick it right there so here if logging is false you look at the bottom of the screen my factorial method turns into this at compile time essentially that's what i get but if logging is true then it's going to go to the other branch and it's going to actually stick a log message before and after essentially and the vowel result equals op and it kind of remembers what the result was and it returns it at the end after the log message goes out that's the secondary log message so you can see that here this is if logging was set to true now i it just replaces that with all that logging junk i get and result equals the body that was passes off so that's how inline works um it really is basically well named it's inlines that's what it does and then there's another feature called transparent inline which is um what happened to white box macro basically what happened to macros that can compute types uh that was uh useful but very dangerous and very hard to understand and just it decided was too well so this is a more disciplined way to do that so what works the way this works is um so i have an example here there's an animal and a bear that extends animal and the bare meth the bear class has a roar method but the animal class does not um so i made a transparent inline choose method so this is an inline method it's going to do the substitution thing but what what's different about transparent is now the the result type of this method which is animal it's said is animal it it a compilation may be narrowed to something that is a subtype of animal so it will always be an animal but like normally when you say it returns an animal then at runtime it can return an animal or a bear or a cat or whatever but what transparent inline means is that at compile time we may decide it returns tight bear and it's just inferred so if you look in like the body here just says if c is true then i'm going to return a new animal that means the result type is going to be the result of the substitution the code that's substituted is exactly animal on the other side um and that's like val a equals choose true basically that's the true side and the static type of a is animal um if c is false then it's going to say new bear but because bear is a subtype of animal the actual static type of that substitution when you do the inline substitution is going to be bear not animal and so val b equals choose false that the the um type of b is is bare so that's what's new and um you can all they can also be recursive and you know you could do all kinds of stuff and if you look at a dot roar of course that's a compiler because there's no row or an animal the b dot works fine so that's called a transparent inline and the other thing the last thing i was going to show you this is like one little slide for this massive subject but it's quotes and splices and i think the a really useful way to think of this is it's like string interpolation but for source code uh so there's this thing called an expression expr which is like an ast of source code an abstract syntax resource code you can think of it and that's like you know i'm writing a program that writes programs this is my data structure the you know i'm going to manipulate these trees called expr's expressions and the quote is like double quotes on a string i'm i'm saying this is string but here the code is it's code it's an expression that's the data structure for it um and then if i have a string interpolator say dollar name like say hello comma dollar name in my streaming interpolated string it's going to take that string or call tostring on that guy and then splice it right in there so what happens here is message expert is an expression it's an ast it sort of evaluates it excuse me it just puts hello there oh man i'm anything it's my 30 minutes just finished about 30 minutes just finished right when my voice ended let me finish one more minute um so what what happens here with the doll sign is just like plugging in when you do string interpolation except in plug instead of plugging in a string you're pogging in the you're evaluating the expression you're plugging that in so what you'll get there is just the hello not the expert hello because i have the dollar sign there but because i have the quote i get an expert of print line hello so you can say show and that just shows you this what the source code would look like this print line expert is actually an expression of type a unit um but uh but it really represents if someone wrote in the source code print line hello and so that's how you do you know if you make an inline one with this that's how you do macros now so anyway that's that's my 30 minutes quick overview so i can uh take q a i just want to mention i did make a a discount code for you guys there's a fifth edition of programming scholar that does cover scholar three not all this advanced stuff yet we're going to move another volume for that but um uh there's sbtb pins five that's um discount code if you wanna use that through november 15th and i'm ready for q a you