Scale By The Bay 2021 : Dean Wampler, Lessons Learned from 15 Years of Scala in the Wild

Scale By The Bay 2021 : Dean Wampler, Lessons Learned from 15 Years of Scala in the Wild

Recording: Scale By The Bay 2021 : Dean Wampler, Lessons Learned from 15 Years of Scala in the Wild

all right thanks a lot hey everyone uh thanks for tuning in it's it's great to be back at scale by the bay uh i think 15 is round is a round number maybe it's not quite 15 but uh hopefully uh that doesn't matter too much um i will post the slides for the conference later and i'll have them on my personal website later as well but they're not up there yet um just a little bit about the photos if you've ever seen my talks i like to put in photos from trips backpacking and so forth this happens to be a road in nevada actually all these photos are from nevada and you can barely make out a little dust devil under the uh the d and wild just barely i don't know if you can really see it but there's a little dust devil out there um i work for domino data lab which is actually a heavy user of scala it's it's basically an integrated platform for data scientists to help them manage their work sort of like the way we use intellij to kind of integrate everything together so we don't have to think about a lot of details for managing your assets and so forth um so check us out if you're into data science or data engineering helping data scientists we are hiring we've got a lot of openings for scala kubernetes even go and that kind of expertise so please check us out dominodatalab.com it's not domino pizza anyway um and uh yeah i finished the third edition of programming scala uh i think was early this year actually i'm kind of losing track now um you can find out a little bit more at programming college.com that's a personal website and um i'm going to show a little bit of scholar 3 syntax and talk about why i think it's it's kind of a great evolution of the language addressing a lot of the the things we've learned over this last uh really scholars basically about 20 years old now um but i obviously can't go into a lot of detail so please check out the book if you want more actually there's also some other reference information at this uh website that goes into a little more depth than i can have time to go into today um so first uh you know let's let's talk about how scholars evolved a little bit um and in response to what we've learned about how to use it effectively i'm going to talk about these three points um you know greater clarity um you know moving from the concept of implicits to a re you know sort of a refactoring of it to context based abstractions and then talk about a few of the improvements in the type system and i noticed earlier that that m is cut off and i forgot to fix it so maybe you didn't notice until i pointed it out anyway um and then i want to talk a little bit about enterprise scala and some of the ways in which i think it really has rejuvenated at least for me the life as an enterprise application developer and some of the specific things that are maybe hot button issues for me i'll vent a little bit if i'm allowed to do that i'm sure that'll be appreciated by some of you um so let's first talk about greater clarity how scala is helping us have um you know code that actually more uh it represents the intent that we're trying to write is more concise even though scala has already been famous for being concise and how it's become even more concise in scholarly and so forth and i'm going to start with one of the uh more controversial uh choices that was made in scala which was to actually enter three that is which was to introduce an option to use uh no braces more like a very python-esque kind of syntax uh so on the left i have uh what may be familiar to you if you've worked with scala before or java maybe uh where you use braces it to mark off blocks here i'm declaring a trait for a monoid all right you know we're talking scholars we always have to get into like category theory concepts or whatever a mono is just a generalization of the notion of like addition or multiplication where i have a zero and i have some way to add two things together i'll actually show you a good example a slightly more sophisticated example shortly um but the point being here notice the differences from the left to the right on the right hand side we use a colon to mark the end of the name of the class or trade in this case and then indentation to indicate methods and then inside the methods they would be further indented we'll see that as well i also threw in one example of what a you know pattern matching looks like um when i heard about this idea i really thought we don't need this why is this being introduced has yet another difference but i've actually really come to love it i wrote i used the syntax almost exclusively in the book um and found that i just really liked the way it makes things concise and one of the interesting ways that makes things concise besides just getting rid of a little punctuation is you know if you've ever written like python or any of the languages that use indentation style syntax you realize that it you can get lost pretty quickly there is actually a mechanism in scala to deal with like marking the end of a block but i think also it kind of really encourages us to be concise to write concise methods to extract methods uh rather than have lots of nested code and so forth so i really like the cleanliness that this introduces and you could call this pythonx syntax and scholar 3 if you like the other thing that's happening is this notion of more intentional constructs and this is the idea that you know if we actually want to be able to reopen a type and add methods to it the way we've done it in the past is shown on the left which was implicit type conversions and i'm actually showing you the exact definition of the arrow with source object or class that's in the standard library which is a way of being able to write like a arrow b and get a tuple back and if you've ever written any of these yourself you you get kind of used to how you idiomatically use this mechanism called implicits to define this but if you step back and put on like you know the beginner's mindset or whatever beginner's eyes it becomes completely unobvious what's going on here um and yet it's a very powerful mechanism so scholar three actually introduced true extension methods where you you just say extension for some type a in this case and then here we want it to be flexible for any type and we're going to define i use tilde arrow just to you know be different than the regular arrow and then we just define the method that we want to exist on this type a whatever a happens to be and then we take an argument for the b that returns then a tuple target named is an interesting little feature that's new too this is actually the name that will appear in the byte code that's generated you can't call this method arrow to or use it with that name you have to use the tilde greater than sign but if you were calling this from java you could actually invoke this method using arrow 2 it turns out so that's uh that's what the target name is all about but um a couple things also not only is it more intentional it's really saying exactly what we want to do which is open up this class and add a method to it but it's much more concise which is really nice um to be fair there's uh you know i put in the full definition on the left so there is the second version of the method that uses the unicode character that has been deprecated so maybe if i took that out it wouldn't be quite as uh dramatically a dramatic a difference but it does kind of add up over time you get a much more concise experience using this new mechanism and that leads us into some general changes that have been made to make implicits both more intentional about what it is you're actually trying to do as opposed to having a general mechanism that is very flexible but requires you to master the idiosyncrasies of how to use it and the other thing that this these changes reflect is thinking about how people have overused implicits you know they're very much like a very powerful weapon that you can either use effectively or you know do a lot of damage um and so there's been some movement towards making it more deliberate and how we use these things for known good patterns and de-emphasizing patterns of use that really aren't as good so uh here's that mynoid example again but maybe made a little bit more sophisticated where i'm going to start with the thing that's called a semi group and all that is is the generalization of this sort of addition operator and then monoi just adds the notion of a unit or a zero or something um but just uh walking through a little bit more on the details here there's a couple of interesting things going on uh in the trait uh what we're going to do is use this to add extension methods that's the only way we're going to use this semi group so we're going to define some combined method that is currently undefined it's abstract and then it will have to be instantiated for integers strings etc and also we're going to define this operator syntax with a i use the sort of the darth vader spaceship kind of operator for plus in this case this you'll also notice a new keyword infix this actually tells the compiler inline this method do not make an actual method call and it turns out this is a really powerful tool also in combination with the brand new macro system so you can get extremely uh performant code that is where things are decided at compile time and then a lot of code for the inline obviously you can abuse a mechanism like this but it's a really nice tool for keeping your code concise as well but notice that the combine and i'll just call it the plus operator are both going to be extension methods but when we get to monoi notice that unit is not an extension method and that's because we only need a like a one unit per type um you know the zero for integers is going to be the same for all instances of the monoi so this is actually basically going to become an object method called the monoi unit for each of the instantiations for type t so again implicits are a mechanism that are is very powerful but you had to understand the idiomatic ways to use it and what we're trying to get to is something that's more intentional and we're also going to talk about the notion of how you pass what used to be called implicit arguments they're now called using clauses and the givens are the implicit values that are in scope so if we want to string monoid we we declare it with this syntax given string monoid and this is a tight declaration um that is going to uh be an instantiation of monoid with uh the definitions that we need for string so the unit string is an empty string obviously and then we define the extension method uh you have to have all these keywords that you're seeing that we even though they were already you know given if you will in the original definition uh and for strings we can actually use the plus uh the real plus operator which is a bit of a cheat but we've you know that was kind of a convenience that is in scala and java and other languages and then if you use this you know in a shell environment or whatever it behaves exactly like you would expect you know the the a lot of the usual algebraic properties of addition are satisfied that's shown in the first example here and then if i add uh to the unit and notice how i'm referencing it as like an object member um if i add one to a string monologue unit and and or reverse the order then you know i get one back so it kind of behaves exactly like we would expect and kind of a fun one is to uh define i think uh let's see this is not working there we go i hit it couldn't get the uh build to work um for numeric monoid we could define monoid instances for you know ant double et cetera et cetera but we can also use numeric and this is how you can do things like type parameters in the usual way so this will work for any kind of numeric type notice the function called summon that's uh the new version of implicitly so rather it basically behaves exactly the same but they just defined a new function with a new name so we're going to summon the in scope context for uh numeric of type t and then we'll grab the zero value from that that's how we define the unit here in this case and then with the extension method we also need to summon the uh the numeric instance in scope so we can call the plus method and now if we we can do this uh with that one definition we can do you know addition with integers doubles big nts and so forth just kind of like the previous example um i'm going to come back to this point a little bit later but you know obviously one of the things that we scholar developers end up doing is learning to master type parameterization and how to use types effectively and the power comes in into play when you see something like this where with what is it five lines of definition on the left i can i now have something that applies to a wide class of types and i could even create support for my own custom types if i define a numeric uh equivalent for them so these are the kind of things that really really pay a lot of dividends and lead to a code reduction even though maybe the sophistication increases and that has major implications for enterprise development as i'll discuss at the end just checking my time here um so just to complete the thought here uh what about passing arguments like context parameters where we used to say um you know it's an implicit argument list now it's going to be a using clause so i just sort of sketched out the idea i'm going to pass some context i'm going to declare a trait and then i do another given to create an instance of it and then i have some method that's going to do some processing where i pass in a string and then a using clause that takes one of these context objects and it's going to summon the value again and return a string so it's kind of a silly little example but the point being that you know i can get the benefits i've enjoyed in the past of using implicit argument lists when i call process as in this example um but there are some things i'm glossing over that are advantages it turns out if i wanted to explicitly pass that context argument i would have to add a parameter list that begins with using as the keyword on on this right-hand example i can't just pass the value what that does for us is that when we see this in code it very unambiguously indicates that something is being passed explicitly even though it's an implicit argument so that's kind of a cool uh way of removing some ambiguities okay there's been a bunch of improvements to the type system both to make it more sound and also to make it a little bit more convenient to use a number of ways and even to address some pragmatic issues like performance and that's the first one i'm going to talk about i'm not going to survey all the changes that's they're huge but just to give you a taste of some of the cool things that have been done one of them is a new idea called opaque type aliases and here's an example i think i stole this from the scala documentation um a way of defining logarithms in a way that doesn't add any overhead over what double has an overhead in other words there won't be an allocation for logarithms they'll just be a double in in the byte code uh and in the idiom that you follow or the i guess the practices as you declare this opaque type logarithm in this case you declare some methods for constructing instances of them and i show two here one is that we just assume that the double passed in can be uh you know converted to a logarithm now that you know that that method math log is not defined for all doubles i think it's at zero it blows up um if i remember my math correctly uh but we also have a safe version that yeah anything great anything zero or less any any um any number that's valid has to be positive that's the way i want to put it so this safe method actually does a check and it only attempts to create a logarithm if it's a valid value and then it returns an option so that's our constructor methods or equivalence thereof sort of the analog of the usual apply methods um and then if you want any methods uh you have to define them as extension methods you don't automatically get all the methods available on double so here i define three uh one to you know extract the double um and notice we we do the exponentiation of it to get it back to the original double and then plus and multiplication so that's a cool way that affixes some of the limitations of previous mechanisms for um like the any vowel types um that lets us get the performance of doubles while still getting the convenience of sort of an object-oriented notation like logarithmic um one of the cool things that everyone's been talking about is intersection and union types and these are the last two that i'll mention in the talking about type systems um intersection is really uh sort of what we've been using all along with uh with like if you define something that can be reset um so it has a reset method or i can grow it you know it's like an extended you know add new things to it then we would you know a type would be something like x's of type resettable with growable that's typically how we would do it but what this actually changes is a couple things one is that those types now can uh commute so uh something of type resettable and growable notice the ampersand is also type compatible uh with growable and resettable now there's some subtleties about this that it's it's not completely compatible in the same sort of sense but i really don't have time to show you uh some of these subtle differences it mostly has to do with deciding which methods to call in a type hierarchy that actually is still dependent on the order of declaration but what it does allow us to do is to pass arguments that are of type resettable and of type growable so if you think about a venn diagram of all the instances of resettable all the instances of growable there's some subset where they overlap and those are the only allowed values that you can pass to this because i'm calling both methods in this case in this example so that's a you know a more concise way of dealing with these kind of intersections of types the other one is unions of types so um this is a way of extending kind of the idea of what we use either for you know either left or right uh to be either an arbitrary number of types but without you don't get the same sort of magnetic behavior with this but sometimes this can be a very useful thing so notice what's happening here i've declared a you know some case class to represent users and i'm going to get a user from a database and the return is either going to be a string which will represent an error in my case or i'll return a single user or i'll return a sequence of users and the way you have to work with these things is actually using pattern matching because you don't know what the type is when you call in so what i'm doing here is i'm you know sort of mocking out the idea i'm going to run a database query where i'm going to select all the users for id equals something maybe that's not a unique you know single record so i'll look at the size of the result set and if i you know if it's zero then i'll just return that string which is my first option which is an error message if the if it's one then i'll just return uh i'll assume that the result set is like a sequence uh whatever it is but then i'll just return the first element and maybe i have some conversion that converts it to a user type is what i'm mocking out here if you will um otherwise i'll just map the sequence to a sequence of users you know whatever the result type is that comes back but notice that pattern matching has to be used here uh in well in this case it has to be used to make sure i return the right instances of something more interestingly though is at the bottom where i use pattern matching to analyze the result of calling git user so i immediately do a match on it and then you know is it a string is it a single user or is the sequence of user and then handle those three cases separately that's the only way i can do it you know you could do things like as instance or is instance of which is of course not the way we normally do things but if you're using union types like this then you usually use pattern matching to figure out what you actually got okay um in my remaining time i'm going to talk a little bit about um sort of at a larger perspective if you will of where things have gone right where they need to go better in scala and how we use it and how we can you know i wanted to be very direct about this sub title how can we unlearn enterprise java habits that i think often drag us down even when we're writing code and scala and i speak from you know years of experience working in various scala projects and i'll just talk about three areas um obviously most of you that are watching this talk are probably already either into scala or into functional programming maybe you're using another language and you you may believe that object-oriented program is garbage i don't believe that personally i think scala does a really good job leveraging the best strengths of both but i really do try to emphasize functional programming in the following senses one of the uh sort of mental exercises i often go through when i'm starting a new use case or user story or whatever it is i'm implementing is i like to think if it's sort of data centric what would a sql query look like that would implement the whole thing could i write a sql query that could actually do this whole thing without all this scholar java whatever code i normally write because it's hard to imagine anything more concise than a sql query now if it's more like a process that i'm building then i might i often think of it in terms of like a bash script which for me is kind of you know about as concise as sort of a procedural kind of problem can be written but um you know when i think about something like this a trivial query here then uh you know everything else is kind of boilerplate in a sense i mean i do need things like security and you know failover and logging and all that stuff but a lot of the problems i see in code basis today is really forgetting how concise things can be and adding a lot of unnecessary ceremony or not knowing where to cleanly separate the business logic from the required ceremony that wraps around it um the other thing that i think is very important is parametric polymorphism i wish i had a little more time to talk about this this this is a blog post i wrote where i talked about its virtues but i actually talked about it as a way of drastically reducing code boilerplate uh and not so much about what i'm talking about on the slide although i mentioned it at the end what this slide is about here is if you look at the signatures of these two functions and you ask yourself i have i can't see the body of these functions the names don't mean anything obviously what are all the possible implementations that could exist for these two functions and if you think hard enough about the first one it turns out the only reasonable assumption is because t could be anything this has to be the length method or size there's really no other reasonable choice i could make and what's beautiful about that if you look at languages like idris more advanced functional languages they can actually generate the code for a signature like this because they can very precisely determine the only allowed choices and they could generate tests for it or whatever as well whereas if you think about the second example i'm the paradox here is i'm being more specific about the type that is coming in but i'm actually losing a lot of this sort of reasoning power about possible implementations you know this like this could be the first element the last element um the you know the median value there's all kinds of possible implementations that would satisfy you know in some reasonable sense what the signature is as well as the size method so the paradox again is you know by making the type more flexible i actually constrain the allowed choices of implementation and that actually gives me a lot more precision and and avoids a lot of ambiguities that are possible so i really like to think about you know how can i use my types to set the proper boundaries of what what this should be able to do here's something i've been wrestling with a lot i don't necessarily have the right answer for it but when should we really faithfully statically strongly type everything that we work with and the example i'm wrestling with at the moment is if we work with a lot with kubernetes domino so i just just grabbed this sample yaml from this kubernetes website and it's very tempting we do this a lot in the enterprise jvm world to have exactly this schema represented in code because we're going to stuff in values for some of these fields and then we're going to spit out yaml so we've got also this ceremonial code for you know reading and parsing yaml and generating yaml from a map or whatever data structures we've declared and it really does seem like a total waste time to me so one of the things i've been thinking about a lot lately is maybe we should be reading templates of these files and ironically the word template happens to be in this uh this yaml example but we should have kind of the way we do interpolated strings let's just load this in and have the minimally sufficient information in the code itself and not have to know about everything because as soon as we change the schema you know a new version like v2 of this api comes out then we we end up modifying code that maybe doesn't really change anything that it does of use so i'm really thinking about ways in which stuff like this ought to be either replaced with maps where we don't worry about the types we just treat them as strings or cases where we we just have the minimal amount of type information for the business domain and the rest passes through as data instead of code um so anyway something i'm thinking a lot about the last point is that if if we apply all of these methods all of these um lessons that we've learned that we get a lot less code and that has enormous benefits for us here is a complete uh a spark program this is taken from a talk i did maybe five years ago it does the inverted index calculation that's not important for our discussion well uh i highlighted uh the met all the methods in yellow here and those of you that you know write scala you know these are all friends right map flat map filter reduced by reduced by key is a i think a just an extended method in the spark api nevertheless if this is the whole program then you know do i really need a lot of the other stuff that we typically pile into our environments do i need a dependency injection framework of some kind do i need fancy mocking libraries for my tests because my dependencies are out of control because i used a dependency injection framework to let me avoid solving the problem of my bad dependencies do i need a lot of the design patterns like factories and adapters and so forth i'm not a lot of functional people uh sort of like to bad mouth design patterns i actually think they're a great idea it's always whether they're used properly or even necessary that's really the issue and then if we don't have a lot of code maybe we don't need as many microservices except for just failover or scaling and maybe we don't need to split things up because our domain is has become less complex as a result of just focusing on the core abstractions of the domain that we have to implement in code and let the rest be data so anyway that's my view on reviews if you will on how scala has made me a much better programmer but we've still got a long ways to go to make enterprise jvm development uh you know as optimal as it can be uh the links here my twitter profiles dean wampler you'll be able to get this talk from the conference website and and from my website um uh probably later tonight and then i also my blog is on medium now and do check out domino data lab thanks you