Scale By The Bay 2021 : Adam Fraser, Next Generation of Scala

Scale By The Bay 2021 : Adam Fraser, Next Generation of Scala

Recording: Scale By The Bay 2021 : Adam Fraser, Next Generation of Scala

[Music] all right so thanks for joining today i'm here to talk to you about the next generation of scala so first a little about me uh my name is adam uh as i mentioned i'm a car zeo so uh maybe a little bit uh biased there but i will uh try to disclose that at least uh i first learned scala from barn naderski's coursera course like some of you may have and i remember from the beginning i always loved the beauty and the yellow elegance of the language i felt like it let me say what i wanted to say and not have to say anything more but just say what needs to be said and i also got a lot out of the type system as i find whenever i try to do something in a python or a language that is just not strongly typed i have many runtime errors and then i'm very glad that i have the scala compiler so i am i'm a big scala fan and uh normally i talk about particular features in xeo but i thought for this talk i would step back a little bit and uh try to share at least my perspective on where scala has been in the past and where it's going and i think this is a really good time for that because i think the future for scala is really bright right now with both scala 3 coming out and really doubling down on some of the things that make scala scala as well as some of the libraries that are coming out right now that are really just helping solve practical problems in a in a very powerful way hopefully zeo being included in that but at the same time i think we've had uh periods in the past where we also felt like uh the future was very bright and maybe sometimes that hasn't been fully realized of i think even if we look at the name of this conference of scale by the bay it used to be scala by the bay and it's great to have more friends and more languages and look at things together but i think that partly reflects the fact that scala maybe didn't grow as quickly as people thought it would to support a conference conference just being scala and i think you can see that in other ways where i imagine many of you may have used scale in the past and may not be using it uh today so i i want to spend some time looking at where we've been and where we're going and hopefully encourage you if you are not working with scala today that there's never been a better time to come back so with that where have we been well i think to answer that we've got to start with the language of scala and what makes scala special and there's this quote from mar nodersky from one of the talks on scala 3 that he did that i thought sum things up very well and he says that the essence of scala is a fusion of functional and object-oriented programming in a typed setting so he's saying there are these two different things there's functional programming there's object oriented programming and scala is going to let us get a fusion of them so that sounds great right we get the best of both worlds we can use these very high level functional abstractions we want to and we want to we can manipulate arrays and we can use each of those tools when we want to for the problem so sounds really good what's the problem well the problem is that choice creates uncertainty that if if you just tell me like hey i want black coffee okay i can get you black coffee but if you say well i want some coffee i want some milk does that mean i want coffee with a splash of milk or does that mean i want latte as a milk with a splash of coffee and we've got to kind of figure out where we want to be on that spectrum between object oriented and functional programming so we can be more object oriented which we might call a better java we might be very far on that functional programming side of what you'd call a poor man's haskell hopefully what we want to be is what you call distinctly scala that sweet spot in the middle where we're taking the best parts of both of them and putting them together in the right way but what exactly where exactly on that continuum is that uh sweet spot and what does that look like not everyone agrees on that and that creates challenges for teams because if you're an organization you can probably be productive with any of those styles but you probably need to be consistently using one of those styles and first thing you have to do is just decide where you want to be which can be particularly challenging if you're a new organization adopting scala because you don't necessarily know what you don't know of what the trade-offs are of where you want to be on this continuum between object oriented and functional they need to onboard new team members and you need to get them on board with the style of scala that you want to use in your team which can be challenging because they may have their own style of scala that they're more comfortable with or they want to use and you've got to maintain consistency there which can be challenging i think a pretty common thing i see is people have someone come aboard it really wants to do things a certain way they do part of the code base that way and then they move on for whatever reason and you're kind of then you're stuck maintaining that and then the other thing is the users you've got to decide which libraries you want to use because you may have some situations where there's a certain library that is the best point solution for this part of your stack but if it has a very different style than what you're trying to do if it's very functional and you're trying to just do better java right now or it's very imperative and you're trying to be super functional neither of those are going to be great for you and the same challenge exists for library authors so as a library author you might say well what style should i write my library in and at one level the answer could be whatever you want you're the your library author if you're taking your time to do it do it however is interesting for you but you might say okay well that's right but i want to do it the best way for my users what what is that and may not be entirely clear and then you also have this issue of how do you integrate with other libraries because just like users have this problem you're going to have an issue that if you're taking one style you may have a challenge integrating with libraries that pursue a very different style and all of this creates risk that as a programming community is either a company or as a group of open source developers yet decide where you want to fall on the spectrum and there's risks to being too far in either direction so if you're too far in this better java direction you can make things that are really useful but there may not be a reason for people to do them in scala versus in another language on the other hand you can go very far in this functional poor man's haskell direction and you can create some things that are really really interesting but that are maybe not that useful for at least the average programmer and hopefully what we're doing is we're getting to that distinctly scala sweet spot where you're developing something where people in other languages are saying boy i need to use scala just to get this software but finding the right place to land there is challenging so let's take a look at a couple of examples of this so i think akka is a great example of something that falls on that better java end of the spectrum it's it's very useful so it takes this actor model that was very useful in languages like erlang for some very complex challenges or things like telecommunications and it brings the jvm these actors encapsulate mutable state so they let you work with immutable state in a concurrent environment uh in an otherwise imperative framework that's probably more similar what you're used to in a way that lets you manage that state better and actors are also really interesting because you can think of an actor as a microcosm of a node in a distributed network and so if you build some system with a set of actors that are communicating that naturally scales up from uh concurrent processes on one machine to a distributed system in a way that may be less obvious with other frameworks so really useful there which is why it's had the success it's had on the other hand [Music] at some level you could say you didn't really need to do it in scala the fundamental signature of an actor is a partial function from any to unit right it says hey send me some message can be any message at all i'll pattern match on it and decide what to do with it and then i'm going to do some kind of side effect based on it don't know what it is now that's completely untyped so you clearly don't need to use scala's type system to do that since then now there's a type there are more types here but the fact that akko probably was most predominant in a period when it didn't have types indicates people weren't going to active because they wanted pipes people were going to acca for all the reasons that we talked about on the previous slide and so the implication of that is that over time now you don't need to use akka in scala you can use it on java and that community has grown away from skull a little bit which makes sense because there are lots more java developers and if you don't actually need scala then it makes sense to broaden out to this wider market another example of this would be spark so spark's incredibly useful great software solves a variety of problems in distributed computing of if you're working with data and data flows in a distributed setting it solves a lot of problems for you this collection analogy that spark promoted where you get to think about these distributed data sets largely as collections and work with them with operations on scala collections is extremely powerful and i think a really good example of abstracting over complexity for users and it's probably the most impactful scale project industry so there's a lot of model out of credit to go there at the same time it's another one where there's not necessarily a reason to do it in scala and that's why lots of people don't do it in scala and they do it in things like python with pi spark and so now when i talk to data scientists they're like why would i do it in scale i do it in python and so skal in a way is great for smart contributors of they get to use the scale language and its features to be more productive and i think particularly in the early days of spark there was more of a differentiation between java and sky there so that's wonderful but the fact that you to really use it you generally don't need scala uh has led to people not doing in scotland so there was at one point this idea that spark was going to lead to scala becoming the lingua franca of data science and that never really happened which yeah again it was great that the software exists and it was developed in scala but maybe if we if we're big scala fans we might say well you know it's a shame that it didn't end up bringing more people to skellin getting more uh development just in scale itself so that's one extreme the other side we can be on is the uh go very functional or what you call the poor man's haskell side and i think you could think of scala zed or cats as going in uh in a little bit of that direction and so on the one hand they brought some really interesting ideas into the scala ecosystem so these underlying concepts have have some deep beauty to them at least at least to me and i remember the first time i learned about monoids and foldables and i saw how many collection operators you could implement in terms of them and then how that could generalize to more probabilistic structures like loop filters and sketch and how you could aggregate these across distributed systems like there there was a deep like beauty and structure that i thought was like really cool um in addition to that these were the uh first libraries particularly like scala zed was the was the first time you could really do functional programming in the large in scala before that the sweet spot was definitely you would use functional programming for like your collections but then when you wanted to interface with the network you would go on and you would uh do it imperatively and this was the first uh tools that let you program in the large in the period of context it's in a functional context so a lot of really interesting ideas that they brought to the scale ecosystem at the same time there are some aspects of this approach that i think make it less useful at least to a broader group of developers and i think maybe the the first thing is just the goal a little bit i think of these libraries was to let you do functional programming in scala which is great but in a way that's a problem you have if you're a functional programming fan and you've just got to work in scala because that's the job you've got that's not a problem that organizations have right companies have problems like i need to build this distributed system or i need to ensure a very high degree of correctness in this thing that i'm building and functional programming can be a great means to accomplish those ends but it's a means it's not an end and these libraries could often have very high barriers to entry where like i've included above the signature of an http route from one of these libraries and so it's this classly of an option t f of a question mark of request and response and this whole thing is actually not even valid scala 2 syntax you need to add a third party compiler plugin to make this compile because the actual signature of this would be too ugly for me to show you on this screen and if you you know if you've kind of read a book on functional programming then this makes sense if you haven't read the book then you're going to have a bad time and you have no idea what this like placely your option t thing is or why this question mark is here and what this f is and so it just creates this like fear and entry where if you're in this like small group of people who like doing functional programming in the style it's like fantastic for you like this is the tool for you but if you're kind of an everyday person who's like hey i just want to like deploy my like server this is like okay now you've like added a whole bunch of other stuff i need to do before i can actually just deploy my server and it can also often just lead to a poor developer experience because even once you overcome that barrier there's just often additional um boilerplate there of kind of like even here you've got like these three layers of nesting and going through them and using these different operators to do it where uh it's not as easy as it could be and i think one thing that kind of jumps out here is this like classy thing of you might wonder like what is this chrisley thing and it turns out it's not it's not what is cliche but who is spicely so he's a mathematician he's a category theorist he there's this thing called cleisly category that's associated with him so the name is kind of to pay homage to him i guess which you know is is nice it's a very nice thought and if you are someone who's coming from that category theory background it's probably going to be very clear what this thing is to you but the flip side of that is that no one else is going to have any idea what this thing is and admittedly there's already some complexity of like what this is i mean it basically adds the ability to have context to some other mode ad which represents some kind of effect uh but yeah you could try to pick a word that would like convey that in some way you could call it contextual or sometimes reader t or something but this is like no you know we're going to really focus on like this one group of people and we're going to make sure it's very clear to them but it's really not going to help anyone else as far as naming and i think this gets a little bit to one of the the challenges that i think we have to like catch ourselves on as people who who really do love functional programming is what i call the learning fallacy which is this idea that well like they should just learn about it like if that like classly of option t thing of look complex then you should just learn about monads and known ad transformers and once you learn about those things your life is going to be wonderful and in a way that's coming from a good place but there's a problem there that it essentially assumes that uh other people don't have other things that they're doing uh yeah it's a great thought of saying like oh you should learn about category theory but the reality is you probably already had a full plate you were trying to solve your original problem maybe you're trying to learn about other things maybe you're trying to learn about machine learning maybe you're trying to learn about distributed computing maybe you want to actually get some exercise in maybe you want to spend some time with your family or your friends so when someone says well you gotta learn about category theory now which of those things are you gonna drop because you're supposed to learn about category theory so i think at this point we've had the better java of useful but maybe not a reason to do it in scala we've had the very very functional of kind of a poor man's haskell of bring some really interesting ideas in but maybe not in a way that's accessible to the broader group of developers so where are we going from here and i think the answer is that the future the next generation of scala is libraries that focus on being distinctly scalar and those are going to be libraries that play to scala's unique strengths so libraries that leverage scala's type system which is probably the most powerful of any mainstream language to create type safe dsls libraries that take on complexity so users don't have to and libraries that are relentlessly focused on developer productivity because the real differentiator for scala is letting teams of developers be very productive so again from my bias perspective i think one library that is an example of this is zio which is a framework for building cloud native applications in scala that lets you declaratively describe parallelism and concurrency and gives you powerful guarantees around resource safety and the ability to see how and even whether your application can fail at all so here's an example of what a snippet of code written with zero would look like and so here we're calling zeo dot for each par so we're saying for each of these files in parallel we want to do something and then we're saying for each of those files we're using this acquire release with constructor which is the concurrent version of try finally so this is going to open this file and we're done with it it's going to guarantee that we're going to close the file and while the file's open we're going to process it and it's going to do this whole thing with a level of parallelism of eight so we're only ever going to have eight files open at a time as soon as we get done with one we'll open another and then we'll timeout this whole thing after 60 seconds and so just this pretty first bit of code does a lot for us so we're processing these files in parallel we're making sure we never have more than eight files open at a time we're stopping on the first error and as soon as the first error occurs we're going to automatically interrupt all of the other files we're working with and we're going to make sure that those files are still safely closed no matter what and then we're going to time out the whole thing after 60 seconds if we pass that timeout where again we're going to shut down the whole thing and we're still going to make sure all these files are closed so we're getting these very powerful guarantees and really describing a lot of logic at a very high level and i think this is at least from my perspective very much reminiscent of what got me attracted to skal in the first place of each of these lines here says exactly what it's doing and i think about as close to plain english as you could for code and there's there's nothing else here there's no f there's no implicits there's just a concrete data type that we're doing these operations on that are describing what we want to do and in addition to being useful zio uniquely leverages scala's type system so the core data type in zeo is this z-o-r-e-a which represents a workflow where r is the dependencies of our application so this might be we need a logging service or we need a database and e represents how our application can fail and a represents the success type of our application and because of scala's support for variance which is a feature that scale actually has that for example hassle does not have because hassle doesn't have subtyping doesn't have variance these dependencies and arrow types naturally compose and we'll see what that means in a second and so this is something that you can't get in haskell you can't get in java you need scallop for this and so here's an example of what that composition looks like with the error type where here we have this domain error type of a password error and it's got two subtypes of an i o error or an invalid password error and then we've got these two different workflows so this first one's going to get some user input and it can failed an i o error and the second one's going to validate that input and it can fail an invalid password error and if we put those together if we get the input and then we validate it we're automatically going to get an error type of password error that is the combination of these two error types here and that's using scala support for variants to do that and you see the same thing here with the environment type where here we've got this method here that requires some kind of bloomberg service to get some stock prices we've got this method that requires a logging service to do some logging and when we write a workflow that does both both of these things we naturally get an inferred type that requires both of those dependencies so again using scala's type system in addition to that zio actually does have this deep algebraic structure so we can we can take those ideas that we got from those very functional uh very kind of haskell libraries and we still have all those properties here so zio is a monad and an arrow and an index monad and actually has an even richer structure that can be described by those because they don't have the ability to describe three different type parameters and how they interact with each other but also powerful laws about interruption resource safety and how those relate to each other and we get all this without any higher kind of types type classes so we don't have to kind of have that just to have that structure we can have the structure and have a very concrete accessible representation and there's also a growing ecosystem around that so libraries for configuration http server serialization persistence stream processing and lots more and there's even a lot of functionality like login metrics that are actually built into co in xeo 2.0 so all this is leading to more and more companies adopting zio and some of these companies were using scala now using zio some of these companies weren't using scala before now using scala with zeo none of this is specific to zeo those same principles can be used by other libraries so i think a great example of that is caliban which is a graphql library that follows a similar philosophy of purely functional on the inside but that's not what they're about what they're about is making it very easy for you to derive and validate a graphql client server based on your data model you just define essentially just a hierarchy of your sealed traits and case classes describe your data it'll generate it for you and it's massive time saving when you use this and this is the type of thing that if you're a company and you're looking to deploy a service that's based on graphql you might say like hey i need to use scout i want to use scala because i want to be able to take advantage of this library and that's ultimately where we need to go of course there's nothing magical about this you can do this with your own library so i think the key ideas here are being both deeply principled and deeply pragmatic using functional programming as a tool as a means rather than an end and letting users focus on solving their problem so you solve your concerns so they can focus on solving their problem they don't have to just kind of learn the internals of what you're doing and if they want to they can at their pace but they don't have to so in conclusion there's never been a better time to come back to scala there are more and more libraries being developed focused on solving real world problems in a very ergonomic way and libraries to make you want to use scala and you can be a part of helping to build this next generation scala so actually a later today we're going to be having uh this year's zeo hackathon uh so i'll post a link to that on the discord after we get done with the q a uh but i really encourage you to check that out there's gonna be a great uh talk introducing zeo if you haven't worked with it before then talks interesting about the xeo libraries and we're going to be hacking together over the next couple days so a really fun opportunity if you're interested in looking involved but i hope you found this talk helpful and uh wanted to thank the conference organizers for their time uh eric here for his introduction uh alex ali anastasia as well another lot of other people involved i did interact with uh want to thank all the other xeo contributors uh all of the zeo users uh feedback from our users is incredible and continuing to improve what we're doing uh john degos in particular for being a mentor to me personally and then finally i'd like to thank all of you for taking time out of your day to attend today and listen to this i hope you enjoy the rest of the conference and thank you so much [Music] you