SF Scala: Joan Goyeau, Scala 3 -- What Does it Mean to Me
Recording: SF Scala: Joan Goyeau, Scala 3 -- What Does it Mean to Me
so our first of two talks will be by dawn of netflix on skull of three what does it mean to me over to you John I heard there's a to be there is a section not on Netflix okay I said yeah today I'm gonna speak about Sky Tree and what it means to us so before I start let me tell you a bit more about what my team does and what we use color for so my name is Ryan I'm a data engineer in industry meaning data science engineering team and so this is what most of my friends tell me Netflix is just streaming movies I have no I don't know what else you are doing there so let me just nail this question right now because I'm gonna get this question all the time so what this friends tell me is that basically the eastern us that that we are a bunch of servers and we are streaming movies to his TV so to put that in context we are responsible for 15 percent of the Internet traffic worldwide and in us at Big Time it goes all the way to 40 percents so if we take back this the schema and it would probably look like this so you may know we are fully all stood on a dress and you're probably connecting to internet by your ISP like I put Xfinity here I'm not doing advertisement for them but and so we are communicating through the backbone and so to your scale if we were doing that it would look like this and so obviously the ice P wouldn't be too happy about that because both both metrics and the ISP would have to pay a lot in transit cost and probably the ice P has to pay for all this infrastructure so yeah there's a class here so what we do instead so what we do instead we have this open connect CDN so it's it's a CDN that we call open connect and basically we just put those boxes directly into the network as close as possible to your ohms so that we push only wants the movie and then you can stream from there and so both the SP and Netflix get back their transit cost and even the the latency is much lower for the users so that's one thing we are doing this is the holy grail of Netflix at least for my team so it's think of it as a CAPTCHA room so it's like you you know let's say you want to start first your movie as fast as possible you'll have to either cut on high quality or minimize maximum or read buffer if you want to have high quality you'll have to first do an initial buffer or to read buffer during the video and it works on all four ways basically and so in my team we believe that we can push that as much as possible always further and further and so to do that we're doing a lot of IV testing so we are data driven company so for those who don't know what is a be testing is just we take for example a million users and we would take a new version and we will try that on 1 million user and see how it compares compared to the baseline if there is a higher conversion rate if people are more happy or not and we are doing that not only with UI but we were doing you know we've been able to improve codecs so this is a comparison with iTunes probably iTunes is not the right one to compare to because they'll come up next year with the laptop that is three times better so yeah they want you to through you know those those movie as fast as fast as possible I mean by the laptop that is faster to load this content so we do the reverse and we also a/b test like anything the trailer even the images so I don't know what you think about those two but I preferred the second one I think I would pick more on this one so there is another type of test that we are doing I don't know if many of you know the cows monkey so this is a Netflix pang so this graph the top one shows the number of stream starts per second so it's the number of people starting a movie every second I had to blur the numbers because they are not public and the lower tree rats are the same as the top one but per region the tree regions where we are deployed and we can see that in EU West the monkey just shut down the whole region for half an hour randomly like that and so then the two other regions are taking over the traffic from us from EU and there's no impact on on the global on the global start stream per second that's how we know that there is nothing and so if tomorrow the continent like Europe goes down or like us will you'll still be able to watch Netflix that's what you think all right so you probably didn't came here for like just listening to the Netflix stuff so Skytree I said just to clarify there's a lot of talk about Scala tree Pro like most of them by multi notice key or like the pH decisions we we I mean Netflix is not using Scott tree neither Dottie we're just users and I thought I'll reverse things but and talk about the issues that we have we saw to that Scala tree might might resolve basically so it's kind of like not listing the features but more listing the issues that we have right now that Scala tree will I hope will resolve so I'll do it by episodes so I'll start with limited animations an easy one so that's how most of you I mean not most of people but this is a way to create an immersion in Scala too yes and so yeah I guess you're probably familiar with it but you just extend an immersion and then you can have there is a little bit of type whirring wiring here and then you can point here yeah and then you have your three enemies let's say values and so the good thing I'm looking for here is I can I can get like values all the possible values of this enemy and I can also from a string when I decide from maybe from Jason I want to get the enemy from a string that's probably the main feature of extending animations some other people do with the still trade and case objects this way and that that has the benefit of being able to extend it let's say you want to have the description you can next you can answer extend this way but the issue is that you don't have you don't get the benefit of values and with name that's the main drawback and obviously there is some libraries like anime written that are that brings basically with some macros those values and with weight value but you have to kind of introduce macros and bring in this library so this is how its quadtree aims to resolve this with a new keyword ennum and cases and you get you get the names that's changed a little bit but you have any ending values and and you named and so if you want to extend it as I did before with descriptions you can do it and so I have there's a bonus point where basically if you extend Java Java langham your ending will be back compatible to Java so if your library is used we have libraries that are used from Java at Netflix unfortunately not everyone is on is on Scala and we often have issues where we have to fall back to Java enemies in poor Scala code just so that we can return that on the API that's pretty good all right so either and co-products we're using quite a bit of them so let's say we have those two function one is like a simple one audio so you give it a movie ID and it returns the audio for this movie as a an area of byte and it might fail because not an exception and so either you throw an exception but this is not very functional so what we do most of the time we will either either for either the you get the exception or an area of fight a bit more complicated the subtitles for example you give ten upsets it can fail in to two different way either giving a miracle ego argument exception or a not found exception and we can already see that in the type we lost quite a bit of information we just know the most common type exception so so I was thinking about Union types and this is perfect actually so if I turn that into Union type that we I think my number one use case where yeah that function would return either an array of bytes or and you can see you can see here that we can return multiple exceptions so we have full documentation on what this function does so I a little bonus thing I I don't know if you know the library refined but so since actually it's it's a feature in in 213 but we have singleton types and so we can do things like you know we can already do that in Scala too right but you can refine and ensure that thus this function gets an A+ in space so you reduce the number of ways it can fail so knows the billion dollar mistake so I guess some people do that and then it blows with a nullpointerexception and so what's the solution to that just don't use neol I think everyone keeps telling this but still it happens so well one way is use options right I guess most of you are using options the problem is that I have colleagues that doesn't and I'm also using Java libraries that doesn't do so in I believe this is gonna be enforced in scar 3.1 not 3-0 directly but basically right now the the keyword nil as the type no with uppercase M and that type is actually a subtype of all the types like nothing and so this is this makes it possible to assign and know to a movie and I believe they're gonna they're gonna remove this this subtyping thing and so you won't be able to assign no to a movie first of all here it would it would just fail at completion and so if you want know again you need to use Union type and you say okay this can movie or no and then you can't call duration on it and so if you want to call duration on the movie you need to match to pattern match on it and you know if it's a movie then do the duration otherwise do something else I don't know what [Music] and so yeah I found it interesting that you can actually combine those things this can have a movie No so it's optional and it can have also an exception alright extent annual does who-knows-what extending Annabelle does what four people that's it okay so I think I don't need to explain why this is an issue so let's say we have this record view function basically it takes a movie IDE and a user idea and basically records in let's say in a database that this users so so this movie and so you would call this way right you give it to user ID we took to the two IDs the movie ID in the user ID and if you recall that and so it would be nice if you know instead of using strings like here it would be nice if we were using something a bit more documented and to do so we can actually use the type but unfortunately type is good for like documentation but I don't know if you saw in this slide there is a bug the user ID and the movie ID is walked around and it combines perfectly fine it happens like I mean it happened to me couple of time copy and past from somewhere else and it was swept so so the solution to that some people do wrap the values into case classes and so we can see that it doesn't compile at that point because you're using actual types for it and the problem of that is that is you're creating classes for that so you have another an overhead at one time so to avoid though the runtime overhead you can extend any well and it will basically inline everything it's like if the case class didn't exist at one time and you get the benefit of compilation compile time safety and runtime benefit so in Scott tree there is a way to do that without a bit more explicit than than extending any well it's opaque types so it's just like types but it's not an alias it's opaque and so it's opaque on to the scope as it's not it's basically not opaque to the scope where it's declared and outside of the scope it's it's opaque right so you opaque what I mean by opaque is you don't know it's a string behind it in this case and so that's why I put an object here a companion object and I put an apply method to lift the string into a movie ID inside the scope so that you can lift it outside of the scope into from a string to a movie ID or user ID and so somewhere else you call the apply function here and you get movie IDs and you get the compile time safety and so we can go one step further you know the refine thing I showed you you can do it here so I know a movie idea shouldn't be empty so I can make it non empty and obviously here I need to handle a bit more you know what if I get a string an empty string I need to you know maybe return an exception here and when I lift it I need to handle the exception that's why I put the three dots here I I don't have too much space on my slides limited right I'll go to the next one so implicit classes I have a friend who who told me he's learning Scala and he's speaking to me like he's when I saw I saw implicit that blowed my mind when I saw implicit classes I was like wow okay I was like yeah I mean this is this is probably going to go away so don't learn that so basically I think most of the time we're using implicit classes to extend types that we don't have control on I think so most of the time so here is an example that is actually in the standard library and so it's extending the long here and adds the function seconds on it and that's how you can call five dots second and pass it to to a function that that gets a duration basically so on the long like five you do dot that's that's that seconds and it gives you a duration so that's probably how you you can implement it and so in Scala tree there is extension methods which is I mean if there's any people doing cuddling but it's pretty much the same thing where you can yeah you can extend anything here and add a function to this type duration like the second direction so type class hack this is my preferred wanna thank you so I think in the for for people I think I need to explain a bit more like type classes because last time I did this talk people didn't know what is type classes so if you think about how how would you do an encoder in Java you would have a tread anchor level and let's say we have this class the skies class movie and you want to encode it so you would extend encoder encoder ball on it which will force you to implement encode and then somewhere else if you have the same function that gets any incredible it can encode it and then send the data right that's how you would probably do object-oriented programming and so the problem that is some like you come up with this then a colleague comes and say oh I need to compare those x times compare and then someone else I don't know he needs to clone it somehow and then obviously were you all using spark so serviceable and then you just become mad so so there's there is a pattern here it seems that we have the data structure and the data manipulation that is mixed up in the same place we call to have a way to separate those two also we can't extend anything that is out of our control I use the names here but all the dependencies that we have if we're using a library we can't we can't make it serious able how many times I came across this issue with respond so so that classes are another way of doing and you would have a tread encoder in color of T and then you would have your type classes implemented here so for a movie you have an encoder for movie for a name for whatever you want and you can extend it you know you can have them wherever you want implement them and then bring them in scope and then your function your function will be I can send any T any data of type T which can be anything as long as I have an encoder in scope for it for the T and then I can encode it and then send it so so one thing also we can add you know use implicit kind of like for dependency injection thing like at the client here the HTTP client and send it right and so there is an issue here where there's a clash between two Scala features the implicit and the apply methods you can see here this function returns the sequence of string and so it would be nice you know I would like to get the first element of this string it returns and here if you do that it won't compile and it will tell you oh I got an int but I was expected expecting an encoder so basically it calls the second argument the second group of argument of the function instead of calling the apply function so the fix to that is to explicitly write a PI which is which is a bummer I think or like to two features are just like contract like contradicting each other so this is how I believe the key word didn't change the give given so this is how you would do it in Scott in in Scala tree and so you have the skipping keyword which kind of replaces the implicit and you would have a given for that type class the implementation and then here just like the implicit you have I can send data given an encoder of T and notice here that ice kept the name you can you can skip the naming here and notice also that I combined also the previous the previous feature the extension methods if you put it directly inside the the type class here so this is like the type last is the is the track you can call it tricky on the data you can do two J's in Turkey say again so if you need the instance if you need to catch it there is actually so I think it's got yeah in Scala to you would do implicitly write implicitly of encoder of the type that you want you can do it in Scala 3 and it's called the th e I don't pronounce it well yeah I know that there is some benefit over the implicitly I don't know too much you can ask the question to set and so yeah coming back to my previous example you can also add you know a HTTP clients as a yeah there we go i I did the example you were you were talking about the death so I catch the ICP clients and then I call sand on it if there is a same function and so here is another example with the name if you don't want to use the function you can you can name it if you want it's optional that that's it so untyped equality so I did this is like this code me not for not long to go like two weeks ago I did a refactoring I'll show you so basically it was not that exact example but again class movie and you create one the black murmur one and see that we have a duration in int here which is whatever and you'll like so I there is there would be like some other work mate who would come and say hey what is this direction is it millisecond minutes hours I have no idea so I would be like okay you know maybe we can do something here so somewhere else in my code I have this function s duration five-second and basically gets a movie and tells you if it's five seconds that's it so from here you can understand it was millisecond right but it's not very clear so I come back to my class here and I decide to add to change the type from int to duration so I scare and now we have again like five second I just did that committed see I pass compiles and you will tell me you didn't have tests yeah I didn't but but yeah and it keeps me up so much that it just passed here because it compares duration with int and this is clearly never gonna match so this doesn't make sense and so this is due to the fact that in Java you equals takes an object X any so with that flag with that flag on the compiler that wouldn't match basically so you'll have to refactor it to trap the type that matches exactly so trade parameters when I saw this future I was I was wondering why would I ever pass like parameters to a tread and I found a use case so for those who are using cats and cats a packed maybe you often have to do a service and I ended up trying this and it didn't compile because when you do this this is this is basically an implicit here you're creating an implicit and that materialized in in a parameter to the tread and you can't apart the tread parameter space basically so so yeah that's kind of a bummer and you have to yeah because it combines to this basically and so in scala in scala treat maybe that would look like this actually forgiving and yeah even without the name it would look like this right and it works I tried it and it's so good I can't wait for this because there is hack stuff everywhere that this yeah comes back to this so temple deconstruction in functions that's a that's a short one basically whenever you have a topple in a list or an option whatever and you map over it and you want to get the actual elements in there you have to give a partial function in it and you need to match actually I don't know if it's impossible partial function but it's a it's just a match you match to deconstruct the table just like you would deconstruct their case class and so in Scala tree you don't need you don't need to match on it it will be constructed for you yeah and yeah you don't just we don't we don't have to change those brackets I keep like every time like I need to go on the left change the bracket into a curly bracket go on the right it's just annoying it's like when you do 20 times this it's like last one so the do people recognize themselves here so cross-compilation mess so you might wonder what like this is cool all those features but when are we going to get them it's already messy when 2:13 was out how many people onto 13 you are can't believe this so I guess it's gonna be even like bigger maybe with Scala tree so basically the issue right now that we have is if you have your project on the right that is on to 12 and you depend on a library that is on that is that has been compiled into 12 - you can depend on it and then if you upgrade to to 13 basically you you break your binary compatibility so you cannot you cannot upgrade until your library that you depend on as upgraded - and so what your library is doing because not everyone has is upgrading at the same time you would have they would have to cross compile basically everything into multiple versions usually it's two versions and so and so you might you might think basically it's going to be the same thing for Scout tree but since so the plan is that in - 14 sorry - 14 will be able to compile to a an abstraction called tasty and so from tasty basically it's a format tasty means typed abstract syntax tree and why we have no idea why and so from from from that at fracks industry we will be able to work on to reconstruct the bytecode basically and so we would have four and backward compatibility so you would be able to depend from Scala tree to scholar to a library that is compiled with to 13 to 14 sorry and also the other way round it works also so if your library maintainer you would be able to support to 14 but again set as more information on that that's it so I just encourage people to have a look at the feature on dot EPFL that's eh I played a lot with on sketchy and it's actually really accessible and I was surprised at some point I tried some stuff and didn't work as advertised I went on the Gator just said hey guys is it working and I saw multi node is he coming oh sorry I'm gonna fix that Wow yeah obviously we are hiring just like everyone thanks great thanks so much Shawn really appreciate it thank you [Applause]