Scale By The Bay 2020: Julien Truffaut, Monocle 3: a peek into the future

Scale By The Bay 2020: Julien Truffaut, Monocle 3: a peek into the future

Recording: Scale By The Bay 2020: Julien Truffaut, Monocle 3: a peek into the future

[Music] so hi everyone and thank you for being here today in this presentation i would like to talk about optics optics often describe as being the equivalent of jquery or xpath but for all immutable data so with such an impressive comparison you would expect optics to be very popular right well this is the case in haskell for example but in scala we rarely see optics in real world code bases why is that why are optics not popular in scala today i won't not only answer this question but also offer some solutions to the current problem so let me introduce myself my name is julian ruffo i'm a scala back-end developer and i also teach functional scala at fptower.com i recently released an online course that focuses on the practical side of functional programming and if i have time i would like to say a few words about this project at the end of this talk now the most relevant aspect for today's presentation is that i'm the creator and one of the maintainers of the open source library monocle which implements optics in scala and it's part of the type level ecosystem okay so what is the agenda for today first i would like to talk about optics uh you know about what problems are they trying to solve and to do this we'll go through a few realistic examples that i'm pretty sure you're all facing in your code base then i will give a quick introduction to optics what what they are and the most important parts of the api but you know this section will be very brief finally i will present two problems with the current version of monocle and how we intend to fix them in the future does that make sense okay great so i don't know about you but you know when i learned programming in languages like c or java pretty much all data structures were mutable we can replicate the same behavior in scala by creating a couple of case classes where all the fields are marked with var so for example here i created the class user containing a name and an address an address is itself a case class with a street number and a postcode case classes are extremely convenient because we can access the data inside them using the dot operator for example we say julian dot address dot street number to obtain the value of the street number field for the user julian which is 12. now the benefit of using variables is that we can update an instance of a case class as easily as we access the data inside it for instance we can say julian dot address dot street number equal 16 and this will modify the street number field to 16 as you can see here so this syntax you know is very concise and easy to understand so you know this seems to be a great option so what is the problem well you know the issue is that in fp we mostly use immutable data i will quickly go through you know the main reasons for that choice immutable data offers some nice features like versioning or the possibility to roll back since we never destroy or erase anything for some reason immutable data simplifies testing we can take an object apply a few transformation and then compare the result with the initial state this initial state is still available and hasn't been altered in any way since the data is immutable finally and and most importantly in a concurrent environment it's extremely difficult to work with shared mutable data either it leads to race conditions meaning that some threads can access the program in an inconsistent state resulting in some very tricky bugs or we get deadlocks where nothing gets done because all the threads are waiting on each other while you know it is theoretically possible to avoid these issues it's notoriously difficult in practice i recently watched an interview with mozilla's jim blandy a co-author of programming rust in the interview jim mentioned that an engineer that mozilla decided to use a threat science as a tool to look for data races in their c-plus code base and he found that in every case where firefox used threads there were data races not just some or most cases but every single case those bugs were you know eventually fixed but it illustrates how difficult it is to deal with shared mutable data so if we agree we should use immutable data structures one big question remains how do we change them because you know at the end of the day we need to encode changes in our applications for example when we receive a salary we expect the software that you know run our bank account to increase the balance right so how do we change something that by definition cannot be changed i have a short story to illustrate this situation imagine you're a developer that lives in an immutable world and say you're at your desk when you realize that the light bulb is broken how can you change it in a normal world you will just you know unscrew it and putting in the new one but here you can't because the world is immutable unscrewing a light bulb and putting in a new one is a mutation so what can you do you are a clever fp developer so you have an idea let's create a new planet you know from scratch with nothing in it then let's copy all the continents oceans mountains and forests then let's copy all the cities roads and buildings then let's copy all the rooms and all the items inside it including the lamp and the only change we make is we put a new lamp we put a new light bulb in the lamp easy peasy you know when i heard this story for the first time i thought it was a joke why on earth would you go through so much trouble for such a tiny modification it turned out that actually it's not that expensive to create a new world and copy everything inside it because we can safely share the data that hasn't been touched between the different copies of the world for example if i change the light bulb in my office in london i can reuse the african continent from the previous world since i don't live there this is called a structural sharing if you want to google it by the way i would like to thank impur pix for making those images and i encourage you to check out his website impurepix.com where you can find lots of fp related infographics that are both funny and you know really instructive you will not be disappointed okay so now if we agree that you know we need to use immutable data and we need to copy it in order to change it the next question is how do we copy things in scala because this is a language we are using this is where i have three examples for you that will be very straightforward first one let's go back to a very simple use case where we have a user and an address but this time every single field is immutable now how do we change a string number using vanilla scala you know like a scala from the standard library it's actually quite straightforward we use the copy method from case classes so we can say let's take the current user julian let's call copy and we specify what we want to change so we want to change the address and how do we want to change the address we take the current address julian.address and we copy it and change the street number and as you can see it works so you know it it's uh copy is very simple it's a bit boilerplatey but to be honest if this was the only use case we had of you know modifying immutable data then unicopy will be completely fine there is no need of something more fancy the problem is that in reality we have more complex data quite often what we have is enumerations so let's say now we want to add a payment method to our user so and let's say we only support two forms of payment methods a paypal account which is basically an email or a debit card which the card number an expiration date and a security code so enumeration in scala are encoded using a seal trait and every car class extends this cell trait but you know what matters here is that we have either a paypal account or a debit card okay so now let's update our user we need this user will have a payment method so now i created two users one called julian who has a debit card and one called anna who has a paypal account so now the next question is how do we change the expiry date of the debit card using vanilla scala okay so first let's try the same strategy let's use a copy method so let's take our current user we copy it we change the payment method and we change the payment method by using copy on it fairly simple except that it doesn't work and the scala compiler rightly tells us so it says that copy is not a member of payment method meaning that the method copy doesn't belong to the trade payment method which makes sense because copy is only available for case classes so the more general question is how do we modify enumeration in scala and you know the answer with enumeration is almost always the same use pattern match so now this time we'll look up our current user or current payment method using user.payment method and we patent match on it and here there are two cases either it's a debit card or a paypal account if it's a debit card it's very simple to change it because we can copy it it's a case class but what do we do if it's a paypal account you know we are a bit stuck because a paypal account doesn't have an expiration date so we can't set all this new expiry inside it and we have to return a payment method because we are within a copy method so this is the expected type the only other thing we could do is throw an exception but this is something we try not to do to avoid in fpe so in fact here we only have one option is to return the current payment method unchanged in other words if you know if you call update expiry on a user who has a debit card it will update the the expiration date but if you call update expiry with anna who has a paypal account it will do nothing now potentially we could change this method to return an option on either but this will require a significant refactoring so that's why i haven't shown it to you here now i have one last example that to slightly complexify this example let's imagine our boss comes to us and say you know what we actually want our users to have several payment methods you know they can have several debit cards several paypal account and you know they can annotate it with a string with a name just to to make it easy to find them out so you're like okay that's fine let's slightly change our user to have this time a map string to payment method okay and let's take a user who has a paypal account annotated as business and a debit card marked as personal so now how do we change the expiry date of this personal payment method so it's going to be the same but slightly more complex now we'll have to use get from from the from map to look up this payment this payment name inside of the map now we have three cases either it's none where this payment name doesn't exist or it's a paypal account or it's a debit card now for the same reason as before if it's a non or a paypal account we we can't do anything so we have to return the payment methods unchanged but if it's a debit card we can copy it and update it and then insert it inside of the map using the method updated so you know this code is not like rocket science it's it's not too difficult to write but it's not pleasant and it's very like low low level kind of and uh when you look at this method it's not really clear exactly what is going on so this is where optics come into the picture optics are trying to simplify this use case so what's an optic an optic is an interface you know a trait which has two type parameters a from and a two so from is kind of a big type like let's say a user and two is something within a user so something we target so imagine you have an optic call name that focuses into the username we have three methods uh streams these are the three main methods so we have get that says you know if you give me a user i will give you back its name replace that says you know if you give me a user and a new username i will give you a new user where this new this new username will be inserted everything else stays the same and modify which is kind of a convenient method that combine get and replace that allows us to you know apply an update function to the targeted field so for example here we can uppercase the username now you might have heard optic in plural like optics and you may have heard that they are fancy names like lenses optionals traversals prism to be honest it's kind of a detail we don't really care about it right now why we need several optics is in order to include different kind of relationship between a firm and a two the main difference is around the gate method sometimes the get method will return a two for example when we know for sure we have the data sometimes it will return an option of two when we may or may not have it or even a list of two when we target several items at the same time but you know it's not the most important point what really matters about optics is that they compose meaning that if you have an optics from from to two and an optics from two to next you can kind of squash them together and get an optics from from to next exactly like function composition and this allows us to drill down inside of an object let me show you an example so here on the left hand side i drew a user so a user has three fields a name an address and payment methods payment method is a map so it has different kind of keys and you see this highlighted root in this object this represents an optic that kind of zoom inside an object the path that it goes through and you see that it targets the expiration date at the end and each segment in this uh in this diagram represent one optic so for example the first segment is called payment methods so it's an optics that focus into the field payment methods inside of a user then we compose it with the optics index that allows us to zoom inside of a person a particular key inside of a map then once we have a payment method you know it can be a debit card or a paypal account so we use a debit card optic which will be kind of a you know a safe donecast into a debit card so if the payment method is a paypal account we'll discard it and finally we use expiration date to look up the part this particular field inside of a debit card so i really like optics because you know it is very powerful you describe a path within your data structure and once you have this this complete path you can access the data or modify it as you want it is as simple as that so now the main question is why optics are not popular in scala because you know here i put on the screen the vanilla scala implementation of update expiry and the optics implementation i know i'm biased but i would say that the optics implementation is much nicer of course you need to know about you know optics and in a bit of the api but you know once you've done this the uh the api is quite clear you know you see that you define a path within your data structure and then you call replace to update the expiry while the vanisca implementation is very much lower level so why everyone are not why everyone is not using optics in this case i think there are mainly two reasons for that i will present you these two problems first i think i i kind of lied to you because uh yes it's nice to have this update expiry uh function using optics but you need to define these tiny optics inside it in the first place they don't come out of thin air so we need to define payment methods debit card expiration date and indexed now you could argue that index is very generic so it's an optics that focus into a map so we can define it in the library in monocle but the rest payment methods debit card and expiration date are optics about your business model so we can't do anything about it you have to define them yourself and to make things worse each definition of an optic like that is very bowler platy it's very boring and it follows always the same pattern if you want to create an optic that focuses into a field of a case class it will be always the same code if you want to focus into a branch of an of an enumeration it would be always the same code so it's very repetitive on top of that you need to know uh different the different kind of optics so for example you need to know that payment methods and expiration date are lenses but debit card is a prism so if you think about it from the point of view of a new user of the library in order to start using it you need to define tons of boilerplate code and you need to learn a lot of things about the internal of the library so it's very discouraging so in monocle 2 so the current version of monocle what we did is that we define a couple of macros to simplify the creation of optics so we have gen lens gen prism genizole and which we know simplify the code you still need to define them in the first place but at least you know the implementation is much shorter and this is where macros are already made for you know when you have code that is completely stupid completely repetitive macro are exactly made for this use case the main issue remained is that you still need to know the underlying optics you know which object you need for each use case which is you know annoying especially for new users so for the next version of monocle we had an idea to create one macro to rule them all and we called it focus so focus is going to work very similarly to gen lens so you first specify the type you want to start your optics so in this case you want to start from a user and then you define a path within your data structure so you can say dot underscore dot address dot street number and you know your ide will support code completion out of the box and in this case it will generate a lens as an end user you don't need to know that it's a lens you know the compiler and the library will tell you that and you can just use a method available on lenses if you decide to zoom inside of a map using for example index then in this case it will return an optional because potentially this data may not be there but again you don't need to know about it the library and the macro will will let you know so i think this is great like as a new user you will be able to write optics out of the box uh it will be supported by your ide and you know it will be fairly very simple very concise we even plan to add some kind of cute uh operators not many but you know a few for example asterix will mean that you know you you zoom into every single item inside of a map or inside of a list we may add also a question mark to zoom inside of an option these operators are are very common and you can find them in other live other libraries like xpass or other kind of technologies so i think this is good because it will make the code of complex optics very concise so it's great for power users but it's even better for beginners because it can take you know the normal paths that they know in other kind of language and reuse it straight away in scala and it will be supported by the compiler so i'm really really excited about this this focus macro and i would like to thank healing way who prototype an early version of focus inside of monocli so monocli is a repository a playground we are using to prototype ideas for monocle with catastrophe so focus will only be available for scala 3 because sk3 has a completely new macro system and it will be like too expensive to try to do it in sky 2. okay so this is the most important point now there is another point i would like to bring like why optics are not popular and this point is much more subtle so it's about lawfulness i hate that word lawfulness if you have been teaching scala or functional programming for a while most likely at some point you told someone like you can't do that it's unlawful and most likely at that point the person in front of you like laugh and say what do you mean it's unlawful if there is a kind of like secret lambda police that you know will come to me and put me in jail because i threw an exception no no that's not what people mean it's just unfortunate name it means that you know we are dealing with interfaces uh and and these interfaces have unexpected behavior or contract and so we we just want this contract to be satisfied and let me show you in practice what does it mean so let's say we create an optic that focus from a user to the street number let's say for example using the the focus macro now if you use replace you will be able to change the street number to 48 you see like before it was 12 now it's 48. perfect it works but how do you check that it works you could write a unit test but this would be very specific to this particular optic another way to do this is because you know optic has a get method so why don't we use it in order to call get on the updated user and we'll verify that we get back the same data that we just replaced you know this will be the most standard behavior that the least surprising behavior for an optic and we can automate this process using property based testing with property based testing we can generate a random user a random street number and do this operation replace and get and verify that we get back the same thing and functional libraries like monocle they they package all these properties in in a module and offer it to the users so like this you don't even need to know that they exist you know when you create your own optics you can just run the test suite and and you can check if your optics you just created is valid so it's very convenient and there is a fp library design principle that says that you know when you create this interface from scratch using like a very low level constructor it's up to you the user to verify that the optics you just created satisfy the expected behavior so in concrete term it means you need to run the test you need to verify that they are correct it's your responsibility but but fb libraries the very nice features that they tell you is that once you've done this job once you have solid foundations if you combine these these optics together using you know functions we provide in the library then you're guaranteed that these optics will be correct by construction this is a very nice property because it means you can trust you just need to verify the bottom layer and everything else you build on top of this will be safe so this is a great property the problem is that sometimes we have some combinators some methods that can potentially in very weird cases be break the test so let me show you an example zip so zip is a combinator a method on on an optic that that you know zip two optics that starts from the same type so imagine you have an optics from a user that focus into the username and another optics from a user that focus into the street number when you zip them together you focus both at the same time into the street number and and the username and it does what you would expect you know if you call get on it it will give you back a tuple julian 12 and if you call replace you will both you will change at the same time the username and the street number so it's very convenient it's been years and you know users of the of monocle have been asking us can i get zip in the library and so far we have always refused because there is a tiny chance to make that zip is unsafe let me show you where so imagine you create an optic called name that that focuses to the username and you zip it with itself now there is no reason no use case for doing such a thing but you can you know there is nothing blocking you from doing it now what happens if you do it and you call replace you you will specify two strings you will say okay bob and alice but you know there is only one username so we can only select one of these so we'll create let's say a new user with only the last one alice but the problem is once you call get on this updated user you will get back the tuple alice alice because this is the current username so in other words in this very weird use case where you will you will zip the same optics together you break the properties that you should get what you just replaced now this is very unlikely to happen there is no reason for it to happen but because it is possible in monocle we said okay we don't want you it's too risky you could potentially hurt yourself we're not giving it to you i think this you know this is why in we in fp we have this uh reputation of being on top of an ivory tower you know i think some like it's very condescending to our users to say that there is a tiny chance that you screw up so we're not going to even give it to you i think if something is useful and there is a tiny chance that you know there is an issue we should offer it to our users and just document it you know we'll use a scala doc we'll use blog posts we'll use our website to document this use case and you know probably some people will get it wrong but you know it will happen once or twice and then they will learn the lesson but you know zip is very useful so we should offer it to our users okay so to summarize monocle currently has a steep learning curve and a lot of accidental complexity in my opinion these are the main reasons why optics are not mainstream in scala in the upcoming version of the library our goal is to focus on improving user experience and making it more friendly for beginners first we'll implement a unique macro called focus which is in charge of generating the right optics for a given path this should simplify a lot the onboarding process offering an api that is you know very similar to what people are used to coming from xpath or jquery next we'll adopt a softer stance with regard to the so-called laws this means that if we find a useful combinator that can potentially be misused we'll add it to the library but we'll try our best to document how to use it safely in practice this min will add new powerful methods like zip filter or with default that were previously rejected from monocle and there are lots of other features that we have in the pipeline but this would be the subject for another talk on that topic we do have lots of work in front of us and very limited resources you know we share the same problem that most open source library have we have dozens if not hundreds of companies using our code but zero financial contribution now i'm not expecting that to change in the near future but i will really appreciate it if more people will take the time to contribute and ideally during your office hours you don't need to be an expert in optics to join the team and you don't need a phd in category theory either we are more than happy to mentor you and fill any gaps you may have but we also need help with other topics such as the build tools technical writing testing even you know commenting on pr and tickets is a huge help so don't hesitate to reach out to me or any other maintainers and please join or get a channel so i think i have a minute or two left so i would like to briefly present my course foundations of functional programming in scala it's a self-paced online course that focuses on the practical side of functional programming you know it's not about the fancy type level things but the dawn to the earth fp techniques that allows us to write robust and easy to maintain applications the course has no prerequisites other than basic understanding of the scar syntax so it's a perfect fit for a new sky developer or someone with an intermediate level but who wants to fill in some gaps in their fundamental fp skills so if this sounds like something you might be interested in you can find more information on the course web page or just send me a message after the talk i also created a 15 discount code for people attending the conference using the coupon sbtb15 so don't wait too long because this offer is only valid for a week so thank you everyone for attending this presentation and i hope i will see you soon in the monocle guitar channel cheers [Music] you