SF Scala, James Earl Douglas: Putting Functional Programming to Work
Recording: SF Scala, James Earl Douglas: Putting Functional Programming to Work
uh so thanks everyone for coming out um and skipping the giant sports ball match thing that's happening um i'm going to try to keep this fairly short and i want if there are any questions during the presentation feel free to jump in this is going to be kind of walking a line between fairly deep concepts but trying to keep it shallow enough that it's still accessible and consumable but the main motivation behind this talk is if you go to meetups like this there's often um very interesting and compelling presentations on functional programming or category theory or uh monads and all these kind of exciting and sexy topics uh but they're they're very frequently uh sort of sandbox to like a very academic environment or academic application and it's by that i mean it's sort of hard to take these ideas and connect them with code that you actually write during the day for money so i want to try to can bridge those two worlds a little bit and use an example from some code that i actually write at work at versailles where i work and hopefully show you how you can do some of the same kind of things so the the scope of this is functional programming which i'll define in a second promises these things like safety and composability and reusability and lots of fun buzzwords and we're going to connect that with actual code that can do real things and by the way this is an adaptation of another talk i did where all the code was written in javascript and this is a scala port of that talk so i highly recommend the scala version of this over the javascript version as we'll see types tend to help out okay so i have a couple of asterisks on the previous slide um functional programming and real world so for the purposes of this presentation when i say functional programming um i basically mean referential transparency um how many of you are familiar with referential transparency okay cool so most of you uh in a nutshell it means that uh if your code is referentially transparent or if an expression is referentially transparent uh your code is identical whether you replace that expression with the result of evaluating that expression or with the expression itself so if you say val a equals one plus two uh then anywhere in your code that a appears you could replace with one plus two or vice versa and the code would not change in meaning and and where this uh starts to be useful is if you had val a equals println hello world one plus two uh then that goes away because uh referencing a uh well i guess if it were a function referencing a has this side effect where we're printing to the screen and then returning the result of one plus two and that's of course not equivalent to just saying one plus two uh so we'll get into that in a little bit and then when i say real world uh i i couldn't actually take code from our production environment and put it into slide form and have it be readable so this is a an analogous example that i hope is still uh real world-esque so our example a real world sort of like um problem is an atm and so we're gonna write some code for operating an atm that will let a user do things like check their balance and deposit money and withdraw money and so forth and yeah atm examples are kind of common and maybe a little bit boring but they're useful especially in this case because we're talking about functional programming and we're talking about state and handling state in a consistent and predictable way which is a little important when you're talking about money and then a quick aside so for again since this is kind of a simplistic example uh when we talk about a bank account in this example we're really just talking about a list of transactions that have happened so my bank account is the list of all the deposits and withdrawals that i've ever made uh and so the type of an account will be a list of floats and of uh individual contribution or deduction from the account is a single float um so where are we coming from what's the the inspiration for this uh this is sort of the non-functional way that we might write of something like a deposit function so off somewhere in some global context we have an account and it's this globally accessible reference to a list of floats and in this case i've made it a var so that we can change it and when we call the deposit function we say i want to deposit 20 and so the first thing we do is grab that account and update it so we add we append a transaction and then from the function we return to the balance of the account by just summing up the those list of floats uh and so it seems nice and straightforward but it actually has a lot of problems so we've got a lot of mutability in here we have this account reference which is a variable reference a immutable reference and so basically anyone can come along and set account equal to a new list of floats or well that's basically it they can they can change the assignment of account and you won't know that that happened necessarily we also have this kind of imperative looking deposit function so when you when you reference the deposit function uh and and you apply it with an x in this case it will actually execute this code so when you say deposit of 20 we will run account equals account append x and then we will run account.sum so the code kind of happens as soon as you touch it and then finally it's kind of inconsistent uh it might not look at like it at first but there are actually what three references to account um in this function we're we're looking at the account we're assigning it and then we're looking it up again uh and as you know in kind of a multi-threaded environment there could be many people referencing the same data structure and so when i look it up the first time it might be one thing and then when i append to it and reassign it it might have changed uh in the meantime by someone else and then finally when i look it up again and take the sum it might have changed again behind the scenes and so we really have no idea what the consistency of this data structure is and so here's how we can take state as a side effect which we had in the previous slide and convert it into more of a computational effect so this is basically philosophically this is doing the exact same thing except we're now representing our deposit function as this sort of callback looking thing so when i call deposit of 20 rather than doing anything so we actually oops we actually are not touching account we're not modifying account nothing really happens other than we construct this function and return it and so this is a function that once provided an account knows how to calculate the balance and knows how to create a new account uh that's the result of appending a new transaction to it so we've gotten rid of the mutability when we eventually do run this function that's on the second line the that account reference is fixed so on the left on the right side of that arrow we say account.sum and account append x that's the same account reference and there's no way it can change behind the scenes uh assuming it's an immutable list we also have as i kind of alluded to we have this sort of declarative structure uh where instead of uh like on the previous slide where when we call deposit we're actually running our our business logic in this case when we call deposit we just return a function that we can hang on to and run at a later time once we know what the account is and then finally it's consistent uh so within the body of this function uh we are certain that the account reference is the same for every access so what i want to do is come up with a way to represent these kind of stateful actions and and then we'll add in some some nice functions that we can use to compose these things together but let's start simple so let's create a case class called state and state just wraps this function run run is a generic function that provided some external state will do some kind of computation and it will return a tuple that's uh contains some arbitrary value and then a potentially new state so in our previous slide you can see that our function takes an account which is that list of floats and then it returns a tuple of the balance of the account and a new account which is account with one transaction appended so just generalizing that a little bit that would be this run function where s is a list of floats and a is a float so now we have this case class and we can stick a sort of a state action function inside of it now that we have it we can start to compose it with different things so let's say we have uh well before i get into an example uh we want to uh come up with a way to take our state action uh which was deposited in an earlier slide and potentially do something with the output of that function so let's say a run function returns a tuple a s and we have another function that can convert the a into something but this function f is called pure because it doesn't really care about the state coming into or going out of our run function it doesn't care about the account essentially all it really knows how to do is manipulate the in this case the balance in some way so all we really need to do is compose run and then this little case partial function where we we run our state action and then we take the output of that which is this as tuple and then we apply f to the a and that gives us a b and then we return the same s because it hasn't been changed at all and so an example of this would be uh if we have oh and and for simplicity we're going to call a transaction this is just a type alias so it saying tx or state of list float are identical but for cleanliness we'll say that a transaction of type a is a state where the input is this list of floats it's the account and the output is some arbitrary a and so an example of that is called balance so balance is a value which is a function wrapped up in a state and so balance says i'm a function that when given an account i know how to compute the balance of the account which is this account.sum and then i'll also pass back the account unmodified and then if we have a pure function that knows how to take a balance which is a float and convert it into some readable string like your balance is such and such we can apply that function to the balance without touching the state by using map like this so we have a report which is itself a state function wrapped up in a state case class that's the result of calling map on this balance state class and then applying this pure function to it okay so that's composition with a pure function which we use map for when we want to compose with uh another state action uh we have to use flat map and so the the difference in type signatures between map and flat map is subtle map takes a function from a to b and returns a new state flat map takes a function from a to a new state and returns a new state and so in this case uh our function f is actually potentially performing some computation that depends on the state and maybe returns a new state um but our composition looks almost the same so we will run our embedded state function and we'll get an a and a an s out of that um and then now we have our a so we can run f to get our new state uh uh our new state instance um from the from uh the input of a um and then once we have uh our state of sb uh we can't just return it because we would have a state of a state which is a little bit too nested so on the state that's returned oh i've got a laser on the state that's returned by the evaluation of this function which is this state here we call its run function so so this state has its own run which will be s to bs and so we apply that function with the same state that came out of running this function so an example of that it's a little bit more code but i think we can get through it so first let's create a state so we'll call this one deduct so when i call deduct and give it some some value it will return a transaction remember a transaction in this case is just a state of list of float and a where in this case our a is float so this says that it will be a state action that when provided an account will return a float and then a new account so inside of our state action we take the account and if the account has sufficient funds um that is if the the sum of the account is at least as much as we want to deduct from the account then we go ahead and do so we return a tuple containing the amount deducted i think i might have said balance earlier anyway we return the amount deducted and then we create a new account representation where we've appended the deduction to the account otherwise there are insufficient funds to make this deduction so we return a tuple that says nothing was withdrawn or nothing was deducted from the account and then pass the account straight through because it hasn't changed so this is a this is a single state um or transaction so now we're going to take that and we're going to flat map it with something so let's say we want to be able to deduct but then also charge a fee you know you go to a foreign atm or something and it charges you three bucks to get your own money uh so in that case if we call deduct with fee we pass it the amount that we want to withdraw so maybe deduct with fee of 20 bucks uh and then the first thing we do is we call deduct of x so we're getting this state here with the argument passed in so this this will either deduct nothing or it will deduct uh 20 and then return the new account with the new transaction so from that we've got a state and now we can call flat map and flatmap will take an argument which is in this case either this zero or this amount deducted and with that we'll create a new state so this is a function from float to state uh so we'll create a new state that given an account and in this so it's maybe a little bit tricky but this account here will actually be what's returned from this state action so either this guy or this one so this state given the potentially modified account will create a fee and then it will return a new tuple where we're adding the amount deducted and the fee so it will be the total amount uh deducted from the account and we're also appending a new um a new transaction to the account so when we run this whole thing or if we were to run deductive fee with a value and then provide it with an account it would possibly deduct the amount that we want to deduct if there's enough funds and then it will charge a fee and so what we'll come out of here is our original account potentially with this minus x appended uh and then with another minus fee appended so actually it occurs to me that this one is kind of a bummer because we're always deducting a fee even if we didn't uh deduct any money but uh maybe it's a it's a bad bank i guess yeah so this could this could give you an account with negative three dollars uh which would then maybe charge you for insufficient funds or something yes in a real situation you actually want to have really one and only single account right that to me seems to complete the original intention right so the question is where is this account coming from and how are we making sure that it's we're not just creating new accounts all over the place which would not really reflect reality because you only have one account per person and actually nowhere in this code yet have we created a new account so all we've done so far is build up a bunch of functions who have this type where s is the initial account to provide well that's a new instance of a list that that has a sure so i i is your question about what do we do now that there is one account reference here and another one here i'm trying to point out that using that state format we are actually having multiple instances of accounts around the place sure where the semantics of what we wanted originally to do assuming the real application was to have one and only one account that is uh continue to be updated right it's a good point and we'll we'll actually get there um in a few slides but so basically the concern is we have one account instance here this creates a new one this doesn't but this also creates a notes we have potentially three lists that are considered accounts and does that create a problem um and it it could potentially be problematic if we don't have good semantics around controlling access to the single the real account um but yeah we'll get there in a few slides okay so now that we've built our state case class and we've written map and flat map we basically have what's called the state monad uh and so it looks like this but this is unchanged from the the previous slides so what can we do with this thing uh well i want to come back to some of the claims i made at the beginning that functional programming is safe and composable and reusable and so let's look at how state gets us that so for safety here's another example of a transaction an instance of the state monad if we write a function called contribute which is kind of the opposite of deduct it takes an argument that's the amount to contribute to the account and it returns a new state action that when run with an account returns that nothing was withdrawn from the account or nothing was deducted and the new account is has been appended with this new transaction and so we've we've solved some of the issues that we saw earlier where we have no immutable references uh there are no vars in here nothing no no references to our values can change we're not using any mutable data structures so our well i'm asserting that we would only use a an immutable list uh in in this example which means that we can never call account.append uh to modify the account in place um this function necessarily returns a new list uh which you pointed out so our state actions are i say necessarily atomic meaning you can't you can't you can either apply this function or not there's kind of no in between so when you apply the function you you tell here's the current account run the function and the output of that is the new account the new um canonical account uh and and there's no kind of middle ground and then finally uh we won't get too much into this but these things can optionally be transactional uh but that then gets into uh the way we run these things is with an interpreter which we'll see in a few slides oh so for the purposes of this thing i'm uh i want this tuple to indicate the amount um deducted from the account um so i guess i could have put zero there in this case the reason it's unit is mostly because there's not any useful information that we care to provide so when you call contribute it's sort of like it's sort of like calling a set method on a java class you know set x to one the output of that is is usually void like you don't really care there's there's no useful information coming out of that uh we could put maybe a confirmation in here true it worked or put a number saying zero dollars were were deducted or x dollars were contributed it really depends on the use case but in this case we don't really care about what happened because it's so simple okay and and so these are composable as well um we in a few slides ago we wrote uh deduct and contribute uh did we also write balance yes we did um and so contribute uh which we saw in the just in the previous slide had the type transaction of unit balance had the type transaction of float so when we call balance we get a state that computes the sum of the account and returns that and to put these things together all we have to do is stick them in a four comprehension and the reason we can do this is because we wrote map and flat map on our state case class so when i call contribute of x what comes back is a state of list of float and unit and when i put this little arrow here scala will de-sugar that into this thing dot flat map and then a function that takes an argument that we don't care about and passes it to evaluate balance and then again balance returns a state which we flat map and take b and then we will just map that into a result actually this will be a map not a flat map so our our total function here deposit is this nice composition of contribute and balance we say hey i want to deposit x first thing we do is contribute x to the account um and the next thing we do is get the balance and then we yield that zero dollars were deducted and b is our balance and the reason this works and kind of i think this is what i find so cool about this code is that we have this implicit account being passed through all these functions but you don't see it anywhere in here so we don't ever have to say account dot append this amount uh and then call balance and pass it the account and and you know get the new account out of it like all of that is nicely abstracted from us by map and flat map in our state uh monad and then similarly we can we can write a withdrawal function that does kind of the same thing it calls deduct which is a state uh case class instance takes the uh takes that that first tuple argument which will be the amount deducted uh and then we call balance to check the balance of our new account that comes out of evaluating this thing and then we just return the amount withdrawn and then the new balance uh so the the point of this is that we can make these more complicated or larger state instances by building them up from smaller ones and it's it's handy because uh this looks imperative right it looks like we're calling deduct and we're calling balance but actually just like before all of this code is just returning a new instance of state so it's kind of this deferred action or this callback it's just returning a thing that when you give it an account will run all this stuff and then tell you what the new account is so there's kind of no side effects it's not like we had print lines in here that kind of do stuff on the side and we don't really realize that's happening when we call withdraw no accounts are changed nothing really happens we're just sort of ready to go oh and then finally reusability which is basically in this case the same as composition we can take deposit and withdraw which we wrote in the previous slide and then put them together and so we can keep we can build up this big library of all these different state actions and put them together in different ways for different use cases and it all kind of fits together very nicely okay so how do we make this thing go when we have a state action uh we have one of these transactions the only way to to make it run is to provide it with an account and when we give it an account it gives us back some value plus the new account um so let's write an interpreter uh called run so run says hey if if you give me uh one of these state actions i will run it for you and then give you back the thing that it returns plus the new account and it does that by taking x which is our state action x is an instance of state of state uh and state has a run argument remember state is just a case class with a run function uh so we call run and we apply account which is this list of floats and then the output of that is kind of hard to read but it's uh here's our a and then here's our s so a is this tuple with the amount withdrawn and the new balance and then s which in this case is a is the new account um so we run with our existing account we get a bunch of data out including the new account we update this reference so we're setting account is now the the updated account with all the new transactions appended to it and then finally we return this data so we return the amount withdrawn the new balance plus the new account list which is fitting this structure here so where did this actually get us because i see a var there and i thought the whole point of this was to get rid of vars so yeah there are problems with this if we're writing our code like this we have a var which means we're not thread safe anyone can come along and modify this thing we could potentially have two state actions running in parallel and they're both trying to reassign account which is a problem and we also there's nothing in here about transactions so what if uh our our transaction state instance was something that had many steps like get the balance contribute some some funds compute the balance again uh withdraw some funds whatever and during one of those steps there's some database error like we lost the connection uh then now we're not in a consistent state like we've committed some things to the database and not others so uh this does nothing for transaction management but it's actually not as big a problem as it might seem so what we've done here is basically this represents the the far edge of our program so this is like our maiden method or our servlet or you know whatever uh tooling or framework you're using this is like the outer edge that finally makes our state uh our state action that represents our program to run uh run and so synchronizing access to a var if that's the pattern we want to use is maybe not so bad or if we're using a database like a relational database and not a this var to track accounts uh we would then figure out how do we want to do transaction management in that environment um but for the purposes of this slideshow it's we can't really cover all of those uh possibilities but i will say that uh in practice uh we at my company we use this pattern and it actually works pretty well uh we we represent all state-changing actions as this uh serialized queue of things to to run um and so that way we can synchronize access we can synchronize updates to this uh data model um and then because our data is immutable so account is the the reference is mutable but the data itself is immutable because we're using an immutable list uh we don't have to worry about people who are just getting information about the account to access that that variable so it's only these sort of state-changing events that we're concerned about and that we have to synchronize access to and it turns out that it works pretty well um then i have a little demo i don't think it's going to be that interesting without a keyboard since i can't click on anything with this uh i guess that's it are there any questions i kind of ran through that really fast yeah so unfortunately i had to really sort of simplify this example but this the pattern is exactly what we're doing at my company we have all of our business logic is represented as uh instances of these statement ads and we just combine them in different ways to build the different so like if you want my company builds an education platform so there might be an action that represents modification of a course and that might have many steps like looking up who the author of the course is and making sure you're doing some authorization to make sure that's you uh and then looking up the data of the course taking the data that you've provided and doing some kind of update and then persisting that back to the database all of that is composed from these little you know smaller actions uh to build up a bigger and bigger state instance uh and then once that thing is kind of ready to go we uh if there are updates required we stick it in a queue and it runs kind of in order um if there are not updates required if it's just something like you know looking up some metadata or something we'll run it right away with whatever the oops with whatever the current state of the world is um but yeah it's so we don't have uh we're not building an atm but it's basically the same thing we have this data model that represents the current state of the world all of the in our case it's all the courses and all of the users and all of the organizations and kind of other data that we have and we update that using uh this pattern and we read from it directly in your real world implementation or utilization of this how did you deal with errors and validations did you bury that in the state and let it and then inspect that when you need to yeah that's a very good question so how do we deal with errors uh like i said there could be a problem in one of these little buried state actions and we wouldn't really know about it and that's another reason why i had to simplify this for the purposes of presenting we're using uh we're using state but it's rather than being a state that returns this tuple of float it's actually returning an either type which is either the failure or the successful value and our state actually is an estate it's a state t which is a monet transformer but uh basically it's a state of another monad which is either and so all when we write when we write our primitives uh things like this are even more primitive things like this uh when we're when we're doing this update step that's where we're likely to get an error especially if it's a database insert or any kind of database interaction um so this function we'll we'll go ahead and try to do this and then if there's an error the thing we return will not be just this unit it would be a left of the actual error otherwise it would be the right of the success and then by putting them together as soon as we encounter a left which is captures the error we basically stop and that's the thing that gets returned and so i think i know the answer to this but i'm just curious whether in your implementation did you draw upon any of scala zed or just write it off from scratch we're using scholars eating a lot yet so we back using either instead of instead of validations or we were using we started with our own either implementation um and we quickly found that it was uh that scholarly already had way more uh than we really wanted to write so yeah we've converted to the i think we're using their disjunction not their validation but it gives us the semantics that we need all right