SBTB 2019: Paul Chiusano, Unison, and why the codebase of the future is a purely functional...
all right everyone I am Paul I'm going to be talking about the Unison programming language this work I'm presenting it's joint work that I've been doing with my colleagues Aria who's in the audience here also my colleagues Byrne are and a number of awesome open source contributors so what is unison it is an open source statically typed functional language very much influenced by a few languages Haskell Erlang and this interesting research language called Frank unison itself has been a kind of a research project for several years now but we are late this summer we started alpha testing our first release you can help out with that if you're interested and we're really trying to get unison to a point where you could use it for real stuff and so that's very exciting so why I guess why a new programming language why another programming language so overall the goal are sort of the guiding philosophy behind unison is we want to try to make programming more fun simpler easier eliminate needless complexity and do that by being willing to rethink pretty much anything and everything about how programming currently works so really just be willing to start from first principles and like rethink things and say like hey how do we want this to work and so that's been kind of the guiding philosophy behind the development of unison but early on in the development of unison we sort of hit on this one core idea core technical idea which I'm going to spend the next two slides trying to explain clearly and this one core idea ends up making so many other things much simpler so these next two slides are like the most important slides of the talk so all right so what's the idea the idea is that in unison we are going to identify code not by its name but by a hash of its content mm-hmm so if we have a function factorial factorial then is the product of the numbers 1 through n it actually doesn't matter - unison whether we call that function factorial or whether we call it blah it also doesn't matter whether we call the parameter to the function N or whether we call it Z we can compute a hash for this and the hash is J 1 e dot now in reality these are going to be like 512-bit sha-3 hashes which I'm not I'm not showing the full hashes on these slides but so the idea is we're identifying the code by a hash of its implementation and we're also going to store a unison code base in a different way so rather than just storing the code base as this collection of text files where the text is just this totally unprocessed form of your code we are going to store it in this more structured way so we're going to store a serialized syntax tree and and associate that with each hash so the hash of factorial J 1 e we're gonna have say a sub directory for that hash and then underneath that subdirectory we have this ast file which is a serialized form of the syntax tree and then separate from the ast we have metadata like what's the human readable name or names for this hash so ok let's take a look at what is in this serialized syntax tree so this is also important so if we look at factorial it's defined in terms of product range and plus and so what's actually stored in the syntax tree is not those names but the hashes of those definitions so and then furthermore when we compute the hash of factorial we even normalize away like what the local variables are called like the parameter to the function and so forth so those don't impact the hash and so bottom line unlike names a unison hash uniquely identifies the exact implementation of of a definition and pins down all of its dependencies so it's an unambiguous way of referring to a piece of code that never changes okay so is that pretty clear and raise your hands if that was like those two slides were clear okay I guess I did pretty good okay okay great so so we get all these benefits from this idea and I'm going to basically spend the rest of the talk kind of going through them and demoing how they work in the current version of unison so so we get sort of this nice story for doing code base management so we don't have builds anymore you're never waiting around for your code to compile we can do renames without breaking anything we can even cache the results of tests so we don't have to keep running the same unit tests over and over again and then we also get this nice story for doing refactorings and making changes to our code base without ever breaking anything so our code base is always going to be in a working State it's never going to be broken or have you know in a state where we can't run things even when we're in the middle of a refactoring there's also all these other benefits which I'm not talking about today but which are which are very nice that also kind of emerge from this core idea and I gave a talk at strange loop a couple months ago that kind of talked through some of these if you're if you're interested but I'm just gonna be focusing on these first two for the rest of this talk okay so let's look at renames first this is kind of the the easiest one so when we're storing the names as sort of separate metadata and we're identifying code by the hash it's pretty easy to just swap out that metadata with something else so if I want to rename fact to transmogrify all I need to do is just swap out that metadata of factorial and replace it with transmogrify and I'm just updating that information in one place so I'm not to do this rename I'm not modifying a whole bunch of text files and generating this huge diff I'm really just making this making that change in one place and very important is that this rename it doesn't break anyone so it's not like the refactoring or the rename that your hide that your IDE can do where it works great but then if you are a library author and you publish a library and then you later renamed some of the definitions that's going to break all your downstream library users so renames in unison sort of don't break anyone because all the references to definitions but like factorial are not by name they're reference references by hash okay so we get this nicer story for rename so that's that's nice we also don't really have builds anymore so when we go to write new code that depends on you know a definition like factorial that we've already written we well once factorial has been parsed and type checked and added to the Unison codebase once it never changes and we never need to parse or type check it again we can just look up hey what's the type of factorial so the the way that we can do that is just again associated with the hash for factorial we store the syntax tree but also hey what was the type after type checking this this hash and so when we're writing code unison really is only ever having to type check the code that we are actively editing at that moment and it's never having to do like a full rebuild of all the code or anything like that and also very important is that once anyone has written a definition and add it to the codebase no one else who gets that code needs to parse or type check that again so it's this is not just like a compilation cache that's sort of local to your IDE it's actually part of the Unison code based format okay let's keep going so we can also cache test results so most of the time like you you have tests and the results of those tests are not actually changing their deterministic if it's a test that is not doing i/o or calling out to the outside world so in unison you can actually when you define unit tests the results of those unit tests can actually be cached so that you don't have to keep running the same unit tests over and over again well you know I'll show a demo of how this works but the basic idea is is again associated with the hash we store not just the syntax tree and the type but hey what was the evaluated result was it a passed or failed and so it's the the actual unison code based format is a little more interesting than what I'm showing in these slides but this is giving you the basic idea of how we can associate different kinds of information with these with these hashes and so we're getting all these nice benefits from the fact that the Unison codebase is append only we never actually mutate definitions in place that so the definition that is associated with the hash never changes and so we can cache all this information and associate it with the hash and we don't need to worry about cache invalidation so certainly your IDE keeps some sort of compilation cache but it is attempting to maintain that compilation cache while all these text files are being mutated out from under it which is very complicated and often the the cache you know gets out of sync with with the files and you have to like do a clean and you're like even it's just sort of it can be very very complicated to maintain that and so unison we just sort of side set that we just have this a pendulum in code base and we can maintain all these caches and they're always just correct so all right I'm gonna show scary we're gonna actually do a live demo and we'll see how this goes but but before I just like like holy crap like it's it's like it's like we've discovered this alternate reality of pro self-consistent alternate reality of programming where we serve a get are getting all these things kind of for free that are just emerging from this this one idea so I thought that that that was cool and something that was made in unison very fun to work on okay so I'm going to switch to a live demo and we'll see how this goes okay so on the right hand side I have my text editor yeah you can use whatever text editor you want and on the left hand side I have unison which is watching for changes to my text file here specifically any file that ends in dot u and so every time I save this file unison will notice and will parse and type check the file if there's errors it'll tell me about errors and so you can see it picked up the definitions of factorial and product maybe while I was like right maybe while I was writing this I wanted to you know look up a definition like oh I wanted to call like a list function but I wasn't sure you know which one I might do a find I get tab complete I might you know look for a definition like like range I might look for a definition like map and you know I might view the code so I might just sort of interactively be querying the code base for for diff things just as I'm writing the code and but okay so let's so another thing that we can do in these dot u files so we call them scratch files because even though you are editing you're editing the code but in in a text file but they're just sort of the text files are just for this temporary way of sort of entering in or viewing code but you don't really keep them around as we'll see so another thing you can do in scratch files is just interactively evaluate expressions so any line that starts with greater than we call that a watch expression it gets evaluated on every file save if we the evaluation is is just sort of incremental it only sort of recomputes things that are necessary so if you see this I know if you can see this but if it's a red arrow that means it actually had to compute it because it wasn't in the evaluation cache if it's a grey arrow it means that it was already in the evaluation cache and it just looked it up so it's pretty smart about sort of only recomputing things that have changed in your scratch file I guess the other thing I was going to show is that you know it pretty prints the these evaluation results you don't have to worry about formatting them to like you know look nice and it sort of formats them based on available with what's his which is nice and what else can we do so we can actually write our unit tests directly in our scratch file as well so here I'm just checking a few examples of factorial this is called the test watch expression and we can see that it is passed and it also says hey this was cashed I found it in the evaluation cache why is it cash that's cash because I just ran through this example earlier okay but and so once once I've written written the code and made sure it works maybe written some tests I'm happy with and now I'm like okay great I'm ready to add it to the codebase so if we look at this message it says these new definitions are okay to add so I'm going to go ahead and type add and now these definitions are in the codebase so let's we can view them we can if we do a find four factorial you know it autocompletes so the the the codebase has sort of recognized that this is a new definition and it can and it can find it and you can query for it and I'm actually now that now that the code is like in the codebase and I can sort of get it back at any time I'm even just gonna like delete delete that stuff for my scratch file and just keep it nice and tidy if I wanted to get factorial back like say I want to make changes to it I just type edit it pops it or pushes it back onto the top of my scratch file I might do some editing but you know I'm actually it's cool I'm happy that it's there then I can get it back at any time and so I'm just going to leave it there so another thing that so once the code has been slurped up into the codebase like this if we do things like rename like say we rename list dot fold L to something something and we view the definition of factorial again sorry the definition of product again which uses fold L you see it picks up that new name for what used to be called fold out and this also works for so I'll actually we just okay let's undo that because old name is better but this also works for when you write documentation in unison so unison documentation it's actually represented as values in the Unison language see even like build up documentation programmatically but here I'll just show like a little example of this is some documentation for product you basically write your documentation and markdown you can have these links which start with an app you can splice in source as part of your documentation you can do all kinds of neat things and when we view when we view the documentation these are actually links to the definitions and so if I were to do that rename again and then view the documentation it picks up the new name so your documentation links are never broken it's not just like text files that like gets out of date you know you actually have links that are that are by hash and that when the Doc's are displayed they pick up the latest names for things okay let's go ahead and undo that change again and cool so the other thing that was gonna show is okay when I run the tests so there's a test command you can see it's like okay here are the cached test results I didn't actually need to run anything I'm just looking up these test results from the test cache you could blow away your test cache at any time and it would recompute but you know generally you don't have to like keep running the same tests over and over again okay cool so and that that test cache I guess that works even if we say rename if we rename factorial to something else you know the test cache isn't impacted because the test cache is keyed by the these hashes okay I'll go ahead and undo that all right so that's kind of showing the basics of kind of how you edit and interact with the Unison code base and it's kind of a sort of a low-tech minimal environment you really just have like one text buffer and unison running in a terminal and and that's that's all you need to be able to write unison code okay so let's jump back to the slides so there's this big question of like okay in unison definitions they don't change right if we if we factorial it has some hash if we change the definition of factorial if we say factorial then is 42 we're actually just introducing a new definition with a new hash which coincidentally is just also called factorial and so all the code that references factorial is referencing it by hash and so it's still by default referencing the old hash and so this raises this question of well how do we actually refactor code in the Unison world and this is something we had to figure out a nice a nice way to do so how does it work right now so the way it works right now is we actually we just modify code in place right and that works that works pretty well when we're making a type preserving change so you're not changing any type signature is just changing an implementation you get your that one definition compiling and your code base works and it's all good right okay but then we have these other changes that we were often making which are not type preserving and then we make the change and then we get a bunch of compile errors and our code base is basically in this broken state and you know we're like okay great I'll just kind of work through the compilers and it's it's all good you know I can I can do this you get it down and you you finally you maybe you feel a little bit like a robot because you're kinda like reading through these confusing compile errors and you have to like sometimes you don't even read the compilers because they're just so misleading you're like you know I'm just gonna like look at what line is pointing to and see if I can figure it out but you're almost back to having a working code base so you're like okay do this and I'm I'm okay and then you fix that last error and there's like 37 more errors and you're like what okay what's even happening here like well the initial list of 22 errors was maybe misleading it was maybe covering up other parts of your code preventing other parts of your code from even being examined by the compiler and you're like okay 37 more errors like all right all right I guess I can do this like it's okay breathe you know work through this 37 errors that's not that many and so you like work work work and you finally get it down to that mm-hmm that last error and you finally fix it and then there's like 26 more errors or something and you're like what like what the like what's even happening here like am I even making any progress at all I'm just like you know sometimes the number of errors is going up sometimes it's going down sometimes it's going left right sideways like I feel like I'm adrift in the sea of errors and just I don't even know if the information that I'm getting back from from the compiler is even accurate in any way and a lot of times I don't know this has ever happened anyone you might be like hours into this like long refactoring and then you realize you have this this terrible realization that the refactoring that you're working on is like it's not even correct it's not even gonna work and it wasn't until you like slogged through like you know hundreds of confusing compile errors that you were able to realize like you know what this was just the wrong approach and you're just like you're totally demoralized and you're just like you know forget getting anything done the rest of the day because you just you spent alright anyone is that resonate with anyone [Laughter] all right great all right so let's not do that but uh but what what's actually going on here so it's like if your codebase is like a skyscraper and you're like you're trying to go from version one of the skyscraper to version two of the skyscraper like the way we modify a codebase right now is we're like yeah let me just try to modify the skyscraper in place well just like pull out this one little foundational column and it like collapses into this pile of rubble slash compile errors and then you attempt to reconstitute a standing skyscraper from the rubble so so we're not going to do that in unison instead we're going to build a new structure off to the side and this is just like what we do with a functional data structure we don't actually mutate functional data structure we create a new version of the structure that shares a lot of the same content as the existing structure and we're gonna do the same thing for our code base so here's an example we have Thanksgiving dessert depends on pecan pie pumpkin pie both the pies depend on pie crust and pumpkin pie depends on pumpkin filling and so what we're gonna do is like if we want to introduce you know a new and improved version of pumpkin pie is we are going to just do that off to the side it can still share the same you know so you can still depend on pie crust and pumpkin filling so we don't need to make copies of that and we could just incrementally build up this new versions of things off to the side and sort of so the idea is that we will we have an existing code base and then we have this incomplete but also still working code base that we're building up off to the side and at the entire time we're going to keep the code compiling we're not gonna have anything broken we're gonna be able to do other things and it's gonna be great all right you ready for a demo all right let's see how this actually works okay so I'm going to do a this is you can do a type base in Indian since I'm just doing a search for what are all the things of type ingredients and you know maybe I want to look at some of these there's my pecan pie and pumpkin pie and let's actually we're gonna do a refactoring and we're gonna edit pumpkin pie and okay I guess the first the first change I gonna make is like a very straightforward one I'm going to make a just a local Thai preserving change like let's increase the amount of pumpkin spices from one-to-one units to 11 units whatever that means alright so you know kick it up a notch here with the with the pumpkin spices okay so we get this type type checking and I'll go ahead and type update and unison tells me that great I've updated this definition you don't have any refactorings that are in praat and in progress because basically i've been able to propagate that change automatically for you so if we look at what's happened here so so it's actually recorded this change in what's called the patch and the patch is basically a mapping from old hash to new hash and so we can see that it is the patch contains this mapping from this was the old hash of pumpkin pie and now it's being mapped to the current new version of pumpkin pie and if we view if we view the definition of pumpkin pie its we can see that yeah it's picked up that new amount of pumpkin spices okay so that's kind of a simple case of you're just editing a code it's type preserving and it's sort of just as just as much work or just as little work as it is right now when you're just modifying things in place so I think the power of this approach approach comes when you are making changes that are not type preserving but so I'm actually just going to undo that undo that change so so let's make a change like adding a parameter to pumpkin pie so we will I'm gonna actually make it parameterised on the type or on the pie crust and you can see it picks up that new type and now if I do an update so it's not going to know how to so let's look at one of the dependence of one of the dependence of pumpkin pie Thanksgiving dessert it's not going to know how do you update that that dependent for me so I'm going to need to tell it how to how to do that update so let's see how this works so I type update and rather than it sort of presenting me with like a list of compile errors that I have to work through it is just telling me a like here are some definitions that I don't know how to update yet and I recommend that you work on them in dependency order so I might type and actually I'm running out of time but I'll just show this very quickly what I'll actually do is I'll change this to I'm actually just going to bind that dependency right here so I I've added the parameter to pumpkin pie and here I'm just binding up parameter and so this makes Thanksgiving dessert I'm preserving the type of Thanksgiving dessert and so any dependents of that can just be updated automatically and so if I type update you can see I don't have anything left to do the refactoring is complete and it's all good now if if at any time during this refactoring I had some other work I needed to do I needed I wanted to like evaluate some other code or introduced some new definitions I could do that having a refactoring that's partially completed doesn't prevent you from doing other things your code base is not broken so yeah that's kind of the basics of of how you can sort of make changes to your code base in this very structured controlled way and without having this big list of compile errors so great so if you are interested in this stuff we are currently alpha testing unison if you'd like to help us with that you know test it out report bugs like tell us what you think could be better used in web dorg slash Doc's to get started we also have a slack channel you know for like asking questions and things like that and also if you just want to follow along with the project we are on Twitter and we have a blog and just again thank you to all of our open-source contributors and also you know we have lots of open source work and and different projects so if people are interested in contributing to unison you know come come find one of us after and we'd be happy to chat so thank you [Applause] [Music] [Applause]