Scale By The Bay 2018: Paul Chiusano, Programming the worldwide elastic supercomputer with Unison
yeah so super excited uh I'm going to be giving an intro to uh the Unison language this is uh very much a team effort been working super closely with uh my awesome colleagues uh Runar and Arya Runar do you want to stand up raise your hand all right uh Runar is on a panel later today uh that y'all should go to as well so um okay so just quick disclaimer uh this this is you know the bleeding edge uh very much under construction uh so just kind of bear with me you're probably going to have tons of questions and I'd love to hear questions after uh you know come come talk to me or Runar so little background on Unis Unison is a statically typed functional language it is open source um it's been an R&D for a while some of you might have been following along with the project uh now we're actually kind of more in engineering mode we're trying to get a release out fairly soon and yeah like why why a new programming language uh so when I first started working on Unison I had this goal that I wanted everything about the uh process of building software to be delightful or you know at the very least reasonable uh so not insane um it should be you should look at sort of every aspect of our programming languages of the tooling and feel like wow this is this is done thoughtfully this makes sense um and the reason I was so motivated to to work on improving this is that I just I think the essence of programming is like so much fun and um I sort of wanted more more of that when I'm actually doing my work um I think programming can have sort of this almost Universal appeal um so I actually I have two kids uh this is Ariana This is Nico um Ari is four and a half years old now and I've been trying to teach her a little bit about you know what does Papa do for work and like Papa's a programmer and like what does that mean well Papa teaches computer how to do things and he has to use a special language for teaching the computer but uh if if you use this special language you can get the computer to do whatever you want and it'll do it faster than you can possibly imagine and she's like okay like like what does that mean um so I was like all right I'm gonna I'm gonna show her some stuff so I you know crack open the the hasal repple I'm like all right Ari I'm going to tell the computer to say your name and the computer said her name and she was like not that impressed um all right Ari I'm going to tell the computer to say your name six times and the computer said her name six times and like the wheels were starting to turn in her head she's like whoa Papa all right Ari what's the biggest number you can think of 92 let's tell the computer to say your name 92 times and the computer said her name 92 times and she was just like oh my God this is insane and like let's try saying mama's name 92 times and like that worked too and it was just like totally mind-blowing for her it was awesome and I was sort of reminded of the things that I found so captivating uh when I first learned about programming and I was I was like oh my gosh you can actually get paid to do this stuff um and this is this is fantastic right and you get into industry and it's like things start out I think they start out good actually um as long as you're working within kind of the confines of of a single OS process we have these nice programming languages that sort of like give us like Lego blocks for computations we can snap computations together we have all this nice abstraction composability it's great but then you realize like oh man there's all this stuff outside this box and like every time you cross this this boundary to do something to interact with something else it's like super painful and and annoying and um kind of the way this might play out is like okay you start out you're things are good you're building your little microservice in your favorite language hle Scala whatever it is and you have all that nice composability and it's great except that's not the whole story you have to talk to other stuff and the way you do that is by like sending these Blobs of Json you know out into the into The Ether and like man that's kind of annoying you've got this maybe of parsing and serialization code at that boundary and then maybe on the other side there's some other service it's written in Java and like it's got its own layer of parsing and serialization and then you might want to elastically scale up or down the number of instances of of these services so you're doing things like setting up an auto scaling group connecting that to your elastic load balancer pretty soon you have a whole bunch of these microservices they're all communicating you've got Json blobs and yaml files flying around left and right and you assemble all these different Technologies into this crazy rub gold machine and like yeah maybe you kind of SAR to get it to work but jeez like this is just so far from like the simple fun like beautiful essence of of uh kind of what programming is about I think um at least that's that's how I feel and so I sort of wanted wanted to do something about that and um so kind of the there's been this transition that I think our programming languages like haven't caught up to it's like before the internet um our programs were were kind of confined to this box of of the OS process but that was kind of okay because there wasn't really much going on outside that box right okay then the internet came along and that sort of raised this possibility that we could have computations that were occupying multiple physical Computing devices and in fact we might even have computations that were elastic that were changing the number of uh computers that they're running on at any given point in time and like what we sort of realized like the past maybe 15 years or so especially with the cloud becoming a thing is like distributed elastic computation is really more the default it's like more the kind of computation we're interested in these days um and yet like we're still kind of using the same sorts of programming languages um that are still sort of Trapped in this little box of an OS process so all right so the big idea behind Unison is that a single program should be able to describe an entire elastic distributed system and I'm going to try to give a little bit this is like a super short talk so but I'm going to try to give a little bit of a taste of of what that could feel like uh when you're programming um so I'll just start with some basics of of the Unison language um okay so here is a function definition in unison um we could leave off the type signature unison's uh would infer the type otherwise um the file Unison file can have a mix of uh definitions like this and then these are these lines that start with a greater than are like uh watch expressions and every time you save the file it'll it'll parse and type check the file and print out the results of those expressions so you can kind of use a Unison file a little bit like a spreadsheet um okay so that's just basic syntax of Unison so let's look at a distributed program in unison so okay we have a function here it's taking in a whole bunch of data um the first thing it's doing is it's actually spawning a node which we're calling Alice here you can think of a node it's a location where computation can happen uh it might be on the same physical computer it might be halfway across the internet might be somewhere in the cloud uh same interface regardless uh and the big thing that you can do with a node is you can move a computation to a node and start it running there um so you can just say okay at that Alice node go ahead and start evaluating this computation and that's going to start this computation running on the Alice node in the background in parallel um so this is going to sort of return immediately uh and it gives us back uh this handle to the result a future I'll talk about that in a sec um then we do the same thing on another node we don't even bother to name the second node we're just saying hey at this other node that we've just conjured up uh go ahead and start running this other computation so now we have these two computations they're running in parallel on two different nodes and we're going to wait for each of them to complete by calling force and then uh we're going to combine the results all right so this is like to me this is like the dream right I mean this is a nice simple interface to constructing a distributed program and you're not like uh opening up TCP sockets and and sending Json blobs around or anything it's just sort of very declarative uh so what's the type what's the type of this so it's not this um it's not just a pure function that takes in some stuff and returns a natural number there's this there's these other things going on um that require so we say that this function requires the remote ability so unison's type system has this feature called algebraic effects which I won't talk too much about but um in order to use the spawn at and force functions we need to declare that this uh function uses the remote ability so and if we look at the the types of Spawn at and force you can see spawn it's going to give us a node but it requires that remote ability at is going to take a uh a remote computation and sort of start it running on another node and it gives you back this future and then to actually get back the result of the future basically to wait for it to complete you call Force um okay so this is like a very very simp this is probably a little bit too simple but I'm just trying to give give the basic idea here um so you might ask like well where where are these nodes actually coming from and I think the right way to think of of these Primitives is it's more you as the programmer are annotating your program and you're saying here are the places where distributed parallel execution could make sense here are the places where elasticity of compute can make sense and it's up to The Interpreter of the remote ability or the Handler of the remote ability to actually decide what to do with those annotations so you could actually have different different interpreters you could have one inter interpreter this I'll actually show a demo of this um that just simulates execution locally on your current just one one computer you might have one that simulates execution locally with failures and faults injected um and then you might have one that actually connects your Unison program to like an actually elastic source of cloud compute and runs it at whatever scale there and your program would run completely unchanged regardless of which interpreter of the ability you're of the remote ability you're using cool so I'm G to show a quick example a distributed merge sort in 17 lines of code Let's Do It um okay so merge sort it's based on this function uh merge as you probably recall that takes in two sorted lists and returns a single sorted list so here's the merge function I won't go through it in detail but this is using some other features of the language like pattern matching you know it's got if if uh if statements it's it's okay so we've got our merge function nothing distributed happening yet um so let's first look at a regular uh single single node merge sort not a not distributed version so it looks like this we uh we take in a a list we see if the list the base case if it's size less than two it is already sorted we just return it otherwise we divide the list in half we recursively sort both the left half and the right half and then we merge the two sorted lists using that merge function we just wrote and okay this function works great um okay now let's look at the distributed version of this merge sort so it actually is going to look almost identical um the only difference is we are going to sort the two two halves of the list in parallel on two different notes so we're uh spawning a a recursive call to sort the left and at the same time we're also spawning a recursive call to sort the right and this works too um I'm just simulating its execution locally but um you know it's uh the idea is like this would be the same code you would run even if you were connected to some actually elastic source of compute um so yeah again just the difference is is really just in that this recursive call I don't know how legible okay um yeah it's that sort of sequential recursive call to sort versus that in parallel uh starting that recursive call running in parallel and then later calling Force to to combine the results cool all right let me show a super quick demo um okay so this is you can see these are the definitions of sort and desort um I'll go ahead and save the file um you can see it actually you know it type checks the file tells you all the definitions and um you can see it actually runs uh runs it and yep gives the same results um one other quick thing I was going to show is that like this run local function you know it's just a function anyone could write um so it's pretty simpler to simple to write these handlers for for the these abilities so you can have different ways of of interpreting these effects pretty easily it's not a lot of code cool all right so I think the big the big idea here is that all these things that we now consider as being separate from our programming languages are really part of the programming model so uh provisioning like okay spawn is saying hey provision a node for me deployment when I say at when I move a computation to a node it is dynamically deploying code to that node um it's not a thing that's happening sort of out of band outside of the language It's Just Happening all the time as part of execution um and yeah things like sort of autoscaling like fail over like these are all things that can just be libraries in the programming language and um by by bringing these these concerns into the language it means we have all the nice tools of programming languages to manage the complexity of that and and um the idea is like yeah instead of it being this crazy rub Goldberg machine you know it's back to this nice Fun World of like Lego blocks and like snapping things together and you have types everywhere and it's like great it's happy it's fun um so like you should have that same feeling when you're you're building a huge distributed system as you do when you're just building a thing that runs on one node and um yeah I think a big benefit is like okay now we can talk about distributed programs that are actually built in a in a truly compositional fashion where you actually factor out duplication and do all the nice things we're used to doing um so for instance like this distributed sort it's like oh man I look at this I'm like oh jeez hey I bet I could generalize This and like have work for like any monoid like I can just do a gen totally generic distributed reduce operation and like oh man that turns out that's super easy to do great and now I can like reuse this for all kinds of stuff not just distributed sort um so cool um the just just to kind of wrap up yeah we're we're trying to get an actual release out that you could use for real stuff uh spring 2019 um there's going to be just like a ton of stuff to do and like like the library ecosystem is like totally nothing exists yet so if you are like interested in distributed systems and like want to try writing cool libraries uh in unison like you can just have a huge impact on on this ecosystem and um yeah I think it'll be a lot of fun so yeah so that's that's all I got um thank you everyone [Applause] um cool yeah I guess yeah we we have some time for questions so let's do it um I guess Martin two questions one is you had this quot character what was what did that mean oh yeah sorry um that's uh just like a delayed computation it's it's you can think of it as uh just like unit to a okay yeah that's right the second question is would it be feasible to just use essentially maybe a normal programming language or a language with your ideas I WR effects and Implement things like spawn and act in the library so does it have to be in the language or could it um I guess I never want to say it's impossible uh to to fit an existing language um I think it would be quite difficult or it would be more more complicated than the way we're doing it um there I guess I didn't I didn't get to talk about this but yeah I'd be happy to talk more after but there there's sort of some key core language decisions that that make it super easy to do that uh Dynamic deployment that I think would be hard to kind of add on to an existing language but I never want to say it's impossible because yeah yeah sure yeah yeah we can chat chat after uh Paul so uh I love the scale out story I wonder if for one of your backends you given any thought to targeting something like um and you know scaling on GPU sort of parallels story there um because I think we're in sort of this weird to me weird world right now where we have things like tensor flow which are basically libraries which are C++ using intrinsics to Target vectorization and or GPU native code and it just seems like it's sort of you know a Target Rich environment for something that cool uh yeah the question is about could we have uh use this for like GPU programming like spawning up nodes that like run on the GPU maybe something like that uh sounds very interesting it would be super cool to to do something around that um yeah one one thought is like I think you might want to have nodes with different uh different abilities so uh you might have like a GPU ability where there's certain operations that are uh super efficient and you might be like oh I want a GPU node and then it gives you access to these other abilities um so yeah I think there could be some really interesting stuff there but kind of TVD yeah if you have ideas like come come chat with us all right so is that a good ide support good ide support yeah um yeah that that would be great U yeah I mean we've been thinking a lot about how to make it um so it sort of fits because there is a lot of stuff that's new about Unison but we're we're trying to we'd like it to slot in as easily as possible to existing kind of programmer workflows and tools that people are used to using um so yeah all right um yeah definitely come come chat with us uh after and you know ask ask questions uh so thanks everyone [Applause]