Function-Passing Style: A New Model for Asynchronous and Distributed Programming

SF Scala: Heather Miller, Function-Passing Style, A New Model for Distributed Programming

Recording: SF Scala: Heather Miller, Function-Passing Style, A New Model for Distributed Programming

so my name is Heather Miller I'm a PhD student at epfl um I work with Martin Odi on Scala a lot of the time actually aot all of my life that's all I do um and uh I mean I might have been responsible for annoying documentation so if you're annoyed by documentation it's probably my fault um if you're annoyed by lots of different things I could be involved and you know you being annoyed sorry in advance um since I got involved in all the schala stuff I've been working on like lots of different things um I was involved in the Futures and Promises Library uh like all the documentation and uh lately I've been working on on uh trying to make Scala a little bit nicer to use when you're moving data around basically or when you're operating on a bunch of data like that's sort of like the thesis statement of all the things that I try to do but I try to do it from the language of the level I'm sorry the the the the level of the language um and so today um I I I've given this talk like once or twice before um it's about a research project that I'm currently working on it's not um it's just like a bunch of cool ideas something that we built and we're playing with it and we've implemented a bunch of things using it and it seems to be pretty interesting and useful um it seems to have its own specific sort of like you know Niche use cases um but you know this is actually like a a a project that's currently like underway in the in the middle of like the design phase so details are changing um and uh you know like so what you see um you know could change in a month or something like this um but this actually um oh it works this actually brings together uh two projects that I've recently worked on that you might have heard of before one is this uh serialization project called uh Scala pickling and uh the other is uh this thing called spores which I'll talk about in a second but the serialization project is actually um It's actually kind of an interesting project because it's the idea is that you don't you do serialization totally differently than usually you do like in the jbm you try to do as much of it at compile time as possible then you like inline sort of the formats and whatnot that you actually want to use at the use site where you like you know are actually trying to serialize data like you serialize it at compile time if you can you can't always serialize at compile time um but we try to do it way the most uh and so it's really performant and then it's also really flexible because you can change formats and stuff it's a it's a pretty cool uh framework and it's also like all based on type classes so if you know what type classes are that's great if you don't doesn't really matter the the nice thing about type classes is that you can like compose them and do cool things with them make them like do all kinds of stuff like in your programs and they're kind of a powerful tool if you know how to use them correctly um this is actually like one cool thing about this framework is that this this this this it's built using these type classes but at compile time so the actual like when the code is compiling the compiler is like composing type classes and that makes it makes it possible for you to add your own serializers that you like write yourself that are pretty small if you want to just like you know um like escape from java serialization and then like patch how Java does it well or well how this framework does it not Java necessarily but also uh you know this is this is how you you input your own calization format and then the other thing is is spores uh they they're a funny name um they are a closure like abstraction that uh are safe and guaranteed to be sterilizable and don't grab crazy object graphs and so the idea is that you should be able to send these these these things over the wire and use them in situations where you want to move some function to some data somewhere like that's the goal of this so and also there's a there's like uh research that we've done in this direction uh there's a re there's research that we've done in the direction of like adding type constraints so you can actually control the environment and all kinds of things that these closures capture so people can't capture actors or sockets or things like this um that's more like research but the the portable sort of serialization um sorry seriz closures thing is actually not super researchy uh so these two things exist um and this talk actually kind of brings these two things together in some way um so it builds on the the the basis of these serializable functions to provide some sort of substrate that if you were to build a system yourself like some I mean if you're if you're going to build spark for example I mean or something else where you're making different two different machines communicate or you're you're you've got two data stores you're coordinating between um the idea is that you want to uh you know like maybe bring these two things together to uh uh to to well actually the idea is to bring these two things together in a nice programming model um that that is based on this idea of uh of oh am I ahead of myself okay yeah I can't I usually have a I can see what I'm talking about okay so uh yeah so okay so the idea is that you have this nice stateless programming model that uh like makes it a little bit easier to recover from failures or handle failures yourself and also to to maybe put keep stuff in memory and also do a little bit of debugging on the on the sort of like the the you know debugging communication and the level of like distribution between machines um because if you if you actually like hack around in in in a bunch of you know like Frameworks that do communication between machines you find that like a lot of type information goes away and then you kind of get like lost with what you try to send over the wire or what something is that you receive on the other end so debugging and and types like at that level are pretty useful uh and the whole point is that this whole thing is is actually uh very like clean and functional it's totally stateless and it's all based on like you know IM mutable sort of data structures that I'm going to show you in a minute and again uh it's a research project it's we're working on a paper right now details are changing of the design so don't like you know etch this in stone or anything and data bricks let me do this for you know probably 20% of my time there so that was really nice of them so I have to say thank you to dat for letting me spend time on this um but the fundamental idea is to actually kind of invert the actor model so if you I'll show you what I mean in a moment but this this model is kind of like a dual to actors if you look at it that way um and it actually can work nicely with actors if you if you play with it a little bit so you might okay all right what does that mean um whoops if you look at actors actors are you can think of them as as something that exists on one machine that typically kind of protects some sort of state that you know uh and makes it easier to reason about things happening concurrently like these these agents kind of or actors rather talking to each other um there's a message Handler that the actor has that represents sort of all the stuff that you're supposed to be doing to that state right and you're sending data between these actors the whole idea is that uh when you receive a certain message uh then you do some action based that's sort of implemented this this this um this this event handler or sorry this this message Handler uh and and you know if you look at it like and you squint your eyes you're you're you're what you're doing with an actor is you're exchanging data or commands whatever you want to call them through asynchronous messaging right um that's sort of the the like if you zoom out and look at it um this function passing stuff is basically the opposite so rather than keeping uh you know your your functionality stationary on one machine uh we keep your data stationary on one machine we don't let you mutate it and then we let you you know you pass these functions around to actually uh to do bunch of Transformations on your data so think about it kind of like you know how we work with lists or or things like this you know we we we we chain a bunch of transformations to something and we we have a bunch of sort of like functions that we apply to high that we pass to high order functions and you wind up with this this you know like sort of transformation on an original piece of data so that's the idea like that's the the basic idea um and so I also would like you know so people I say oh this is something where you're passing these functions around this this actually not a new model of like concurrent processes uh it's just it's just like a new a new means of working with distributed data in in like an inverted way so this don't think that there's going to be some you know interesting calculus or something about this this is not not up that that alley um actually I was going to make a point about this this is actually there's this cool paper um that was written in 1994 it's actually like an essay it's not a paper it's like a discussion uh and and you know it it lists a whole bunch of things that we don't really do nowadays even 10 years later when we build systems there's a bunch of sort of design um uh sort of design constraint or rather design decisions that people who build systems like don't always follow um or or Frameworks for you know doing things in some distributed way um and and like the the the one kind of quote that I like from this paper uh it kind of it identifies uh like four things latency memory access partial failure and concurrency and not like you know making this part of your your your uh it basically alleges that if you don't make this part of your computational model or the programing model the programmer uses uh it's just it's a bad idea uh and I mean there's a lot of reasons why you can read the paper um but what they suggest is that well a better idea is to accept that these things exist don't pretend like they they don't exist don't try to abstract over them and say they're not there don't say that latency doesn't exist uh don't say so I can just read the quote a better approach is to accept that there are irreconcilable differences between local and distributed computation don't pretend that you know something that is simple to reason about on one machine should be you know uh like exactly transparent and the same in all other machines right instead uh be conscious of those differences all these four things at all stages of the design and the implementation of distributed applications and rather than trying to merge these two objects local and remote Engineers should be constantly reminded of the differences between the two knowing it is appropriate to use each kind of object and be sort of exposed to these these things like latency and whatnot like know they exist I bring this up because um these things uh are kind of also like if you look at the actor model I mean these things are kind of exposed and visible if you're trying to use actors you have some notion of latency you have some notion of concurrency all these things are not hidden from you uh and and we try to do the kind of the same in this in this model where we're sending these functions around um so uh I I that's that's that's a bit of an introduction um I made a bunch of little illustrations to show what it looks like so you can kind of get some intuition about it um so the first thing there are two concepts um so like I said your data is stationary stays on one machine and it's mutable um and you have these functions that you can move around uh so you can you know apply these functions to data on different machines uh I have a really stupid name because I started with the tradition of naming things stupidly after spores I have to name it silos right now if somebody wants to suggest something better please let me know if you're better at naming I do computer science so I'm bad at naming um and the other thing is you know are these spores things so so The Silo the idea of The Silo is is something that holds this immutable data okay so what are they first I'm going to show a picture of how you use them so there are two things actually a silo exists kind of like an actor do you does anybody is are people familiar with actor systems do you use actors ever like can I see a raise of hands like show of hands okay so an actor ref you guys know what that is okay cool all right so you use these things to to to to to basically on one machine to talk to a remote actor uh and that's the idea so we have the same kind of idea uh it's just that they're called I mean for a lack of better name of course Sy ref and uh forget about master worker it doesn't matter these are just two different machines on one machine you might have a silo ref which is the thing on the or on the right sorry and on the other machine you would have this this big chunky blocky thing it would be like the actual data container the thing that holds the data thing we call a silo um and the idea is is that you build kind of a data structure by operating on The Silo so if I start applying functions to this thing that represents a piece of data I mean just think of it like an option or something else you you you know you apply some you get something out of it you apply some function to it you get another option or something out you know um you can kind of like make a tree of of things that you've done to this original piece of data like by applying functions to it in different ways so um it kind of like it oops no it's going to do the whole animation now okay well now you saw the animation all right so the idea is that you use this this this representation of this data that you have distributed elsewhere it doesn't all have to be in one machine actually these these these Silo things can also exist on other machines these Silo refs just point to where they they actually live and you do all of your operations on these these little Silo ref things so you're you're basically applying functions to your pieces of data and that's how you you use it on one machine um and and and basically what happens is you wind up so these little dotted lines like represent the sort of like the reference to this distributed or this this object on another machine uh and and the arrows the solid arrows are basically kind of like a tree of Transformations that you might do to it so like you know if let's just imagine that this is a list of like a silo of type list of something um sorry like maybe this is some map map you know I don't know another map or filter reduce or something like you you have like a tree of of Transformations or operations that you do on this piece of data that's like the basic sort of picture uh and again I I highlight like that you know you can have this guy like on some other machine and don't worry it all kind of still makes some kind of sense it's just that you know that you're dealing with a silo ref and you actually never touch the actual Silo um and and that's the basic idea of The Silo thing you have a few Primitives as well three right now uh I I'm I'm trying to make you know trying to make sure this stays as small as possible um oh okay so again Silo ref points to The Silo so you might have a silo of some type T these are all have type parameters so Silo ref of type T points to a silo of type T right and you look at the the The Silo ref thing as the Workhorse the thing that you do all of the the work on to you know like get some job done right um there's this method apply send and another stupidly named thing looking for better names called pump two um so there this this this apply method is actually uh lazy and I'll show you what that means in a minute why basically the idea is that you can build a richer logic by composing functions together that that do sort of complicated things um but maybe you don't always want to like kick off that computation right away um so I mean you can I I guess one like way to think about it is if you use spark you don't you know you're not like eagerly doing like every map or anything like this you know like only when you may be like actually need to kick off that computation will you actually do it it's the same kind of idea you don't have to like this this is lazy it like it figures out how to apply functions together it like kind of cues them up and then there's another oops oh no uh so basically it cues them up and it it defers it defers the actual application of these things to your remote data until you force it to be sent until you send it over the network uh and that's the send operator that's that's actually eager oops oh yeah so the send thing um so so with apply you know you apply a silo I'm sorry a Spore um in this case there are two we're working on like figuring out whether or not two or one are necessary doesn't matter so much the details why now I can talk to you more if you want to know why later but um it returns The Silo ref that represents the the the new the new piece of distributed or remote data that that will one day eventually be materialized so you can keep on operating on it uh it doesn't actually have to be materialized yet it can be materialize later but you can chain all kinds of operations to it and then only when you push like when you when you push sort of like the the send button when you force this uh do these functions get sent over the wire do they get applied and then eventually uh this this uh this Silo rep that points to your piece of data like it all like you know connects and completes and I'll show you in a second again another illustration to maybe make it a little bit more clear um and the send operation actually just does the the real send and you get back a future representing uh the result type that you want to get back from your distributed data or your data on another machine so this is all asynchronous uh whenever that work gets done eventually you'll receive it back in the form of a future and it'll be completed and sometimes you need to move data so if you think about like uh some kind of distributed group buy or something like this um you need to move data around to do these things uh so that's what this thing is about this this pump two thing it's a way of of actually uh of of of like you know shuffling or moving data basically so you need some way to do that um could you push the thing yeah thank you uh so sometimes you need to do that uh and uh basically the idea is that it defines how you should like so there's this this this this Spore this function that should Define how to transform each piece of data and and then you know where to send it to basically so this is important it needs to exist if you ever want to actually do any sort of you know real building over a system um I'm going to show you another illustration again so maybe like I I can like fill in what The Silo ref and like what these messages kind of look like now if you send them a function so let's just say you have some Silo of type t and a silo ref which points to that Silo of type T Silo is also Silo is also of type T um and let's just say you want to do something to your data like I said maybe whatever you want to do represents like a map maybe maybe there's a list inside of your Silo and you want to do a map on all the things inside of it doesn't matter what you do is you there's you know you you can use this apply function plus the send function basically to you know you define your function how it should be applied to the data inside and you just kind of do your send and you get back this new stf so this is kind of like a list right um but it's all like lazy or rather it's all asynchronous and it happens between machines basically so if you if you have a so let's if you pretend like this is a list you have a list of T you do like a map on it you get a new list out of T same idea right like you're kind of kind of you know doing these like you know Transformations on on collections if you're familiar with these kind of things okay when this function gets to this machine it gets applied by the runtime so there's something that like receives the messages and applies them to the data that they point to and all that like know that's part of sort of like the runtime that's running eventually a new piece of data Silo of Type S appears and this this uh this The Silo rep that points to it uh you know can actually start getting new messages to that piece of data so you can actually be using the silo of s before the The Silo of Type S is materialized the runtime cues the messages and then sends them whenever The Silo of s is actually materialized is that clear I mean if it's if it's not like I am happy to explain things like 20 times it's totally fine where is the S when is the S happen so I just I just pretended like you know I I did apply and then send so like let just say apply this function do send now and then that's kind of like the illustration and where is the value that is returned by the future so this is just like applying in oh actually good question I didn't I didn't uh I didn't illustrate that back there's another example you'll see that don't worry okay oops oh and another Point again um this all doesn't have to be in this like you know this machine and that machine mode you can mix these things you can have Sil on some machines with silos on that machine that have no relationship to one another and you can mix them in any way you want just like you can put actors in any way on any machine and have actors with other references on other machines oh yeah there see okay so um I will get back to your question when I talk a little bit about what these Spore things are and then I'll show I'll I'll answer by trying to do like like again it's it's I can't show all the details but I will show you how to do kind of like like a little map reduce kind of thing on uh on these things and you'll see the the the result coming back so now these four things does anybody like are you familiar with these I just don't I want to know how much I should ramble on about them does anybody know about these things Okay cool so I can actually give you a bit more detail so these things I I like to keep using this definition they are small units of possibly mobile functional Behavior they are not act they subass functions uh but they are not functions in the eles they have extra I mean they are functions they contain a function um but they are a separate abstraction because you need extra checking you need some extra work done to make sure that you don't get a crazy object graph and all kinds of other stuff happening when you're trying to send these things around um oh also this this I mean if you want to like go into gory details I recommend you like look at this uh it's a it's a Scala Improvement proposal it talks about all the stuff that is valid to be inside of a Spore thing say and what's not um it's at the bottom of the screen but I tell you now in case you want to write down anyway uh so yes there are closure like abstraction for use in distributed or concurrent environments I can explain how they're useful in concurrent environments right now we're talking about sending things over the wire so I'll focus on that um but really when we built these or we came up with this idea the idea was uh to have these have well- behaved closures with controlled environments controlled environments that can avoid all kinds of hazards that end up happening and there are like numerous really inelegant stupid ugly hazards um and these these include memory leaks uh so accidentally capturing something you didn't know you captured in in a closure and keeping it around forever um raise raise conditions due to capturing mutable references so if you capture a VAR of something and someone some other thread can just you know keep on mutating it and you think it means something it changes over time right that's not a good that happens actually and uh uh what I care about a lot lately is these runtime serialization eras that occur to unintended capture of references so that means like you accidentally captured perhaps maybe you're enclosing object or something like this but you can't really tell from looking at the code it's just you know because we translate these things into into objects and like you know there's some sort of ownership chain or rather chain of things that that are are outer to it and accidentally things can get pulled in basically and that you don't know actually if you had any idea what any of your object graphs looks like you guys would all be disgusted because there they're so confusing um anyway so this is like the long hand form this is what they look like and this is like when they're ugly they can be shorter and prettier but this is everything that they look like completely all of them right now so you have this this thing called a like I call it the Spore header there's two parts uh there's this thing right this thing called helper right now and then there's the actual function okay so that's that's those are the two parts this represents the environment and this is the actual function that you want to send around so bottom I the the basic idea is that this code will not compile if anything inside of here reference anything outside of this four block what that means is that you you have to explicitly say what your environment is basically so that means you can't build a crazy object graph um there's there's more to it you can have four comprehension and all kinds of other things this gets more like to be a little bit more detailed but um that's the basic idea these things can also be implicitly converted um so if you have a have a let's just well I'll show you in a second actually um so there's also uh so basically everything that so so the the rules or the guarantees are that everything that you capture has to be declared in this header thing and that and the way that this all works is that the initializers of the things that are captured are executed once when you create the Spore like when you create that like local you know value thing okay uh so that means that that what you actually put in there can't change over time they're all vowels and everything so it can't change meaning uh it can't be like later initialized ized or like as your program is running and something else in your program changes meaning like when you create it and use it it's like kind of fixed basically uh and then references to captured variables do not change like I said um also very important to note like I'm not changing how things get evaluated here there's nothing special I'm not like you know changing the rules of how stuff gets evaluated at all uh if you remove that this this thing here if you remove this guy this is just a block and it behaves exactly the same so if you if you know what a block looks like in Scala you make some curly braces and you put some stuff inside of it and that just like is just a block of something that gets executed right this is the same um yep and also they're related in that while one is a subass of the other but uh you can also implicitly convert uh you can implicitly convert uh spores I'm sorry functions to spores so if you have a an API like some method that expects a Spore a user and some user has written a function then it can be basically converted and then checked and then you get a helpful error message that says you captured something horrible you shouldn't do that so um that like it's it's like very simple and you know kind of ugly maybe but it can be made prettier uh it's actually like does a lot for you it makes like you know it sort of drastically changes like the the kind of stuff that you can be passing around in some sort of system if you're passing functions around right it like really makes everything that you're sending over the network sane all of a sudden so since captured expressions are evaluated upon sport creation you can think of this all as like you have a function with an immutable environment going around everywhere like that that environment has nothing to do with one machine like you kind of make a local copy of that environment and you're sting that environment to uh and and also the environment is specified and checked and there's no accidentally capturing you have to you have to write your own environment so you don't accidentally capture what media um if you if if you think about this uh like you know a cute little example here is like maybe if this wasn't a Spore and this was just a function like you know you could do like wikipedia. length or something and like maybe you accidentally capture all of Wikipedia and then try to send it over the wire but I only want it an integer but you captured Wikipedia I mean these are kind of things where you have to explicitly say like put Wikipedia in my function and you know what you're doing um yeah and you don't want to do that right that's common sense okay so I even drew a picture cuz I like pictures uh so if you create one of these things and you create a Spore like you put them next to each other like I said all the initializers everything that you end up sort of you know like referencing that all gets called like right when the thing is created so if there is like you know so so some objects or whatever like this is all specified in the environment and it's like sort of solid in there whereas closures you can have all kinds of vs in them something that you're calling could be unstable it could be just like a method that changes over time whatever um this you have no idea what it is from the time that the the program starts to the time that it ends basically and it can change it can grab all kinds of other objects over time like at one moment in time it can point to something another moment in time it can have like point to 20 other things you don't know and spores always just kind of stay the same and that's like you know what's what's what's useful about them um is working yes oh yeah in your stuff drag on your okay so um to reiterate uh sort of the benefits of these things are that the the captured variables are declared explicitly fixed and that you can actually also uh statically ensure that everything inside of it that you've captured is serializable so I can also um this is another detail you can actually you know you can require that they're pickable basically that that the serialization framework knows how to serialize them so then you get statically like you know like like static errors saying hey that was a bad idea to serialize that thing because I can't serialize it um there's a research paper about this uh that we published this year um and it goes a lot more into other details in other than just like this shape and making sure that object graphs don't get crazy and whatever uh this actually goes into there's even a way to enforce that certain like logical properties hold true of the stuff that you capture inside so uh there's all kinds of like idea like you again you don't want to capture an actor maybe you don't want to capture a socket you or if you want oh well not you but let's just say you have an let's just say you're spark let's say you have an API that you you you give to users you say okay pass a function to this thing and we're going to send this this thing over the network a user can pass anything into it um but maybe you want to say okay well don't capture the spark context or don't don't capture you know sockets or don't capture something like this so This is actually a way to to prevent all of that stuff from being captured using the type system and you can compose these things together you can compose these functions together and they keep their type constraints so if you make some rule about what should not be captured and then somebody composes that with some other function from somewhere else that constraint like holds and stays uh so actually that's kind of interesting thing but that's that's like well beyond the realm of of like you know like everybody in this room probably you don't want to do that necessarily um okay so I told you I was going to give you an example now again and it's going to be another one of these visual ones uh uh so this is this the idea here um you're going to have to use your imagination because I can't put everything on one slide I want you to just like get the intuition um let's just say that you want to invent your own like you know distributed list thing uh and you want to have something that represents like a map and a reduce operation like you want to just apply something to every element inside of your list and then you want to like like somehow combine all the elements in your list and you want to implement these two pieces of logic okay uh and you have let's just like you know like let's get down to the actual user code you have a silo of list of int like let's just see how this works imagine for now that you have this map and this reduce thing implemented in terms of these Primitives that I gave you it does doesn't matter so much exactly what they look like now just imagine how they work uh important to note these things are also the spores so this logic is implemented kind of as a function oops um and okay here we go let's just say now you want to apply like this this this map function that you've implemented yourself that exists as a Spore and this F thing here this this F guy is uh is the user provided function that you want to apply to all the elements in your list so I don't know let's just say since it's a list of integers this is plus one on everything okay so this is the logic that you provided in your framework this is the logic that the user gives you okay and I'm just kind of like queuing this up for now like I I called basically you call apply on The Silo ref and you you give this um oops and maybe you do another map and you you know multiply everything by two who cares uh so you do map map basically use your your your logic that's implemented as map uh and then you know now maybe you want to do a reduce so you have your reduce Spore thing you have your function that you're going to pass to your reduce and now you're actually going to want to get a value back right so this is when you call this send thing so what the send thing does is it basically composes these functions well beforehand these functions are composed and all of this stuff is composed and sent using send it's that's then eager like right when that send thing is called all the stuff get sent over the wire to your the data that represents this or the stle that represents this list of integers okay so this is what it looks like like the moment you hit the send button basically so you have this this this Silo ref that was always pointing to some Silo existing on another machine your send thing happens so the runtime sort of receives the the message uh and it knows how to apply it to this this piece of data like it knows where it is and how to apply it uh and then oops these things get materialized like this little graph gets built does that make sense now is when the basically whoops is when this this integer gets sent back and the future is completed with the integer is that clear uh and that's the that's the basic that's the basic idea so this program model just like an active framework handles like the messaging and uses like NTI and whatever underneath and it does all that crap for you but you're you're reasoning about things like You're applying functions to data and another important point is um you can actually get rid of these intermediate data structures by you know doing the like you know composing these functions and like not calling so many applies like this is the logic this is this is really meant for people who want to build something that's distributed right and so you can do these optimizations yourself like you know using like these lazy ideas like I'm just going to whatever have something that that just takes stuff and then uses apply only until whatever you have some other method that uses descent something like this um and yeah like I said this this these things can be trees so you can make like graphs of computation from this like it's not just like here's one thing you transform you get a new thing out you can make these huge graphs of all kinds of different you know like that represent a bunch of Transformations that you do a piece of data and that's cool because if you ever anything ever like craps out and dies you can always just apply all of those functions again to it right like this is how spark works this is why spark is so smart um so the idea is that you're actually building like a persistent data structure between machines so you have pieces of data on one machine you got like the the little um a cute little data structure that represents all the Transformations that you end up applying to that data and you're basically like making like like you're replicating uh all of like the logic that you wish to apply and it's all serializable so all your functions everything that you're doing is serializable so you can just write all of this stuff to disk or you can send it to someone else if you ever like you know oh just just in case I die here here have like you know this all the stuff that I have done all you have to do is Traverse this data structure collect all the functions and then send them over does that make sense is that clear like why this is I mean this is just you just kind of can restart everything you don't have have to worry about you know some weird Corrupted State that happened you have just some persistent piece of data and then you replay everything on top of it so one question sure if you have a sequence of 10 operations they're all really expensive and the 10th one fails y run it are you going to do the first n again well actually I don't help you with that this is just kind of like you know actors are like hey if it fails we'll like notify you but good luck you know I mean I would I would like to whoops there's a button on there if you push it again it's the little play button wait wait wait I pushed it wait oh no it doesn't it's it's not like it doesn't like it sorry uh okay I'm sorry I totally got distracted what was I oh yeah I don't I don't do any of that like for I mean this is something so the idea is that you would kind of put the code in in the middle to manage these things based on whatever is important to you so uh you get it yes but can an act actually tell whether it has been run sorry oh like okay how far it's gotten no not necessarily you can so basically right now we can notify you if something doesn't exist anymore if it died or something but we don't do right now I mean that's a good question and it's something we should consider but we haven't like we just trying to figure out how to restart things if they die uh yeah so basically said all these things already so cool um so this is all I really wanted to like cover with this this function passing stuff just to give you kind of like the basic idea of how it works um to you know reiterate you have these pieces of data called syus that are that are actually a persistent data structure um and all the operations actually that you apply to them are all serializable so you can kind of save stuff if you want to uh and and taken together it it you know you can sort of it makes it maybe a little bit easier to recover from something dying maybe just by Design uh and so um I wanted to show you some stuff on my laptop uh but uh we couldn't get the the thing working there's like some air Apple TV thing so um I will maybe go put this up after I I I I like walk away or answer some questions but um we have there's actually so like I said we're working on a paper this thing is implemented it exists um it it's a right now the like the the current status of it is that we have um uh well right now it's built on top of like netti and we also have it built on top of of of actors just to see as a proof of concept to do the networking but it's built on top of netti basically and that's really the only dependency um we have uh a cute little uh I mean we got some like little demo applications like there's one um it's called we have this group group by key example where I can actually I I could I could I'm not going to show it because it's like you want to show Stu off your laptop have to yeah no it's okay I it's like I I should have maybe prepared it nicer it's like just some code that's hard to look at so I I can explain it uh more simply and you can look at it if you want online um again this like these apis and all of this and like sort of the semantics of this thing is changing so um like if you think that there's something stupid or ugly should let me know um we have a there's also in this if you go to this Google group thing there's also a design document that uh talks about The Primitives and why they're necessary and how you can you can Implement things like unions and joins and all that using them so there's more detail if you're like ah this isn't enough um it's if just go to this Google group thing and you'll see the the the sort of like working document we're working on a paper about it right now um and uh and and yeah so I think that's that's most of it um this also dep oh the other dependencies I'm sorry there the spores and the pickling and these things are are uh released and usable um for purposes right now no type system stuff on the spores uh because that stuff is I don't want people like shooting themselves in the foot with it um so I keep that secret if you really want to ask me um and that's currently the the status of any of these things if you have any questions about this stuff or uh you know like the function stuff or the serialization stuff or the spores stuff uh or any other random like I hate this method and that thing why is it that way I mean I could try to answer it but probably don't know but you can ask me any random scholar question if you want I think that was all I was hoping to say tonight so does anybody have any questions sure up um Can a data structure exceed the size of one machine without something bu into uh no no not yet that's important can you repeat the question good I'm the worst of that yeah okay it's okay I'll remind you I'm so bad at that what was question so the question was what happens if a piece of data gets to too big for one machine or even better uh what if you build some crazy graph that's too big and then you know I don't do anything to help you yet because I'm just trying to make it work right now um but yes good question we need to do something about that because that's something that then then you if you want to have one operation on all those things that that I mean that's kind of a lot well that's the sort of the reason why you have these two like you know ways to apply functionality you have the lazy and the eager so the idea is that you could at least you know try not to make too many copies of Wikipedia and memory on the machine that's distri like on another node so that was like one reason for this uh because you don't want too many intermediate data structures so they're like views if you know is anybody are you familiar with views okay they're buggy good um but the idea is good the idea is really nice um the idea is that basically you you can get rid of intermediate data structures by kind of like like lazily applying stuff and then only like you know making the data structures that you need so in the case where you do like map map reduce you actually only make like one data structure that represents the the result of the two maps together and then you can like do a reduce on that so you reduce the stuff that's in memory so that's why things are designed that way but uh we don't do anything explicitly to like manage me we're not like doing any sort of management of memory or any cluster anything we're just sending stuff to other machines and making sure it actually works okay thanks yep so when when you have um yeah hey uh so when you have like the um materialization of those views and you have the Sol send it gets a future but when that feature is complete does that data actually like flow back to that yeah so you obviously don't want like if your T is you know a wikkipedia graph again you know you want to but at least you should see that future of Wikipedia you know then you're like maybe that's expensive so yes um ideally like like the the sort of the nice pretty use case is like if you're going to do something like a reduce and you want just something small back you want to keep everything that you've mapped on those machines you don't want to move them what if you have data on two machines and you want to do something with them and get the result on third machine yeah so so okay so the question is like what if you have data on different machines and you want to do something with their results so that's why this funny pump to thing exists um so there in the in this repo there's this um this group by key example so the idea like I can just illustrate the example really quick the it's it's very simple let's just say you have um four machines and you have uh like hundreds of thousands of people on those machines and they all have an age and let's just say you want you know machine a to have people under the age 25 then 25 50 whatever you want to do like this you want to like put them on the machines dependent on their age um basically there's you know we can there's this this demo that shows how you can use these three these three Primitives this apply send and and this this pump to thing to to do that um and a way that looks kind of like you know high order functions on Collections and stuff so it's possible but I didn't illustrate it yep the U data processing example G really sounds a lot like the way spark works today in terms of you map operations not evaluated right away can you kind of highlight the differences between well spark is a distributed system and this is just like a aute little like coordination mechanism between machines so if you wanted to like you know have like some database that or not a database but like something that maybe accesses a database or like you have two different machines with two different pieces of data on them some way and you don't need like this you know like heterogeneous kind of like I'm sorry homogeneous like you know distributed list thing and you wanted to build something like of your own that somehow coordinates between two pieces of data or or you wanted to like make your own spark or something I use spark as an example because everybody knows it and and you know this this is like uh like a generalization of kind of like the spark kind of or more like more spark but also kind of map reduce like model in the form of like something that's uh the granularity of like an acurate framework so that's the difference it's this tiny little thing that does communication for you and it also it it makes communication more tight that's very important yep there is something I didn't understand about capturing the environment and getting it back for example if you were to capture anatomic integer or something that then can mutate do you completely freeze it okay so that that's a good question so the question is basically there's the notion of like shallow and kind of like deep immutability so I make sure you don't capture anything that like where the reference can be like mutated and changed so vars um you can't capture those but if you have like you know something else that's implemented in terms of all kinds of mutable stuff um I can't prevent that thing from being mutable I don't deeply like ically check that everything inside of this thing is is is mutable actually that's to be honest it's actually not the best idea because most data structures that you use that are effectively mutable as you see them like the list and Scala this thing is you know implemented in a mutable way everything is is actually mutable so if I actually did do that checking you couldn't use anything basically but no it doesn't do this deep checking where it figures out whether or not everything inside of it is mutable I think that's too restrictive but you can do it actually if you read the paper we show how it's like some way of defining something that exists as a property and you can connect Frameworks that do static checking basically which is how it works but how do one use silos get garbage collected good question good question how do sils get garbage collected they don't because they're all pointing to different things and you know you kind of want to keep them in memory because you're probably doing something with them for a while that's like you know the idea um we're currently like discussing well what if you wanted to do some streaming thing what if you wanted to uh I don't know maybe on the machine that The Silo exists on get rid of the intermediate data structures if you only need one we could do that we can make that possible problem is then that like you know the the whole making a like restarting the computation on something that exists that that goes away all of a sudden so then like being able to recover from some given given like all these random like this comp like this composition of functions that you have plus your piece of data like you can't go from that to to the result anymore so that's currently a design discussion and if you have ideas Alexi be loud so I just want to I want to build up on the previous question about the difference in SP and obviously um this a data bricks so what does data bricks want to see nothing this is just this is a resarch project and um I was just doing this on the side and having design discussions with people about it so this is not like anything that they have any so it's not like interesting spark 2.0 no not necessarily no hey um The Silo refs are they uh like locations of data on a particular machine on a network like yeah they they basically just contain like you know information like I mean just like you know IP address and like whatever report and all that like when you so I didn't show how to make a silo but you have to like you know it's just similar to actors at least in scholar okay so like in Earline right you have the ability to at least globally register her name you say you always talk to her name and then you're not bound to talking to an IP right um because I'm thinking like in the case The Silo is not available yeah so we don't have any like we don't do any sort of global so we don't do any sort of like Global like keeping track of all the actors that currently exist it's like you you create one on one machine it points to one on another machine uh and if you want to somehow share the knowledge of that actor ref existing I'm sorry that Silo ref existing you can serialize it sent it somewhere else but that's what we let you do basically okay thanks yeah does anybody want to know any other random thing do you have like other random like schola you know questions or anything else that you don't feel like asking tell me now I can't answer all of them by the way don't think I'm like some sort of know at all uh I was just wondering related to spars what would happen if in your closer you took uh you took like as an example a database connection and you send it over the wire what will happen then wait a database switch connection con something that's not me to be CER yeah so that's when you use the the the type system extension um but I mean just as it is right now it's not going to stop you from doing that but that's what the that's the whole point of this type system extension and the idea is then you can also compose these closures that have rules and stuff so like for example uh like imagine the scenario so so people use AA I've seen like lots of hands uh did people use Futures have you guys used them together have you ever had a problem okay yeah I mean so so I I can go into the detail why there are problems but um basically if you imagined okay well hey the reason why there are problems is because basically uh a future is like a closure it captures something that changes over time like that's the reason why it's a problem um doesn't really matter you can say ah wouldn't it be cool if we implemented Futures in terms of these four things it would fix all kinds of people's problems right um actually wait what was the point I was going to make there there was there was a point I'm not wait oh yeah okay I remember now that was the back on to the question of the of the of the constraints so if this happen uh let's just say you know okay well we know AA exists and like let's just say that uh actually does AA still ship Futures anymore or because I mean they're in the they're in the standard library but like they were shipping their own their own library next to it for a while I don't think they do anymore but at one point they did let's just say they all existed in the same in the same like you know distribution Library package okay uh you know that this type that you have in your library should never be C caped then you can always basically put on in in the type system uh where you're like you know defining what a what a what a what a uh what a future should be that it should always exclude in the type system uh this other type that you know about in your like library in your world so then you would then get a static error saying okay somebody tried to Capt one of your users tried to capture an actor basically uh and then um again these things can compose so if you have uh forget about a future now if you just have a function that captured something or that that didn't capture something you can you can compose these two functions together and the the the composition of the two functions uh like you don't lose constraints you can add constraints and stuff like this yay oh wait no more question so there is a sip U there was a sip outstanding for fors why do they need the S that's a good question so this is this like okay so this is this sounds to me like a scolar question yes I will so it sounds like so let me ask if this is the correct question then I will repeat the question um so this sounds to me like a like a process evolution of language kind of question like you know Community contributions how this all works like what does epfl and types have to do with things and blah okay so you wanted to know uh there's a sip why does it need a sip blah blah blah um well the whole point is to you know like let people see this not for it to be like some random surprise uh let people you know like argue about why it's a bad idea or a good idea uh show situations where it sucks or you know it could be better people have actually contributed quite a lot of ideas that have made it nicer um and and that's why it exists uh it also exists because we don't know if it's a good idea yet I mean it seems to be that everyone has sort of settled on it being a good idea which means it should be eventually shipped in some kind of distribution in some way um but this is you know to enable sort of other people other than just like six people to make decisions right or to weigh in on a decision being made um in terms of like process and how this all works and like you know oh I make a i here's a an improvement document okay World review it you know uh when will it be in like there is no answer on any of that because um you know there are a lot of different things that can delay something being included I mean you know uh type safe like binary and Source compatibility agreements for example or it can be uh just like we're not sure about the implementation or we're not sure it's a good idea yet so like there's no in terms of time there's no like you know in six months this will be in and it's it's just a way to put stuff up let people think about it if they want it to keep it or if they want to throw it away they can say something about it and then we can weigh decisions on whether or not to include it in the distri later distribution that way but I mean I'm wondering in terms of language so for AA does not require SS is is it the library or do do you actually need language change implementation so sips should be so uh currently the opinion about sips is that Library like anything that would be shipped in the distribution whether it be a change to the to like the semantics of the language to the syntax of the language or to just a new library or a new module or something that everyone's going to have to pull in all of a sudden this all needs a sip uh why AA doesn't get one um well I I right I mean they're officially they're like a separate project you know we do ship an outdated version of the library with the standard distribution um but I'm not I mean that's just because we include we pull in like a dependency which is AA uh it's it's it's really they're still a separate project they have their own sort of way of making decisions and we try not to basically break their Library I guess yeah anybody else all right big round of applause fore