Scale By The Bay 2021 : Kévin Rauscher, Functional streams for factorians

Scale By The Bay 2021 : Kévin Rauscher, Functional streams for factorians

Recording: Scale By The Bay 2021 : Kévin Rauscher, Functional streams for factorians

[Music] thank you hello everybody so let's start so we are here today to talk about two things i like functional programming and factorio a very quick note about me i'm a senior scholar developer i've been sewing with scala since 2016 and i'm actually joining zendesk very soon so now about the topic let's start so what is factorio so if you have not played factorio is a game basically to give you a very quick summary uh you crashed on an alien planet the local phone is not particularly happy about you being there and um well you need to basically build a factory to be to build a rocket to get out of this place so in a sense it's rocket science the interesting part about factorio is that in order for a machinery to continue producing stuff you need to have place for storing its output or else it will stop it's really practical because it means that we will not flood everything with uh with in unused production and it's also very practical for me because it's kind of look like some problems we actually have in in programming uh which we i will just talk about like right now um which is back pressure so what is back pressure well back pressure is when you have a very fast producer on a slow consumer or just a fast producer that actually produce stuff faster than the rest of the system is able to consume it can be a lot of things it can be different micro services it can be um different threads anything really and since since you are producing value faster than you consume them well at some point your consumers are the problem and since we are working with uh with bonded systems you have infinite memory you have finished cpu you have finite time um usually usually the issue is that well when you have these kind of problems you eat some limit like for example your system crash because it gets out of memory right um one of the the main one of the main solution to actually under this kind of issue is to basically slow down production production so it's not always possible but uh when it is it'll actually allow your system to continue operating in a at least relatively normal way even maybe slower but at least relatively normal at least it will not crash and that's why we are talking about this because uh well there are actually several libraries that actually under this kind of stuff but there is one i really love and i really want to talk about which is functional stream for scala right so functional stream for scala under this kind of issue by having a pool based model meaning that you actually don't produce value you pull value out of something so if you have a db for example you will get the necro next row on demand so this is a really well thought and well designed library and that's why i wanted to talk about it so let's get started so let's start with something simple here we have a simple mining drain right so this mining drill is uh we will do uh really not so much um it just repeat an operation which is the operation of let's mine something right and but if we do it over and over again until there is nothing more to mine or there is no room in zero so let's represent that using scala and using functional stream for scala so um here we have so basically uh two operations we have the first one which is mine iron which is basically mining one chunk of iron right so um for those of you that would not know about io basically in the in the context of this talk you can think about them like futures uh basically they are reproducible future when the the only difference is which is important is that when you run twice on io it basically produces two values while your future is cached so mine iron will basically do something it will basically sleep for two seconds wait for two seconds and then press produce a value right then we'll basically do this over and over again so i've not implemented the part where we actually run and run out of resources but it would not be very hard um and basically it will evaluate the i o so basically called mine iron over and over again so stream that eval would basically call an io and give you one result and repeat will basically repeat this operation over and over again and i've just added a small uh small method that basically allowed me to run my program because when you describe a stream or describe an io you basically are not actually doing anything you are just contracting translating a value that represents what you want to do which is really uh practical um and but uh but no we want to actually run it so basically we actually take 10 by 10 elements of the stream and we basically compile compile it into something that is runnable drain it of all existing value in the stream so all 10 values and run it and if we run that well we can see that every two seconds we are actually producing something right so that was a really simple example uh it's just for getting started of course let's try something a bit more different a bit more complex um so we have a furnace which is basically able to smelt iron ore into iron plates if you look at it it's some sort of function in the end it's a function that takes time and iron ore and turned it into a iron plate so in a sense uh it's a function that you can basically in some sense apply in a map right and there is actually a concept in fs2 that you can use for this kind of thing which is basically pipe so basically a pipe is a you can think of it as a map for streams right it transforms a stream of a into a stream of b right relatively simple again um so let's do it so again we have this uh this function that operates again only on one element so basically turning iron ore into iron and i'm i want to insist on that because there is duality with between operating on one element and operating of on all the elements it's usually very um practical to describe operations on only one element except if you have good reason not to and then only apply uh so basically in the detail you go for one element and in the large you go for a stream right so um so we can we smelt iron on basically what we will do so if you have played with scala before you know you know map probably which is the function that turn a collection of a into a collection of b provided a function a to b well we will basically do the same thing we will uh turn uh use eval map to turn a stream of iron ore into a stream of iron the only difference between map and evil map is that even map will take an a to a io or any kind of effect of b rather than a to b right so um this allows us to basically run this and we basically use through to take a stream and pass it to through a pipe right and you can see that well now we actually um are producing iron plates but if you look closely you will remark that um there is a small difference between the picture i gave you the picture of a furnace and the uh the cli that is the this uh going on and the difference is that uh well when if you look at it when you produce one iron ore you actually produce one iron and then and then when you have produced this iron ore you will try to produce an iron plate which is not what is happening here because if you look at it you can see that um the production of iron does not stop it continues on and on and on because actually the furnace has some sort of input buffer which allows it to smooth issues with production so let's change it a bit and we we can improve it and say okay right now um i would say okay my input i will prefetch 50 values prefetch and 50. so basically what it will do is that it will create a buffer in input that basically is able to hold 50 value 50 elements and it will allow the input and the output to actually run in parallel and here we are and you can see that now i'm not limited by by uh by iron ore production i'm limited only by uh by iron iron furnace okay so now you can see that um well actually since it takes approximately twice the time to actually smelt iron than it takes to actually mine iron i can have two furnaces that operate on one single uh ion mining grain right so let's go for another another structure let's go for the transport pellet so the transfer belt is a bit more complicated because what i want to do is be able to emit somewhere and actually take elements from two different uh from two different consumers take the element of one producer right so basically i want some fan out and to do that i will use something that is called q so q is a concept that was uh originally in a functional stream for scala now it has been moved to cat's effect but since it's really useful for a lot of things um let's let's still talk about it today so a queue is basically a very simple it's pretty much what you would expect it's something that basically has four methods so basically you can take try take offer and try offer take and offer are actually synchronous so it means that if you try to take when there is no element uh it will block and wait for an element to be available if you're trying to offer while the queue is full uh it will block until you have room in the queue and try take and try over are the asynchronous ones where you have to actually handle when something goes wrong like for example there is nothing to take her you cannot offer right now so now we can implement uh this transfer belt and we can say okay um so i will create my queue so with this like always in this kind of scenarios basically there is a and i um basically when you create this kind of thing since it's a side effect most of this library will be into an i o so what i do is basically i create my my the buildings and the basically the i can't build the appropriate of the i the oh i will create a queue in iron belt and then i in iron pipeline actually create the the blueprint that will start by creating the cube and then i have basically three separate processes that what the first one will be the drill that will basically uh keep pushing things into the queue and the two furnaces that basically uh repeat the operation i take to the into the queue i take from the queue and then i go i pass it through the to the pipe so here i'm using repeatable which is basically the same thing that eval.repeat and the interesting part is that now i need to basically run all these streams together and i need to do it in a relatively smart way so basically what i want to do i want to get a stream of iron so basically i want the output of furnace 1 and furnace 2. that's why i will basically use merge on the two streams so furnace one merge furnace two which means they will run and give you a result the indent in deterministically from the first one or the second one and then i still need to actually run the drill stream because if i have nothing that is offering to the queue well basically my queue will always be empty and if my queue is always empty well my stream does not work so that's why i will basically add this concurrently drill and current currently drill will basically run the the stream in the background so it will basically run the stream in the background and doesn't care about the actual the actual values produced which there will be none since belt.offer does not give you any interesting values but it will continue running and you can see there that basically now i'm limited so i actually melting a creating one iron plank every two seconds which means that i'm capped now by the the rate of the creation of iron ore which is what we wanted um so okay so now we have we have this done and so now we want again continue building on this kind of stuff and start with again more complex or in this case not necessarily more complex scenario but which are assembly machines and assembly machines basically are really close to furnaces so um the idea of an assembly machine is that you basically take some element and give you back another item right um or it can take several elements and give you back one item so we basically have two recipes in this inside we have the copper cable recipe and the electronic circuit recipe the particularity of the cable recipe is that it basically takes one element and gives you two elements and the particularity of the electronic recipe is that basically it takes four elements three of one type and one on the other and gives you one m so let's start with the copper cable recipe so for assembling copper cable well basically i take one copper and i give you back a list of cable right it's a list of size two i could have used the tuple but there are several ways of course of do this to do this let's do it with uh with uh with a list because it's way simpler and again i will so basically my cable assembler is a pipe it takes an input copper and gives you back cable i again again i'm using prefetch n to desynchronize because to uh to decorate two so because assembly machine have an input buffer the only difficulties the difficulty is that basically well i end up with a stream of lists of copper right basically when i do input prefetch n50 eval map assemble copper cable i have a list a stream of list of cable and so the easiest way to actually handle this is basically to say okay i will just flat map my stream in uh giving it a stream of uh of cable and basically i can from a from a list of cable i can get a stream using stream.emits stream.emits will turn a list of a into a stream of a basically nothing nothing extremely complex there [Music] let's now talk about electronic circuits so basically electronic circuits so again same idea that basically i need three cables and one iron plate so my my functions that actually under the production of one circuit will not uh embody uh will not take much logic in it so i will just break if i have an incorrect amount of cable so here you can see that there are there is a chunk of cable the chunk of cable is basically is a so chunk is a specific collection from fs2 you can mostly think about it as a as a list in this context it's approximately the same um so um since i've got a list of cable and an iron well i can do my secret assembler again the circuit assembler will not handle a lot of logic in it uh i will do the same things and with the previous assembler basically saying okay prefetch n eval map etc etc but here the interesting part is about the circuit pipeline right because basically what i need to do is to basically bundle a stream of copper cables into streams of chunk of copper cable and to do that uh you can actually it's actually really easy to do because you can use trunk and method which will basically uh so here it's checked free so it will give you chunks of three elements and the second parameter is do we allow of having chunks of uh low of less than three in our case uh no because we don't we really won't have a hard constraint on the three elements cable and then well we we've got the iron pipeline that we defined earlier which is basically something that gives you iron plates and we basically zip the two so here uh so zipping a stream is the same as zipping a list basically you have a list of a a list of b you zip them you got a list of tuple a b right here's the only that is is that you we if you want to be coherent with the game we actually need to parzip them because uh zipping exists on a stream but zip what zip will do is that it will pull from the left side and then when it has an element pull from the right side um that's not what we want to do we want to basically wait for both in parallel so a par zip will wait for one of the two when it gets one of the two sides it will wait for the other and then start again and with prefetch it will actually take a regular element and you can see it works so when it will restart you can see that at around 8 seconds we will get our first electronic circuit so here we go here's the first one and you can see that everything is running in parallel so i didn't mention it before but uh we i have actually used a copper pipeline that is basically the iron pipeline i define it's a copy paste of of the iron pipeline it's exactly the same thing but so let's go for one other component let's go for splitters so splitters are are really important part of factorio it basically does a lot of things we will not cover all of them because it's actually there are actually several functions and which are pretty hard to implement actually we will only carry care about two the fan-in function so basically you get two you have two streams and you merge them into one so here is pretty pretty easy because well as you you've seen before well actually the functionality is already embedded in fs2 it's basically the merge functionality so nothing much to to say about that but it's more about it was more about introducing the component however splitters have a way more interesting property is that they can actually filter elements um they can actually say okay i have a stream of several kind of elements and i want to have a end up with two streams one of element a and one of the element of all the other elements or in our case it will just be element b so this is actually a bit more complex to implement right so here we will basically aggregate a lot of concepts with uh with seed and we actually will actually use two uh to cues to uh to simplify that so basically i'm representing uh the left output and the right output as two queues right and basically uh the stream the output stream so and the the left output stream and the right or top stream would just be take from the queue um as you can see i'm returning something that has a big a bit barbaric in its time signature it's a stream of i o of stream of fire iron and stream of io copper basically the so the encapsulating stream represents the the creation of the cues basically it's the effect of creating a cube it's necessary uh at the end i will explain wine so basically now that i have two cues so you can see that i have basically a bounded queue so of size zero uh so size zero basically it means that i can actually uh only have one element ongoing in the queue um and then so basically my filter will take the input and uh just depending pattern match on the input and basically send it either to the right or to the left um so once i have that um basically the idea is that i just need to give the left output to the right output in the output and in the stream and then i will i still need to run filters so basically i need to concurrently run filters so basically filter will run in the background for the same reason than before if i don't do that basically nothing will get inside the queue and if nothing gets inside the queue well basically my filter will do nothing and then i can run this so when i run this i need so i'm it's a bit harder to run because i get back two strings and actually need to do something with these two streams so what i will do is that i will merge them back but i will only take two elements of the copper stream or the uh the left string and let's see what it does so what you can see is that at some point it just stopped filtering well that's annoying but the reason the question is why well actually so the issue here is that since i'm taking only two elements of the copper stream basically i have a blocked output since i have a blocked output my process cannot continue it will continue uh producing values inside my all my buffers because well they don't know about the rest of the process but uh since my uh my copper stream is blocked well if you go back here you can see that my right.take or actually my right that offer will end up with a full uh full cue and it will just block indefinitely so that's something you need to take okay to care about when you are doing streams there are still a few subtleties that you have to deal but you have to deal with them in any library that's actually on that stream okay so uh here we go and let's let's start let's do something a bit more complex now that we have all the concept um required right so i mean here's this four factorial is a pretty simple simple element um yeah no we will not do that uh because not because it's necessary hard to do but because um we reach the limits of how much code you can you can fit in a slide or in this kind of presentation well um to conclude um well i hope that so basically now you have built your factory you have launched your first successful rocket you're happy so you are now a master of rocket science um so my i hope that i encourage you to try a function that stream for scala it's a really well designed library um it's really interesting and it's basically the same ideas and with functional programming uh you basically define well well uh well behaving components and you basically assemble these components and that's what you've been doing and doing and doing um basically assembling components that basically not know nothing from one another but are still composing well because they are basically sane um and so you can really build pretty complex system with just small components that you have cobbled together right so well uh i guess that's that's it so thank you for your attention uh i've got the source of the code are available uh we're following the link and the slides are also available following the link so i thank you and if you have any question i'm available for a few more minutes and i will be available in spicy chat just after that thank you yeah thank you very much kevin for this really excellent presentation and it appears that we don't have any questions uh in the discord at the moment so i think we can wrap it up okay perfect see you in special chat thanks again [Music] you