SF Scala: Steven Grady, An Introduction to Erlang

SF Scala: Steven Grady, An Introduction to Erlang

Recording: SF Scala: Steven Grady, An Introduction to Erlang

I guess we can get started uh I'd like to thank Alexi for inviting me to talk to you guys it was quite a uh was a nice experience to meet up with him in uh the beak coffee house which I can also recommend great por overs in the East Bay right next to rockid Bart I'm there just about every morning um so uh I had overheard Alexa uh talking about organizing a scholar conference and I my perked up my uh thank you I perked up my uh ears and and uh start introduced myself and he said we have scholar people who I'm sure would be fascinated to learn about earling so I was happy to uh volunteer some time to tell you about it so uh I do there's going to be a chance to ask some questions this is going to be a little bit longer talk than the last one it'll last about 45 minutes and then there'll be a 15minute Q&A period uh so my name is Steven Grady I'm a senior architect with earling Solutions that's the premier earling Consulting company uh I've been with the company about 6 months uh the company's been around for uh many years they are primarily European presence but we're starting to expand into the into the Americas uh Mark sugiyama and I are the two engineers in the America in in North America for uh erlang and luckily we're both in the Bay Area uh you may have heard uh about erlang in a couple of different ways they made the news a few uh months ago when WhatsApp was purchased by Facebook for $19 billion that is a company with 55 employees total of which about a third of them were engineers working on the back end and they did all their work in erlang and they managed to capture a pretty large Market segment and so people were certainly very interested in this and that ratio of engineer to uh to billion was intriguing to people the other reason you might know about it particularly as scholar folks is that you uh know that it's a it's one of the earlier functional programs that uh programming languages that established a uh a presence and uh an interest in from people uh so I'm going to go through this talk a little bit quickly this is basically an introduction to the language in general uh with a focus on some of the elements that might relate to scholar programmers uh distributed programming uh and concurrency in particular but uh so I apologize now if I'm going pretty fast but I've got about 40 slides to get through um so erling started as uh a project by Ericson and in fact they still are the ones that develop the software and distributed it is open sourced but it's under their control they're the ones who uh produce it it was created originally to solve Telecom problems because Ericson is a telecommunications company uh the ISS the Telecom systems of 20 years ago were particularly complex but they were not allowed to have any downtime they needed to be scalable they needed to be maintainable and it was important to be distributed in particular because if you have your machine that's handling lots and lots of phone calls and it goes down you don't want everyone's phone calls to fail so uh and but meanwhile they needed to develop new features quickly and so uh some folks with Ericson created a new language called llang to address this so the ancestors of erlang uh included a VAR had a kind of a complex history it was influenced by a variet of languages on the concurrent side Ada and modula and chill on the functional side ML and Miranda this was essentially preh hascal uh and logical languages like prologue which in particular it took its syntax from so we're going to I'm going to talk about a various uh parts to erlang the declarative elements the how it does concurrency and it's approach what makes it a particularly robust language because you'll often hear discussions about how earing systems generally don't go down and that's as a whole any given part might go down uh how the distributed systems work uh how the hot code loading Works in particular which is pretty distinctive to erlang how it deals with M multi multicore support and OTP the open Telecom platform which is basically all of the libraries and uh Frameworks that come along with it that are not the language itself so let's start with the uh basic syntax and and roughly what the language itself is so it is a declarative language it's a functional programming language you guys are pretty familiar with functional programming I imagine uh and it's got a high abstraction level uh pattern matching is key to it uh by the way I have a a small background in schola I've done a little bit of schala programming a few years ago and so anything I speak will be not true but let's pretend it is uh so pattern matching case classes are somewhat uh you know similar to the pattern matching which is used all over the place in our line and it gen generally you get very concise readable programs so for instance let's suppose you wanted to write a fa factorial function because this is the canonical example you always give when talking about functional programming and recursion and so your mathematical expression of it might look familiar and so the implementation is almost a direct transliteration of that this code is actually a complete module if you stuck this into a file and compiled it with the airline compiler that's all you need so it's declaring that there is a module called ex ex1 example one it exports one function called factorial the factorial function has two versions and it does pattern matching effectively to say is this a zero or is it not a zero and if it's not a zero then let's return n * factorial of n minus one so there's a couple of things about this it's uh like I said it's expressive it's concise uh it's not a difficult language but it is very different I mean I'm sure all of this fun this syntax looks quite unusual to you if you come from a Java or Scola background or pretty much any of the the C derived languages that have lots of open curlies and closed curries hopefully um so in this case functions are defined with arrows uh phrases of a function are terminated with semicolons and complete definitions are terminated with periods variables are capitalized which is particularly unusual again that's one of the key things that comes from prologue uh also uh it has no pointers uh there's also no mutable values so in even as pure a functional language like hascal has mvars and such but llang has literally no mutable variables whatsoever there is a way to have mutable data but it's not based on any form of variable that you can change uh it has earling has support for higher order functions um also I'll point out one other thing which is interesting about this example there's no case for a factorial when n is less than zero and that's actually intentional and I'm going to come back to that okay so another uh oh sorry and if you wanted to oopsie if you want to compile it you can do this from inside the erlang uh interpreter so you just say C compile the name of the module and then you can just invoke it with module colon function argument uh here's another example this is uh a quick sort function actually not the true quick sort if you're going to be pedantic about it but a nice concise implementation of a a quick sort that uses list comprehens iions I believe the there's a rough equivalent in Scala and so this is basically saying uh there is that the thing highlighted is a list comprehension and what it's saying is let's take all the objects y from the list called rest where Y is greater than first uh rest and first come from pattern matching where it says a list consists of the first element and the rest of the elements uh again you can simply compile it and you would get be able to invoke the function uh I should mention that uh I think I forgot to mention this earing is dynamically typed key difference from Scala uh so you can so this function will work as long as uh comparisons work you less than or equal and you pass in a list uh another interesting uh relatively unique element of earling is the fact that it has support for uh bit sytax and this is particularly useful when dealing with network protocols as in this example where the first line or rather the first um expression which starts with these less than less than and goes all the way to equals segment is simply an assignment from the segment which would be a binary uh that contains a TCP packet and it puts it into all the individual variables so they're after that line Source Port is a variable of 16 bits destination Port is a variable of 16 bits and then even the uh Flags which over on the right say Flags colon 8 is an 8 bit segment that can then be broken down as well into the cwr ECE and so forth flags and so now you have all these variables and that's in two lines and that's implemented in the VM and so it's a very fast match okay so that was uh kind of the introduction to some of the syntax let's talk a little bit about the concurrency of ouring so it has either uh transparent or uh explicit concurrency and that means that you can either uh use functions that will spawn off uh processes processes by the way is the earling term for essentially a thread it's roughly the same in that it's extremely lightweight uh and so processes or you can explicitly spawn off new processes and I'm going to show examples of that processes are very lightweight uh you can they can be created in micros seconds and a normal orling VM running on a uh standard post can have millions of CPUs I had uh had a nice experience or an interesting experience when I was taking a class in a framework called Cowboy which is a very high performance low memory use um uh HTTP framework and the uh teacher was saying how would you do such and such and so various people said here's my solution and he says well that's fine if you've got a million simultaneous connections but if you have you know when you start getting up to 2 or three million that might start being too slow and I don't know about you guys but I had never really considered a single h DP server that could handle you know 2 million simultaneous connections and this this is written uh the the code would be written in a very natural way in earling where each one is its own process I.E thread handling its own input and the reason it can do that is because uh earling processes are so lightweight and it's very and it's highly scalable so because you tend to use lots of processes because they are very lightweight you can then uh easily uh you can scale in various ways if you're on multi-core machine you can get more processes running on the different cores if you need to get it onto multiple machines then you can have some of those processes running on those other machines as well and I'll talk about that more as well so uh the basic way that you use concurrency is the spawn function and so spawn creates a new process so in this example we've got a module EX3 that has a function called activity that has three arguments and so we invoke the spawn function taking the name of the mod the name of the function and the three arguments and we assign the result to PID a variable that stands for process ID and that will start a new process running that function these processes don't uh spawn basically doesn't fail if there's something wrong with the function then the new process will fail but the um the spawn will just continue uh when you have these processes you communicate in what people have referred to in a variety of context as Earl Lang style concurrency so that means that there is uh no sharing of data all process intercommunication goes through messages uh all of the data is immutable so that means you never have to deal with locks and mutexes uh people have implemented this erlang style uh concurrency in a variety of languages obviously AA being the significant one in the scholar world uh but it'sing is kind of the progenitor of this style it's so messaging message uh a few more points about this so This by the way is we've got one process that is going to invoke PID where PID is the variable pointing to this other process and it's going to send it a message that says data comma 12 comma 13 which is a tuple of three elements and the process goes to the other excuse me the message goes to the other process and the other process is listening with a receive statement saying if I see a tupal called start then do something if I do a stop do something else if I see a data XY which is what this will be then it's going to do that third case so all message sending is asynchronous if you want to block and wait for a response what you do is you make sure that it's written so that the receiver sends a message back to you and then you wait so you do your send and then do you do your own receive so you have to do explicit synchronization generally speaking you actually don't want to block very often in good or on code and I'll talk about that later so most so the asynchronous by default model is a kind of a key element of the design and philosophy uh by the way if you send a message to a non-existent process it just gets thrown away so you don't get errors if something's gone which also means if you want to know that it didn't get received then you're going to then you are going to need to have a message that notifies you if it was if it was received or not uh and finally if the receiving process does not uh is not ready to receive it for instance suppose I sent a message Fu to that pit and it's sitting that receive that'll that message will just sit in its mailbox and so later it might do a different receive where it says did anyone send me the foo message so it's that has its own implications for how you design Earline code okay so let's talk about what makes Earl Lang robust so there's a few elements first of all it's simple and consistent uh and excuse me simple and consistent error recovery the error recovery uh approach is standardized and I'm going to be talking about that quite a bit or you know uh more than uh nothing how about that and that's the key is the idea of supervision hierarchies and so let me step back for a second the reason why erlang has almost all of the features it has is because it was designed to be robust this idea that you can't have your telephone system go down leads to well what if various things what if the system has a crash okay well let's make sure we have have multiple machines so that something can pick up if the other one uh fails what if the programmer writes a bug well let's make sure that we can recover from that case cleanly and ideally uh let's recover uh in a way that's invisible to anybody uh to externally what if the we need a new feature let's make sure that we can add new code without bringing the system down and so forth so this is a particularly uh distinctive and important element of the technical it's not the language it's the libraries that come with the language called OTP the idea is that as with the pre previous slides these yellow circles represent processes and the processes uh are going to be communicating or not but one of the key questions is if I the process that I a process that's part of the system crashes how uh what are we going to do about it and the answer is you make sure that the other processor processes or at least one other process knows it about that so what you can do is you can create links between processes and when you create a link between process a and process B then a will now be notified if B crashes and B will be notified if a crashes and so this is you can do this by either invoking a particular uh F function when you create the process called spawn link or you can create the link after the fact the existence of Spawn link is important because it means that you can automically create and Link a process in one statement the handling of the links again is completely inside the earling VM so the idea is that if a if one of these processes crashes then it will send an exit message to all the other processes that are linked with so in that case the process that crashed was linked to one and then but that other one was linked to more and so those crashes will propagate and then the termination will propagate to all of the uh processes there's another approach which is that you can trap the failure of a uh process and so in this case we'll say that we're going to catch process failures so when that one crashes they various uh processes uh that are not catching things crash but this one says wait a minute if i i instead want to receive a message and so it uses its receive function to say if I ever get an exit tell me which PID which process ID it is and then we can figure out what we're going to do about it and that idea is the basis essentially for the robustness of llang because what you can do with these linked processes is create a robust a robust system by layering the various kinds of processes so in particular what you have is two kinds of processes supervisors and workers and the supervisor's job is to just watch the workers and if they crash to do something the workers can do whatever you want they can be HTTP handlers they can be doing CPU um calculations what have you and the the concept is that if you if if a process crashes then probably something went bad Pro went wrong we don't know what it was and rather than trying to recover directly let's just restart that process with the same with with a known State because a lot of bugs end up being effectively ephemeral in that they're due to something it might be a weird race condition it might be um bad data coming from somewhere it might be a bug in in some other part of the code that on unsanitized data and so if you can just restart your worker then you can by getting it back into a known State you can quite likely pick up from where you left off this works particularly well uh there's plenty of anecdotes in the airline Community uh about people who have developed relatively complex systems that heard about this and maybe were a little bit new to oring and so they hadn't necessarily done all the things they should have in terms of setting up monitoring and then they said hey this is great my system was working just fine and then they eventually think to check their logs and they find that there's been workers that have been crashing you know every hour for months but the functionality of the system didn't change at all I read another anecdote today actually that Rea which is a database built on top of earling a distributed database uh somebody was testing react among other databases and they wanted to say well what are the edge cases let's uh with the various distributed database systems what happens when you kill off a node and they were getting confused because when a the react post was killed it kind of dipped performance-wise briefly and then popped back up and everything was fine and it's like there was no real evidence that anything had gone wrong and again that's because the model is always things might fail but let's just recognize it let's log it let's restart so uh the next thing is that because of the erlang style model of uh actors and and sending data around rather than sharing data between threads it means that you don't actually need uh to do anything particularly different when you're decide that you want to scale out to B to multiple machines because you're just sending data around and so as a result you can get very uh clean distribution also the VM and the language support the idea of sending a message to a remote host as easily as sending to another process on the same host or in the same node and so you can e explicitly set up your distribution or you can do it implicitly by basically saying send a message to this process wherever it is I don't care and the runtime system is Network aware so it knows when new n new nodes connect it you can find out a list of nodes and things like that as again part of the basic language so the basic way this works is you might have a couple of processes running on uh one node A and B and then you if a wants to send a message to b it just says b bang whatever the message is if it wants to send a message to C C it does exactly the same thing but this time because the runtime system knows that c is on a different host it says let me handle this I'm going to stick onto the network I'm going to serialize the message and then when it gets received on the other side it gets deserialized uh but it's completely transparent and there are various libraries that come with earling to do they're built on top of this basic model for doing RPC uh parallel mapping and and so forth another feature of llang that is uh really stands out although I'm going to give some caveats to how useful it is is the idea of hot code loading and again this was developed in the environment where you have your telecommunication switch you need to have some new functionality but you can't you don't want to interrupt anything including the existing calls so the idea is suppose that you have some module with your code and you've got version one and you've got a process that is running version one and then you say wait a minute we need new functionality let's develop version two so what I mean when I say the process is running version one is it's probably in a loop it's sitting there saying am I receiving a message oh I've got a message let me handle it great now let's go back to the top and receive another message and so that Loop is presumably running relatively often as it's dealing with whatever is coming in and so so but when we've brought in version two we've now got a uh new module that's also part of the VM at the same time as the old one is and so we if we ever create a new process for that function for that code it will use version two so now you've got version two with one process running its code and you've got old version one with one process running its code but once version two is in place when version one gets back to the top of its Loop it can basically say all right I want to use tail recursion to invoke my function again and when it looks up what my function is it's going to be the version 2 one so the top of the loop is just going to keep on going so it's like version one version one version one version one version two version two version two and the process never stopped and now it's running the new code it does mean that your code has to be structured the right way to take advantage of this but that comes again standard with OTP these standard libraries uh let me give you uh I tell you another anecdote of something I found this on YouTube a few months ago I just found it delightful someone as a way of demonstrating this concept of hot code swapping had decided to implement a flight controller in a quad cop drone so it was like sitting on their table and he deliberately introduced a bug so you could watch the quadopter Drone going it was supposed to be stable in flight but it wasn't stable and then he fixed the bug uploaded it and the quadcopter went and it just became stable so this was upgrading the code of an inflight uh vehicle so that's again something that I'm not sure I would trust very much in other languages um all that being said oh by the way there's a a garbage collection of the modules themselves so once there's no more processes talking to version one version one will just go away um all that being said I said that there was a bit of a caveat and that is if you are depending on the style of system you're developing if you're creating say a web scale backend and you've got multiple servers you need to have an entire host be able to go down and the other hosts take over and if you've got that in place correctly then if you just kill the erlang VM without bothering to do any of this hot Cod hot code swapping then presumably your functionality will do the right thing and spread the workload to the other machines and then you can just copy in your new code arst it or what have you and then start it up again and then if you've again you written your code correctly it's all going to migrate back as as appropriate so in despite the cleverness of hot code swapping the I've never actually used it in a project because it was because all the systems I've worked on have had to been had to be robust enough that they could deal with shutting this system down or shutting a given VM down Okay so let's talk a little bit about multie support this is relatively new uh in the sense that it's a few years old as opposed to 20 years old like most of earling uh the key idea is that it provides linear scalability out of the box and what that means essentially is if you have a standardly written llang application that has let's say a few 100 or a few thousand threads or maybe a few million if you've ended up being ambitious and have scalability going on then when you put it on a machine with more CPUs those process will processes will process will just go to the appro uh various uh cores and it's you watch your CPU and you went from 100% to 700% it's I have never spent any particular time on earling systems figuring out how to take advantage of this of the multiple course it's just it just happens uh and that's again because of this the approach to concurrency it uses where we don't have to worry about locking and there's not scalability issues that come up with oh now you've got more contention for the locks Etc of course the inside the VM there's plenty of that stuff going on but that's where all of the engineering has Focus has been in the development of earling uh let's see is there anything else I want to say uh not about this uh I do want to talk about how it works though and this is actually a key distinction between the earling VM and the uh jvm or some of the jvms so the idea is that you have every process uh effective well there's every scheduler uh every core gets its own scheduler how about that and each scheduler maintains its own run que and there's also uh and and and so the processes just go to what uh to whatever scheduler is currently available and so the idea is that again you have more cores then you just get more schedulers and each scheduler is running on its own core and when there's something new to run it runs it the uh the other thing is but the key thing I'm sorry is the distinction between this and the jvm is the garbage collection so every process has its own garbage collection essentially independent of all the other ones that's there's a little bit of a lie in that but let's say that that's basically true what the heck's going on um and what that means is that a processes that are dealing with a small if you have lots and lots of processes that each are dealing with a relatively small amount of data and again there isn't shared data so uh the context is going to be constrained to that single process the GC is going to run very quickly and you basically don't get any stop the world we need to do garbage collection you you can of course have garbage collection issues if for some reason your code is written poorly or you and otherwise need want to optimize things but the nice thing is that when you have those problems what it means is you're not garbage collecting enough and you see your memory usage go up and that means you can anticipate a problem as opposed to waiting for something bad to happen in the you know a more common story in the jvm world is that you the garbage collection is suddenly very slow and now you're not getting the performance you were expecting but you find out after the fact rather than before it uh okay so let me talk about OTP briefly because it's actually a pretty significant part of the uh what you get when you buy into erlang so technically OTP is the entire erlang distribution so when you download erlang what you're really downloading is OTP and earling the compiler and The Interpreter comes with it uh it was open sourced 16 years ago the latest version is 17.3 and it is considered stable uh it has a lot of different libraries including things for XML SNMP SSL and so forth of course nothing compares to the number of java libraries I'm not going to say we compete there but it's pretty close to a batteries included kind of a distribution um one of the key elements of OTP is that design principles uh are implemented as code essentially that is to say uh if you want to know how to write a good a a good gen server in good early style you actually excuse me a good server in good early style one that listens to not not Network server but one that listens to messages and responds to them rather than figuring out how to write it on your own you should probably just use the standard gen server which stands for generic server and there's generic finite State machines and it's uh and this whole mechanism of saying there're supervisors and workers you don't have to write any of that yourself all of those things are supported as libraries they're standardized they have been um maintained uh and uh refined for many years and so there's just all this stuff you can you can pretty much think about the problem you want to solve and not worry about all the details so when I talk about all these nice things about how Earl works that's already been encapsulated in these libraries comes with a lot of useful tools Amnesia is a distributed replicating database uh dialyzer so I mentioned that earling is uh dynamically typed or uni typed if you prefer but diyer is an external tool that's not part of the compiler that uh implements what's called success type which is roughly like static typing it's a little bit looser version of static typing but basically you can find out if your functions are uh if if you got type errors in your code uh Observer is a real uh uh it's a graphical interface that provides real time access to all sorts of things about the earling uh VM so you can see you can look at all the processes that exist and what their current state is and how long your message cues are and so forth uh eunit is a unit test framework there's a documentation framework and so forth of course the entire system comes with it's it's fully documented there's an entire Suite of reference documents for everything that comes with it there's also a suite of user guides that correspond to each of uh the U subsystems and it's continuing to undergo development so uh Ericson releases a new one like I say or if I didn't say um every year and it continues to get better so for instance in the most recent version uh they added an entire new data type which was Maps uh used to be that you would use a library to do a uh to do any kind of map and now it's part of the language uh and other things that came in the most recent one is new optimizations better unic could support uh entire new language features like named local functions and so that came out in April and version 18 will probably come out in April or something of next year there actually other implementations of earling by the way there's including one on Zen and uh the Z that implementation is interesting because it's so low level that it can really make use of the some uh good lowlevel optimizations and one application that's been implemented on top of that is a software switch that can actually switch half a million packets per second which about the only system out there that can do better than that is written in C and you know sells for you know half a million dollars uh and then another part nice part about RCP and earling is that when you start it up you get a shell or you can uh you can get a console you can uh always get a for for default implementation of an earling system you can always uh get a console by uh coming in through a network port or through a local pipe or you can so you can you can interact with your functions and create new functions on the Fly and uh query your system and debug and turn tracing on and so forth all with pretty much any Earline application you write you don't have to throw anything extra in to get all of that functionality let me talk for a minute about Java uh integration so the way if you are writing because you can use uh if you for some reason earling seems like the right solution but you have to do some sort of interoperability with Java the normal way to do it is to it it extends the normal in erlang model which is you do it by sending messages so there's a protocol based uh interface called J interface and so for example if you want to send to erlang then you say here is the node that I want to be let's call it Bingo and then we're going to create a mailbox something so that we can uh receive and send messages and then let's send a message to Fu at Local Host which is going to be the name of another of an earling node and what we're going to send is an atom or an OTP earling atom that consist of the string hello if you want to receive messages from llang then you can register a name I I didn't talk about registering but the idea is that processes can either be named or unnamed or registered or unregistered if it's unregistered you actually need a very aable that has the process ID but you can also register a name and then you can just say I want to send this to for instance my print the print server process so in this case we can register a name called server and then we say inbox. receive and then that's going to return with some object when it comes back and then you can use instance of to determine what type it is uh there's a plenty of other serialization Frameworks however if you don't like that uh it comes with asn.1 there's Json message pack protocol buffers XML all these parsers and serial izers and deserializers um or you can just go to straight networking and not using uh serialization you can of course it's very straightforward to do TCP HTTP htps amqp zeromq so so I'm going to talk for a couple minutes about various myths of erlang so if you have been exposed to earling in the community you might have heard someone say hey I wrote my earling system in 4 weeks yeah not so much um it's certainly possible to write something in four weeks but if you're going to have a real system it you need to be do just like any other development uh of a robust system you're going to need to document it you need to uh make sure that you know how it's going to be supported uh Etc and so all these extra things you know they always require work it's not something earing provides a lot of help for these things but if you're going to do want SMP interfaces then you need to Define your MIB Etc uh but you can get your prototyping going very quickly and quicker than just about any other system that I've developed on uh another thing I was saying like hey upgrades during runtime are easy well not so much you can do it but it's not really easy I mean it's easy if you're going to do simple patches uh it's easy if you say oh I want to put this new algorithm in but the state is exactly the same but if you have to do things that are not backwards compatible you need to migrate your database schema uh if you are suddenly changing the state of your process remember I talked about that Loop which goes back to the top well if the loop is expecting to have a different state in version two than version one then you need a way of figuring out how to migrate that information uh and in you're when you're in distributed environments you have to figure out how to make that you have to think is this going to be transactional or what have you and of course you always have to test this kind of thing and so it's again not easy but it's possible and it's easier than other other systems another one is the nine nines of availability so Joe Armstrong one of the creators of earling actually wrote his PhD thesis about this and he was able to achieve this by lots and lots of interesting and careful techniques but it wasn't easy uh Mark and I actually worked at a previous company where the engineers jok that they hadn't achieved 9 NS but they had achieved 88s um but so not so much on the other hand um but are some but it still is pretty darn good like uh there's anecdotes that like they say there was this quote was from um Ericson that they have a risk that their Engineers are not learning how to do field maintenance because the system just works uh by the way uh I should to put that in context 9 NS is about 1 second every 30 years so that's pretty ambitious but 59s actually is very possible and very common which is about 5 minutes per year um and you can usually achieve it at the fraction of of the effort of what it would take to do the equivalent thing in Java or C uh but keep in mind you know where do those five minutes come from well upgrades are not always uh smooth and there's always non-software related issues if you have power outages and everything's collocated that can be an issue and so forth I wanted to talk for a few minutes about the erlang best practices uh so the first thing and this was actually one of the first things I learned when I started on earling which I thought I think I said I been doing it for about seven years uh is that you should follow OTP so I thought well gosh I understand the basics of how do you write a server why do I need all this extra Library stuff and then at some point I figured it out because I wasn't getting a lot of the functionality that comes for free if you use those libraries uh and OTP is not just some Engineers throwing it together this has been refined over and over and over again with lots of feedback from the community so you know imagine take your favorite library and think how would that be if just about everyone in the community schola or what have you was using it and they were providing feedback and we had 20 years of those iterations so that's where OTP sits um Let It crash as a key one this was actually because kind of blossomed as the motto of the irlan community although it goes back quite a few years I will say I was uh I'm not sure what my reaction was to discover today that let ITC crash.com is actually an AA site um I kind of feel like it should be an early sight but you someone uh missed the vote on that one uh so the concept behind let it crash is essentially rather than doing defensive programming and checking that you've got the uh that you're getting the answer back than you expect assume that you've got the the correct answer and assume that things work and if it doesn't then let things crash so this was my example when I mentioned about the factorial that we didn't check for n as less than zero that would have crashed if you passed something less than zero the key is that crashing is not necessar a bad thing when you're talking about an llang style uh set of processes and supervisors so in this example I've say well suppose you have some function that says let's try something that might fail so it's bad to just do that if there's an if it's and let's presume that try something which can return okay or an error of some sort so if you're not looking at the error at all that's bad because if it returned an error you have no idea and now you've got a state that you don't really understand because when you tried to do the thing it didn't and who knows what happened um a a traditional way to do it is let's make sure we check our errors so we're going to do case try something of blah blah blah and if it's okay then great let's keep going if it's an error let's log things and so forth but the best style is to say okay equals try something and that is actually doing a pattern match but the pattern is a constant pattern so it's going to say this must we're going to take the result and we're going to match it again okay against okay and as long as they match we're fine we keep going if it doesn't then that's a it's not a bug but it's a crash and the and the VM will effectively crash that process which will cause maybe exceptions to be handled higher but really what's likely in airling is that there will be no exception in the entire process will crash the supervisor will get notified because the supervisor is linked to it and will get this message saying hey I just died and it will log something in your logs and maybe send other kinds of triggers and then it'll restart the worker so and then the worker has restarted in a known State and maybe the thing that was supposed not supposed to fail uh will not will no longer fail uh by the way supervisors are reasonably intelligent if they uh restart something too many times in Too Short a period then they will give up and then they can even fail and you can have higher level supervisors which is why it's an entire tree but generally it doesn't come to that oh and by the way if you're thinking what happens if the top level supervisor crashes again 20 years focused on essentially one piece of code it does not crash um so given that your code is going to uh crash Anyway by which I mean you can't guarantee your code is perfect even with really strong type checking and so forth you may have some bugs that are dependent on external circumstances so allow your code to be written in a way that says okay we might crash sometimes let's make sure we catch and deal with it uh another thing is that you if you are going to let things crash you don't want much to crash this is another part of how earling has a nice approach where you use lots of processes that each do small things because if that process crashes only that small set of things is affected so for example if you're dealing with HTTP requests and each request is in its own process if one pro if one HTTP request fails for some reason it won't affect any of the others because that process will crash but all the other million processes won't uh and then you let them reset to a ownn state because again uh if you if this is an important worker then let's get it going again and let let get it going maybe in the initial state it started with and then it's going to be nice and clean again but you of course you need to make sure that the errors that do happen are logged and then you can look at the logs later and you can debug it maybe the next day rather than at you know 3: in the morning on a Saturday uh and but you do want to watch out for thrashing which is if something is crashing a lot your system might spend more time dealing with that and restarting it so you do have to be aware of your system you don't simply want to let all uh crashing systems slide because hey your system was working reasonably well because you can have consequences of that but the basic model is let it crash and then let it clean up but that that would be a longer motto um so another best practice is basically synchronization in a distributed system is hard so you've got race conditions uh you know you don't have race conditions in terms of threads and and uh data shared data structures but you can if I'm sending a message uh and and and I'm expecting that the two NOS my node and the other node have the same view of the data but meanwhile something in comes in and maybe it's going in the opposite direction you may have a bug that's due to a race condition um mirroring the data that says you know I'm relying on the fact that this you know I'm trying to scale out my workload I'm going to rely on the fact that that other machine has exactly the same data I have and we're going to make sure we send those messages but there's fractions time when it's not the same and this is why you have the cap theorem right consistency availability and partitioning are not all possible to satisfy so so don't try if to the extent you can use asynchronous messages so you send your data without assuming that you're going to get that you're synchronized on the result or that the other guy has the same data that you do and then if and then that the receiver should probably check and say wait a minute you made an assumption about my data that's not currently true so let me notify you and they have a way of getting back in sync this is not a universal solution uh but it can simplify a lot of problems where it's not so critical like you may not want this for certain banking or or you know flight navigation and such but if you're dealing with a chat system on a massive social network and you've got one you know one time in a million something is not quite right it's a lot better than trying to pay the cost of having everything synchronized all the time but uh what you but when you have a distributed system there's lots and lots of messages going back and forth and your behavior can be a nice proxy for the behavior in a system like like that is are my messages message cues too long are they building up are the receivers not actually dealing with the messages so the cues are getting longer and longer and so if you can look at your message cues you can get a pretty good sense of uh am I sending too many messages am I sending messages that are taking too long to handle and so forth another key concept is the idea of dynamic scalability it's pretty common these days for people to write systems that say oh we can have we can scale this out horizontally so we can do 10 hosts now oh what happens when 10 hosts isn't enough all right well let's bring it let's figure out when we're going to bring it down reconfigure Etc but it's that's you don't want to bring things down right that's the airling world so instead you need to be able to expand your capacity dynamically you also need you may want to contract Your Capacity if you're using ec2 instances or something then you don't want to be paying as much money it's like yeah we're idle for you know most of the evening because we're only running in North America all right let's use fewer instances during that period but both of these problems are a little bit tricky because uh when you have an basically because of um hot caches if your cash is cold which is always going to be in a new system then it the new host may not be able to handle the workload that would be shoved onto it when it takes over say 10% of your entire workload similarly if you have a host that goes down and all the workload migrates back to the smaller set of machines the cash for those uh Parts is not necessary necessarily hot and so you have to be careful on in both directions but luckily there are tools to help I mentioned earlier there's this distributed database called reac that comes from the company baso it's open source and uh it's actually a very nice tool because not only does it do all this stuff nicely but it's actually written as an extensible framework so you can uh use it for doing Dynamic scalability of things that's not about databases for instance if you want to have a chat system and you say I want to be able to add new nodes that have new rooms and then it'll just find the new hosts and there's a way that it trickles the work over and so forth uh use processes freely use uh if you think you want a new process or something go ahead and give it a try don't feel like oh I can't have too many of these or it's going to cause a problem for the system that's almost never the case if you start running into a billion processes you're probably going too far you know but but a million processes is absolutely fine um anytime there's you've got concurrent activity the concept that well this could be happening while something else is happening I'm dealing with two HTTP requests at once use to two processes for them try to have processes that have only perform only a single role so if you've got something that's uh doing that needs you've got some of your functionality that needs to be a client to some service out there and another one that needs to be a service to various clients out there probably don't have the same process for that split them into two again that relates in part to the let crash philosophy so if one thing fails you don't bring too much down uh and just as a matter of coding keep that main Loop of the process in one module don't have it jump all over the place doing receives and you know block blocking weights in one bit of code and over here it's jumping in a function where it's doing blocking bits blocking weights in another bit of code it's just confusing and again the OTP model clarifies that the way you would do most uh standard servers by the way in early uh using gen server the OTP standard framework for this is just as call backs it'll say it'll you have a call back for when you get a message when you get an asynchronous message when you've got a call that needs a response and so forth you don't actually write the loop yourself at all um monitoring is key in all this we're talking about relatively complex systems earling does not really um make a an ideal solution when you're talking about like small scripts that you might write for various reasons not the least of which is it's basically in some sense an operating system all of its own I was very mythed to real Iz a few years ago that you cannot write code that catches signals under lineing so I want to catch control C you can't do it which is kind of shocking you of course can do it by extending the system but it's not part of the language or the evm and so but it is very appropriate to larger servers and uh larger systems that cross multiple machines and so forth and so the more complex the system the more transparency is going to be necessary uh so you want to be able to find out what's going on like the the previous talk was giving great examples of what you really want to be seeing you want to be seeing these events you want to see them as graphs you want to get triggers when certain things happen you want to be able to analyze your historical Trends am I getting more messages this time or less you want to be able to see things at the not just the uh application Level but the level of the VM itself and what's going on in the host and what's going on with the network as a whole and because in this complex system there will be failures of various sorts and the better you can understand what the system looks like in your steady state as well as in exceptional circumstances the better you're going to be able to get a handle not just on what's going on but how to address Your Design so that you don't keep on having having that in the future and again there's plenty of tools out there that uh that can provide really good monitoring and I'll provide a little plug here that earling Solutions sells a tool called wombat that is specific specifically for this it provides great transparency into what's going on with an llang but it hooks into the non-l parts of your system as well as um other tools out there that are have their own own ways of gathering data uh so just I'm going to wrap up here so uh there's a earling factory every year in San Francisco they generally happen in March and so the most recent one in 2014 had all these various participants uh there's plenty of ways to find out more information there's uh here's a few books that I we can recommend the programming by Joe Armstrong who created the language uh earling programming one of the authors Franchesco cherini is my boss and the founder of earling solutions uh and earling and OTP in action kind of focuses on the OTP side of side of things but if you're looking for more there's a variety of others uh and so if you want to do something interesting you can try it out go to.org and download it and when you get a when you finished installing it type Earl and you're in your prompt and you can type hello Plus+ world and you get a result if you want to learn it then you can among other things go to learn.com thanks very much are there questions yes um could you expand more about garbage collection especially um lines were going up with you were talking about and I missed it uh sure what I was saying was that the that's right we had that was the interesting um lighting Fiasco of 2014 um the the idea in garbage collection and earling is that each process and again you have millions of them uh is garbage collected at separately and so what that means is that normally a single process because of the idea of isolation and so forth probably doesn't have a lot of data so it can be garbage collected very quickly and so you never see a stop the world garbage collection event what you see is if the only real problem you'll see if your code is not written to appropriately deal with uh the garbage collection behavior of erlang is that your memory would start building up because you've still got a handle on memory that is not actually considered garbage by the VM and the nice thing about that is you can see that problem happening before it actually becomes a real problem in other words if you watch your memory go up you can say uhoh if we reach a gigabyte there's something wrong but you can see that when you're at a quarter gigabyte as opposed to say a stop the world event in a typical jvm experience where the uhoh moment is when it's already too late because it's suddenly been not reacting for a few seconds and your you know plane is starting to fall does that answer your question yeah one more thing you said there's a way to know ahead of time with how is that done um you don't get the question was can you find out whether uh garbage collection is collection is going to happen ahead of time there's no notification beforehand but there's quite a few hooks into it so you can have Express various controls of the uh garbage collection parameter themselves but yeah you don't get an opportunity to intercede uh before it happens yeah in the distributed case of promote processes how much control do I have over where everything is how much keep track that the question is whether uh when you're doing distributed uh management how much control you have over it you have up to complete control so you can express that you want a particular as an example there's a an RPC mechanism where you can say I want to uh have this node run this function or you can say I want to send a message to this node or you can just say I want to uh one one process can say I'm going to register myself as the global you know print server was my example before and then somebody else can say okay I'm going to send a message to the print server um I mentioned parallel Maps so you can if you wanted to have if you had 50 machines and you wanted to run the same function on all of them by default you can just say pmap and then give the function it'll be sent to all of them and then the results will come back in a list so you can have as much or as little explicit control as you want yeah on the back you said there was some mechis for handing some sort of State what is that um the question is about how you do mutable state in llang so there's something called etss the llang term system and it's a very high performance key value storage where the values are tupal and so it's got uh along with the IDE of saying I want to store this value with this key and then being able to look it up very quickly you can also do pattern matching and some other higher level constructs and that term storage is store is available potentially to all processes when you create it you might create it so that only your process can see it so in other words you have some visibility control on it but one of the ways you can use it is to say I'm going to stick this data in here so that other processes can see it now of course at that point you're talk you're not going to have any um crashes uh excuse me you're not going to have uh memory corruption as a result of multiple access to it because the VM controls that so it's all transactional but you do need to be aware that if you might be looking at a value and right after you look at it someone can change it yes is there any AA Airline interoperability is there AA Airline interoperability um so like I say I have had very little experience with scy itself much less AA I literally have never used AA the closest I've come has read tried to about it a little bit it's actually pretty impressive how much of the uh how much they've achieved in particularly in a language that is not designed for it from scratch I would be surprised if there weren't some uh somebody had done it I haven't looked into it obviously like I say you can use this J interface to talk to the Java version of things but if you really want the um the case classes sent by Scala itself I don't know how you would do that sorry yes do you have an opinion on Elixir do I have an opinion on Elixir um so Mark actually just talked a class last week in Elixir and so I've got a little bit of exposure to it I haven't used it much um I'm glad that it exists because the main obstacle to earling it seems for a lot of people has been the syntax and elixir has this very nice Ruby like Syntax for people who don't know uh Elixir is a relatively new language developed really over the last year or two that runs on the earling VM and it uses a ruby be like syntax but it has access to all of the features of earling everything I just described is available from Elixir as well and so it's relatively immature I wouldn't want to do a production system on it yet but a lot of people are coming into the earling world because they come from Ruby and said oh let's try doing this Elixir thing so that's great plus it has a lot of features that airling as a language doesn't have it's got um I won't go into them but at one point I went through and tried to figure out what was different and came up with 30 different things most of which were pretty nice and I'd think oh gosh I wish I had that in Earl like for instance using um so earling by default uses lists to store strings so each character is its own integer and that's not very fast there's other ways to do it but by Def but you have to know about them to use the binary storage instead of the standard list of in in storage um elixir does it by default this as an example so I'm I'm happy for it and and that's I'm glad you brought it up because I can say if people were interested in this and just felt like the they first of all they want to be on The Cutting Edge something really new and they don't like the earling syntax check out Elixir instead all right thanks very much