SF Scala: Alexy Khrabrov Interviews Marius Eriksen

SF Scala: Alexy Khrabrov Interviews Marius Eriksen

Recording: SF Scala: Alexy Khrabrov Interviews Marius Eriksen

Technology Series and this is our engineering blog where we meet interesting people who Define the what's going on in the back end who do interesting Technologies which we also use and Technology of interest to to new kind of startups which are growing in in s and elsewhere and today our guest is Marius Ericson uh Marius is a lead engineer at Twitter he is the creator of uh finagle which is the networking layer Twitter uses for all kinds of services he a Scola uh expert he's the author of effective Scola uh guidelines which are used by uh beginning and and experienced scholar programmers everywhere for you know of their organizations uh he's currently the organiz uh one the corechair of uh commercial user of function programming conference which will be in Boston uh September 22nd to 24th uh so and his GitHub is uh reposit is full of of yummy repositories so basically he is allaround uh functional programming guy uh and he is originally from Norway uh and we're basically going to talk about the Technologies uh he develops and and Twitter uses and um uh some lessons from it uh for uh all kinds of companies which want to scale and basically reach this uh amazing level of scalability what Twitter is right now so uh let me just ask you first some a little bit about your background because we know your engineering side but uh so you're originally from uh from Norway that's right yeah what does like what is it about uh Scandinavia which makes so many good programmers and actually programmers have a functional twist to come out of there like a lot of AKA developers are from Sweden when a lot of functional programmers are in Finland uh and generally there are schools of kind of rigorous thoughts and academic uh departments which do this kind of work so is there anything about geography which makes it think the long dark and cold Winters you know but why why don't they spend them WR in JavaScript for inst uh well I mean I I I I I don't know the precise answer to your your question I'm sure uh you know the way these things often develop is that sometimes uh there's an initial group of people that sort of take hold and set u maybe an academic direction for a particular University or group of universities and so on um from which you know a sort of style like this can can spring um in terms of Scandinavia I think and I was actually never part of this particular sort of demographic or scene but one thing that uh was uh very popular in Scandinavia and still is I think is the sort of demo scene where you um where where people have these competitions to say well use the least amount of resources to do the most sort of impressive Graphics demos say um or uh you know try to subvert you know various games and crack them and things like that uh there's a lot of um there's a lot of uh interest like that I think in Scandinavia generally um and uh I think I'm I'm not sure how that relates exactly to functional programming uh or to sort of more rigorous computer science U but I think that's that that's actually better sort of characterization of the style of of work that goes on there um I think in terms of functional program versus not uh Europe actually has sort of generally had a more um rigorous and more math-based uh or math biased way of thinking about uh computational Sciences okay um in fact uh if you if you ever peruse uh Edward dexra archives um he has a whole set of essays about this and why you know two different schools of of computer science are evolved in the US versus Europe uh and so I think that part of it is maybe more Europe versus America so this General high level of math education in in Europe kind of academic mindset it it may not even have to do with math education but more what the um you know so I think one of the things that Dexter points out is that uh you know in when computers first came out this is sort of like you know right after World War II uh Europe was impoverished was getting rebuilt and so on nobody had any computers and so they had to they had to sort of theorize about them okay uh whereas uh in America there was much more of an engineering tradition and discipline around these things computers were used for you know manufacturing and Automation and you know obviously things like rocketry um and especially around Silicon Valley where there was a lot of Defense Contracting work um uh and so one one of D essay sort of goes on about how uh you know maybe the reason why there's this sort of split is because uh in the US we actually had computers that that we could work with them play with them and there were sort of tangible things that we could experiment with uh whereas in Europe you were left to sort of the theorize about them and you have to sort of formulate models and and the interesting work uh in computer science uh becomes you know thinking about things like you know complex algorithms and how logic relates to Computing and things like that and so that's one possible explanation I'm sure uh I'm I'm sure that's not the not the most accurate one but it's it's it's a possible one and I mean I can can associate with this because originally from uh Russia which was USSR and actually there was a similar uh question of lack of computers and a lot of mathematicians so basically people are theorizing so a lot of I think those was actually early crossover from I mean computer people had to come from somewhere so they came from Linguistics philosophy physics and math and and U math people theorize and K mov and all the kind of computational complexity uh and uh I think some data structures like AVL trees actually I thing we developed uh in the Soviet Union so uh they thought about algorithms as a math uh math problems um so okay so uh and then basically uh uh you can you outline sort of your your career uh in in in computers so I know you were at Google and you at Twitter maybe you sort of can uh show like what was The Logical Evolution sure so uh in I've always had a sort of deep fascination with I guess how sort of anything works but in particular computers um and that sort of very very early join me to work on you know things like operating systems because at least in software it's you know about the lowest level you can go uh and uh it gives you sort of perspective on understanding the whole stock of software always in terms of closes to Hardware but there is a lot of questions and Architectural issues um and so um you know I I I uh I worked a lot on an offering Sy called open BSD um for for a while and I was doing a lot of work in F systems and memory management and so on um and and that was a lot of fun um now my own progression sort of follows that of of academic progression in the sense that what we call operating systems today in Academia is really distributed system if you go to conference like sosp which is nominally about operating systems it's really mostly distributed systems um and so I kind of follow the same trajectory and have ended up sort of working mostly on on distributive systems um and uh for the reason that uh you know sort of operating systems uh have kind of been in in some way sort of fully explored uh that's maybe controversial but um uh and and there's a lot of interesting work in a similar vein and distributed systems right that are Happ this is actually interesting view I I didn't think about this but in fact this is true right so you can actually see finagle as an operating system or kind of a piece of an operating system and now the Twitter is basically running on an operating system both both pag and also a lot of the other things that go around is things like zookeeper uh things like we use system called mesos to do distributed scheduling all these things are a sort of connective tissue of of a modern uh distributed distributed system and the and what you might consider the sort of basic services that are required uh to construct meaningful distributed systems uh and it sort of fills the role of an operating system uh in the distributed environment now of course uh one thing that that I find very interesting and and and uh I think is a a sort of problem in domain that we don't have good answers to yet is how do you sort of meaningfully uh write programs that are targeted to these sort of large distributed systems um systems like fago systems like mesos um and services like zookeeper are sort of a partial answer to that but they sort of assume this tradition model of um you know programming where you you know you write you you you write an application that issues system calls do things like iio you have uh you can map memory and do things with memory and so on and so forth um but fundamental is sort of the same thing except that in a Distributive setting where you have multiple of these things that talk to each other hopefully VI some discipline approach like finagle um but that's not to say that's the best way to program a a a data center right um and so you know other other sort of answers to the same questions are things like AA right and the sort of actor model that uh largely sort of fell out of the work and llang uh that's that's also very interesting uh there are you know there are trade-offs that you make when when uh programming system is that way um but I don't think that we have uh you know answers that are that are good enough yet right it's still even if you use AKA even if you use finagle even if you use whatever else um it's still diffic more difficult to program these systems than it should be and it's difficult to sort of reason about the system as a whole uh when you're working on an individual component right um and that that has to do with everything from just the sort of mechanics of deploying these applications you know if you think about how you develop an application locally you write some code you maybe compile it maybe you don't maybe you just interpret it and you just sort of run it you can't really do that with a distributed system right and so everything from sort of how you work it um to uh how you relate uh you know sort of actions in in the various parts of the system how you what what you consider the sort of module boundaries to be how you distribute work and so on uh are are questions that we have a lot of answers to but I don't necessarily think those answers are are uh good enough yet yeah it's actually more complex now right because like if if if we look at this through this Paradigm that we have an operating system in in the cloud now we still have an OS in our desktop and and basically all these issues are continuously commingled right because we are using memory here so you have to now know both Stacks very well to understand what happens uh but I us like this this a lot I think this it gives kind of a general uh General View and I've seen some some things like an operating system from coming from a camel Community where they look talk about Services uh and just talk this the call an operating system uh so maybe for those who are not familiar with mesos and finagle maybe you can sort of give a brief description of what they are so let's start with mesos so um mesos is um a a system that allows you to sort of slice up resources in a cluster so if you have a if you have a cluster like a data center with say a thousand machines for talking purposes um what mesos allows you to do is sort of treat this cluster as one pool of resources and you can divy it up any any way you like um and Mesa itself is sort of very very kernel with this it's roughly like just a resource manager so what you can build on top of mesos though are things that make use of these things and so for example uh you can imagine writing um or modifying the schedu in hop to instead of say install hop on these on these 100 machines you instead ask mesos for resources like I want I want a slice of resource that has you know 2 gbes of Ram uh 5 gbt of dis and uh at least two CPUs um and then mes will give that to you and sort of give you access to uh to the machine if you will um and so Mesa itself is sort of very lowlevel and sort of it's the resource manager now now uh at Twitter we have uh a schedule sophisticated schedule that's built on top of that that allows you to say uh I actually want to run uh you know end replicas of this uh of this executable um and never have more than you know K replicas down at any given time and so on and it'll sort of dynamically schedule and reschedule as it goes on and so what it does is it sort of subsumes the um part of a traditional operating system that schedule schedules processes right you can it's like keep alive for a process like it it does that it does that as well uh but but more fundamentally it manages the resources you have and so instead of deploying an application to say 100 static machines you instead uh deploy an application in sort of 100 slices um and those get scheduled dynamically throughout the cluster and now you're giving msos the sort of freedom to schedule that where you can most efficiently use your resources and so for example if you have an application that uses a lot of uh memory but not a lot of CPU you can schedule that together with one that uses a lot of CPU but not a lot of memory and so on so it allows you to sort of more efficiently use the resources that your Avail but then also um but then also sort of overcomes a lot of the operational challenges with distributed systems or deploying distributive systems for example uh let's say you have static configurations where I deploy um you know 100 100 replicas of of my system uh to 100 different machines well now if five of those machines are down for maintenance you have you have to go out and five maybe you have to keep some um some sort of on standby or whatever whereas with meso is because you have the sort of funable resource pool um the the all the sort of inhabitants of of those resources everybody that uses those resources uh can sort of take that tax together and you don't have to individually think about it so it removes a lot of sort of operator responsibilities from uh the individual sort of deploying things on the on the cluster uh and so that's that's something that you know both allows us to use our resources much more effectively uh but then sort of makes it fundamentally simpler to deploy things to the distributor system so you always talk in terms of CPUs me memory and basically so you have this actually OS level abstractions for resources so you're not talking about in terms of Nod like you don't you don't say you know I want a VM essentially right you actually say you know I want this compute power but you make an assumption that this compute power resides next to this for instance right that being said I mean there's the what we what we do is we we sort of delegate uh these resources along these Dimensions right um there's no reason you couldn't use mesos to ual virtual machines as well of course the virtual machines would uh would need to reserve uh resources along the same Dimension so it's really just this distinction is between a process and a virtual machine it's not really a meaningful distinction other than uh you know the kinds of trade-offs you're making operationally okay um and so so yeah so so so that's mesas um and and and that certainly makes a lot of the operational aspects of distributive systems a whole lot easier to deal with um what finagle is is is the following so um one way to think about finagle this is sort of the connective tissue of of distributed uh of distributed systems and so uh when you write when you write a distributed system you usually have a number of uh processes that need to communicate to each other and finagle uh does two things one it makes that easy to do in a robust manner um and uh has sort of a principled way of uh reasoning about and abstracting what those modules boundaries are so uh I can say I have a a sort of a widget in in finagle that represents some logical remote service right and finagle takes care of the details of like how you do lad balancing and Resource Management uh and what the application topology uh looks like and so on um and then secondly um it sort of encourages or presents uh a concurrency model uh that we believe is very easy to work with uh and also results in sort of robust and efficient software um and that's important in in uh in in distributed systems because uh they tend to be highly concurrent so you if you look at a a simple web server uh they can handle thousands if not tens of thousands of requests at the same time um and often uh it's even the case that uh th those requests are related in some way like maybe they you know share some caches uh they definitely they definitely need to be Multiplex s on to the same Connections in terms of dispatching requests and receiving responses and so forth um and uh what what we have is we have a a sort of concurrency model that's based on uh Concepts from data flow programming that allows you to instead of sort of specifying uh in order to compute uh let's say a search and so uh to give a little bit of background you know in modern in in sort of modern search engines you typically divide your data up into several chunks you have to quy all of them and sort of gather the results and then do some processing on those results and pered um and so uh the way you would do that in in a traditional sort of concurrency model like let's say uh using Java threads is that uh you might allocate a thread for each of those requests uh issue that request use some sort of shared INF some sort of um share data structure to communicate the results of this request back uh and then you also compute the Aggregate and number response to that query uh the kind of model that finagle encourages uh is a little bit different uh instead of sort of specifying the execution mechanics of doing that you're instead just sort of specifying the relationships between the data and so you say well in order to satisfy a search query I need uh rather the the the result that is the search uh that is a search result is a combination of uh the individual results to the individual shards um and you express those relationships through what we call combinators and you let the underlying system finagle actually deal with how those things are scheduled how the resources are shared and so on and uh when you assume that kind of programming Model A lot of the kind of synchronization concerns are sort of subsumed by the data flow itself and so uh in the sort of traditional example you have to worry about for example synchronizing around the shared data structure to uh well first of all to Signal when the thing is done uh to actually uh you know Reserve slots for each individual result and so on whereas uh if you're just sort of uh phrasing this problem as uh relationships between individual components uh that produce data um a lot of the cons of concerns like synchronizations are sort of subsuming by that so is this uh what is done in in future combinators in right actually it's it's interesting I I didn't think about this is actually a data flow Paradigm I thought about it as basically like pretty uh simple pipelining and kind of glue right so that that was the view which which you know I got when when when looking at it and we just did our first uh service in finagle uh we did the transcoding service for ver because we use a lot of educational videos and and we accept videos from uh from uh uh customers in in various formats and then we transform them to the most optimal format for us and actually we implemented it in finagle and it's beautiful it's it's it's very expressive but for us it was basically a simple service which interacts with Amazon elastic transcoder and uh to me it was very interesting to see how um how it works um but you know it's actually I think it's a small scale it's it's basically Sone Service as we get more of them we'll probably uh uh understand how data flow works better but that's actually very interesting so so one way to one way to view it is that through these combin areas you can actually sort of construct arbitrary dependency graphs right um and so you know in the example that I just described it's sort of a graph that starts with um you know the simple query and then you know fans out to however many nodes they you know that of shards you have to to gather results from and then fans in again right and so any sort of dag you can really shape um with with these combinators but you still have to sort of I mean there is some some some logic which went to the design of this topology because you know where each chart is right uh well well and ideally so so that's another thing that fago gives you is that you know fago will for example in this case gives give you a service per Shard and you might keep that in a you know sequence or whatever and map over it um but the details of of dispatching those requests and so on are handled by fego and this actually gives rise to another interesting uh sort of benefit which is um so everything that I've sort of described with respect to data flow the sort of data flow style of programming um means that the way you sort of Express the result of uh a query to a server for example is very much in a sort of declarative fashion right you say uh well in order to do search I first need to you know query all the chards and then combine the results in this way and so on and so in some way it's an inversion of control right you're not saying U make five queries uh take the results of those combine them together rather than reply uh you're sort of expressing the data dependencies and uh an additional advantage of that now is that finagle itself can take advantage of this um and uh Implement an execution strategy that is sort of optimal for that particular problem right so in this particular case you might say well maybe we want to optimize for minimizing Contex switches and because finagle is responsible for both dispat requests and multiplexing all the work onto several you know sort of IO uh it can for example guarantee uh that all the work that you receive in a given thread will only be will be handled exclusively by that thread right so you can do you can do things like this which is sort of interesting too uh it's sort of analogous to having a a runtime with a jet right and so uh we haven't we haven't done much sophisticated work around this yet but you could imagine that finagle could in a way observe and profile your application and then adapt its execution strategy accordingly interesting which is another uh interesting thing that uh that which again future work that you might be able to do with systems like that it's actually interesting because we're currently looking at elastic search and I think uh from what you describe it would be fairly easy to do a finagle replacement of elastic search because what it interrupts a Java and very wellknown Java application weene right basically so each elastic search node has is is a Shard basically which is with its index and what elastic search itself is is basically the rest API which which this this and it has obviously its own custom logic but I think that's probably uh like if finagle has the same functionality that maybe can be an interesting heon project to take you know a bunch of Lucin and and harness them all with with finagle uh but you know it brings kind of this interesting uh uh uh question about program languages so Scala as we all know is basically the flagship language at Twitter I think became known a lot because of Twitter and uh what is it about Scala which made it the choice at Twitter and and made things like finagle possible what was the route by which it took over Twitter and sort of grew with this yeah I mean so I actually wasn't around when this happened uh at Twitter but uh in initially um Scola was used first at Twitter for a small project uh that was sort of orthogonal to or could be separated out from the rest of the the stack which was written in Ruby at the time um and there were only a handful of Engineers and they sort of needed a a a replacement for a system they already had uh which was getting pretty slow and turns out that uh you know scol Scola is a very expressive language it uh you know took somebody just a couple of days to sort of come up with a new a new system that did the same thing uh and because it ran on the jvm um it performed a whole lot better um just because of the quality of the runtime basically in this case and uh that's how it it sort of started and then people saw the value of that and so we continued using it um and there was no I don't think there was any sort of conscious uh effort at any given point it was sort of more serendipitous you know somebody had wanted to wanted to try for this particular project and it turned out to work really well and then people contined using it but something happened right because I mean obviously if they would have picked basically you know it's very easy to beat a long running process in Ruby with the jvm right because garbage collector uh in jvm is so well tested so basically that could have been addressed with Java uh so but but U what happened later it basically gave rise to all these kind of libraries such as finagle and obviously there's a lot of features in Scola which make finagle possible so I'm just wondering uh but on the other hand there are things like airl and and uh you written quite a a bunch of a camel for instance so which um in many case actually fell back onto as well because it's kind of a similar language right so what for if we consider uh features in scull there is a lot of features right and and and some people consider complex still it was used to build uh uh the a lot of systems at Twitter and it clearly is a demonstrative case that you can have a gigantic system written Scala and it works so I'm just wondering is this uh is there any intrinsic properties of Scala Scala Community or Technologies which basically make it sort of suitable uh so I mean it's definitely good Legacy technology Twitter now right so so but I'm just wondering like what is the is is it now a compound in effect that choice of Scala and as as a language which has this obstructions for concurrency and function programming features is it now sort of facilitating faster iterations at Twitter or like uh Twitter folks using other things like I'm just wondering what I mean so that's always hard to tell because you're not doing an actual control experiment uh we uh I don't know if anybody really done that to sort of set aside part of the organization like you're going to be the control and whatever else um so so it's kind of hard to say I think in a in a large system like Twitter um the the kinds of things that matter for iteration speed actually usually aren't the language right uh it's basically everything else that goes around it everything from like you know the Bild systems to how it is to deploy things into the into the distributed environment all the kinds of things that I talked about before to sort of just the sort of friction of of dealing with large scale distributed systems um is is sort of what I think slows iteration more than the choice of any particular language um now that being said uh I'm personally a big fan of the sort of functional approach to uh to to to languages um mostly I think because it it it sort of encourages you to write code in a way that sort of emphasizes things like value Transformations over M uh over um mutable data um and uh sort of encourage you to think think in a more rigorous fashion about sort of how to exploit an variance how to express your problems in in in sort of different ways that at least personally I feel uh leads to code that's sort of simpler and easier to reason about um of course this is this is sort of after you uh learn it right quite overwhelming initially um to to to sort of phrase problems in terms of recursion and to think of higher functions and compositional different things but once you internalize these Concepts um I believe that it leads to simpler systems that are easier to understand um and uh that's partly I'm sure an aesthetic preference right uh I don't think there has been a number of studies that have try to sort of disentangle these different things and try to show that you know language X or this style of programming X is is more efficient or effective or simpler than why and nobody's really had any sort of compelling results there right because I mean it's a social process right I I think it's in related to the set of people that's part of it too and and there's also all sorts of different trade-offs you make right and so um there are sort of short-term versus long-term costs right and so uh certainly if you're hiring um a sort of standard profile engineer right um take any engineer from a big company even like like Google um they will be more immediately productive in a more traditional imperative language like Java uh if they don't have any sort of previous exposure to functional programming um and so from a from an organizational point of view uh you know once you once you get to be a large company you don't you can't only think about individual productivity anymore you have to think about the productivity of the whole organization and so you end up making different sort of trade-offs for that right uh and this is one of the reasons for example why why why Google is extremely strict in the kinds of languages you can use in production it's basically Java or C++ uh uh python is mostly for S operational things I think uh and U and that's it and probably for Google that is a is a good decision right um and so it's just very difficult to make non make make really sort of good non- athetic arguments uh about functional programing versus non-functional programming uh it's my own experience that after you've learned the basic concepts after you've become comfortable and and working in that style uh that I think it's much easier to work work with I think it leads to more correct code um and I think it's also in some ways more fun um and but but I can't I can't point to numbers that back that argument that right right I don't think it's it's it's you know it's numeric but uh it's interesting actually you haven't mentioned uh static typing but I think you know it really does help in in production and interation speed if you catch a lot of Errors no sure but I mean that's that's not the exclusive domain of functional languages that's right that's right so uh so how is uh current scale how does Twitter sort of deal with uh this uh transition of new people to functional approach and uh it's it's it's very interesting right that you have this scale of a company so what what's your opinion like internally where T is at and how does it work at scale people scale of Twitter employees at this point right and so um so there there a number of different things so so one is that I think uh even by simply using um like a language like like scholar functional language kind of sending a signal to um job Seekers I guess that uh we're doing things slightly differently um and also the kind of things that are the kinds of people that are sort of attracted to that tend also to be fairly high quality engineers and so there's a there's a kind of selection bias going on for one which is which is beneficial I think for Us Again difficult to measure completely uh qualitative assessment uh in terms of bringing people up to speed uh we've done a couple of things so you mentioned before that Scola is a large language and indeed that's true it's can be quite overwhelming so one thing we do is that we're very careful about the kinds of features you use with Scola which is the same which is the the same thing that people end up doing with things like C++ C++ is also even even larger language than Scola is um but but anyway we um we uh and this is part of why I wrote this this document called effective Scala uh we sort of are careful to define the sort of subset of the language that we're comfortable with the kinds of techniques that uh you know we consider kosher versus not um so as to make our code both uniform as well as sort of simple in construction right and so basically what we're saying there is like well you know you could you you probably get you know three or 4% you know better uh better programs if you were to use uh things like uh you know encapsulating effects um but uh the the sort of complexity to a programmer would would rise dramatically um and so we we sort of choose to or we we isue using those kinds of techniques um and so you know we sort of down and say well what are the kinds of things that uh we believe gives you a lot of power power without you know too much cost and they end up being things like well obviously you use static typing um uh very heavily and and and use the type system to your advantage but then obviously things like you know first class functions um are high up there uh we choose not to use uh Mixon very heavily for example which is um you know one of the things that are at least in in the early phases was s touted by Scala uh because uh it makes at least the kind of code that we tend to write more complex and sort of uh prefer sort of traditional composition um or we're using mexin as as a simple example um and so so so that's one thing we do so we we we try to make our our code look or be uniform and and also simple um and then we also have a bunch of internal um sort of developer boot camps I guess about these different things we have people that U that teach uh scholar from the gr up basically how long does it take so let's say you know you absorb a new Java programmer who's a very good Java programmer who was doing jaob for 10 years but you know they didn't do call it you you can you can have a person like that write scholar code in in in in a very short amount of time but they'll probably be writing Java in scholar Define short amount of time uh like a week's time okay that's that's a short and however uh it takes longer to um to sort of indoctrinate them to the the the ways in which you're supposed to work uh are most effective effectively working uh in in in these kinds of languages um and so so what we do there is that well we do some of that obviously in the um in the education programs that we have uh but another very important component is doing a lot of code reviews right and so uh you're not going to you're not going to be uh say a SC A J Java developer uh doing Scola and then like writing your own product from scratch you're going to be working with other people who are experienced um and they'll do code reviews you'll be doing their code and so on um and that way you sort of be get exposed to the kinds of techniques and methods um and sort of idiomatic use of the language uh and that sort of seeps in right and then eventually you become a proficient scholar programmer I was going to ask basically how effective scholar is enforced is it kind of now a Bible so you know in code reviews you can say you're violating chapter you know 14.5 uh it's not not not not quite that strict um but uh it it definitely does get used for justifying certain things yeah mhm um so it can be can be in in other words can it be using as operation manual for a new company let's say there is a a Java company and now a lot of them trying to Pivot to schola can they take effective schola and make their sort of internal code review with a like a core of uh you know scholar uh uh gurus let's say right so can they basically say you know we're going to teach people based on effective scholar and then we're going to do code reviews and we you know we'll refer to basically to different things which people may violate or not follow and can this be basic adoption guide uh yeah I mean I I I would do that sure yeah I heard because actually I heard from some companies in the space doing this because recently there's been Trend actually that a lot of uh sort of Hardcore Java companies are are pivoting to to skull sort of in another question I had about uh scull and Java I think stems from this famous ymer controversy uh because I think for many companies Twitter is actually in this similar position so let's say you have a high performance process with you very tight inner loop right so yammer's choice you know if they profile then inner loop produced by Scala was inefficient so they answer was you know gosh let's you know we really need some you know speed here so they switch to Java and I'm sure that the kind of situations like this might happen uh at Twitter procing you know huge volumes of messages so I mean first off I'm sure the Yammer situation was more nuanced than that because on the surface that sounds like an overreaction um I guess a few things so so I I'm I'm a pretty big adherent to uh canut Maxim which is you know premature optimization is the root of all evil and so forth um so however what I like to do instead is sort of focus on having good profiling tools and so uh we actually built a suite of tools that allow you to profile um everything well the sort of traditional Dimensions you care about which is you know CPU usage lock contention and um memory allocation and we have these profilers and they're embedded in all of our processes and so you can be running a service and you can run a curl command and get a profile so you can say I want a 10-second uh CPU profile sampled at you know 100 HZ okay and uh the profiles you get are compatible with Google per tools I don't know if you used these uh what they do is that they use graph B to produce this very pretty stack call graphs for you and uh usually if you have a process that uh you suspect is slow uh the uh what what you do to profile it is you run one of these profiles um you run it through Google perf tools that gives you like an SVG file with a graph in it you find the biggest box and your answer is right there uh this technique uh basically Works 95% of the time um and so for things like this we we can expose these inner Loops very very quickly right okay uh so I think the first thing to do is well first of all don't don't prematurely optimized and then but but have good ways of of profiling U processes that are live in production meaning you know receiving meaningful load and so on now when that happens I mean you can write Scala in a sort of low-level way uh that uh where you of course have to to isue a lot of the interesting features U but you can for example write while Loops in Scala and use mutable variables and so on and so the kind of um the what what what we encourage is you know write it in the sort of simplest and cleanest most idiomatic way first uh then profile it uh and then if anything comes up in when when you profile it um you know replace parts of that with with sort of more performant lower level less idiomatic uh versions of the same thing and that that tends to work work very well how often does this actually happen like how often do you need to drop down to this you know kind of assembly level scalar yes so so actually in most of our in most of our systems now I speak mostly from the sort of runtime systems perspective the serving systems that we run not from like say analytics and so on uh where they do a lot more sort of things that are traditionally number CR M um the kinds of problems that arise most frequently actually relate to garbage collection okay right and so uh you know we we have uh we have a lot of these very high throughput systems that receive a God awful amount of of careers every second and uh most of which are extremely short-lived um but they're also very latency sensitive right and so we have we have systems that uh you know need to reply within a few milliseconds uh otherwise they have an outsize outsize effect on on the on the total latency of of the request that sort of is at the root um of um of of the query that you're satisfying and um one way to one way to think about that is is is the following so you know in isolation um if you have a a process that under goes a minor garbage collection let's say every 30 seconds right and so um with the the sizes of heaps that we use minor garbage collections are somewhere between 10 and 15 milliseconds uh that means that every half every half a minute every 13 uh every 30 seconds you're going to have say 10 milliseconds of pause right yes which means that um during those 10 milliseconds every query will take 10 Mill unacceptable delay for this slas also in in isolation that's fine right it doesn't matter if your your timeline Cory returns 10 milliseconds later however uh because most of our quers are very high fan out which is to say when you make a query to Twitter um that typically re results in uh us needing to query and and agate hundreds of different servers right um the probability that any one of those is undergoing garbage collection increases very rapidly right um and so if you're in garbage collection 1% of the time right uh then if you if you always issue 100 queries to 100 different machines then every single query is going to encounter a garbage collection right right and uh and so in that way uh the these large fan out systems amplify garbage collection other other also other kinds of variants um and uh so that's why garbage collection is extremely important to us right and that's kind of trickier because with garbage collection you have to be very you know you sort of have to first of all you have to know what affects garbage collection what doesn't uh things like object lifetime start to matter uh of course the the rate at which you which you allocate new objects and so on matter as well um and so we have a number of Tools around that too so we do things like allocation profiling where you can see you know where your we we sample allocations and you can see where your objects are allocated um but then we also have a number of well we have one tool in particular that allows us to effectively run garbage colion regression tests um and so what this tool does is uh over some sort of test load we measure um how how many alloc you know how much allocation you do per request say and then we also measure the histogram of lifetime of that so then we can say well you add you allocated the let's say 10 kilobytes of of memory to handle this reques out of those 10 kilobytes um you know uh 5 kilobytes were um were not needed anymore were were garbage by uh you know by uh five handling five more requests so you measure time and number of requests that you serve um and so so that gives you effectively a histogram of of um longevity of of allocation and uh that allows you not to do effective kind of regression test where you say well oh I might introduce something that doesn't introduce more allocation can in fact reduce allocation but maybe make some objects more long LIF and those things you can pick out you know quickly from this tool so so for sort of crucial critical libraries like finagle uh we run them through this kind of regression testing as well right as do you point back to actual source which like can you so does this profiling shows you where in your Cod no no no it doesn't do that it just it just gives you the that would be too expensive to do in in a meaningful way uh it just gives you that but if you use it with regression testing you can compare you know change by change right so you can say oh this change introduced this what does it take you then to link because we know that Scala does produce a lot of objects right so clearly it's it's a tricky it's a tricky choice and some things may produce much more intermediate objects than others so I think the beginning programmer or even you know uh uh an experienced programmer who doesn't normally care about this kind of stuff may not have any you know like intuition about this so I'm just curious you know let's say what does it take them from you like if you now see suddenly you changing this histogram you know so you know let's say you decreas the of object how do you go back to your Source how do you find you know like is it well I mean so so in this case if you run it in a sort of regression testing uh environment then you can usually point out a single change that cost the because you know what change in the source you use Jenkins or something and Jenkins will either do the binary search for you or whatever yes um and so if you use it in that mode then at least you get a sort of atomic commit that that gives you the the regression um so I remember from your uh talk at uh uh scal up early this year you were doing work on um on introducing queing features to protocol level and finel so you can drain servers is it related to this kind of problem that basically would you drain a server when basically it's garbage collection deteriorate because it's just you know was just living for a long time and uh is this related to yeah so actually one of the things I'm experimenting with now is uh related to that which is the following so if you so one of the things I'm I'm trying to figure out so so one one of the one of the sort of challenges with um uh that that sort of exists that the module boundaries between distributed systems and so if you have you know one node in your system that's a client to another node which provides a service to that um when so so if if you sort of bring back the analogy of well how how would this work in a sort of traditional operating system well that would be like a sub routine call right and so what you're doing now is effectively your program you're making a sub routine call into you know to a library I guess uh but what you're effectively doing is that you're giving your CPU time now to that library right okay um this is not happen in distributed systems right like the resources that I have uh I can't give to whoever is doing work on my behalf right and so uh one of the problems that come up and and and one of the things that make RPC systems challenging um is how you deal with um well things like uh priority inversion in particular so you might have some some system that uh has allocated a lot of resources to but it might depend on a system that has fewer resources uh so that you can't fully utilize the resources of the larger system for one um but also that system might now be capable of sending more requests to the the the downstream system than it it is capable of handling right this is where you need things like Hing disciplines and so queing discipline can say well you know make sure that I I Only Have You Know This Much outstanding work at any given time and start reject things you know only only for this this long time this many items and so on otherwise drop and reject work um but uh in a the way this has been traditionally done is that you sort of statically configure this you say right uh this client I'm going to send it most as many outstanding queries to or for a server you might say I'm going to handle at least this much work at a given time the essentially so so so this is the sort of uh in the sort of traditional Java role this is uh like how you for example choose to size your thread pools for request like things like that um and Fino allows us to adjust all those different things right you can say um you know B basic basically consider tune the curing discipline one thing that I want to do is is make that more dynamic in the sense that um the the downstream systems uh you you should need to configure in these different things and um you there should be a reliable way to effectively communicate back pressure throughout the system so that the slowest component effectively governs the rate of which you can submit work at the head right I see um we uh without doing any sort tuning in between one of the things that uh one of the things that um allows us to work on that is that we've introduced a new RPC protocol called MX it's also open source in pagle uh and what MX does is uh it has a a separate control plan that allows you to send control messages um but it also allows you to do things like you know rejecting requests uh before they dispatch on the server so that a client can safely retry them elsewhere and so on uh which you can't do in traditional traditional Thrift setting um but one of the things that that that it does is that it effectively um allows you to implement a kind of flow control just like TCP does um and so a downstream server can communicate to its Upstream server um that you know I'm only able to accept this Min request for example or here's you sort of here's your window of requests at this particular time and uh one way in which you can use that is you can inform your flow control by your garbage collection right and so you can say things like well uh let's consider the sort of bottleneck resource on my server to be the uh number of bytes available in my Young Generation okay right and that's my that's my primary resource when I run out of that I have to pause for 10 milliseconds yes right so that Bally you can make a strategy on receiving new requests depending on so then that effectively becomes your window that's effectively the amount of capacity available in your server at a given time and you can sort of relate that to uh flow control and use it to inform Upstream server um and then they can in turn use that for load balancing for example right um and so there's a lot of interesting opportunities here where uh you can effectively use this mechanism to Route Around garbage collection right because you always have you always have more than one replica in a distributed system uh otherwise you're in trouble and so uh having this kind of mechanism um allows to potentially Route Around garbage collections when they occur right um and uh that's one possible latency C mitigation strategy that's very interesting architecture and surely will will uh uh explore it uh so I have endless amount of other systems questions uh you know which I hope you know we can address in in our future episodes uh but for now I want to ask you about uh the CU conference right because you're the co-chair of this conference I think it's a kind of story conference uh so there is a general functional programming conference and C is is basically the kind of pragmatic uh part of it which which which uh is uh basically commercial users of functional programming and uh maybe you can tell us a little bit about uh this conference about you know the kind of uh companies and talks you uh you'd like to see there because you know you know in the organizing committee and uh I mean I think the agenda is is is is is already said but generally what would make in your view a good c conference what first of all like what is your interest in in organizing this what do you want to come out of it what kind of people you know would you like to meet there right so so um so uh cfp is part of uh icfp which is the sort of big ACM conference on functional programming and icfp is actually most famous for his programming competition so they have every year they have this competition where students from all over the place sort of tackle very difficult problems um in a limited amount of time and so on uh the the original idea behind that uh competition was that the sort of a to Dem demonstrate that functional programming Superior because uh you know the teams that win will be using functional programming languages I'm not sure they always do I think the last couple of times they've been like C++ anym but um but regardless so icfp is one of the probably the premier sort of conference on functional programming uh cfp is a workshop under the egis of of icfp and uh comprises two things so the one um there's a day of talks um and I'll I'll get into that and what follows is two days of workshops um and the workshops uh this year we have some really excellent workshops um it's everything from sort of we have two a two-day workshop on Hull like starting from from um not knowing anything of has to you know being able to write say a web server that does meaningful things um we have uh workshops on on erlang um and Scala and closure um and the interesting thing about these workshops is that they're often taught by you know the people that literally wrote the the various books on on these languages so they're very high quality uh affairs uh now uh the talks uh is what I'm personally most interested in and uh it's it's a day of talks it's a long day of talks and um there are effectively sort of three kinds of talks so one is um the kind of experience reports that uh where where people relate um experience with using functional languages or functional techniques in a commercial setting um and you know we've had talks we have a lot we've had a lot of interesting talks historically about this everything from uh you know people embedding scheme into like you know PlayStation gaming engines uh to people using uh has for doing uh or hll derived language for doing uh verifications for like you know government contracts you can imagine what that means uh for you know for for crypto systems um to um to things like prototyping one of my favorites actually was uh was somebody from from uh AT&T Labs or losing Labs now uh prototyping a uh sort of new Next Generation antenna system in hll uh and they did it in like you know a week and it was it just turned out to be beautiful fit for the kind of programming you do uh for for for data processing or signal processing and so that's the kind of experience reports um and uh I find those very useful because it it sort of uh demonstrates uh the applicability for one of functional languages uh but then it's sort of we hope to sort of encourage an honest assessment of of how the effort actually run and we've had we had some talks uh that where where they basically said well we try this for a year and then we canver the Java because of these things and they're they're very valuable things for the community to learn um and then uh there's a number of talks that are more technical in nature so that's sort of introducing you know technique um that are probably not of interest to the academic Community but are probably very interesting to um uh to commercial users that's sort of you know how do you in practice sort of structure large programs in scholar right like things like that thing would interesting especially you know like from like T because they obviously learn some some applicational functional programming at scale right um and uh and and and so so that's very useful as well and then thirdly um uh there we've also had a number of talks where you know people introduce like new new products and languages and things like that that are functional in nature but are uh perhaps geared towards a more commercial audience right um and uh it's one of my favorite to go to there's a lot of not only are the talks really really good um and they also tend to be very Lively you know people ask a lot of hard questions and uh you know I was actually introduced to cfp by giving a talk there uh and I remember giving this talk and uh uh making some claim and then Simon pton Jones immediately interrupts me and say well you know can you back this up with this and that right and so it's it's it's an interesting place because uh we tend to get these high-profile people from the sort of academic Community mesh with with uh actual practitioners using functional programming um and hopefully uh if if we if we arrange this in the right way uh they'll end up influencing each other right yes and and and so so so it's sort of an explicit goal of the conference to be a meeting point of of the two communities because they traditionally have been fairly far apart I see right um and uh so yes uh and if you're all interested in in functional programming uh especially if you're interested in in applying it in a in a commercial setting uh you should definitely go I don't there's know you don't need to have read like uh you know Benjamin P's book to attend this thing it's helps but it's not necessary uh it's it's meant to be uh it's it's it's meant to be sort of comprehensible to anybody that um is a programmer basically so uh you know like I think uh SCP and and C are fairly traditional conferences by now but recently we've seen some new entral such as strange Loop and this year actually strange Loops is a few days before the and and maybe I mean like I see the sort of mix of of academic and and Industry Community a distinguishing point in C uh do are you aware of any kind of other uh stylistic differences because you know like strange loop I think it's in similar in theenda in terms of closure hoscale Scala right and uh like what would be your take oner of uh well I mean cfp is a more uh more sort of focused conference right the sort of explicit goal is well as as the title say commercial use of functional programming which where I believe things like strange are more sort of General technology technology Trend conferences right okay um and so the it's it's much more focused right okay um that being said like I've never been to strange but it always looks excellent to me yes uh yeah so hopefully uh all of you guys who are woring this can make this uh uh this year and we thank marus for coming and uh hopefully we'll cover much more topics in our future episodes thank you all right good job