SBTB 2014, Marius Eriksen: Functional Systems
Recording: SBTB 2014, Marius Eriksen: Functional Systems
alright so as the lexi said i'm marcus ericsson and i work at twitter and I wanted to talk today about sort of how we employ some useful techniques from functional programming and what I'll call systems programming so these are the kinds of you know back-end systems that power large web sites like Twitter and and many like it so I think it's very important when you're listening to a talk that you're very very skeptical about what you know what the speaker says and because everybody comes from a you know a particular perspective and my particular perspective is that we're supporting over a thousand engineers working on a very large scale internet service and that scale manifests itself not just in terms of Sid amount of traffic it's also the amount of data and users and in our case tweets and so on and so forth that our systems need to handle right and it's important to note that I'm not trying to make sort of unbiased statements here what I'm about to talk about is something that is sort of very much relevant to us but I hope is also more generally useful so another thing to sort of keep in mind is what what is our sort of design value system I guess what are the things that we require in our systems what are the kinds of properties that we consider good and I like to boil the sound to you know five different things and so one just functionality well obviously if it's something doesn't work or isn't correct it's more or less useless for us scalability is very very important we can't build a system that we can't grow right because our business the number of users we have the amount of data that we're dealing with is continually growing and so scalability is very very important for our use kids it's also very important that our systems be operable and what I mean by that is when they break because they do break all the time it must be simple to to diagnose and fix the issues and so they must have a large degree of sort of operational flexibility it must be able to change them as we go along I must be able to figure out very quickly what you know what actually is wrong and in these large-scale systems it's often very difficult to do so these systems also have to be efficient and the reason that's important has partly to do with the fact that we want our systems to be fast but maybe more profoundly that they have to be cheap to operate its large businesses like like Twitter and many like it sort of operate on the on the fact that we are able to or rather exist only because we're able to do what we do very very cheaply we don't make a lot of money sort of / you know / action or operation that we do but as long as they're cheap enough we can do enough of them and still come out ahead and finally our systems must be flexible and this is important because it's often very very difficult to tell what the systems are as opposed to do in a year or two years or three years from now and when we're putting a lot of effort and investing a lot in the systems that we build it's important that they remain somewhat durable and in order to do so they have to be they have to be flexible they have to be changeable so that sort of this gives you any idea where I'm coming from this is nothing to do with functional programming so far but I think it's important to keep in mind right that these are served goals and motivations so I wanted to keep this this grounded and discuss sort of three specific ways in which we use functional programming techniques for you know great prophets basically and I try to come up with sort of three fairly pithy sounding you know mottos if you'd like one is your server as a function the next next one is your state machine as a formula and finally your software stack has a value now it's fairly ambitious I think to try to cover three of these in one so you might only get to two but hopefully you'll kind of get the point so and also because time constraints and I wanted to cover a fairly large amount of material I'm not going to go into a great amount of detail in any of those but hopefully enough that it that will allow you to sort of derive some interesting lessons so so first your server as a function so modern server software is kind of a tricky beast servers need to be highly concurrent because they need to be able to handle many many requests at any given time in order to to be efficient basically most servers are part of a larger distributed system and so they need to interconnect and talk to other other system software and so on and so forth which complicate things a great deal we don't live in the sort of world of a neat lamp stack anymore the operating environment for modern server software is fairly complicated we're dealing with asynchronous networks and among other things that means we can't tell the difference between a slow node or a failed one we're dealing with partial failures Iraq may fail independently of the rest of the data center a machine in that rock made fell in the penalty of the rest of the machines in Iraq and so on and so forth and finally we keep building really crappy machines because that is economical things the thing to do and we're able to sort of scale out our computing capacity in that way and finally because we live in this sort of very hetero-energy heterogeneous world of open source software we need to talk to lots of different software that operate in different protocols with different conventions with different operating models and so trying to unify all this all this into one is a fairly ambitious thing and it is what we tried to do in our software systems and so what I'm about to talk just briefly about I actually wrote a paper about and that courage to check it out it has the same title as this section of the talk this paper is about a year old but it goes into a great amount of detail and motivation that I'm not going to be able to cover in this small amount of time that we have here today so the first thing I mentioned was that our systems need to be highly concurrent and this is a central theme in in modern server software and so I think it's very very important that we have a solid flexible model for dealing with concurrent software and for that we use futures now futures are now also part of the Scala standard library in fact it came out of an effort with Twitter and types of in a few others and I'm going to describe to you Twitter's futures and how we use them they're more or less the same as scholars futures but you'll see in the end that I discuss some of the differences so a future is you can think of it as a container that has the most one element and a future is in exactly one of three states it can either be empty and that means that there isn't yet a value available it can be succeeded where a value is available of this type T or it can fit or it can be failed where the thing that was supposed to produce that value actually failed to do so and communicates the exception instead right and that's a fairly standard thing you have some sort of handle on a computation and the results of that computation can be yet either of those right now we use futures to represent all sorts of asynchronous operations like for example a pending RPC I make a call to another host and I'm waiting for the result things like this guy owes are good candidates for being represented by futures because discs are very very slow and you shouldn't you know block awaiting their their results also things like timeouts you can represent our model as an asynchronous operation in this manner and the important thing to note about futures is that the sort of producer of the value is entirely decoupled from the consumer the consumer merrily has a handle on one of these futures and the producer independently ie asynchronously is responsible for doing the computation that's necessary to fulfill that future and futures as a percent of them and as witnessed by their API is read-only construct and what we call the thing that is actually allowed to write into future is a promise promises as a type of future that is also writable so I mentioned before that you can think of futures as a kind of container and indeed they are and in fact when you think of them that way you can define operations over them that look very much like container operations and so you know one of our favorite kinds of operations is map where we say we have a container a function that takes some value of T to some value of U and we get back that container of now value you or type you and this is a completely valid thing you can do in futures if I have say an operation that returns an integer and I want to make a string out of it as long as if a function that takes an integer to string I'm good to go similarly you can define things like filters for example now what does a filter mean for something like a future where you have at most one value or rather exactly one value or pending the way we've interpreted that is that if the predicate fails the future also fails now the probably the most interesting operation you can define in fact on any container and in Scala is this flat map operation where instead of and I'm sure many of you are familiar with us wear instead of defining a function from T to you you define a function from t2 future of you in this case and what that means is that you're given back the future which behaves as the inner future once that function is applied to the the value produced by the outer future right and it's important to note here there note here that we can obviously only have a a value of type T if the outer future succeeded and so if the outer future fails we have to you know fail the whole composition and I think the use of them like flat map is perhaps best illustrated by an example so let's say I have a method that authenticates a user so given an ID and a password I i get a future of user object right and so this gives me a promise effectively that at some point in the future I'm going to this user object and I might have a method called get which given this user object gives me a future of some user data right and so these two things might actually be represented by two different software systems where I have to actually dispatch our pcs I might have to query a database whatever it doesn't matter the basic idea is that the operations themselves are represented by these these future values if I now wanted to combine these two things together in in one method which does authentication and gets me the user data it becomes fairly obvious that you know sort of flat map is the operator of choice sir so if i authenticate the user first and i take the result of that and then retrieve the user data that fits exactly the pattern that we saw in the in the flat map method signature and so one way to think about this is that it's a kind of dependent composition I first need the result of the authentication to get the user data and so forth and now I have this composite operation which represents this operation of actually authenticating and getting authenticating the user and getting the data pertinent to that user and the interesting thing about this is that I haven't really done anything about sort of describing how i go about this I've kind of just to clear that what it means to do this operation is to first do the outer operation and then given the result of that do this interoperation and I haven't you know started new threads or had to make a decision about exactly how these things are scheduled and so on and so forth and this gives rise to very certain nice and declarative programming your model next so so far I've described what we can do with the data that the futures return obviously and most systems is often very very useful to also operate over errors for example you might be able to recover from certain classes of errors and so futures also allow you to do this where there's a Combinator called rescue which is sort of dual to flat map except that instead of operating over values that operates over errors and so whereas flat map is run whenever you have a value to operate over rescue is run whenever that future fails and it gives you now an opportunity to recover from failures it's exactly the dual thing except we operate over error values instead of the contained value so as a simple example we can use rescue to for example retry an operation so imagine if the authentication operation is a is issued by say RP seeing to some remote system that memo system can fail and thus the operation can't i'm out and i might encode timeout errors and this say exception value and then use rescue to try to retry that operation basically and so now we have again basically said well f represents the operation which attempts to authenticate the user and if that fails for the time up attempts as a attempted again right and again this is very nice in the claritin model where I'm a moral lesser stating what F represents rather than how to how exactly to go about executing the thing that F is supposed to do another very interesting thing that you can do once you start representing these concurrent operations with futures is that you can do this kind of composition over multiple futures as well so for example the collect Combinator turns the sequence of futures into a future of sequence and so if I have a number of futures that represents a number of concurrent operations I can wait for all of them collectively and get a future back that that succeeds only when they're all complete right and so this is very very useful when you're doing this or operations for example and the failure semantics are very much like we saw in flat map if any of the Constituent futures fail the outer future will also fill which most of the time is what you want so this is how you might implement a very very simple sort of segmented search and so a very common pattern in distributed systems is that you partition your data over many different systems and in order to say query the whole data data set you have to issue queries to all the partitions and then combine the results this is very common in search engines for example and so in this example let's imagine you have a method that queries a particular segment identified by an integer and it returns to you a future of result right and that's the search result then I might use the following to query all the partitions and combine the results in this case just by a flattening is set right and so first I issue all the queries so for the number of segments I have query each of them and this gives me now a sequence of future of results then i can call future collect on that which now recall gives me a future of sequence of result right which is this and then i just flatten the set and return to thing and again note this kind of pattern again I'm sort of I'm not saying anything about the sort of execution plan of this I'm merely saying what it means to search is to query these segments combine the results and so on and so forth I haven't spun up any threads I haven't specified anything about how to synchronize across these different things I'm merely saying that what it means to do this is a combination of these other concurrent operations one way to visualize this is the sort of collect operator is kind of a confluence point in a data flow graph right and so you can look at programming with futures as a if you squint real hard as a kind of dataflow programming system and this is how this would look like if you if you were to lay it out graphically so we talked about concurrent programming now another or rather a model for one model for doing concurrent programming another very important thing in distributed systems and at least the kinds of service that we create is that there's a uniform way to represent what if server is right and recall this a service and if so this is simply an asynchronous function and so it's it's not too too too much of a stretch to say that a service is merely a function from some request type to a future of rear fly right and so you can think well if I have an HTTP client then that's more or less a function from some HTTP request type to an HTP reply type if I have a rightist client that's more or less a service from a like a Redis command to write as reply and so on and so forth you can see how this fits nicely in with with this way of thinking and this is also sort of would give rise to the the title of this this particular section now the interesting thing is that these services are symmetric right we can use the same type to represent both clients and servers so whereas in a client you have one of these values one of these functions that you can call if you're a server you're responsible for supplying that function right and so I might say HTTP new service and give it some parameters and get an HTTP client whereas if I have an HTTP server I need to create my own service right and the nice thing about that is servicing clients are completely symmetric so you can imagine that if I use these services to represent some module boundary it doesn't matter whether that service is a local thing or something that's injected from a say over and over an RPC protocol so services represent applications effectively and so I if I have an HTTP service that represents a particular application over which I can issue HTTP requests for instance however there's usually a need for a lot of or service agnostic behavior in the service that we create so examples of this includes things like timeouts and retries maybe you want to keep statistics which is very very important for doing diagnostics things like authentication and logging are all just a few examples of the kinds of things that we want our servers to do but are completely agnostic to the particulars of that server and so it would be nice to be able to separate those two not baked this functionality into the service that we create or the clients that that we use so this is what we call filters and a filter again is more or less a simple function and in this case the function goes from some input request type and a service so you're given an input request in a service onto which I want that request to be dispatched and you're responsible for returning a future of some output type and this is a very very messy type signature but is serve required if you want your filters to be able to transform types right and so I think it's easier to look at this diagrammatically where this is the filter hair you're sort of responsible for taking some some requests and producing some reply and this is where everything fits in so a very simple example of a filter is something like a timeout right and and here you see how in the simple cases many of these type variables are superfluous where the input and output types are exactly the same I'm not doing any transformations of the types of the values right and this is a filter now that says I want to timeout after the given duration right and so what that means is once I dispatch a request on to the service either the service has to succeed within the given time or I'm going to produce an error that is a timeout error in this case and the whole filter is the following again we call them given a request and a service so i dispatched the request recall that services are simple functions so i can use to just use the apply syntax sugar i I call the service with this request that gives me back the future the Twitter future implementation has a Combinator that allows you to basically the timeouts we're given a duration you get a new future back which either comes out within that duration or succeeds before it and so this is the the total amount of code required to implement that kind of functionality again a completely orthogonal way a slightly more complicated example which makes use of the fact that we have the ability to transform request and response types is something like a authentication authentication filter and so maybe I have the need to receive only authenticated request to my service and so maybe you have a part of your HTTP API for example that is required to be authenticated and you might want to encode that in your your static types right so maybe you have a distinguished type for authenticated HTTP requests which is distinct from regular HTTP requests now this is a filter that takes a service that takes an authenticator should be request but the service it produces takes regular HTTP requests right and so what it has to do is it has to attempt to authenticate that request if it succeeds with now in our thrackan so you assume you have some sort of authentication method that can lift the unaffiliated request into an authenticator request then I can dispatch it onto the service if it fails then I short-circuit the request right there right and the nice thing about this is that now we can also use the type system to our advantage where effectively have a firewall between serve an authenticator nothing to get it realm and very very importantly these things can be sort of combined in arbitrary ways and so or stacked and in arbitrary way station say let's say I have a timeout filter and again we call it filters on them on their on their own are just these values that I can combine with other things might have an authentication filter now I can create an authentication and time I'd filter by composing them together just like we can post functions and probably familiar in familiar ways and then if I have a service i can now layer this on top of the service to create a service now which in addition to doing the the serving that it does also does authentication and time up and this seems on lesser need on the page but and actually in practice is used a lot this is actually a short excerpt from our front-end web servers this is a real like real live code which you know you probably all actually have hit and you can see that we use filters and mirrored ways to do different things like you know logging collecting some stats doing tracing doing traffic control all sorts of different things and so these things are really really useful and of course they're really really nice thing about this is that because these things are independent components i can use well first of all used to minerals are the different configurations but they're also much much simpler very very simple components that i can test independently that i can rearrange in arbitrary ways and so on and so forth i'm going to skip this example so you could argue that what I've described to you for you know if you sort of squint it kind of forms an athaki no basis on top of which we can build these kinds of server systems and the interesting thing to me is that as well as I think just being good software architecture for the implementer and in many ways forces the users of these api's to follow the same best practices right because you kind of your kind of forced to to practice that a good modularity and hygiene by using this abstractions however there are some real issues with this sort of very sort of pure way of thinking about concurrency and services in particular there are often many times where we don't want to decouple the producer from the consumer for example if the consumer is no longer interested in an answer right so maybe you have a network client that on whose behalf you're doing some expensive work and that network line disconnects or times out or something like for example you have a mobile phone app if you continue to do that work you might end up in things like dogpiling situations where well maybe that client will not reach why and then times out again and it retries again and so you get into a situation that you can never get out of where you're just doing ever more work and your cues are growing and growing and growing but all of the work you're doing is useless because nobody's interested in it right and so this is one sort of very very real way in which this restrict the coupling between producer and consumer falls short and I don't think I'm gonna have time to discuss the second one it's not as important so the way we ended up solving that in our futures implementation is by introducing something called interrupts and what that allows you to do is that if you're a producer if you're responsible for populating one of these promises which becomes a future you can set an interrupt handler and that interrupt handler can be can be called by effectively the consumer and of course it survives with any sort of combinations that you do and so for example in the data flow graph that i showed before if any other consumers raise the center ups it flows all the way to the producer to all the pretty all the relevant producers and so this leads to patterns like this for example I might send an interrupt handler where if I get a cancelled interrupt then attempt to update my promise with a failure and if that succeeded and this is important to sort of resolve the race between interrupts and actually completing the work then I canceled the underlying operation and go on happily and this is turn out to be very very nice and one way to think about futures is that the flow in exactly the opposite direction from from the values in these if you sort of take this this data flow of you programming with futures and the other really nice thing about it is that any code in the middle does not have to worry about bout interrupts if you use all the commentators and the simple ways that most people use them interrupts are just accounted for you and only the the eventual consumers and the eventual producers need to worry about them and that's been a very very useful thing so the second thing I want to talk about is another sort of technique that we've taken from functional programming and use used to great benefit is dealing with with state machines so the motivating example here is doing something we call service discovery and the way well first of all what is service discovery it's a way for clients discover what servers are available to serve a particular service or systems so if I need to say get you submitted it or talk to you sir metadata server then this is a system that I can query and it will tell me here are all the hosts and port port numbers basically that are able to serve you some metadata for you now we've chosen to use zookeeper to store all this data and zookeeper is a system that is kind of notoriously difficult to deal with correctly almost any time you try and do anything fancy on top of zookeeper you're probably doing it wrong because the the intricacies and beaten and sort of very sort of finicky behavior of zookeeper clients and in addition to that we have to do a great amount of resource management when dealing with zookeeper in particular so for example if I am a client that needs to maintain hosts list for a thousand client or 4,000 different servers services but only some at a time I don't want to burden a zookeeper cluster and I want to make sure won't need to keep track of which of these observations or which of these services are actually useful at any given time right this is just serve a diagram to show some of the state transitions the zookeeper client so the interesting thing about this particular use case is that we can take another sort of hint from functional programming in this case self-adjusting computation to solve basically all these problems in a very very nice way so in order to do that we introduced something called bar and a bar looks very much like a future with a different name and again it has a flat map and so on and so forth the big difference between a bar in the future is that of our can change continuously over time right so whereas a future is sort of a one-shot value or container the value inside of a wire can change over time and furthermore bars always have a value so whereas futures can either sort of be pending or failed or whatever bars always have to have a value but you can compose over them in the same ways right so for example i can write the following if i have a bar x and y both integers then i can create a VAR z that represents the sum and if i update x and y at any given time these z will also update with it right and so I have a way of doing the same kind of you know transformations and combinations over these containers but the computations will update and rerun as necessary to account for the input and the interesting thing about bars is that in order to actually observe it you actually have to observe a stream of changes discreet changes and so every time that the value of the Vark changes you effectively get a stream of what we call event that tells you what the most recent value of the VAR is right so on top of ours we can build something called activity right and so again we call that bar always has to have a value it always has to be defined it can't fail and so on and so forth but when dealing with you know fallible computations like we do all the time it's very important that we are able to model first of all operations that are pending and second of all operations that are failed now this starts to look very much like futures right and so an activity it's actually just a sum that we build on top of our and an activity t can either be as I sort of mentioned in okay with a given value pending or failed with with an exception and an activity is more or less just a sort of convenient wrapper around a bar of activity of of T right and so this now allows me to do the same kind of modeling as i do with futures except that the value is continues and can change at any point right and so if you really want to squint you can say a a futurist to a vowel aspect activity is to a bar right and so whereas a future have a vowel which is completely immutable it gets populated at some point it can fail with a bar I have a handle to this computation that can change in any given time but can also be pending infill right so how can you use use these bars to do interesting or activities which again are just simple wrappers around bars to do interesting things well first of all we can model most operations that zookeeper can do in terms of activity so for example I can take the glob of a path right and this just gives me all the pathnames that that comply to that glob pattern as a sequence of strings right I might be able to get the data of a given path right and then I can combine over that operation for example to say well if I have a bunch of paths and I want to collect and I want to get the data for all of them then I can do activity collect an activity collect exactly like future collecting surf have a bunch of activities it represents the sort of aggregate result of all of the constituent activities now the interesting thing about this is that this sort of hides all the complexity of zookeeper for you right and so by explicitly modeling something like a glob this is like a directory listing basically as an activity of sequence of strings all of the underlying stuff about if you're familiar with zookeeper of setting watches dealing with you know this connects all these different things is completely hidden away from you you're just given this activity which represents that operation right and we can hide away all the machinations of dealing with zookeeper and sort of fire wallet away from you so on top of this I in services by the way sort of what we call our service discovery library now we can easily build a server set implementation right and so let's say I have and by the way just to give you a little bit I guess more details about it the way service that's works is that every host that is there ever able to serve a particular service simply creates a no in a directory in zookeeper right and so the job of a client of that is to enumerate all those nodes parse them into socket or doses basically and then have something that represents that you know that host list now that hostess can change at any time right so the nodes can it be added or removed data inside of the nodes could change and so on and so forth and so what we're what we're building here is something that represents the sort of dynamic view of this operation and if any of these things change as I said before the the result will recompute itself and it will also attempt to recompute the minimal amount of things that has to in order to update the return value for these sort of changing facts on the ground and so let's say we have a method called parse which takes some sort of buffer and gets me a socket or dress right some JSON parsing or thrift or whatever and now i can simply compose on top of the zookeeper client that i have and so my job to get my job is basically well given all the pat all the all the paths that i have so given some sort of prefix so every node in the directory then i can take all those nodes and get all the data now i get an activity of sequence of data basically right and given that i can parse them and get an activity of sets of soccer buses right and this is more or less and outline the entire up entire implementation you have to build on top of a zookeeper client that's modeled in terms of the these abstractions so i see that i'm running out of time there's actually some more details which are very interesting in terms of how we can model say broken so you keep your clients on top of this but the disorder just of the thing remains that we are simply composing these simple constituent parts and declaring that what it means to you know what it means to have entries in the server sets is just you know taking the the directory listing hair and parsing all getting the data from all those entries and parsing them and if anything changes the result is automatically updated for you so for example if a notice removed then only the the parts of the code that is required in order to update the result is run again and so on and so forth and this has been a very very useful technique another very interesting and important thing about the way we have constructed bars is that it automatically takes takes care of resource management for you and so an outline basically what it means is if I am actually watching something if I'm actively observing a result anything that contributes to that result being computed will be live but the moment that I am no longer interested in the result those Constitution operations also closed down and so for example if I'm watching a particular service set and that implies that i have to that i have to get you know particular directory listing or put a you know a bunch of zookeeper watches on particular files then all those watches will be active but the moment that I'm no longer interested in that observation anymore all the constituent operations also will shut down and so what we have is a very sort of principled way of the limiting lifetimes of observations and resource management again is only a concern for the producers and the consumers everything else in between does not need to worry about resource management concerns so I had one more thing and I encourage you to check out the slides or some pretty interesting stuff there but I wanted to sort of wrap it up with a sort of few I guess lessons that you might derive from this or certainly lessons that I think are guiding our own use of functional programming and first of all just a little bit on obstruction I think I think abstraction when used correctly is something is a tool that can serve to increase your precision and clarify your code right abstraction kind of gets a bad name because of indirection an indirection is not the same thing as good abstraction and that's a very very important thing right people will often sort of push back against abstraction but not for good reason is because of abstract bean factory implementations basically right secondly and this is something that I think you've seen throughout this talk is that I think if there's one thing that you can learn from sort of functional programming is that compass it's that composability is like you know your number one friend this ability to define simple parts and combine them in a bigger more complex thing that has emergent behavior is very very important first of all it's very easy to reason about and test all these individual constituent parts and you can bind can combine them in multiple ways and also as I said before in a way it kind of enforced this modularity because he uses these api eyes are kind of forced to comply with you know the standard way of combining these things and i think it's very important that you put a lot of thought into sort of finding your what alcohol orthogonal basis in that you want to find the sort of set of abstractions which are mostly non-overlapping and give you the kind of tool set with which you can construct these systems secondly I think it's very important another sort of very important lesson we can learn is that it's important to decouple systems and separate concerns right and so you've seen this button in both of the examples that I give where we've you know if you want to be pithy about it we've kind of separated mechanics from semantics where when I'm dealing with futures I'm simply sort of declaring that what it means to be this particular operation is the combination of these other operations and haven't made any judgments about exactly how you go about computing that that's up to the underlying library and that allows us to change things as we go along and gives us a great deal of flexibility and implementation several times at Twitter we've changed your underlying implementation and automatically everybody's benefited without having to you know do anything or change your code in any way and I think it also leads a simpler user code in that most of the code that you're right as a user of these abstractions and systems is more or less or approaches more and more sort of pure application logic right and I think that's a great thing and I like to call this kind of like the software tools approach to Engineering us maybe it's like slightly hubristic I think nobody's quite approached the original version of that yet which is you know unix pipes but I think we're getting kind of close I think another very interestin thing to keep in mind it's just keeping it simple so Scala and and other FB languages also tend to be very very powerful and it's very tempting to try to use these esoteric features or or techniques and in interesting ways but but that's not not always profitable especially in a setting like ours where it's a key characteristic of systems that we create that they be understandable by a large population of engineers right and so basically always remember the reader is a very very important I think slogan that you can adopt right and I think to a large degree actually software engineering is the art of sort of knowing when you should make something worse then you know that it sort of can be like when you deviate from your Platonic ideal of a system and secondly I or thirdly or fourth day I don't know which one I'm on I I think it's very important to realize that programming languages are but a small piece of the puzzle em in generic right when we talk about software engineering we talk about so much more right we talk about development practices how the sort of develop our operations aspects of things we're talking about how to make code useful and understandable and and fort for a large population of engineers right some of which may be you know newly out of school have less experience have no exposure to FP and so on and so forth and this is very very important in a setting like ours in particular and I also like to sort of practice where I call real technique and if you're familiar with realpolitik it's basically the same definition but adopted for engineering where and again the sort comes back to to keeping it simple you really want to make sure that you're not creating abstractions for abstractions sake or modeling thing things for modeling sake but rather really really consider what those that are practical and material factors are that you're that you're dealing with instead of keeping it real and it's very important to have fun while you're doing it it's not not a judgment about that but I'll take any questions