Better Models through Metaprogramming at Jane Street

SF Scala: Yaron Minsky: Better Models through Metaprogramming

Recording: SF Scala: Yaron Minsky: Better Models through Metaprogramming

[Music] so I want to talk to you guys about some systems that we've built at Jane Street for improving some of the modeling that we do but before I talk about the system I feel like I have to say a little bit about how Jane Street works because my own focus for a bunch of years at that Jane Street has been on the software side of things and I think people who hear me talk often imagine that Jane Street is completely about kind of systemic trading systems that the way people work is they come in the morning and they build these trading systems they write programs they look at statistical analyses and they tweak parameters on the scale of weeks as they kind of resolve different experiments and think about what's the right way to run the system and that's definitely an aspect of what we do but the trading exchange rate is much broader spectrum than that we have some trading that's like that and we have some trading that involves not fully automated systematic systems but things that are more like robotic arms than like full-on robots systems that are really augmenting the actions and decisions the traders make and tightly integrate with their daily workflow and if anything that's more central to the kind of trading that we do and we even have traders who literally like type a specific concrete order into a front-end and decide which exchange to send and what the exact details of that are this manual order entry is an important part of we trade we even do trades where you pick up the phone and think about how scared you are to trade with a person on the other side and whether they know something that you don't right so there's this enormous range of different kinds of trading and that involves a lot of human elements and modeling is still important there but these aren't the kind of models that you think of from a kind of traditional machine learning statistically driven model perspective where the models are relatively opaque these are models that humans are interacting with in a deep way and need to be able to understand and make changes to and the system I want to talk about is really about how to make that kind of system work well work effectively and what programming techniques you can bring to the table to do that and this is a you know Scala meetup but the the lessons here I think are not specific to Scala I'm really not specific to functional programming I think you can you can take a lot of these ideas and use them in almost any programming language so let's talk a little bit more about what kind of models we have in mind so I think the first aspect of the model is what I was just alluding to is the models need to be transparent they have to be understood by the humans who are using them and indeed are often designed by the humans who use them so there are trading desks that think about what models they want to have and think about what parameters they want to get and how they want to derive those parameters and they're going to set them up and they understand both how they were constructed and how they want to modify them when their understanding of the world changes that's kind of fundamental to what we're doing here they're also shared you may have a whole trading desk that is working together with a collection of interrelated models where many people are looking at it different people are looking at different aspects of it a change that one person makes to a parameter affects what they see but also what other people see so there's a lot of interconnections between the work that different people are doing the models are typically simple when you think about them on their own each individual component of the model like it can include things as simple as applying a currency conversion or doing the sum product of a bunch of fair values to compute the fair value of that corresponds to a basket of securities so the you don't tend to have a lot of subtlety and complexity in the individual models but altogether the system can be quite complicated you are the the full scope of the models is large they it takes a lot of computing power to compute them and a lot of efforts to even express what is in there what that what the models have and finally they're streaming models which is to say these are models that are responding to live market data you're getting streams of data in a lot of it is coming in as just updates from say exchanges where securities are trading some of the streaming updates come from humans entering parameters there's lots of different sources of the data but you need this as a kind of live a real-time system that respond to the world as things change okay so those are the kind of models the think about what the requirements are for a system like this if we wanted to be really good what do we need so it has to be flexible it has to be easy for people to add new things to the system right if there's some new kind of thing they want to trade they need to be able to write that down you want them to have enough power to express it in the way that seems natural to the user right so you need just a lot of expressive power and and I think that's a fundamental thing and I think that's easy to kind of screw up you're trying to get other properties of the system right one way of getting the system to work in other ways is to limit the expressiveness of the system but here the expresses the expressiveness of the system we build is fundamental you also want it to be debuggable and this has to do with a fundamental aspect of trading and I'm not sure debuggable is exactly the right term I might also just say understandable because one of the most important things when you're trading is being able to figure out what of two different worlds you're in so imagine you walk in the morning and you start trading and your models indicate that trades are setting up the the availabilities to trade in the marketplace look good by the way you think about what the what the values are the things that are being traded are and when you see that happen well you could be in one of two situations one is the models are setting up because there's some natural reason for that to happen say there's some external event that's causing people to demand a lot of liquidity right at that moment and that's pushing things out of line and there it's a natural opportunity for you to step in as liquidity provider and be able to make money from being the other side of those trades that's one world view that you might be in the other world you might be in is you're wrong you have made a mistake you have missed some detail of what's going on you don't understand exactly what's going on with the taxes or you missed understood the definition of some security that your trading and the reason that it suddenly looks like you're able to trade a lot you're able to trade in size is because you're angrily stuffing money into someone else's pocket and to be able to distinguish between those two worlds you need to be able to look at something that's unusual like hey my model is setting up by a lot that doesn't usually happen and understand where it comes from be able to break it down and see what contributes to that number that you're seeing showing up so understand ability is fundamental and it comes up in this case we're trying to kind of distinguish between these worlds also just when stuff goes wrong you have all sorts of safety checks in the system like this that says oh that number doesn't make sense so we're gonna kind of error out that part of the model and you need to be able to understand when something errors out why it errored out to be able to kind of look back into the kind of structure of the computation and understand the reasons behind the problems and that's hard right and you need because it's right you think about debug ability as an ordinary software developer and you think oh when a bug happens then yeah I can go and understand it and it might take me a few minutes or it might take me hours or days to understand the details of what's gone wrong and in a system like this the answer kind of almost always has to be that you need to be able to get the answer you need to understand what happened on the order of seconds right minutes is a long time when there are trading opportunities of foot so the kind of understandability and debug ability of the models is absolutely fundamental to what we're doing it has to be performing it has to run fast enough now this is whenever you talk about something needing to be to have good performance well it means different things in different contexts we write systems sometimes whereby good performance we mean oh yeah we can take down a few million messages a second and we can you know consume and emit a packet in a handful of microseconds that's not the kind of speed we're talking about here this is human scale performance so if it takes 10 seconds to for us to refresh the computation that's super painful if it takes a second it's a little longer than I'd like if it takes a tenth of a second it's great and if it takes a millionth of a second I just don't care like that's not an improvement so performance matters but we're really talking about performance on a human scale and then the last thing that you need out of a system like this is scalability and I don't mean scalability in the I can run very big models although you need that too but I'm kind of classifying that under performance I mean scalability in the face of complexity so as you need to specify larger models because in the end these aren't just models for one tradable thing there's maybe thousands of different securities that you're trying to come up with information about and so you end up having lots and lots of little models that tell you about different little parts of the world and you need that to be understandable in the end you need to be able to manage the complexity of that if you decide oh we think we've modelled this whole class of securities incorrectly we need to be able to change the model in lots of different places you want it to be maintainable in the way that ordinary software systems that are well designed can be maintainable and and that's going to turn out to be a very important constraint on how we design the system so this is not a unique problem it comes up all over the place there are lots of financial firms that need these kind of shareable models and there is an industry standard solution it's called Excel that's that is the basic approach that people take to solving this problem and don't snigger too much at Excel Excel is kind of an amazing thing first of all if you like functional programming well spreadsheets are a nice functional programming language right there the expressions inside the each cell that you have there well that is really a functional program and it's more than just a functional program it's a graph structured program right it gives you the when you think about what's really going on in the spreadsheet each little cell has an expression that tells you how it's computed and that computation depends on a bunch of other cells and then what's in those cells can depend yet on other cells and this forms a directed acyclic graph which really represents the structure of the computation the graph is by the way dynamic you can use what are called indirects in Excel to change what a given cell depends on dynamically depending on what data it sees and you can pull streaming data into excel in various ways and you get in some ways a very nice system and that system has about a bunch of advantages so let's kind of score how the system does on our requirements so it's very flexible you can encode all sorts of complicated things inside a spreadsheet the formula language like I have some quibbles the 4-wheeler language they're things that you could make better about it but it lets you do lots of things it gives you basic forms of control flow you can you know have if statements and other control things that let you make decisions about where you're pulling data from it lets you express lots of different things that you want to express it kind of checks the box there the debug abilities story is also really good if you think about the difference between writing a system in Excel and writing a system in a traditional programming language you can think of them as kind of dual to each other in an ordinary programming language the logic is primary you see the code that's the thing you look at and the behavior the data the way the data flows through your system that's the thing that's harder to look at when you think about a spreadsheet it's reversed the data is as plain as day you see it right in front of you and by aunt by seeing that data it gives you a nice way of understanding what's going on in the computation the code is harder to see the code is you know sprayed out all over the system and there can be sometimes surprising irregularities in a badly constructed spreadsheet that are hard to think about but this front this kind of front and center role of data this kind of data first programming model has a lot of advantages in terms of making it possible to understand what's going on and just very simple things you want to understand why a given cell has the value that it does you can just pull up that cell and see what it depends on see what the expression is and just look at the inputs and understand how that was constructed it gives you a very powerful way simple but powerful way of understanding the way in which things can go wrong and and also understanding just what even if you don't know if something is going wrong understanding where that value came from performance is more of a mixed grade Excel has had an enormous amount I can't imagine how many hours of engineering have gone into Excel but a lot of that engineering has gone to good effect Excel is very efficient in terms of memory usage and it's scales reasonably well when you give it multiple cores to run on but there are limits and we have in the systems that we've built really pushed those limits we have some spreadsheets are so large and complicated that you literally have to wait ten or more seconds for a computation cycle to go and literally like inserting a row into the spreadsheet takes minutes right so you like and sums it the way we've run some of these spreadsheets is we just keep on adding stuff to them adding more models adding more data because we keep we want to do that because that's the thing that controls how we can trade right every new model that we can add every new security we can add in there is another trade that we can do and we keep on running the system with more and more data until it almost falls over and then we like run it like that and so performance it-it-it is in fact a good system for doing this in some ways and another thing that's important is another thing that excel is it's an incremental computing model which is to say you have this big graph structure and if only part of the inputs to the computation change Excel is smart enough to only run the subset of the graph that actually needs to be refreshed and so it's got like the right basic mechanisms to be efficient but it's not quite as efficient as we'd want it to be so there's the mixed grid there and scalability is where the whole thing kind of falls over this graph oriented computing model is great in the small and sometimes you want to understand what's going on in a particular corner of the graph it's all very lovely and it's terrible in the large if you have millions of nodes with tens of millions of edges just understanding what the hell is in there is kind of a disaster right you can it's it's it can be enormous ly irregular different things have can being constructed in two ways and there's no clear way to see even see what the structure is so that's the thing that really makes the kind of simple I'm just going to use excel on the straightforward way that model really not work when you want to do this at scale so we understood that this was an issue and we had some software engineers who we wanted to kind of stand in and help improve the system both for performance reasons performance was one of the big motivations for improving the systems that we had but also to improve the scalability of the system and we're software developers we know we want to do when we see a problem like this we go we're going to write a program right instead of using a spreadsheet we'll write a traditional program in an ordinary programming language well I guess if you can't let go camel and Scala's ordinary programming languages and that was the thing that we tried to do our first attempt to fix the problem was to really go into rewriting the thing in oh camel in some ways I think of James Street as a company that makes money through the creative process of construction and destruction of spreadsheets right new idea is coming to the world some quick spreadsheet is thrown together that lets you understand and model what's going on and then you replace it by like a real program that's actually maintainable when it turns out to be important and we just kind of tried to follow that same pattern here and it didn't work very well so let me talk about why that didn't work so well so let's kind of start from the parts that did work well so the scalability story of code is good code is actually very good at expressing big and complicated things in a regular way and that regularity is important it kind of gives you a compression mechanism by you can write a small compact program that lets you construct a very large and elaborate computation right the kind of parameterization you get from code is extremely powerful and so that and and you get all of the ordinary advantage you get from the culture of programming you get natural notions of version control and release management of code review of testing all of these things help you scale your programs and things like type systems help you build programs that are kind of understandable even when they are at large scale so the scalability part is really good and performance is also good right we you know you need the the graph model that I described is fundamental you need some incremental computing fabric in order to build a system like this just to the size of the models is large enough that you can't live in a world where you just recompute everything all the time but you can do that you can layer that into your otherwise ordinary code base and we have all sorts of clever monadic api's that let us in a natural way encode these kind of graph like computation structures in what feels like an ordinary program so far so good but now is where the story gets worst the debug ability or understandability part of the story was much worse so it's not like we didn't think about it at all we had hooks in the system where we could expose information for debug debugging purposes and places where people could dig in and customize different parts of it and explore what was going on in the computation but it was only in the places where he explicitly decided that was we want what we wanted and it just didn't work very well in the end the understandability of the resulting system wasn't good enough and the flexibility story was also kind of poor so again we could express all the things that we wanted to in some sense that was good it was sufficient to express the models that we wanted but the moment-to-moment in the small flexibility of a trader being able to reach in and understand a small thing and modify it that was much better in the spreadsheet world right if there was some cell that wasn't doing what you wanted you could modify the the formula in that cell and get a change in behavior understand what that change meant understand how it fit into the rest of the system and hear you when you want it to make changes you almost inevitably had to go to this much longer loop of ordinary programming it just takes more time and more consideration to write a program than it does to modify the formula in the middle of some large graph structured computation so this story did not work quite as well as we'd hope so let's think about what's going on here so the Excel model it's in some sense very simple you have a computation graph that computation graph lets you build the results the outputs the computations that you want to see the values that you need to be able to interact with and then there's the pro the ordinary probably right an ordinary program and I can get those same results and from a computational point of view in some sense the same thing happens on both sides but there are different advantages to the two worlds right the the computation graph version of the world gives you a nice way of interacting with an understanding in the small what's going on in your program the ordinary programming approach gives you a good way of scaling to large and complicated models and to understand regularities of the system so what we really want is the best of both worlds and it turns out there's a natural way of achieving this which is meta programming which is to say instead of writing a program that directly represents the computation that you want we essentially write a program that explicitly constructs a graph that represents the graph structure computation that we're really going to expose to users so the core model in some sense looks a lot like the model of Excel if a trader who is interacting with the system can think about this very much as the same kind of thing that they might see in a spreadsheet the end model we have has a formula language it's not the same formula language as Excel it a key looks more like a mini o camel it's a small statically typed language with a hindley-milner style type system but that's not really the important part the poor part it has the x4 acidity to kind of build the kind of computations that you need to do much as the Excel formula language does and instead of writing those by hand we write them a hand you have this complicated hard to understand a regular system is hard to understand and harder than an irregular system you can do it where you generate with you are well understood programs you generate the the graphs that you need to interact with so it turns out this model works well and I kind of want to talk about why it matches the different requirements we have so the basic flexibility story comes out of the formula DSL having these explicit computation nodes named nodes you can point out seeing a user interface and interact directly with it gives the people who are who are interacting system the flexibility to make changes they need to be able to make and it's worth noting that it's not like people have the freedom to just modify any part of the graph that they want to the way we construct the graph there are some nodes that are whose meaning is fixed fully controlled by the by the graph generator part of the system and some that are open that the people who interact with are free to make changes to and those changes get saved and persisted over time but the ability to do that is an enormous a - giving people the kind of opportunity to make simple changes when they need to and the button the debug ability is good because we have this graph model that gives us a semantically simple way of engaging with the system its performing in some sense for reasons that don't have anything to do with the programming model that's performed because we underneath it have a compute engine we built our own parallel and incremental compute engine for running this thing on and you essentially need something like that there's another talk you can give about the kind of algorithmic questions behind building that thing effectively but it's a kind of necessary piece for this kind of system and then scalability comes out of using traditional programming techniques the culture of code all the things that we do when we try and build good scalable understandable systems so this is a real system that we've used in production we have multiple different trading desks who use the system for to good effect kind of to express the models that need on some desks it's replaced big old spreadsheets on other desks replaced a whole bunch of systems some programs some spreadsheets and kind of pulled them all together into one easier to understand system we were able to meet the performance goals that we wanted - it's about 10 times faster than previous iterations of the systems that we had that's still slower than I'm comfortable with I feel like there's still work to be done - to really get the performance I feel like there's another factor of 10 in there that we can get to if we really think hard about some of these algorithmic questions and but in some sense the important part for us was really getting the functionality in place and so we haven't had the time yet to really dig in on all the performance optimizations that we know are there and in terms of the simplicity of the system that that is there it's pretty good so we have something like 50,000 lines of oh camel that generates three million nodes and something like 10 million edges and the system as it stands now and just to get a sense of the regularity there this is 50 thousand lines of code maybe for the whole thing but maybe two thousand lines of code generates 2.5 million of those nodes so most of the structure that's there comes from a very small amount of code and this is super important this gives you the regularity that you want and that regulator is both important when you want to evolve your model you want to make changes you don't want to have to change everything in the world and also in a day-to-day way when a trader wants to understand what's going on the complexity of the overall computation graph is limited because it doesn't vary in lots of arbitrary ways it's generated by a relatively small codebase and another thing that's been nice about this is we've been able to build better analysis tools what I'm talking about the graph model being more understandable it's more understandable to humans but it's also more understandable in a straightforward kind of computer science analysis since you can do static and dynamic analyses of the graph structure in a way that turns out to be very useful so for example if we kind of zoom back to talking about doing this in a spreadsheet I said well when something errors out you want to look at the things that fit in and figure out where that error came from well it turns out you can make the debug ability story better by building simple analysis tools into the system and those and how those tools can do things like if they're different given notice in the error state walk back in the graph and find to me find me the oldest the sort of the the kind of farthest away nodes in the dependency structure that are in an error state and have no errors as inputs and that turns out to be a really good heuristic for figuring out where the true origin of a bug is in our origin of an error is in a computation and now that we have this simple computation model in our hands well we have the freedom to write these kind of analyses and we don't really have the freedom to write these analyses on arbitrary code because you can't answer any questions or library reliably about arbitrary code like this it's the old halting problem but you cannot even answer the simplest question about a general-purpose program but something that's a simple graph gives you a lot more power it gives the person who's writing the analysis essentially more power so we're really focusing mostly on the programming model here but there's a lot of other interesting technical components this so there's some issue about the dsl design itself this is a dsl that has its own semantics in the zone type system the DSL actually it set the DSL that goes inside the formulas is actually its own little incremental programming framework which allows you to optimize the execution of individual cells they can take advantage of previous versions of the computation to optimize updates so that further is a further optimization the system supports there's also a parallel incremental evaluator that we had to build which is an interesting set of algorithmic problems on its own and I think I really didn't talk about it all is there's the whole sharing question there's a distributed configuration store which lets you synchronize the activity of a bunch of different traders who work on the same desk and maybe different traders who work in different offices so this is really a distributed system so we have people in New York and in London in Hong Kong who are really collaborating and working together on the same set of models and you both want easy and smooth sharing between them and you want it to continue to work when things are disconnected it's kind of awful if you know you lose the Hong Kong New York link or it goes flaky for a while and the end result is people in Hong Kong or New York or both can't trade anymore well that's not good so that's another interesting technical aspect of the system and another piece is building a UI for this so excel among other things has a very useful and and and friendly user interface you know a lot of things to complain about but in the end it's a really well-designed thing and we needed to build our own UI and we did this in the end as a web UI and by the way that's also an incremental computation story we have a different incremental competing framework that we use here we compile oh camel down to JavaScript and that runs in the browser and there's a bunch of incremental competing tricks to make that be able to efficiently display the kind of fast changing streams of data that come from these models and give people a good interactive experience and lets them explore data and see things when they go wrong so there's lots of things we want to do in the future in this system one of the fun technical problems is the parallel and incremental evaluator and I think this is actually there's a bunch of I think just open questions about how to build these kind of algorithms well we've talked to a bunch of people who worked on this kind of change propagation problem and there's a lot of academic study on this and I think there aren't clear answers as how to build these algorithms to get the maximum amount of parallelism that you can and so that's an area that we're actively investigating another issue is scaling we as I said we have something in our current deployment zone like three million nodes with 10 million edges we really want to get to something where we could support 30 million or even a hundred million nodes and I think the system is it happens right now isn't quite able to do it I think this work we can do in terms of improving the memory utilization of the system and I think also some of the issues with a parallel and incremental evaluation also needs to be improved to hit that that kind of scale so there's a lot of a lot of interesting work to be done there analysis is another important piece we have a few simple graph level analyses that we do but there's a lot more we can do I think we could do a lot of useful stuff to help understand performance problems in the graphs you could get good tracing information out of the graph to help you understand reasons why the computation you've constructed is slow and another thing is when you have this kind of system you have a lot of people hacking on it and generating new configurations and creating new models you have a kind of dead code problem there are parts of the model that no one has even looked at maybe for months or years and you want to be able to detect those so you can go ahead and delete them because the day where your model changed its mind decides to look at that part of the graph that no one has looked at for six months it's probably making a mistake so a lots of opportunity for that and then another interesting part of systems like this that I think again you might not have thought about but turns out to be important is support for what we sometimes call counterfactuals and the idea is you have some this big shared model that everyone's using and sometimes you don't want to just make a change sometimes you want to say well what if I did this what if I made the following modification and be able to kind of work the world for a while and make a bunch of changes and see how they work out and see if you like them and then if you like them merge back in and even more than that sometimes you want the counterfactuals to be permanent parts of the model you might say well I have this version of the system and I'm running it like that and say well what if I make this change what does it look then and then the difference between those two models itself might be an interest thing a kind of ad-hoc derivative baked into the system so there's both interesting there's both questions about user interfaces how you make counterfactuals work well but also algorithmic questions about how you efficiently build counterfactuals into the system so that people can experiment with them in a lightweight way and it doesn't require you do a lot of work either to construct the counterfactuals or to prepare the computation that you make for that so this is a very specific finance problem these kind of I don't think I've heard this kind of problem about these rich interactive models that people need to build a debug very quickly I don't think I've heard about that in any other context but I think the lessons of this design are actually significantly more general than the particular application so if we kind of zoom out and think about what happened here we have two different solutions there's a traditional programming we just write code to express our models in the ordinary way and it has some good points and bad points the good part is the descriptions are concise and the models are regular and the bad side is the semantics of the resulting system are complicated programs are hard to understand because they're so rich and expressive it can be hard to analyze what's going on hard for people and hard for kind of technical formal methods to analyze on the other hand graph structured computations are verbose and potentially irregular where you just have a bunch of nodes that are connected to each other by default but the semantics are simple it's easy to understand for both humans and for analysis tools and the meta programming trick gives us a way to get multiple different views on the same computation and give us kind of the best of both worlds and this idea of giving yourself multiple ways multiple slices of the same thing is a general case so I can just replace the word graphs where the word configs and now there's the same story in fact we use very similar techniques for config management think about it configuration files are typically relatively flat and simple easy to understand in the small but if you look at the whole configuration of a big complicated system the configurations can be gargantuan almost impossible to understand in the large highly irregular if they're done by hand and again metaprogramming is a way out of it you have if you have programs that generate your configs then you can look at both you can look at the generating program to understand the big picture regularities and to make the resulting system manageable and then you can look at the generated configs when you want to understand in the small the semantics of the system that you've constructed and in fact we see the same technique in other places too we also we we've started getting doing more work in terms of synthesizing Hardware doing FPGA designs you know camel and we take a very similar meta programming approach and you get the same design the program that generates the FPGA that can be hard to analyze but the FPGA is a simple actually again graph structured computation and you can throw sat solvers and other formal techniques at it to understand aspects of it that are very hard to understand when you're analyzing a general-purpose program another version of this idea comes up when you think about another version that this comes up when you think about testing so we have a whole set of tools for doing what we call expect testing which is basically a way of capturing output from a program in the context of a test so the details of exactly how it works doesn't matter but the way we use it I think is interesting which is a lot of the testing we do in this context doesn't look like traditional tests it's not like you have a property in mind that you want to make sure it's true some of what we do is just capture program traces it's a way of getting kind of a slice of the semantics of the program that you've written and by looking at those program traces they give you another view on the program that you've written another way of understanding what's going on and that's useful both when you build some new feature by just taking an example and writing down the trace in some particular context gives you some intuition of twos as to whether or not it's behaving in the way you expect and then as the program evolves as you make patches that modify the code you get two ways of seeing it one ways you see the change that you made to the logic of the program but you can also see the diff in the trace right the trick that's kind of the change in this slice of the program semantics that you so I think this idea of being able to construct multiple different views of the same underlying computational prior process is a very general idea and one that it's worth thinking about where you can apply it in all sorts of applications okay and that I think is all I've got thank you very much any questions so why did you decide use the create a new DSL rather than just usual camel so using no camel has a number of downsides so to be clear we the the DSL in question is more or less a subset of a camel which is to say when you write your meta program that generates the cells you just kind of use that kind of quoting antiquity statistics and write things in Oh camel syntax that are really expressions in the DSL okay let me bring up a little example so here you see on the screen two different computations one written as an ordinary Oh camel program and one written as a meta program that's generating things so long that's generating what's called webs syntax the system is called webs and so you see that this it's like it looks almost exactly the same there's just this little part where you write percent webs and instead of writing x times y we write percent at present webs x times y and its uses the exact same ole camel syntax but you're instead getting an AST of some other mini dsl that you have so we try to share a lot with o camel in fact the type system of the DSL embeds nicely in the types of move o camel so you get type errors when you're writing your camel program that tell you when you have type errors in the underlying DSL so you get to share quite a lot but we wanted to have this separate DSL so we had more control over what it was doing and so that we can do it without having to like constantly dynamically link lots of little pieces and so we have more control over the evaluation semantics right again it's it looks like ordinary oh camel but there's a kind of implicit built in incremental 'ti and dependency tracking in this language which you wouldn't get if you were using simple oh camel for the DSL it also by the way gives us the freedom to do things in the future like if we want to emit LLVM say for like compiling down the computation to a specialized computations I kind of both does the computation and emits the dependencies that's the thing that's natural to do if you have your mitts on the tea directly I would be happy to chat about blockchain after but I think it's not super relevant to anything in the talk so let's take that off can she get a little louder I can't quite hear I mean the people who were building this system were developers who were directly living on the trading desk that were that needed these systems and people who in fact kind of were very tightly integrated in there so there's a I think that we had really earlier versions of systems that were this was an evolution from and so we really needed to think hard about the daily workflows and making sure that the people of the people on the software side really understood in depth what was going on inside the system so the integration was very tight is a we have a question of heroes sure hi thanks to the presentation I really liked the ending there where you showed traditional code versus graphs etc pose wondering where do you think graphic eat when a whole might kind of fit into that model I embarrassing to know essentially nothing about gravity well sorry doors Thank You Rachel just the formal education he went for the residence duration from so say oh so the we so we had the operator I think before I sharp had it but we we use it used to be pipe bang and then we switch it to pipe greater than to line up with F sharp these needs be operationalized ah interesting so it's not incredibly complicated so you the the the back end the key value store that keeps all of the configuration is just a Postgres database so you have to be able to install that and in fact we have multiple menus in distributed operations you have multiple ones of them that are kind of replicating information into each other so there's some complexity around that and you need a set of servers that can run the system and there's kind of ordinary boxes on which you run the programs and that's kind of that the the the UI is just a web-based UI so you just need you know a web server to go get the JavaScript from and mote you know the the the client essentially is running in the browser one of the nice things we have is we can run completely arbitrary oh camel code in the browser and that means our internal like binary binary protocol RPC libraries and all of that just work like yeah it goes over WebSockets instead of ordinary tcp/ip but you get a lot of the conveniences of having kind of one system and writing what feels like a single language but you know sometimes it's executing as JavaScript and sometimes it's executing as Native Oh camel code I think I noticed like this idea is so generous I said I think it was around application inside built systems like is it like basil for example recover the graph of dependency statistically described in your files and the DOE system has to figure out people changed and what has to be repeated what a big pile [Music] yeah I guess in some sense there are two aspects of how we solve the problem that are both I think general things that show up in other cases one is this multiple view thing where you use metaprogramming and sometimes other techniques to give you multiple different views on the same kind of computation and then this is other issue of like oh I have an incremental system that works by representing the computation as a graph and yeah that shows up all over the place it shows up and build systems it shows up in systems like this their computational systems like the self adjusting computation systems that I described like incremental work this way and actually one of things I found to be frustrating over the years is that these systems are all very closely connected but it's hard to find a kind of core calculus a set of shared abstractions that can be used to build all of them you end up building a bunch of systems that are similar to each other and share lots of conceptual components but don't really share any code and that's kind of true here we've built multiple different graphs structured incremental computing systems we have heaven save us to build systems that we've built and we have a bunch of different incremental computing systems with different applications that are optimizing for different things and we've yet to find like the nice general-purpose fabric that lets you kind of keep them all in one shared framework should be one Oh moaning oh that no that doesn't help that is that is completely irrelevant which is to say yeah some of these are monads yeah we use monads but that's like nothing that's a very small part of the design there's there are motives are a nice shared abstraction but they're not like a foot that they're like they're good they're pleasant to use but they don't solve any of the fundamental problems right there's all of this complicated graph structure and updating the graphic propagation all of that and the mouna it's just like a skin that you use for expressing the computation it doesn't really help you at the kind of figuring out what the real core calculus that you need to share is nothing against monads like them a lot but it's not it's not the issue here I unfortunately can't I mean its we're saying it is the most boring web UI in the world it's like a bunch of tables and you can click on things to expand and see the details like nothing you know we're still getting used to this whole web thing like there's no so my question is how how does your adoption ESL's affected improved or worsened your ability to reason about the co-pays [Music] let's say you have a maybe through ESL because used ESL sort of so it's interesting like one thing I'll say is that in some sense we use domain-specific languages surprisingly rarely for an organization that's populated with the disturbing number of pl geeks we've actually on the whole found that we almost are never happy when we design our own completely free strat freestanding domain-specific language we've almost always regretted the decision to build those because every time you build your own language it invariably sucks right like you didn't design the semantics quite right and you didn't build any decent tools and the error reporting is kind of terrible so the vast majority the time when we want what feels like a DSL it's effectively some kind of embedded DSL rather than B and the embedded sales often greatly restricted right it doesn't mean that you have full-on closures from from the ordinary source host language baked in there and there's a kind of fundamental design question you have to answer when you build a library like this because an embedded DSL is effectively a kind of library there's this trade-off of power how much power do you give to the user of the library and how much power to give to the implementer of the library and the more restrictive this mini language is the more power and analyze ability goes to the author of the library and the other way you give the people who write it more freedom to build rich and complicated things so I think in this case the the use of this very restricted DSL which in some sense feels like it is it's also represented as an embedded DSL in no camel but it's also a free-standing ast that we can evaluate on its own that the fact that that is constrained and we can analyze it well enough to understand the dependency structure that is implied by the computation that turns out to be really valuable but I feel like contrary to what you might expect James Street is not a place a wash with you know lots of little baby programming languages all over the place we actually think that's kind of a terrifying thing to do when you're mostly trying to avoid it so you do we have time for probably one or two more questions and people have them yeah yeah thanks for the talk really interesting oh I just used a few more examples DSL so I have a couple other DSL examples I can pull up there so I guess this one you can go into a little bit more detail if you just kind of look at what's going on your building I feel like it but I really wanted a little picture like you're building a simple kind of you know join join join join join join join so this is like a very unbalanced little tree that you're building is probably not what you really the kind of computation you really want to build into practice you typically want more balanced binary trees when you're constructing things in which we have lots of primitives to make that nicer let me pull up another somewhat more complicated one [Music] so again the thing on the top is the computation done in ordinary Oh camel other thing on the bottom is done using webs so here this kind of shows how the how the kind of conditionals work so here the you get arguments so that in the first computation the arguments are all simple floating point values in that first function and the return value is a floating point value in the second one the inputs are web syntax values and there's a type parameter there is a key a kind of interesting mix between dynamic and static programming here and that it is a kind of ordinary little baby hindley-milner type system for the DSL but the the standard values that are passed around look more like Excel values meaning it's like a kind of dynamic dynamic language style variant of like it might be a float or it might be a string or it might be an int and that just kind of fit we're going back the question of how integrated was this with the with the desks that needed it that was the kind of computation that people were used to and they wanted that flexibilities that's kind of baked into the language and then here you just see that the that percent webs like in there you have this more complicated you know if this then that else the other and I mentioned before thinking about dependencies one thing that's worth noting is you can in different branches end up depending on different inputs so in the first two cases when you return error values you're dependent well you send all of these are depending on X and from and threshold but if we had say another variable in there so if you imagine that was like let's say if that was X plus Z say where Z now doesn't show up in the conditions then the node that's represented here would change its dependencies but ending on the result of the conditionals right so that dependency tracking turns out to be a fundamental thing that we're getting out of the expression language a thing that's by the way missing from this example is it doesn't show you how you name things this is just the kind of simple meta programming question of constructing a single expression but really each individual expression is put in a kind of cell that has a name and then different computations will refer to values coming from different cells so and this is well this one's maybe too big to look at but you could but there's you can see there's a little bit more structure here and there's in fact the it's a little hard to see given the size but there's there's a kind of module notion where whole sets of cells actually belong to what are called modules and when you refresh a computation what you do is essentially blow away everything within a module and rerun that module and kind of lay it back out it's almost like in Excel blowing away a sheet of the spreadsheet and repopulating it and this is how you avoid building up random cruft over time you do don't let like random cell you constructor before continue to exist you have some notion of scoping for the names and you can explicitly get rid of them when you want to rerun them kind of meta program that populates a sheet so I could go over this with more details probably not it's probably easier to do huddle over computer than in front of the whole room Thanks [Music] sorry not talking what uh what do you guys used to compile JavaScript we use so right so just to explain what all the pieces are buckles script and JSF oh camel are two different Oh camel to JavaScript compilers and they have very different goals buckle script is really focused on existing nicely with an existing JavaScript like development workflow and in particular has the unusual goal for a compiler of generating readable JavaScript esque output right it tries to make the output pretty so that like your JavaScript friends don't get really angry at you when you commit it to the repo and there are some trade-offs there where buckle script doesn't quite as fully get exactly right all the semantics of oh camel but it hits this goal of generating nice JavaScript very effectively GS of oh camel on the other hand uses the whole ordinary oh camel tool chain attack e okay imma has a bytecode compiler and exactly a layer that takes oh camel bytecode and converts it into JavaScript and it has shockingly good precision we have big complicated libraries like our incremental computing library like our asynchronous programming library that play all sorts of dirty memory representation tricks that you take them and you compile via JSON well and it just works it's astonishing like yeah it's a little bit but em scripting let the the difference is them scripting targets like typically things like web assembly or the kind of this sort of very simple kind of numerical core of there and doesn't it's like that's a way of like getting a C program to run in a JavaScript world but in a way that's quite separate from the rest of it where Jo camel lets you easily call back and forth between Java no camel you know in a fairly lightweight way so it gives you better integration than in scripting does but still preserves the semantics to a camel shockingly well so it's a it's actually a very impressive piece of engineering and it's also much easier to keep up to date with Oh camel because it only has to understand the bytecode representation that doesn't change very often so buckle script is great but it's like a few versions of a camel back it's back at 4:02 we're now at 405 in fact 406 is just about to come out where as Jessica camel is kind of always up to date with the tip so there's various trade-offs there reason it's a little complicated and that lots of people use reason and buckle script together but they're like totally orthogonal reason is an alternative syntax for Oh camel and some tooling around that which is meant to make it kind of easier and smoother for people who are outside of the traditional Oh camel world and it has like the syntax is is in some ways just objectively cleaner and better I love Oh camel but I don't I can't especially advocate for its syntax there's lots of weird if you're seeing Christie's in it and reason fixes a lot of those and it also just makes it feel more normal for someone who comes from a JavaScript background like it has all the curly braces you could hope for but really you could totally you could use reason with G so camel you could reason with buckle script one of the nice things about reason is it's extremely interoperable it's not really in any way a new language it's really a different syntax and kind of a different user experience but it just generates it just like D sugars into a camel and so everything is fully interoperable between the two worlds so so for this particular tool that you talking about do you know spreadsheet basically the interface that still adds up to the trainers being like spreadsheet has experience so the the UI so one thing that in some sense losses from the Excel world is Excel gives you this kind of sheet oriented playground where you can kind of do whatever you want and there's because you have one uniform way of embedding the cells in there's a uniform UI for all of that and that's not really how it works in in the in the web system the system is called webs and in Excel you can like either you could imagine creating computations by writing VBA programs that laid out sheets for you or you could just do it by hand and in webs you really don't have the by hand option anymore you really always generate your computations by way of meta programming and sometimes you might explicitly choose to create little you know grid like playgrounds to do things but that's not the norm it's not like you have these big sheets everywhere so the UI is not really the UI of excel it's its own UI it's you know you when you construct a computation you write symbol camel code to say how you want that presented it's not just like automatically presented by just like laying out whatever the grid happened to be sorry and I can see the computation model is like this it has the same graph like a computation model but the UI is not the same I am happy to stick around and ask questions [Applause] [Music]