Chris Yang, Scala Notebook -- SF Scala @Nitro 20150205

Chris Yang, Scala Notebook -- SF Scala @Nitro 20150205

Recording: Chris Yang, Scala Notebook -- SF Scala @Nitro 20150205

surely Alexi thank you so much nitro thank you for hosting us I'm really excited to be here to talk to you guys today about skull in a book ! so my name is Chris yang I used to work at bridgewater associates a couple years ago and Scala notebook was a project that I worked on their sort of a skunkworks thing and they decided to open source it a little bit they decide to open source it and we've been contributing sort of to it every once in a while after leaving Bridgewater I Nick and I started a company called Domino data labs hopefully it will do a demo about that after but this dog is going to be one hundred percent about Scala notebook basically so what is going to a book skull in a book highest level it's a browser-based rebel force kala so it's written in Scala and it's for running Scala code and if you're not familiar with the notebook the notebook format you should think of like ipython notebook or Mathematica but the idea here is that in a notebook format you can interleave code text graphics to to make a rich document that you can sort of read top to bottom and so this is especially useful for data analysis data science scientific sort of workflows as evidenced by sort of the success of Mathematica and ipython notebook but actually in the course of developing skull on notebook we've sort of just figured out having a really powerful rebel that can render things in HTML is actually just sort of generically useful and one of the goals for Scala notebook that we were interested interested in originally I bridgewater was sort of multi-tenant hosting so the idea is that we could have one scala server and that people internally could have their own scala sessions going at the same time whereas i think a night by the notebook now it tends to be more of a developer local type stack so there's some textural security concern implications of that so demo time and so basically this is going to be sort of a was this oh gosh okay so this is a skull and notebook hold on this is a scala notebook so you can sort of tell that it's basically a sequence of cells and cells can be sort of one of two types it can either be texts like documentation and so in this first cell you can tell you know this is just a markdown formatted cell you know we can add another move a hash enter the title becomes bigger but more often than not your cells are going to be code and so here we hit ctrl enter and what that's doing is evaluating this expression in what's called the kernel which is basically the like Scala backends where this user code is running and so the code is here we capture the standard out and then some representation of the of the like Scala object is returned back and so you can do sort of normal repli things you can define a method there then you can call it sort of basic stuff and sort of by convention notebooks execute top to bottom and so you know you define fact before you can use it here but actually you can sort of run the cells in any order it's useful in like an iterative context we're sort of figuring out what's going on in one cell is you're trying to like figure out a function or something just couple nice things if you try and evaluate an expression that doesn't compile then the skull notebook will just give you a line number and say hey this isn't compile and similarly if you are running code that for as an exception so this calls if that error gives you the stack trace back and so just you know not super sexy stuff but you know what you actually need to actually write code in this environment Scala notebook handles HTML natively and so here you can see we're generating an actual HTML table with five rows four columns in each cell is going to be the products of the row and column index so you can see here and then it also handles images so for example here is a gift of a cat um one subtle thing which is an obvious from the examples that are showed before is that we stream the standard out as the cell is executing and so what's happening here is that we're going to loop 1 to 10 we're going to sleep have a second we're going to print the number and so what's nice about this is that if you have a particularly long running cell you actually get some feedback into what's going on before the code actually returns and so we'll go a little bit more into detail about that later but so you can see here you know we got to standard out even before it returned but once it returned then you know we got what we expected and the last thing in that vein is each notebook is configurable to have a startup script that will execute whenever the notebook is opened and so for example here this widgets package object is actually available to your code inside this notebook by merit of and in its dot SC file that imports that package object basically so sort of like a bash RC but for scala notebooks and that's sort of configurable as well okay so getting mixed up as to where my cursor is okay so I want to talk a little bit about the architecture sort of backing everything that you just saw so the first thing to just remind you guys a kernel is the term for the back end where the code is actually running it's stateful obviously as you're defining new classes new imports and stuff so obviously the result of what you run after it depends on what came before it and there's one kernel / running notebook session and so potentially there are many kernels / notebook because one person could have the notebook open several times you might need to restart the colonel if you're modifying something like the class path or the initialization script and then because many users can be running their own Scala notebooks across the same server so you sort multiply that out and potentially there are many kernels being managed by one Scala notebook instance it's my cursor work you guys something that cursor right all right so um the front ends I think I mentioned this but just to say again we forked ipython notebook and basically replace the Python back in with Scala but the the front ends all the JavaScript is basically all there verbatim with like very very few changes other than the CSS styling that you saw and so the the relevant bit here is that in JavaScript there's a code cell object like I said not really touch from my Python notebook that communicates to the Scala notebook server over three sort of open web socket connections there is a oh I'm sorry come back here the calc WebSockets service those are actually a laser pointer this might be more like blinds Melissa okay awesome so the kelp WebSocket service basically knows how to map an open WebSocket connection to a running kernel and the colonel over here we basically use remote actors to communicate between the server and this sort of front at their like API of this kernel which is a replica later which is itself an actor it has a handle to our repple service which itself is implemented by using the Scala I main class which is the built-in interpreter that backs the actual Scala scholar eppel so the basic API is pretty straightforward whenever their user wants to run code they issue an execute request that goes from JavaScript all the way to the colonel via web socket akka message to essentially a rebel function call the colonel can pass three messages back to the front end so first messages you can send back is whenever there's a print Lin from the running kernel from the runnin rebel evaluation it can send a stream response back once the interpretation of that line of code finishes you can either return an execute response or an error response so execute response is basically hey that's completed successfully here's HTML you should stick back in the notebook if it failed we distinguish between compilation errors or sort of like runtime exceptions and then we'll either pass the stack trace or the the compile error there let's see so this is the API of sort of the repple service here as a couple things that we'll get into later in the presentation but the fundamental one is probably what you'd expect which is hey evaluate this code and then we have this little functor that allows us to do the like print lens streaming back and it returns and evaluation results and this string is basically the fully written out fully fleshed standard out the evaluation result is either incomplete which means a compile failure failure here is a runtime exception and success is hey this completed and here's how I want you to render it will come back to why this is a node seek rather than maybe what you think would just be like a string or something we'll talk about that more when we talk about how the rendering works in Scala notebook just to summarize this this was an enormous pain in the ass I mean in Scala library is not designed to have to do this it was a real mess and so most of the effort of giving this to work was a reverse engineering how I main worked and I think there are some improvements slotted at some point in the future and evaluation was definitely easier than the sum of the completion stuff but it was it was still a mess and at the end of the day this is a pretty simple API and I wonder if there are other better rapala services out there we didn't go we thought there would be a lot of juice out of using the I mean but it's totally conceivable there's a better way to implement this this sort of trait so um one thing that may have been a little subtle that you guys may have missed is that the the colonel where the Scala code is actually running is actually a child process like a child operating system process and that is basically because we needed to support this multi-tenancy use case you know potentially our users were have different security levels have different level of access or permission to different pieces of data and so you don't want an attack where because you know they're just running code right like arbitrary code so you don't want them crawling the filesystem trying to eavesdrop on other running skull and notebook sessions and so you know I think something that could be cool is that could potentially run on a different machine entirely perhaps inside a docker container to achieve more isolation there are a couple second-order reasons why we did it beyond sort of the multi-tenancy the first one is that um each kernel potentially needs its own class path and so it would just be quite tricky to inside one vm sort of properly manage the class path of the of the sort of this child rebel and then the other thing is that it's quite easy to accidentally write something that could totally blow up in terms of like an infinite loop and so you need a really reliable way of stopping these kernels and restarting them and you know killing threads and Java is not easy really the most reliable thing you can do is just you know OS kill that process and do it that way so this wasn't without cost there's quite a lot of shenanigans inside the repple spawn or code that spawns the vm with the right remote actor with the right remote settings and there's some bootstrapping to get that inter process communication to work but I think overall it was the right decision but that part of the code definitely be cleaned up and made a lot more reliable the nice thing about having it via processes that stuff like memory leaks at the end of the day you could just blow up the process and restart the colonel isn't that painful and so it just gave us a lot of outs okay we got the basics out of the way the interactive widgets are pretty cool and so we're going to do a live demo of that was this one okay everyone see this ah yeah that's probably better okay so one of the things that was important to us as we were developing this wasn't just the ability to run Scala code interactively but we were particularly interested in in building things like dashboards so that and in particular what you need and that is the ability to you know interact with the notebook as like a almost like a web page itself so like sort of tweak HTML elements and that have that do stuff in Scala basically and so we're going to just go through a couple of these widgets that we created as samples so a widget in Scala notebook is basically interactive HTML that whose values can be triggered by things happening in Scala and so one of the widgets that we have in the box right out of the box is an input box and what that does is create an input text box and so one of the things we can do is we can have the value of this text box be triggered by a scala expression so in particular here we're going to take the a random alphanumeric stream we're going to map it to string we're going to take the first 10 and then this observable that timer basically says I'm going to take a new element every 50 milliseconds I think that's what this magic 5 means that observable then we can bind to the current value of this box and so after this runs and hopefully you can see here so this is a venting from Scala inside the running kernel all the way back to the front end so that's not all so we sort of need bi-directional communication that's important for being able to actuate stuff on the page and so we're going to define a second input box here and then we're going to say that the value of the first input box should always be the same as value of the second input box and so I'm going to run this come here we're going to do a low and so what will go deeper into the mechanics of how this is working but essentially I can sigh here and that'll get mirrored up here we can do more advanced stuff too it wouldn't be particularly interesting if we just wanted mirror text boxes so what we can actually do is hook up the value of the second text box map it through some transformation so in this case I want to take the current value of IB to the second text box and I want to convert it to uppercase and I want that uppercase version of the value in the second text box to be reflected in the first one sometimes it's a little finicky I don't know this is going to work otherwise let go all right so and so we can do sort of simple transformations that way and just they illustrate the point further we can there's a drop-down widget it's seated with three choices so we can see a B and C but the drop-down has to sort of actionable surfaces one of them is the selected value but the other one is the the set of options that um that are part of it and so we can say here that we want the list of options of this drop-down to be the current value of the second text box but we want to split that by commas and so oops oh I didn't actually run the thing sorry and so we change that up here then this is has been changed to do that come back here another option we've added this new option here and so I think the other thing worth noticing is that the connection to the first input box is still there and so it's still capitalizing with strings and so you know widgets can sort of there's a end-to-end relationship between their between their connections so so let's come back to talking about how this actually works so the picture we had earlier is not complete so in addition to the code cell calc WebSocket service rebel calculator rebel there's this parallel observable path that's sort of analogous so you know at its core you know we're using sort of functional reactive you know programming on the Scala side we're using sort of the reactive equivalent inside the JavaScript so we're using knockout we're using the knockout j/s observable and we're using the reactive Java observable method objects over here inside on the Scala side and so basically all of this cruft in the middle is basically trying to marry up a knockout observable in the JavaScript with reactive Java observable in the in the Scala and so right now there are only relationships between the Scala side of things there isn't a way to say hey I want this JavaScript observable to be directly linked to another JavaScript observable and we thought about that you know you definitely pay a latency thing especially you're trying to do something simple like just mirror two text boxes because you have to wait for an entire round trip or that change to get propagated on the Scala side and but ultimately you know people were interested in writing Scala code and so I think that was the right that was the right decision for that so I want to trace through literally what's happening in that text box example so if you'll remember basically the what we want is to be able to type you know a lowercase string over here in this first text box and then have the uppercase version set up over here so first thing that happens is that the user clicks inside the input box and like literally types words and so by mayor of the way that the knockout stuff is set up knockout basically takes care of the if the user touches the Dom how that gets propagated back as a change to the underlying knockout observable and so when I type hello world here let's just call this observable 1111 observable one gets assigned a new value hello world we wire up the knockout in eventing such that whenever there's a Dom change whenever there's a value change here we know to send an event to the observable WebSocket service saying hey observable one got a new value you should be aware of it so this layer is inside the server the observable WebSocket service knows how to map an observable with a particular ID to a particular running kernel it transforms this request into an equivalent akka message and then that message is fired to this j/s bus which is a remote actor and it's also a singleton and the singleton keeps a map of for a given an ID observable one what's the actual underlying reactive Java observable and so when it gets this message hello world it goes and looks up the observable and calls on next on it which is basically the hey here's a new value for this observable because we've run this map subscribe earlier in the code then when observable one value gets changed then it automatically runs to uppercase that tweaks observable tues value j/s bus nose to listen to changes to observables registered to it and so when that happens when it detects that happening it says hey JavaScript's observable to has a new value here it is that gets routed all the way back through knock out gets that setting to hello world and the knockout is actually what replaces the the dom element in the HTML and so it's extremely that's a lot of hops basically let's see let's go here so uh how do you write one of these things the API isn't great but i do want to walk you through it and sort of at the the crux of it is basically there's some juggling of wednesday happens in JavaScript and House of happens at scala but really the name of the game is that observable ID on the previous slide of like observable 111 or observable 222 that string needs to get passed from the back end all the way to the front end and sort of those objects need to get created at the same time so this is the actual code for the actual input box it extends widget widget simplement 8 we just interfaces basically hey if I call to HTML you'll give me back HTML first thing that happens in input box is we use the JavaScript bus singleton to create a connection and the connection is basically the pair of the observable and the observer and so this is the thing that knows how to map a JavaScript event to a skull observable in vice versa and so the input box defines a public value called current value that's basically a function of that underlying JavaScript connection object and what's subtle here and you'll see this codec thing here is that on you really want the type of this current value to be something Scala SH like a string in the case of an input box and that's sort of like a trivial transformation you can imagine for the drop-down box you want like an index or potentially even something like a higher in the abstraction like a country like if you had a country list of countries you might want that dropdowns current selected value to not just be a string but to be the actual country so what the joyous bus creates when you create a connection there something that speaks Jason and so this Kodak this by map codec is basically the thing that tells you how to convert from the JSON value to the Scala specific type so here's where the real dance is basically so this to HTML we create the outer input Dom and then we call scope script and then in this JavaScript gets executed before the HTML gets inserted and this is the thing that sets up the knockout bindings and so here a scope script basically lets you so when you create a Java scripts j/s bus connection that comes with an ID that's the ob's 111 in the previous example and then we basically pass that into the JavaScript via this like map and then so value ID is the name of the JavaScript variable where this string is is bound and we use that to create the JavaScript observable here so that that's how those two things get married up and so there's a bit of nonsense here inside scope to valve which is the actual thing that evaluates this JavaScript to set that up such that this method can look that the way it looks and the other you know bit of trickery is that the this binding inside the javascript is bound to this Dom element out here so you can see here scope is this dot parent element this in this case is this bit of JavaScript and then we call it with the scope and to be honest this is extremely confusing don't entirely remember why we implemented it this way I think it there was some value and having this thing look beautiful at the at the price of a traceability like this is basically a nightmare to figure out if you get this wrong somehow then it's basically a nightmare to figure out how this actually works okay let's talk about rendering and so this is something that I lighted earlier in the demo but is actually a little bit subtle so scala notebook by default has a bunch of helpful renderers and a renderer in this case is basically a thing that can take an arbitrary scala object in this case of you know seek and turn it into something HTML ish and so by default there's a secret door that turns a sikh into takes the first 25 elements of a sikh and turns it into a table but sometimes you want more granular control and so we give you the ability to find your own renderers and these can be things either defined in a custom class path or in the anit script or even just sort of here in the notebook as you're as you're trying to develop it so what if we wanted to render seeks of in snot as a table but as a comma delimited string and so what we've defined here is a renderer and knows how to take a sikh of int and we create a text element would take the first 25 elements we call make string on it and so in this case 1 to 100 actually gets rendered as 1 2 3 4 5 6 etc another useful example is a rendering products so case classes and tuples and so in this case we want to make a row of all the values of the fields of that product and so will define this renderer and then we're going to define a new case class foo as two elements in it and so that gets shown as a row like a JavaScript as an HTML table what's cool about this is that the we'll talk more about the typing and how Scotland up books knows which renderer to use but in this case we can say well actually want to render seeks of products as a table so that you can sort of easily visually compare all the elements of the same in the same column here and then here we'll see a table where each row corresponds to an element and each column corresponds to a single fields this is sort of the value of a and this is the value of the second parameter so this is the main renderer API interface that you need to implement it's pretty straightforward earlier i alighted why the result of successfully computing a rep hole is not a string or an any any ref but actually a node seek and the reason for that is that um the actual rendering code happens inside the colonel and if you sort of squint at that for a second you'll realize that that's the only answer that makes sense the renderer couldn't help but run anywhere else and so the the rendering code actually happens inside the repple as part of this evaluation loop and that's why the result is a node seek because that's um you know that's our representation of HTML skip that a little bit so how does this actually work so I don't know if any of you guys have ever sort of dug into how the the repple works but this is a this is a type of code that it generates basically it defines a new object for each line that's evaluated and then there's a set of imports that give you be able to access previous results and so what we do in in getting the rendering to work is that we'll ask the repple to evaluate the original request that'll get assigned to a value like res 1 and then what we do is we codegen this new objects and tell it to call rendered and so what's happening here is that rendered is explicitly set as a type of widget res 1 could be anything because you're just running in and sort of could be anything and so the magic here is that we actually use an implicit def from renderer to go from this type a to an actual widget and so these are all the implicit objects implicit renders that are defined in skull into a book there's one for HTML there's one for node seeks there's one for Strings there's another one for Sikhs and so basically we use the implicit conversion type resolution of the compiler to actually figure out which renderer to use and so this obey is all of the standard scoping rules of implicit resolutions I guess those semantics might be changing soon but for now this means that an implicit depth that you have an implicit render that you have sort of defined later on we'll take precedence over implicit render that you'd find earlier which is the behavior that you want because if you're like trying to iterate on a renderer and make it work you want the latest version of that render udah to actually be the one that you use okay last last demo so I want to talk a little bit about completions and you know this this is one of those things that's not so sexy until you actually have to use this and then you sort of start missing the fact that you can hit control space and IntelliJ and it tells you what to do and so the good thing about Scala notebook is we've wired so I'm going to hit tab here and then you see there that it auto completed to notebook and then we hit tab again and then this gives sort of the set of choices here that are completa below what this is doing it's not anything fancy is just wiring into the the underlying ruffle completion similarly we can hit left paren and then this hooks into the double tab behavior in the repple to give you the actual like function definition it also works for gosh this is tricky it also works for overloaded methods so here are the two print lens and something that we added to skull in a book that has an equivalent in ipython notebook it doesn't have a cone on the repple is we've complete string literals and so here I've opened a string literal I've typed a letter S and I can tab and so what this is showing me isn't code completion but it's actually listing all the files in my working directory and giving me a little bit of metadata around it and so we originally wanted this functionality because we wanted to autocomplete on things like column names on a sequel table things where there wasn't the type system wouldn't help you if we didn't want to define a new type for every you know sequel call them and so this is an example and there's some nice metadata so for example if we detect it's a directory we tell you if it's a file we give you the size but you sort of imagine this being sort of generically flexible for auto completing all sorts of stuff that you want to that you want to complete so I just want to go through this stuff quickly basically these were the other sort of two methods on the repple interface so for example in that first example we had calmed up you want notes you hit tab that calls completes and that gives you basically hey notebook is the only completion and then similarly with after the dot you know you get a list of these things and the ipython front-end basically knows how to render that similarly with the string completer this is a string literal completer with the file file names it's a very similar API you tell me what to complete and i'll i'll give you some matches what's not shown here is that this match object actually has a arbitrary metadata map so that's how we're passing back the absolute path and the file sizes in their object info is a similar thing here so just that error you get back def error message string nothing yet the signature back it would be really cool if we could inspect the object statically if we could hook into some Scala doc stuff but there wasn't a great way to do that at the time a brief aside on the completion code i mean the the code is littered with comments like turns out this is really important but we could not remember for the life of us why that me to be false there were comments saying just trust me or go read this other line of code and all will be clear when you went to read the other line of code the very first sentence was this gets a little bit hairy and so this is all just connects back to the I made was not at all designed to be like an interface for doing this type of this type of inspection of the running code so water reverse engineering went into this I think we briefly looked at intelligence open source some syntax tree parser and there were some rumblings that SBT was exposing some hook to like give you this type of completion but um yeah if anyone knows what the state of the art is now in terms of doing this programmatically I'd love to know so I think Nick hopefully we'll be talking a little bit about the project the company that we we've started since leaving Bridgewater so he'll go into a lot more detail I think the the the thing to say about it now is you know dominance platform to help analytical workflows help data scientists but in particular we've actually integrated Scala notebook and so let's see if I can find it and so we've actually made a public domino project that lets you with one click spin up a scala notebook session so hopefully that's something you guys can check out and play with it and one of the cool things about Domino is that we make it really easy to to scale hardware and so if you wanted a skull in a book session but you want it to run it on you know a monstrous machine here maybe this isn't so monstrous but it goes up from there with one click you can click hey I want to switch hardware go back here start a new session and then all of a sudden you're running this interactive Scala session but on a very big machine so more useful if you're doing analytical type stuff I I realize now I didn't actually show you it running and so it's a little bit slow but you know this is running on an ec2 instance on our hosted cluster and they hope that returns oh sorry I don't know yeah as if as if the demo weren't risky enough we're going to like integrate it into another platform so um you know that's all the functionality I wanted to just give you guys a little bit of context of like where Scala notebooks it's in the overall notebook ecosystem so obviously we are deeply indebted to ipython notebook and my understanding is they're either rebranding or doing a different thing called Jupiter now this spark notebook ended up being a for I think he's coming to talk in March so that'd be really cool to see but I think Andy forked Scala notebook and basically added a bunch of nice features to make it integrate with spark more easily and to do interactive type stuff I Scala was apparently a swappable backend for running scala inside ipython notebook I think it was dead circa when we were doing this so almost somehow magically revived I don't know I've watched a few demos but I'm really looking forward to the data bricks cloud presentation because they have a notebook type of thing that does specialize in this type of interactivity so i'd love to see sort of the state of the art there and two Sigma's beaker notebook is in a very similar space with a big emphasis on on language interop and so I've seen some cool demos out of them and yeah so um thanks everybody I'll be sticking around to answer any questions we're hiring at domino data lab Scala notebook is not sort of one of our core things sort of just the thing we do for fun but Domino itself is written all in Scala we're always looking for talent so if anything looked interesting here we'd love to chat with you and but you guys should also check out Scott notebook so this is the bridgewater Scala notebook github project and then I also made a public project on Domino that lets you sort of like try it out and one click so it's public you don't need to sign up or register or anything like that so yeah thanks a lot you