BDSBTB 2015: Romain Rigaux and Erick Tryzelaar, Interactive Spark in your Browser
Recording: BDSBTB 2015: Romain Rigaux and Erick Tryzelaar, Interactive Spark in your Browser
yeah thanks a lot yeah so the point of the talk is to show the internals the way we did it ought to mix pack on a dupe in general available in your browser and just quickly before I start is everybody familiar with a dupe in the room unstuck on you witless good so just one side or the overview the goal of you you is a web UI to simplify make a dupe easier to use you can see it as a Swiss Army knife or like the web Excel for a dupe you can quickly cut tables like or within your browser do some quick visualizations with like no spark or like SQL on a dupe export the data shalik share it with your colleague so you don't need to go in the command line anymore and so on this talk so wise pack there was a bunch of talks also already on spark post park is a new way especially in hue on a dupe it opens up Maude's the data science people previously if you want it to clean up data in a dupe you you will use pig or you use a custom map reduce or even some type of SQL now it's much simpler that you can just do it in a regular scalar or Python lacks back screaming is really cool on you can do it in an interactive way press back is in a good way of replacing MapReduce like the engine of a dupe because it's much faster so after wise park while Hugh so this is a scholar conference so I have a little snippet of code or two like the top words in Shakespeare is nice since color I mean in a spark in the spark shell you can type the code on the command line is interactive but you nicer to have a cooler you I where you can do go back on check you can list your tables or even like do quick visualization and just within the same page and so why in you completely so Hugh is really based on top of a dupe to make it simpler to use so if you do some snippets of Spock with like some I've tables we will talk with them for you already you can also see the files Reggie HDFS fights you guys off I browser you can just click on your code you will open it up and is the same you have a bunch of wizards to cut tables so even if you don't know what to use a dupe on one you want to use a DD based on the life table you can just do it in a few clicks on also so Hugh is multi-user so you only need one server on people can log in with ldap user name password you don't need to have one installed like country to ipython for example 1 install of the software by user so it's multi-user and it's evilly based on yarn on web like impersonation in place so all the permission file permissions are related to the user with logged in in the web app on using already in a bunch of distributions so it's already there so it's easy to quick quickly story of the app so we did last year with Uzis which is scheduled for a dupe you could type some code he was slow it was just dumping the output more recently we play with o Allah web server so it was still a bit more interactive but you have to compile your code upload it so not really in Turkey like in the shell and more recently we did the notebook style with a new server that is really like a notebook on you can type interactive scholar Python on Eva ow I will describe quickly the web part on Eric will describe the internal a is on the scale a silver cross back if I have a notebook on the right on some snippet of code like a scholar on I've which is SQL for Hadoop producing you you had like one up by languages so we refactor read it to have only like one API for example if you have a spark snippet in skala you will ask it will send a card session to the you server and then a will call I don't have the mouse but it will call on the left like the levy the olivia has server to open the session if I execute a snippet same is going to call execute on then called the session post a statement on if I have another snippet like I snippet it will same called the exact same API but underlying Hugh the web app will call the ice over to API so there is no need to have like different code paths now with yourself only the specific API on the side I know Drake is going to explain more the internal holiday server morning everybody so Libby is our architecture and technology that drives the underlying heck here so Libby is a it's a rest web server that's built entirely on Scala it allows you to interactively send commands to livi and it will execute them and return the results but one of the the the things that we built on top of and gone beyond the the spark shell that we saw earlier is that we also have ways of doing type introspection and other kind of magic things where we're talking directly to the interpreter in order to extract out things for visualizations the other thing that we we have the ability to run our interpreters inside yarn itself so that Lily can can scale horizontally to support hundreds of active servers or active sessions we also have a local mood for development we also are able to talk to Scala Python are and it's relatively simple to add in different web servers and along the way we've also added a batch mode to so that you can submit batch jobs and upload jars and so if you want to save some workflows it's pretty simple finally it is an open source project it's being completely developed in hue maybe if there is community interests we might pull it out so please talk to us if you if this is something that's interesting to you so our architecture is that we're built on top of jetty and skull etre for handling the rest requests a client comes in and just sends a post command to livi to create a session livvie behind the scenes will talk to the yarn master and this is a synchronous process because you know the yarn cluster may or may not have resources available so we might have to sit for a little bit in order to get slots on some of the nodes but eventually it will reply and we'll have some notes so Libby will then spin up an interpreter running somewhere inside yarn and this instant this will also set up the spark contacts which manages talking to all of the remotes Park workers in order to do the distribute competition once this is finally all set up then we go back to the original client and the client is able to actually cute commands so here is the the actual rest interface hopefully you you think it's pretty simple on the left is creating a session where we just send a JSON object that just specifies the kind and you know one of a couple options there's spark there's pie spark or spark are and then on the right is actually executing the command where we are just doing simple addition here but we can do much more complicated things I'll get to in a second so the interpreter we don't really have too much time to really dive into all of tech technology but we wanted to show some actual skull the code because we're at a scala conference so the interpreters are these pretty simple devices like conceptually what we need to do is you know send some commands or code through standard in execute the code and get output this interpreter then executes codes talk to the spark and does all of the fancy spark stuff but in addition we have to be able to introspect into the interpreter polite information from it and do fancy things so we've designed these so that it is relatively simple like I added the VR interpreter I think about a week ago it took two days so it's it's hopefully not too bad and as we get further along it it will be even simpler to add in New backends so this is the actual workflow that you know the inner loop that's driving the interpreters that you know we receive a chunk of commands we split them and as we go line by line ignore that the magic part for now you know we'll be looking at the execute section where we actually will send a command a line to the interpreter try to execute it it could either succeed and therefore we go back into the loop and we process the next line it could air out or it could be incomplete like you have a statement like an if statement that has Open bracket that we haven't seen the closed bracket yet so we have to merge with the next line and and loop kin so here is finally some of the Scala code this is you know the the interpreter for like the bay straight for all of the interpreters it's it's very simple as you can see we just execute code and eventually we'll get a Jason value out of that after the future completes so to look at one of the real interpreters this is the the spark interpreter where you know one of the great things of being built on top of Scala and you know the spark ecosystem is that there's a lot of really great libraries that were able to take vintage of namely spark like they've done the really hard work of figuring out how to set up the interpreter and how to execute lines and and do all that that stuff and you know our interface is actually really quite simple so here the the real interpreter is this highlighted line the spark I main where you just have to pass into settings and some some buffer that will hold the standard output when the interpreter decides to print something after it's been initialized then we have to create the the spark context that holds all of the spark state and injected into the interpreter which is also just as simple as as binding this value SC that will be then exposed to the interpreter so altogether a very simple operation to do then to execute code I've kind of jumped a few steps ahead because there's some infrastructure that's needed in order to go from like the actual messaging format that is sent to the interpreter but here we're actually looking at when we're actually executing code and so you can see references to that little state flow from earlier where you know first we have to capture the output in case the interpreter decides to print to stander it out and then we execute the command with the simple spark I main interpret and you know we either succeed or incomplete or Arab very simple to do now on to the magic so this is where we pull out information from the interpreter or in order to structure eyes that data so that it can be parsed by by Hugh and rendered or by any of your clients if you want to have more information so this is an in-band magic man so you know the end user will enter in like percent Jason and then a variable name and it will it will convert the it will try to convert the that value into Jason or you know the other option here is a table command where it will try to table eyes the value so Rose columns and each column has a name and has a type that we pass along to Hugh so the executing of adjacent command is is pretty simple like once again we're able to take advantage of the this is a Jason 4s is that the real driver of this it's this execute decompose command that will try to convert a value to Jason or error out very simple and you know we'll return it as a application Jason mime type value we do have two special case our d DS which if you aren't too familiar with with spark and our d DS there are these lazy constructs that have to be evaluated in order for there to be any data so in this case we actually just evaluate and pull off the first ten items just so that we have something to show now to see this in action here's the the Shakespeare example from earlier where you know at the bottom you can see that I've used the percent counts and then on the right side is the responsive will be getting from from Livi you know it's it's pretty straightforward to do now the table magic is a little bit more complicated infrastructure so it's a little bit too big to squash into into these slides but here i just want to show how you know we transformed that data so on the right you know is our header format where we just have names and types it's hopefully very straightforward so that's that's libby like you know it's still in active element and so we still have a lot of work to do on stabilizing and scaling it up but it does work pretty well and we actually know some people in the hue community that are actively using this in their infrastructures and in their production environments so we really want to make sure that we're scalable and stable for them now Clyde area one of clatters biggest goals is to try to secure Hadoop and so as part of that that we really want to one of our main emphasis is on getting security inside Livi so we've already you know I I prototyped out adding Kerberos support last week and we're going to be securing the channel between the livi server and the livi interpreters pretty shortly so you know this is going to be one of our main goals before we go into into general availability and finally we've drawn a lot of inspiration from ipython and the Jupiter ecosystem if you aren't familiar with that it it is that one of the big driving forces in this notebook approach that we drew a lot of inspiration from so we want to emphasize like we've we purposely designed the interior protocols and the architecture to match very closely with ipython and you know how they do things and our intention is to try to grow even closer to that community and eventually be able to use their back ends so that people who have already been doing ipython work that they can very easily import their notebooks into Hugh and livi execute them and get the results back and you know very easily you know go into our system and leave like we think it's really great to work with this this community we might not necessarily see completely aight i but there's no reason we can you know share the same protocol and be able to work together so now i'm going to hand this off over to remain who's going to demo this in action ok so after the attack talk I'm going to do a little demo of living action in the huge notebook you I so this is a quick glad notebook why could you can put snippets snippets of values languages so usually we start with a Python notebook so the first time I'm going to click on the execute button like the TV is going to start a shell somewhere in the cluster unreturned the session ID to the notebook and then we'll just execute the snippet like in a regular shell so we are winning a shell in the cluster on it is wrought by Livy in the little interpreter that Eric described and we just pipe the input on the output so it takes a few SAR going to start but when you started is really fast if I start another example of with the introspection so so for example if I just print a Python list i execute it we get what well we'll have been printed on the shell as a text value but with with a magic commander Libby will return a JSON of the data that we can print as a grid on you as a list of widgets that will take a grid of data I will print it in a in a another widget so that was a Python example if i start up skala example same is going to take a few seconds to start the Shelly remotely on Wednesday Shelly started it would become really quick to interact with so it's like really quick if i play with the Shakespeare example of the beginning of the presentation we can see instead of having if I could show the list of the cons oops so it doesn't twins oh yeah he's a map yeah so it's not really useful so it's for this we use a magic comment on whether series of snippets from Alex color to python and we do the same with SQL SQL is a bit more advanced because it returns by default like tabular data that is easy to be plot on like several widgets and we have a bunch of other widgets except this one on each time we have a grid of data we can select a widget so even if we do introspection of spark of color we can use the same widgets like this on the last example is what we did like two weeks ago is our interpreter where you can also them like pixels so the server is going to return s like the plot of a lexan weblog data except we have a mistake okay I always keep this one and yeah on you can pick the snippets which you prefer if you want to do some pie spice Park or scalar quick so there was a demo thank you everybody if you have any questions are interested in working on this feel free to ask some questions will be air will be at the cradle a booth Iran good thank you