scale.bythebay.io: Roger Menezes, VEGAS: The missing Matplotlib for Spark
Recording: scale.bythebay.io: Roger Menezes, VEGAS: The missing Matplotlib for Spark
hi everyone thanks for coming hello yep I'm Roger I work at Netflix I'll be talking about Vegas it's a visualization library that we wrote in Scala for specifically the notebook use case so if you guys usually use notebooks with Scala like say Jupiter with Scala and have been looking for a visualization solution this might be the solution for it so I work in the recommendations team at Netflix Netflix for people who don't know it's a streaming video service that allows you allows you to watch TV shows and movies like for a small subscription fee and the these are some snapshots of recommendations that you as soon as you log into the Netflix experience you'll see these listing of movies and every aspect of the recommendation experience is machine learned so these are personalized and specifically created for your own unique taste additionally Netflix is in 190 countries so that makes it slightly challenging to incorporate demographics like country and cultural tastes to come out with personalized recommendations Netflix is also present in a plethora of platforms like either there's a client or an app for the Smart TV that you're using or iPhone iPads things of that kind and that also adds a lot more parameters to tune when we come out with these recommendations for you like if you've usually seen that people tend to consume a different set of contact a different set of content if they are on the mobile phone as compared to if they are watching something on a bigger screen so that adds it over there this is again just a screenshot of how the recommendations look at NetFlow recommendations look in Netflix website so each of these rows they serve a specific purpose and they are powered by their own machine learning algorithm and on top of these row algorithms there's a page construction algorithm that tries to reorder these rows in a way to optimize the subscribers content consumption and so as you could see like machine learning is an integral part of recommendations at Netflix and after having done recommendations for almost a day after having done like machine learning for almost a decade and Netflix what we realized was we were spending a lot of our time in the experimentation phase as compared to the production ization phase with products ionization I mean the point when your machine learning models are baked and you know what features you're using and you're using those models to generate recommendations that are served to actual users so whether that be axial production or in an a/b test if you have baked in those particular thing then you're moved into production ization but most of our time is basically spent in the experimentation phase and it looks something like this like a product manager would come in and basically come up with a problem which would be mainly improving recommendations for a small fraction of users that you're looking at so it might be let's improve recommendations for non-english speakers and then we would go try to explore the data try to identify what are the signals that we could feed into our machine learning model and once we have trained a model evaluate the models see how the rankings look like see how the Netflix homepage looks like with the newer model share those findings get some feedback from people go around that particular loop again and what we saw is in the experimentation phase what works really Waze works really well is the usage of notebooks I mean it's a great way of sharing ideas like visualizing the output of your machine learning models looking at metrics slicing them and sharing it with people like really is an easy process plus right now with all the tools that are present like you could do an end-to-end ml experiment see the results then have like then people might provide you with some feedback about I want to tweak this particular hyper parameter it rerun that experiment and see how the results look like it's a very seamless experience in that particular aspect so in the Python ecosystem the notebooks use case is well executed I mean if you search on the web for a gallery of ipython or Jupiter for Python notebooks you'll get like these beautifully crafted notebooks where people are trying to share ideas about some scientific idea that they might have and the two main reasons why it works so well in the Python world is mainly because Python has these well adopted scientific computing libraries like Syfy numpy like a scaler and things of that kind which makes it really easy and the other second aspect is there's a plethora of visualization libraries that are present in Python which makes it really easy to visualize your results but on the Scala notebook side like whatever solution you might be using Zeppelin jupiter-like SPARC notebooks the computing library gap is filling up like a lot more with like even people end up using Java implementations like through Scala or ML Lib is also filling up a lot of statistics oriented techniques that you would want to do Berndt right now there's a huge gap in terms of visualization libraries like so on Jupiter pythons Jupiter Scala site like we don't have anything on Zeppelin they have some inbuilt visualization but you really have to shoehorn your data to visualize the way you want to so mainly to fill the visualization gap over there like we we developed Vegas which is a visualization library in Scala and this is mainly to fill the gap off of visualizing things on the notebook so mainly what is Vegas it's a as I say it's a visualization library in Scala it it provides a declarative API contrasting it with an imperative API so the API is mainly tells the library like what you would wanted to do as compared to how exactly you want to do and that allows the underlying stack to make sensible like pickup sensible defaults for you and visualize things in a very creative and meaningful way and the other part that Vegas provides is like statistical operations so the main thinking over here is your data is present in a tabular format and then you usually just tell Vegas what do you what dimension or what column of the data you want it to be the x-axis what you want to be the y-axis and then in addition to that there are also statistical operations like filtering aggregation for setting I'll talk about what for setting is in the next few slides so it has support for statistical operations Vegas is built on this concept of visualization grammar where the idea is that we provide you a couple of blocks and then you arrange those blocks in a way that you want to come out with your own creative visualizations so I'll talk about and the visualization grammar is I mean it's based on this idea called as grammar of graphics which was popularized by this author Leland Wilkinson and the main idea or there is that your graphs or your charts are composed of these layers that you see over here and you can configure these layers the way you want to come out with the visualization that that you need so at the lowest level you would have your data you would tell the visualization library where my data is and then aesthetics is basically just saying like what the x-axis is what the y-axis if there is a color that you want to encode for a particular dimension that's the information that you provided this particular layer and similar these other ones to where geometries is basically whether you want a bar graph a scatter plot a bubble plot and things of that kind so it just comes down to configuring these layers and then coming out with the chart that you actually want so this slide is just about like what are the levels of configuring that every layer presents you will take a look at a specific example of this so this is how a Vegas chart looks like and there's this concept of channels in Vegas where so channel is any visual variable that you want to modify to convey something meaningful so like say x and y are a channel so you have some data and you have said that my miles per gallon column corresponds to the y channel horsepower as the X column you could also change the shape of the actual data points based on a specific column so you have used the origin column to come out with these different shapes and similarly you have again used the origin column to to pick up a different color for that particular chart so I'm just reusing the columns over here this is not something that you would do for your own charts but this is just to show like what are the things that are available right now and then similarly you could also change the size based on picking up a quantitative column over there so here is here's the same chart like it's it's computed using this particular data frame like you can see the various columns that are present over there and the only thing that you do in the chart is you just say my x-axis corresponds to this column my y-axis corresponds to this column and I want to use the origin column over here to encode the color bits and similarly size and shape so that's what it is so here's just things that you get out of the box by using Vegas like you can get bar charts scatter plots area plots this one is called a steamed graph and bubble plots so all of these are out of the box there's also this concept of trellis plots where you can take a chart and segment it into multiple sub plots and each segment basically zooms in in a specific set of data that you're interested in so we hear you have DVD sales versus worldwide gross and then you're looking at the movie rating segmenting on whether PG pg-13 like how does the data looks like there's also this concept of layers where you could stack a chart on top of the other to come out with even more meaningful ones I'm just thinking I'm doing so over here what exactly is happening is like you just have a bar chart and on top of the bar chart you have created these you have aggregated to max and created that as a layer on top of it and similarly you have another third layer where you could choose the q1 and the q3 quantiles 25 and 75th percentile where does that band look like and superimposed on the chart so that you can communicate even more so Vegas was built for the notebook use case it works great with Jupiter and Zeppelin you could also use it on the command prompt to visualize the kinds of data that you want like say file sizes and things of that kind it uses JavaFX to render graphics on the command prompt this built-in support for spark where where you could basically point to a spark data frame and then call out columns over there that you want to point to X&Y so that's the additional this also I'm just listing out like what's available so there's advanced pinning like you could take basically a column and then bin them into different bins to visualize the data in the right way sorting off scales if you want to use like in addition to linear scale like log scales or exponential scales like you could use those too there's a concept of custom transforms we're so ska Vegas has its own DSL where you could basically it this works very much like spark with the column kind of a concept in the sense you could point to a couple of columns and say can you create me a third column using these two columns which would be the diff of those or some formula that you want to provide so this support for time series like you could have columns which are based on temporal data like date days month things of that kind as I spoke about statistical operations like aggregation max min you could filter data before Vegas prints it out all of the math functions which I used which are available in JavaScript you could use those to transform the columns in ways that you want descriptive statistics like con tiles q1 q3 are also available I mean the way it works is you specify your your code in Scala Vegas then generates an HTML out of it and then it renders it inline in the notebook using an iframe actually I wanted to show some examples over here so if you see so this is a chart which is generated using this data where you have four columns like say the year age sex and people so now if you wanted to generate a faceted wants to say you could do any code so this is what I meant by faceting of charts in the sense like you could segment it on some another column and based on that you can create subplots out of it similarly things like you could add additional color and from a color information based on the year and here's another thing about how so over here like because you said that here is an ordinal column an ordinal is similar to categorical columns just that it has order information in it it chooses these gradients of color to come out with coloring for different years but say if you were to say this is nominal then it would just go ahead and choose different colors for it so it's these kind of declarative specifications that you could do to come out with these charts and and it's very easy to also say I'm just picking up this particular chart over here and then changing from a bar to a line chart would be so understanding the API is way is also pretty easy because you couldn't think about these graphics as various layers that you have and then what you are trying to do is just configure each layer with the customization that you want so say you could also do something like this and it comes out with these kind of ones so when I send out these slides basically there are these there's this sample notebook also that will send it out basically that shows like what's available in in Vegas okay let me just go back so this is this stack for Vegas Vegas is just a Scala binding on top of Vega light this is the actual javascript library which is doing the heavy lifting and so Vegas emits Vega light code or Vega light JSON which gets which Vega light in turn converts it to another great visualization library called Vega and Vega in term actually emits out d3.js code which gets rendered out and so as you go up the stack you'd basically see that the abstraction or the declarative nests of the API increases and as you come down like you have more power but things are much more imperative and you could you need to actually spell out like how you want the charts to be so the small code that you have in Vegas like gets expanded into this were ba's d3.js code that might be there so right now I just showed you like what's available in Vegas but Vega Lite has now introduced different levels of interactivity you like where you could just hover around a chart like pick up a single point and that and interactively you can get more information about it you could also highlight a specific column of use a column of data points there's another where you could basically do a paintbrush kind of an effort like you could just move your mouse your cursor around those data points and everyone in your path would be lit up so that you could take a look at that specific data points that you are interested in this is the list of contributors over here I also have Jeremy from Netflix sitting here yup so that was it Thanks thank you [Music] did issues for being a light they were going to had a pull request to bring in geospatial from they got up to thing of light it's after the two point release if I wanted to help have that to vagueness would I just implement biggest or anything like would you pull in Nevada so if they pull that in Vega light so we go light with every release they they also publish that JSON schema which is the one that we used to generate like the API for us so the only thing that we would have to do is like update the JSON schema and if you're willing to contribute at that level yeah and yeah you just increase the version number get the JSON schema and then you would have a newer version of Vegas doing that for you any other questions [Music] it's people have we haven't tried it on data breaks cloud but people have told us that it also works on data breaks cloud usually people end up using their own custom visualization and data breaks cloud but yeah I guess every notebook solution should work with Vegas we we mainly use it on Jupiter and Zeppelin [Applause]