SF Scala: Marek Kolodziej, Spark and Databricks Notebook at Nitro

SF Scala: Marek Kolodziej, Spark and Databricks Notebook at Nitro

Recording: SF Scala: Marek Kolodziej, Spark and Databricks Notebook at Nitro

I guess this stock is going to be very different even though it touches upon the notebook feature a little bit as far as the data bricks implementation is concerned um I guess this is more of an overview of what Nitro is planning around smart documents and the notebook application is more incidental but it's definitely an important Showcase of why notebooks are important um okay so uh this is just a quick overview of what Nitro does because it's relevant to this particular application so a long long time ago um there were desktop apps and um Nitro still produces one and it's very good it's a very good replacement for Adobe Acrobat but at the same time everybody's moving to the cloud you know we have Google Docs Office 365 Etc right so so that is basically the direction where things are going and the notebook is just one one example of this so Nitro switched over to the reactive platform by typesafe so it's a play and and now we have document processing on the cloud so the benefit of that is that instead of just having the users isolated with their data silos with their individual documents on on the individual machines we have the ability to finally mine this data and and and generate solutions that help the users but to to be able to do that to have any machine learning applied to to documents we had to get the data first right so this uh this movement to the cloud was really critical to this direction in which Nitra is going right now So currently you have documents sharing e-signing collaboration approval so essentially graphs of documents that move between between different stakeholders at the company and this is all cool um okay so where was I so the cloud may be cool right for for sharing PDFs and having a graph of people that need to approve the PD the you know some contract or whatever but essentially this is this is purely a standard engineering problem right how do you scale the cloud but the next the next thing that uh that Nitro would really like to deal with to make documents smart and what does that mean right that's that's just the slogan so let's be a little bit more specific so let's say you get a non-disclosure agreement and most of these agreements really look the same right there's just a lot of boilerplate in fact there's probably some template that somebody wrote many years ago and and everybody just keeps keeps copying and pasting and just changing a small specific section but eventually you probably get to a section of the NDA where you have specifics right so accept that specifically authorized here in and you may have some details as to what that actually means so and if you get a lot of these documents every day and each one of them is several Pages then you would rather Focus your attention on what is specific as opposed to the boilerplate so this is just one idea right boilerplate detection looking at some some NDA or RFP or whatever and and trying to ignore the the repeated text and focusing on whatever is specific to that particular legal contract language detection um You may wish to sort documents or or place them in different folders depending on what language they're in topic mixture detection um you know some documents may be about business and pleasure and you want to understand you know what the what the mixtures are document category classification right is it an RFP or an NDA named entity recognition so knowing that you know this is this is a proper name for example document graph analysis so understanding who sends this document to whom and who approves it might help you know recommend a particular actions to the user um related document recommendations there might be other documents floating around in your organization and and and if you are actually privy to accessing a particular document but you may not be aware that it exists the the system should be able to recommend that to you sentiment analysis right if you have a very very long document you should understand whether whether a person is approving disapproving you know positive or negative about this particular document question answering so um I'm sure most of you have seen how Watson operates so this is essentially this principle and you know to be fair any natural language processing algorithm and for that matter any machine learning algorithm and any probabilistic algorithm has an error rate around it right so it may misclassify it may assign some element to the wrong cluster so clearly it's not a solution that you should be basing your you know legal decisions on for example that said it's still extremely useful because it can be it can be simply a tool to help the reader to guide the reader right so so at the end of the day you might you may have the override writing decision making power right as to whether you're signing this contract or not uh it's it's it's your decision not to read certain sections of the document or read them but if the the overall um application allows allows you to focus your attention on particular pieces of information because it it believes that they're more important well that could potentially save you time and focus you more I mean even if you read a whole document that's 40 pages you may actually miss the relevant sections because you spend so much of your mental energy reading the boilerplate so it's still good to have some some help from from NLP for example um so the thing is you know if you're if you're a cloud platform and if you are getting millions of documents from different people eventually you'll you'll get into problems right because you're not just a data scientist working with r or python you know on a single node So eventually you'll probably have to uh start using spark or Hadoop or you know in the in the olden days it was MPI so big data is actually not that new but whatever the technology is You may wish to distribute your your computation and not just in an embarrassingly parallel way where you know each node is processing something different but you may need to do aggregations which is where the whole idea of mapreduce comes in and Spark is just a fancy new way of doing mapreduce in memory about the the model the computational model is essentially the same so uh should we proceed with Hadoop and the Java mapreduce API well those are how many of you have actually played with Hadoop cool so so you probably know the word count example where you you know extend mapper interwritable and you know and so on and basically right all this ceremony just to get the job started right that's not necessarily a Java problem because uh there were nicer apis for Hadoop like skating to deal with this but nevertheless this is just so much boilerplate um should you actually go a little bit higher if you're focused on machine learning and use Apache Mahood um well first of all it's it runs on Hadoop so so it's unfortunately pretty slow because Hadoop is great for batch jobs but unfortunately if you have an iterative algorithm like logistic regression which takes 100 iteration iterations to converge each of these jobs will basically require you to read from disk stream the data through the jvm and then write to disk again and then for the next iteration you're doing the same old thing so so Hadoop was never really meant for iterative computations and by the way the mahuda API is pretty crappy now they have a nice scalar DSL which is pretty cool they have a linear algebra DSL for example in Scala but and they're actually moving to spark but but originally it was it was difficult to use and first of all it was very slow um so yeah spark through the rescue um I actually have a story from two companies ago where I uh was running kimi's clustering on a particular size data set and it took about an hour and a half and a half in mahut and Hadoop and the same job took about four minutes on spark so that tells you you know what kind of a speed up you can get uh anywhere from you know 10x to 100x but first of all from a from a developer point of view no ceremony right no interwritables and mapreduce bass extensions and whatnot right you just say okay it's just the Scala collection essentially I flat map over it and then reduce it by key and that's it so reduced by key is actually the main distinction between Scala Collections and and Spark because um um the this is essentially a key value operation just like you would expect on Hadoop for example but look at how concise this code is it's it's it could be operating on petabytes of data and you really don't have to think about it well you do if you're implementing a very complex algorithm because you have to know what's local what's not local right whether you're doing um partitioning properly something that that was also a concern on Hadoop but at least from a programming point of view you don't have to think so much about the ceremony and to be fair to Java 8 the spark API for Java 8 is still a million times better than the uh Java mapreduce API in Hadoop and even if you were on Java 7 and you didn't have lambdas it would still be way more concise than than uh than Hadoop um and as I mentioned you know the runtime for many algorithms is is up to 100x faster for iterative algorithms that may not be the case for something that's not iterative like naive bass but for something like logistic regression where we actually have iterative uh calculations uh it makes a huge difference but the thing that really is phenomenal about spark is the fact that it's a whole ecosystem so let's assume you're in Hadoop plan right and you write your Java mapreduce application then you realize okay I want to do relational processing so I want to cut out some boilerplate and I'm going to use Hive then the next problem is graph processing okay I'm going to use graph x then your next problem may be real-time processing so you're gonna go with storm and the problem is first of all there are 10 different apis for 10 different problems but the second problem is you're also not doing data sharing your only data sharing is through hdfs with spark you can pre-compute some batch job using Spark save that data in the cache in memory and say okay now I'm streaming data and I'm let's say doing predictions based on the model that was trained in batch and you don't you can basically seamlessly switch between a batch application and a streaming application and it's all in memory and you don't have to figure out oh am I going to export the model using pmml from the bad job and then do scoring you know using using another easing storm you know it's all very straightforward also mlib seems much more complete relative to Mahood in terms of the the algorithm coverage and it's definitely better suited because of the the performance benefit for iterative algorithms Graphics is also really nicely integrated so again you don't have to think oh I'm going to use graph lab or giraffe or something else for for addressing My Graph processing problems and Spark SQL is definitely much more directly integrated with spark than than Hive was with with Hadoop because you can essentially do a projection using using SQL get your rdd projected or filtered and then you go back to your regular spark jobs um so basically as an ecosystem it's much more tightly integrated and the interoperability is much nicer okay so so um the classic example of of a mapreduce job on Hadoop is a word count but that's kind of boring so let's do something a little bit more interesting so let's say we're looking at millions of PDFs and um and we want to figure out which fonts are are the most popular that's actually a serious business problem because if you have thousands of fonts and many of these fonts are proprietary you have to license them so if some thought occurs you know uh 0.0001 percent of the time and some other fund occurs 99 of the time you really want to know which fonts you should license and which fonts you should you you might be willing to substitute because they're just not very popular and you know that they're close matches so this so their actual business problems that are a little bit like the word count but they're a little bit more complex because for example you have to parse the PDF document which is an unreal problem so let's go through all the PDFs keep account of the characters rendered um in each font in each PDF then aggregate the font counts across all documents and sort by frequency so right I already mentioned the business value so I have both the slides from the databricks cloud and actually have the data X Cloud running so let's try looking at it here um so it's it's very similar to the Scala notebook so I'm not really going to go through uh through all of that here and I'm probably not the best person that I I leave it to to the data bricks folks and there's some really good presentations on YouTube or where they they showcase their product but um um I just wanted to show you how how straightforward that is and and it's very much in in the same um you know vein as as the Scala notebook so you do your Imports here you know Define some functions to actually process the PDFs because you have to extract the metadata from the PDF so here I'm using Apache PDF box um by the way for those of you interested in PDFs you should not use Apache Tikka which is a wrapper because Tika provides a generalization over p over PDFs HTML and many other formats but it provides a match uh more restricted API for PDF so if you want to figure out what fonts you're using for example Tikka doesn't provide that whereas PDF box does Antica just wraps PDF box so you you definitely want to use PDF box if you if you want to get um um access to all the metadata now spark actually has one cool thing which is if you're if you have Legacy Hadoop uh input formats um you can basically reuse them for spark so that's one way to to basically integrate all your previous parsers and in fact I wrote this um PDF parsed box writable which is just like the long writable and introitable and Hadoop and so on and then there's the PDF box input format which is just like the text input format on Hadoop essentially and in the case of of this application it just emits case classes so so once you get to your rdd and and which is the resilient distributed data set which is the abstraction on spark that essentially is is the spark equivalent of a scale collection you just get case classes of these documents so so it's it's very high level at that point so you have to get your PDF stripper that's that's an Apache PDF box API you normalize the font names because some of them are just a little bit mangled so essentially like with any data science problem you have a lot of data wrangling but then again if you if if you think about not just data science but Engineering in many cases you have to do data wrangling too like parsing log files or something right so so there's a little bit of that ceremony there um and then you finally get font stats so you get the characters for the article you know you can use your your nice Scala way of doing things um and uh I don't think I really want to go through all the code because you might not be interested in the details of PDF parsing um but um you basically I made these page stats for each PD document which is the PDF box representation of the PDF so uh let's see yeah so in the in the page fonts in the page font stats method essentially what what you emit is the font and the count of all the occurrences because the way PDFs are encoded is you have the text position where where a word starts and you have the font which this which is is annotated for that text position and um you don't have any more annotations to uh till the end of the word and once you move to the next word you again are told whether it's italic or bald or and which font it is in so you basically have to group these text positions by font and and and that's just like group by in on on a scholar collection and once you omit that you you essentially just just just do uh do summation right and and you do that for every document so that's the embarrassingly parallel part because you do it separately for each documents so you can just do a map on the cluster right that there's no communication between the nodes for this operation because that can be done separately for each document so it can be paralyzed but eventually you have to actually do the aggregations across your collection of all of your PDFs and this is where the reduced phase comes in um in both the Hadoop and the spark case um so I'm going to get to that in a second one thing to to note is we definitely want to Cache this original original data loading the moment the data leaves hdfs or S3 in the case of Amazon you basically want to Cache this data set if you have enough memory because that's where where the spark benefit comes in right you can do multiple queries against the same cache data and they'll be much faster um so basically I just have this wrapper it's not particularly interesting there is my font the count of of of of characters for the document and the percentage of total when I'm doing the final aggregation across all documents and um the main spark job basically folds over all of these documents so so there is a map for each of the document that that shows the font name and the count for that particular document so you have to do a fold over the entire cluster right over all of your partial computations from each node but that's the beauty of spark right it takes care of that for you and you really don't have to think in low level terms it's it just looks exactly like a scholar collection um and then you can do a sort um the the good thing is that because we did these partial aggregations we are actually not doing um a sort on an enormous data set anymore because there's a limited number of fonts so this sort is is actually on a small data set at this point doing a global sort um on both Hadoop and Spark is extremely expensive right because you have a lot of communication to to sort the data across the nodes but this this data set that submitted is actually very small at this point so it's really cheap to do and um and since you actually have the total of number of characters and the total number of characters on a profound basis you can actually calculate the percentages of the total characters across all documents um and uh and since this is a a notebook type interface um you basically have spark emitting um results or basically that this the spark Rebel which is just an enriched version of the Scala Rebel and um okay so we're getting to to um the benefits of of of of the notebook interface here so once we get this um aggregation um actually when you call uh uh let's see when you do this fold it it no longer is an rdd it just becomes a scholar collection and so I need to uh paralyze it convert it to an rdd again because um and and the data bricks World um uh basically you want to register this temporary SQL table that um allows you to query the data and so so this is a very small data set at this point after the aggregation so you can call paralyze and adjust converts it to an rdd um and databricks provides away of of querying your your attempt table just using regular SQL because it's essentially spark SQL um so so it can show you um these percentages and counts and font names um because that's essentially what you got after the segregation and um you can you can view the bar charts or whatever you want right there's there's quite a few uh there there's quite a few graphs that you could do so so this is just an overview of of how easy it would be to spark I assume you could probably plug in spark in a straightforward manner to the skeleton itself and there is a project called spark notebook right which which works like that as well um but what's the benefit of all of this right so so you know many data scientists feel like okay I just want to run run my you know Python scripts in spider ID or whatever or or pie charm for that matter if they're more engineering oriented um and uh and Engineers definitely feel like oh this is not great because I don't have the IntelliJ level of of you know static analysis and I'm not really under proper Version Control it's not good so so so what's the benefit of this well there's this whole trend of reproducible research right now in data science and and you really want to be able to for other people to go in and and very quickly figure out what's going on to have not just the code but the precomputed results of of the analysis and again it's not a new thing because Mathematica had it in the 90s but it's becoming more and more popular because you know data scientists are no longer really sitting in in Ivory Towers right they're they're becoming much a much more vital part of of the Enterprise and they want to communicate these results to business people right and and things like you know microstrategy or you know pentafo all of these bi dashboards don't really give you machine learning capabilities so so what's what's a nice compromise between you know your code sitting in GitHub and and and and and you're working in an ID on the one hand and and say uh a bi tool right while still retaining this flexibility of actually having proper statistical and machine learning algorithms at their disposal well it's it's a notebook and and I think Mathematica got it right in the 90s and you know with IPython notebook and this the Scala notebook and and the data bricks Cloud it it's just becoming very popular because it really has the power of communicating the results that the data scientists produce to uh to a wider audience and um and I basically most of these notebooks have a feature where you know you can actually embed markdown and you can embed latex and and and and images so so it's really kind of a multimedia setting right because uh depending on your problem you may use different languages right so whether it's SQL or markdown or whatever it should all be sort of seamlessly integrated and um I I I really don't think that I'm the right person to talk about the databricks cloud in detail because they're experts who actually built the platform but I just wanted to show how how it could be potentially useful for for the the big data science questions that remain um so I guess that's about it I just wanted to follow up and say that uh so the databricks coil itself was then on the spark Summit last year only God see is the Project Lead and he basically uh agreed to give a talk at this call about later Rick's Cloud architecture itself in one of the future without criticism right so basically there are actually improving the whole product line and they want to share it with us Ali is a great guy so he will come and share the roadmap and are there to integrating all their features so this will change you know in the future yeah one thing that I definitely found nice and the neighbor X cloud is the ability to embed HTML directly so for example and I know that probably some people are going to say okay what about cross-site scripting or whatever but if this is an internal research tool and this is not a you know outward facing website but it's a way to communicate stuff internally I think it's actually very powerful because what if you know the databricks cloud doesn't have some crazy d3js visualization that you want and you want it now right you know maybe maybe uh maybe there there should be an option to disable it but I think it's actually very cool in in my opinion because you can just integrate your new JavaScript library right away and uh and uh you know and and to embed any other features you really want so um obviously any company has a limited bandwidth that addressing every customer request so the ability to actually customize it yourself is pretty powerful but I guess that's sort of the general perspective of the notebooks right that the same is the case with the the IPython notebook you can you can customize it pretty easily and and um yeah I generally I'm I I think it's a really good way of communicating the results um I would rather I would much rather sit in IntelliJ for for like basically developing the algorithm because the ID helps me but I still want to communicate the results and this is the most powerful way to do this so that would be a question today diverse yeah but yeah well I think the autocomplete for example that this that the um scar notebook head is already very helpful um I'm not sure what the general plans are I think it's a little bit complicated when you actually have a rebel because you know the rebel has to essentially compile your code and then do a class loading and you know it's not python right so you're not just evaluating on the Fly it's not going to be very quick because you have to invoke the Scala compiler to figure out what's going on and that can be pretty expensive um unless there is a way to make the rebel basically work the way the way the the fast Scala compiler works that's true yeah right right yeah I I mean it's a question to data breaks I'm sure they're that they already request got that feature request from other people I think that would be great um yeah I I don't know so I have a question for you um you showed us two types of technology for smart documents the Nitro technology the the native bricks technology what does the databricks technology mean for Nitros um business proposition I mean if these are the smart documents of the future how would that you incorporate them into venture right well so so the premise of all these notebook interfaces is just to enable a fast and convenient communication of the research results right and the research results never really go outside they're internal right because if you for example figure out oh okay I know how to classify these documents better or I figured out how to model topics appropriately right this is the kind of stuff that you share internally with your platform team for example as a data scientist so that they could incorporate it into your software as a service platform but this is not really visible externally right so so why not well it could be if you're basically willing to share your intellectual property with everybody else that's different from open sourcing your code base I think because that's where the secret sauce is like if you have tens of millions of documents it's not necessarily the the value is not necessarily in your source code which you could open source but with the fact that nobody has access to that many documents so you could actually learn uh so you can actually let your machine learning algorithm learn more about how documents operate and and of course you know just just being able to ask the right questions featurizing your data appropriately so so this is the kind of stuff that is that is usually part of the any company's ID I would assume and uh so that's why it's more more internally facing right so actually I think we're going to learn a lot from scholar mobile and the difference Club because some of the products we think about this reactor so we're reacting and this is you know our teacher the texture reflects a you know initially attack size tag but it also reflects the nature of the documents themselves right so do you think of the PDF it's a static document which is kind of immutable but you know when you need to correct an offer letter or a contract you actually interact with it so ideally we want to have this fluid interface where illegal conduct some suddenly becomes mutable something you can click on on a piece of it that sounds like a mutate for instance it's a one-way NDA you're going to say in our recommender system actually says a lot of people ask one of the way it is to be changed to two-way engines right and this will actually offer this option to you and you actually should you know see a drop down menu you should update on the fly so it will be a reactive document so the JavaScript is unwielding right so that's how it's called develop around the Prototype how it works in the UI and kind of we've been full stack over full developer right I need to do that uh and and uh we want to have an ability right to to you actually reactive we just don't want to learn from skeletal book Revenue soluble tricks it has a lot of experience I should dream exactly you know you have cells you can easily see how you break it up into this kind of cells so we definitely want to learn from this right it's yeah yeah this technology inspires what we want to do in the UI we'll probably want to do this called JS we want to stay in the scaloscope but the behaviors are certainly the kind of you know the heroes are what we want to use so I would say the goal is a little you know let's go to look like it's the food in yourself right like if we have this Dynamic environment referral documents internally you know and why does we learn how to do this right so so if you distinguish the findings from the data science effort which are probably the company's IP from the inspiration that we can get from these interfaces then then yeah that's that totally makes sense any other questions who wants all that for what well so going back to the font example we want that right because we're licensing funds from other people so if somebody if if use one particular font one in a million documents you may not be willing to actually license it because it's a waste of time and money right um and you may be willing to substitute it but if some fund is very popular then you have to be as true to the original as possible so in that case it would make perfect sense so that's business internal business optimization with regard to things like um like you know Finding boilerplate in documents you know I I I have to read so many contracts right and and I don't even know where to start because like the classic example of the Apple license agreement right everybody clicks agree because they don't have time to read this stuff but if and and and again you know this is not a replacement for human intelligence because um every natural language processing algorithm has an error rate attached to it so it will never classify you know with a hand it won't it will never have 100 recall and precision but if it at least helps you decide that you know maybe this is the section you should pay attention to um you you might choose to read the whole thing anyway but if it focuses your attention that definitely is a Time Saver or at least or at least if you imagine reading 40 pages of legal stuff and then you basically by the time you actually get to the specific section that you should pay attention to because it's not boilerplate your attention span is already reduced because you've read all this boilerplate you may actually miss it even if you decide to read the whole thing so having some kind of um uh you know intelligence in the documents should really help in that case um so okay so so other examples let's let's just not look at PDFs for a second let's just look at something like Google Docs sorry not Google does Google news right so so if you look at Google News it actually shows you the most important news of the day but if you click on a particular piece of news it finds all the related pieces of news that's document clustering nobody actually goes and curates that right Yahoo tried curation in 1995 or whenever and and that was a lot of work right so so so basically this grouping of documents is uh is is done using a clustering algorithm um uh you know obviously from a completely different domain spam filtering right this is a usually even in the 90s it was a knife-based classifier it was it was a now it's probably an svm these are all machine learning algorithms so so so some kind of intelligence and text processing has been around for a while and um and you know there there are tons of applications for business documents that have not been tapped by Google or Microsoft so far they do collaborative editing online but they don't really Aid the reader in any way Gmail is a great example too right because it groups it it groups your your threads and it um you know has pretty intelligent search capabilities so so if you care about making your email processing more efficient and you're still reading the email you're not just letting the system make decisions for you you know there was this um April Fool's Day Gmail video where they said you know they will actually do intelligent auto replies but you know we're not there yet and I don't think we will be but um anytime soon but but if if you expect intelligence out of your email system or um you know many or or you for example have you know Google translate yes that translation is very crude but it can still help you understand a document in a language you don't know um if we expect that kind of intelligence from other documents and and sources of text that we're consuming then then surely there are similar business cases for for the documents that actually matter because you're signing a 100 million dollar deal and you have to read a document right or or you know you're just flooded with paperwork and you have to figure out how do I optimize my time use so I don't have all the answers the you know that this whole office just got started recently so so we're actually looking at these cases but uh you know there are definitely lots of opportunities like that or actually provide suggestions if you have a problem that you're dealing with then we'll try to implement it you have questions yes guys that's fine