BDSBTB 2015: Thomas Gerber, Index of all US Small and Medium Business with Scala and Spark
okay so hi everyone as I said my name is thomas i'm a big data engineer lee that radius and today we're going to discuss how at radius we created a graph of all US businesses in spark and scala and have a confession to make nine months ago I didn't know much and actually nobody outrageous knew much about spark and scada we were a Hadoop and Java shop and so this talk is really about how we moved from Hadoop and Java toward spark and scatter and let me start right off with the key takeaways I would like you to go home with the first one is spark and scada have an amazing synergy and if you consider using spark I encourage you to use it with Skala as opposed to Java the second thing is if you are a current Hadoop user you might consider transforming your next big Hadoop refactor into a migration towards park and scada you might get a lot more out of it as what you get today with hadouken java including more performance more scalability more capabilities out of the box we spark but also you might notice that it will is a new project in sparking Scala is easier to maintain you will ship feature faster and generally speaking you can have more engineer working on that project so a couple of words about my company and why we are trying to create or we're sorry why we do create this graph of all US businesses radius helps marketers discover new segments acquire a new customer and measure success and it does that but provide by providing a big sorry a data science engine that does perfil predictive analytics and segmentation based on our customer CRM data now radius was founded in 2009 we are over a hundred now we recently closed 50 million in CSD that's over 125 million toll and our customer includes dipoles American Express I mean customers that need to sell to a lot of customers and for which CRM data is obviously critical we are very happy in that science and Big Data technologies now back to the business graph what do we need it for remember that what we're trying to do is enable CMOS and VP of Sales finds the segment's where there will be performing well and one naive approach I would say would be to do predictive analytics and segmentation on their row Salesforce data unfortunately that is only going to be as good as their CRM data and as you might know CRM data is not very good because it's not up to date it's not accurate and it's usually very poor in terms of signals and so the way we approach this problem at radius is 3 steps first step we create that list of businesses in the u.s. that graph and we try to keep that graph up-to-date so that means weekly builds accurate and exhaustive and with as many features as possible like what the revenue of this company what's a head count of this company is this company on Facebook is this company and Yelp a lot of things that you wouldn't find in a typical CRM and so really this is what I'm here to talk about this list its graph of businesses then step two we match our customer data with that list and now step 3 we do all our predictive analytics and segmentation on our version of the customer data which is richer which is more accurate which is more exhaustive and all of a sudden by construction the accuracy we get on proactive analytics and segmentation is way higher so that works if we manage to build this graph of us businesses right and so I'm going to talk to you about how we went from doing that in Hadoop and Java and how we migrated that in January to spark and scallop so before in the Hadoop and Java world this is how we would do it so first as an input we have we had half a billion records those records would be things we crawled onto web data that we paid for public record data as well and those half billion records represented roughly 20 billion data points and so first we would standardize it and then we would group records that we think represented the same business together and create a business out of each group that represented roughly 15 million businesses this approach worked but it had several limitation the main one was that our code base was large and with a steep ramp up we were using cascading and cascading helped a lot but even so it was challenging and we only season java developer could really work on that project and that is after as I said a long run rabbit time before they could actually contribute anything the project has a had a tie up keep maintenance costs and we were not shipping as many features as we wanted and so we knew that a very large refactoring was looming at the same time we were looking for two other things we knew that instead of just grouping records together when we think they are about the same business we wanted actually to be able to model the relationship between those records and so we started looking around for graph processing framework also we were imputing classifying doing regression on more and more data points to improve our graph or list and we know that we also needed a machine learning framework and so very naturally well we looked around and here was spark that was December and we thought hey let's give it a shot and so we started to do migration pilot and what we found out was very nice first off spart brought a lot to the table and delivered pretty much on all the promises it delivered performance I delivered scale we had no problem at all processing the data that we already that we were already processing Hadoop and it also delivered with graphics and MLA which are really outstanding libraries and since spark has seamless connectivity with your Hadoop ecosystem we didn't have to jump through hoops to get started we would easily read from a three right to a three so far that was our use case fair enough and then something really cool happened spark also brought us Scala and to be honest at first we weren't a hundred percent sure if we should use spark or sorry skala or Java we spark and we were at that time as I said a Java shop we only had Java developers and well for the pilot we decided to go with SCADA because Park recommend skala so that's what we did and what we found was really cool because scala as you know is a functional language in the Java ecosystem and that meant direct benefits from us first off because it's a functional language so total number of lines of code that you write a way lower that means that the potential for bugs is lower that means that the amendment cost of our project is now way lower also from a spark perspective from a big data engineer perspective skala is very WYSIWYG in the sense sorry I'm boring the term from front-end engineers here but the idea is as a big data engineer when you design a pipeline in your head well then it is in essence if you spark the implementation as well if you know that you need to do a map than a filter then I'd join the net map well that's four lines of code it's a map a filter a join and a map and so that means several things like faster shipping faster prototyping and so forth and you're still in the Java ecosystem so you still have the reliability of the JVM and you still have access to all the libraries and connectors that you used to and so Scott I really felt like the right tool for the job it really felt like a hammer to hammer down that nail as opposed to a screwdriver you can still do it with a screwdriver but it's not going to be as effective and I would argue that even with Java 8 and lambdas I still think Scala is a better choice because you still get the ripple you get case glass case classes you get object and so forth we saw a very fast adoption of Scala outrageous first off big data engineers didn't have any trouble picking it up because it allowed them to express very simply the complex transforms that we're designing so no problems there skala developers that we started hiring also don't have trouble we speak data and spark in particular because they can treat an RDD just as another collection where they do filter flat maps and maps and we saw this kind of virtual cycle virtual cycle starting where the better you got at scala so better you got out spark and the better we got at sparks the better you got at so that always really really nice they feed off each other and a couple of tips I have for companies that start doing skala if you want it to if you want to have a fast adoption is number one keep it simple meaning try to avoid developers that are clever you don't try to be as simple as possible for newcomers meaning avoid using implicit keep your SBT very simple and so forth the other thing is there are very there's a couple of very important in diems in the functional language world and in the scallop world they're not necessary is that complicated not necessary that numerous and so well it's you should have your developer learn them it's actually quite entertaining and some thinking about options as collections I'm thinking about pattern matching and so forth once you have that you will notice that usage of spark within your company especially on the big data side will grow very quickly now for us moving this project of creating this graph of us businesses the migration took three man months we did that in January and three months to migrate over a hundred thousand lines of code they think that's pretty cool and the resulting code base was five times smaller for the same thing and when I say migration and means everything from implementing to testing that it was working properly so that's great and if i look at the rewards we have now we have five times more engineers working on the code base and before and that includes data scientists think about it before we required season java developer to work on the same hydro project data scientists that don't have a formal developer training can now help and contribute to the project awesome for us also we measured that engineers are now releasing twice as many features as before now in in spark and scallop and graphics and ml liebe helped us improve the quality of our graph dramatically the best testament i think to the success of this migration is that the the teams that builds that graph of us businesses is now in the same really cycle than the team that does our web app in the past we those two things were separated you had the data teams that were creating that graph that worked independently of the web app guys that creates a web app and the backend that allow a user to go through segments and stuff like that and the reason we were separated it's because first off we were doing what we thought was the lowest hanging fruit I would say but it was based on constraint of the environment now once we moved to spark and Scala we were so predictable in how we were doing releases we were so fast and and so accurate that everything has been consolidated into one group I would say and with one road map with one release cycle and that's amazing for ages because it means it can focus its effort a lot more than before finally all of our big data teams we have several moved to spark and Scala for two reasons one we're very successful with this project and second we also started using data bricks as notebooks to help exploration of data and stuff like that and that helped convince everyone that spark and Skala we're really the right tool for the job and when we look at what we do now in spark and Scala first of all we have because of the scale allowed by spark we process ten times more records now than before we have four billion records that represents 150 plus billion data points and meaning we now feed every single historical records that we have about the business in the build to get even richer output and number one we do more fine-tuned standardization this is possible because we can have a lot more engineers working on that project and that allows us to do very smart things around string similarity functions around address standardizations I mean a lot of complex problems like that second instead of just grouping records when we think that belong to they represent the same business we can actually build a graph and model model eyes the relationship between those records this is very powerful because it allows us to do smartest things in when we cluster records and third because of ML lib and the easiness to impute classify regress data point we get as an output businesses that have way more data points than before so let me say again so takeaways for this talk ah spark and Scala have an amazing synergy I really urge you to consider using Scala which park as opposed to Java you will get much more out of it and if you feel right now that you had to project has a high maintenance costs or you're not happy with the rate with which you can shape features I encourage you to try to migrate to spark and Scala thank you for your attentions radius is hiring a but we have a lot of fun things to do and if you are interested you can come and see me after the talk and if you have questions please don't let it tight thank you well awesome Thank You Thomas we're running a little behind so