BDSBTB 2015: Vladimir Bacvanski, Developing Big Data Applications with Scalding

BDSBTB 2015: Vladimir Bacvanski, Developing Big Data Applications with Scalding

Recording: BDSBTB 2015: Vladimir Bacvanski, Developing Big Data Applications with Scalding

so the purpose of this talk is to give you an introduction to skaldak and in this talk I will not focus on featuring all the coolest and fanciest things that you can do with coding but I would like you to give you a good start so you understand the key ideas you will see some sample code that you will understand in detail and then you will have a pointers were to go on your own also the conference there will be another talk that will go into some more more deeper elements of a scouting so I said that I'm going to do some drawing actually so you will see in the middle of my talk I'm actually going to switch my computers i will start with my drawing computer and then i will switch to my mac to show you some code in the ID so let's see what is the origin of scouting and why would we need a thing like this so imagine your goal is to write some big data programs and the question is how would you do that and you have a couple of choices one choice is you could use the Java API it is a fairly low-level API it gives you really good control the problem is that this code is a little bit for both so you will need to do a lot of work in order to write your program so that you're not going to make us very happy in many organizations even when you mentioned that you are going to use some programmatic or Java API the data architects and similar they're just going to frown and they want to have their beloved sequel or something that would be similarly easy and the National parts of development of developing such applications is the invention of hive with the hive you have the language that is very similar to sequel it is not exactly sequel but it is pretty close and that the data architects get all very enthusiastic about it and then eventually they start hitting problems and that is the hive is actually quite limited in comparison with real sequel and you end up writing a lot of user-defined functions things worse you're writing this user defined functions in yet another language the Java with rather clunky API or a Python another problem with hive is when you write your blood queries they're extremely difficult to debug and sometimes you have the problem that you have to write this square is in a rather declarative fashion and you run it and then in the third hour of your Hadoop run something goes wrong you have a nullpointerexception you have no idea how to troubleshoot it so people figure out we need something that will enable us to a right program step-by-step so we can really understand what is really going on and that is big and a pig also has a problem that it is relatively limited you don't have any control structures you end up writing again a lots of udfs but it gives you a nicer somewhat programmatic way of development to order your queries however there are big problems so after the honeymoon is over when you start developing a real programs you start to desire certain things first of all you start desiring test-driven development your programs are getting larger and small examples don't work anymore the other big problem is that you would like to include your programs into conventional DevOps and this is just not working very well with either hive or pig so as a result of this consideration so we had the invention of scouting and to understand how scouting works you actually need to see what is its foundation so at the lowest level we have the foundation which consists of Hadoop now we have seen that using Java is very clunky with the native API is that we have four MapReduce but then we would still like to use a real programming language where we can use the test driven development then we can have the full DevOps and the right our big data applications so that's resulted in creation of a Java library called cascading and this is the fundamental part of scouting so in this library we have a mechanism that will allow us to access the Hadoop api's but in a very pragmatic very efficient way and read rather few lines of code of course from our point of view scouting is still somewhat low level we would like to have something nicer and what is very nice we have heard our scholar collections now scholar collections can be implemented so that they actually met their work to the cascading API i will tell you love is more about it in a moment and this is the scouting so the scouting will be using cascading api and basically we will have that the scholar programs will be translated on the fly so that the user cascading api which in turn uses the low-level hadoop api's now what is interesting about this today is that cascading does not run only on top of ordinary hadoop mapreduce but also runs on top of spark and what is particularly exciting for us today there is a project that is porking cascading on top of flink who has heard about apache flank the batch of link is the new approach to implementing big data systems where you have the streaming paradigm as the dominant paradigm so it's not the batch where you are then doing streaming with micro batches but it is the streaming paradigm which is the foundation and a stream that is a finite becomes your bad job this is very exciting so one of the very exciting things for us today and why scouting is so interesting is that it enabled us selective independence from our implementation infrastructure and this is interesting for some of our clients for example when you're looking into Hadoop and tez you would say that this is a very good choice where the data is significantly larger than the total memory then that you have on your cluster if you have a different setup so the memory is big enough you perhaps may want to go with Park so with a scouting set up we can change our backends by retaining the same code now one of the interesting things that you will see when I run some code is that the programs are organized in a paradigm that is using the pipes and filters paradigm so we have a tab which is our source and then the data from here is going to be piped you can imagine that you are building the real piping and each of these is going to be the element in your program you have a pipes that will just carry data you're going to find that there are some elements that will act like valves but you also have some other elements that are going to do for example filtering of some data and just pass certain things that satisfy your criteria and eventually you are going to have your sinks now what is also very interesting about these things is that you are not only dealing with HDFS and plain files but you have a variety of different inputs so for both tabs and sings you have HDFS you have also jdbc you have HBase you have Cassandra MongoDB and so on so what is really interesting is that you can build your applications that are actually very heterogeneous they consume the big data but they can also merge them with the other data that you have in your system so this is very important for many of our clients when we think about these elements that are actually doing the processing we have various capabilities that include things like filtering projections joins and so on and one of the really interesting things is that when you run this program the program is not going to be directly interpreted like we have seen in various other approaches instead what will happen is as you start executing your program the first phase is going to be what is going to be to build an access plan and that this is something that you're going to find missing in many of the Big Data technologies something that is perfectly normal in the relational world and then this knowledge was forgotten so what this compiler is doing it is going to construct the pass it is going to calculate what is the optimal access pass it can do some optimizations and then it is going to run this pipeline so this is very interesting all right so at this point I'm going to switch my computer and go to the ID and we are going to see some scouting code we are going to run it and we are going to see what are the things that are happening there okay so it should come up in a second there we go so I have prepared a couple of examples and here we have a rather ordinary scouting project and that the first project that i'm going to show you is the obligatory word count so what we have here is a word count programming scalding it begins with the class that is going to accept certain arguments and this class is going to extend a class job which is taking the arguments in this program we are going to first start consuming our input in our case this is the input that you're getting from the text line and what we have here is the mapping so when i run this i'm going to pass in the input parameter and i will be using the data set which is the book alice in wonderland so this is going to be my input file and then i will be using the output which is the ellis word count the text so what I do here is I say text line with arguments input this is going to open my input file and it was going to create what it's going to create a pipe so this is something quite interesting in the skull API you are working with pipes that are going to look like scholar collections and then you will be applying different transformations like in a completely ordinary scholar collections so the next thing that we are going to do is where we are going to do the flat map here now there is something interesting here something that you don't have in the ordinary scholar collections and this is this mapping so the line is what is known as the input field this is the text that we are getting and that the output of our flight map is going to be known as word and then we have the function that is going to take the line which is a string and just we are going to split it using the spaces separator so what we are going to have here is from the book we are going to have a collection that contains words the next line is interesting we are going to do the group by word and then what we are going to M it is the size of this collection so for every word you're going to have the list of all the words that have occurred in the text and then we are going to get the size and eventually we are going to do the right tsv tab-separated values and we are going to write this into our output so very simple a very similar to normal color collections there is one small difference that you're going to find and that is you always have the name of the input field and you have the name of the output field so when you add that you have functionality that is very similar to the planes color collections all right so let's see how we would run that over here i have prepared an SBT task that is going to execute our program so i'm going to copy this and you are also going to see something interesting something that is different from normal scholar on so run that and i'm going to open this and zoom in a little bit so what is happening here is when you submit your scouting program it is going to run using a job runner job runner is a small utility that is providing this initial functionality and then what you can notice here is that you have some info information so this is coming from the cascading this is coming from the java library the version that we are using and then you have the information about the pipe and over here we see what is the source it is a file type there is a sink which is a file tab parallel execution is enabled starting job and is a small example running on one thread what we are doing here is we are running scouting in the local mode and this is very cool because we can develop our programs locally we don't need to deploy it on the Hadoop cluster we can get feedback in a matter of seconds and later when we are ready for the actual deployment we can go to the three actual Hadoop deployment so that is a typical run of a scouting program and notice what is happening here we have immediately information about the source and sink and this is because the pass is first built and then eventually optimized and this info information is coming from the cascading before it starts actually executing the program a couple of other things that we can see as an example so another very simple program that shows this pipe lining in action is a simple file copy program that is going to copy one file into the other so we open the input as a text line we open the output and these are pipes and then what we do is we are going to start from the input pipe read from it and a result is pipe that you are going to write in our output sink now what is interesting here is that when you run this on Hadoop this is actually going to be the parallel copy that is happening across multiple machines but as we are using it we have the image of a fairly a simple program the other thing that we have here is a slightly more complex workflow where you can see some of the common elements of such programs so here we are going to show a program that is going to do a variation of the word count but it is only going to count words that are coming in from another file so we are going to count only selected words here i have hard-coded our input so input is a text line what you have here is a pipe selected words is another pipe and then we start building our code so we say selected words pipe we start from our input then we are going to read it we are going to rename the line which is the default name for the line in a file we selected word and we do that because in this file each word is on the separate line and then we are going to project selected word so what is interesting here is that as you read in the file the scouting is going to have the first column which is the line number so we are going to get rid of the line number and retain only the word we prepare our output the text line and then we start doing our join so for our input we are going to do the flight map as before we do the group by and then I'm going to execute here join with tiny I expect that the set of selected words is small and what is going to happen here is that when i run this on hadoop this small set of words is actually going to be copied to each of the machines in my cluster so I'm doing the join and eventually i'm projecting the word and the size and i write that in the output so see it is very easy to construct the follow it is a very natural for scholar programmer who is familiar with scholar collections what is also interesting and that was actually quite surprising you know normally you don't expect data architects in companies to to be enthusiastic about scala you know even when they see java they they just don't want to learn that they're looking for something simpler but when you demonstrate the scouting program they see that it is first only couple of lines of code it is fairly easy to understand and we had in one organization where very traditional dbas and date architects started learning scala just in order to use scouting and sometimes in organizations you are going to hear complaints that ask allah is way too complicated that you cannot have scala in this setting and that is not true when you think of scala and its introduction in such environments you need to think about three levels Scala is better java and this is mostly what we are doing here with very concise code then you have the introduction of the functional programming for that you will need some training otherwise the people are going to struggle and then there is the level where some people complain that scar is complex and this is for the development of complex libraries and then you can see quite elaborate signatures but what you need to understand is that there are very few people that will be operating at that level and they do it for the purpose that everybody else has easier life and have type safety so we have been introducing scala in in financial organizations and in order to be able to be productive scholar programmers we put developers in a four day training a normal java programmer comes in and that they come out as the scala functional programmer and then it is very easy to add a little bit of vodka to that and make them very so this is an example of the scouting program so what is nice here is it is concise it is easy to follow and this particular program here is using an API which is called the field API most of the scouting called other is using this API but there is a newer API that is not preferred the API and this is the typed API typed API is giving us better type safety and it is now the recommended API you cannot is here and that in the variant of our word count I'm defining what what is the content of the pipe that I expect to have so I say words is a typed pipe that has strings inside and then when I'm talking about groups I have groups of strings and strings and over here when I'm looking into the output typed tsv I have a string for the word and I have longed for D for my word count now scouting does not end here Scotty has additional libraries and one which is very interesting for us is a library that comes really handy for machine learning and data mining and similar and this is the metrics API so you have the matrix library that operates on data that are coming from for example HDFS files and this data can be sparse matrices and the matrix API provides all the normal operation that you will find in matrix libraries only it scales and it is handling the data in the in the HDFS scale and just to give you a taste of that here is one example where we are processing graphs we are using the tsv matrix that has inputs of users and how are they connected with each other so very simple and we are calculating out degree so what we do is we are going to read in the matrix and we are giving the names to the columns so we have a user 1 user 2 and the relationship that exists between them we are going to read it we convert it to matrix with these fields and then we can use operations from this rich matrix API and then eventually we write it out so I'm very powerful powerful a set of capabilities and finally scouting is a very active project it has been developed at Twitter the github is quite active particularly wiki is improving and you will see that one of the weaknesses of using scouting that we had before and this is the documentation was not always complete it's significantly improved within the last year wiki has much more comprehensive set of tutorials and it becomes much easier to start with the system and again one of the situations where you would really like to use coding is when you are using ordinary Hadoop and you would like to have the full programming language power instead of dealing with began hive deficiencies and the other reason was causing a soil interesting today is it allows us to decouple from the underlying platform so we can run it on plane Hadoop on spark on flink and we can easily survive the change in the infrastructure so i will put these slides on the github and the code example so by tonight you should be able to find them there by tomorrow at the latest and what questions you have there is a microphone behind you how's the error handling done in the pipes is it like excellent question so how is the error handling done in the pipes so one of the ways to handle the errors is that you can create a special pipe that is going to deal with exceptions so all the exceptions will go into that pipe and then you can put that you can either do some processing so for example you you receive the values and then you can notify somebody or you store them in the file so you can imagine that there is special a tap that is the exception that at the other thing that is quite related to that which you will use as you are developing your programs is debugging and that is let me just go to my code here what count job one thing that I can do is as i'm writing this I can insert the debug statement and now when I run this you will be able to see the output of this particular pipe and that is very useful especially the beginners often don't understand so how does group by exactly work and we should cease now a bunch of output coming from this pipe there you go so you can see here I have inserted the bug and the result of group is coming out so as you are interactively developing your program the local mode you have a good under standing and we have found that that eliminates lot of errors and you get your feedback very very quickly so very practical thing thank you you're welcome any other questions awesome wolf that's it thank you so much for a beer me thank you join me in thanking him