BDSBTB 2015: Brad Rubin, Introducing Java/Hadoop Developers to Scala/Spark
Recording: BDSBTB 2015: Brad Rubin, Introducing Java/Hadoop Developers to Scala/Spark
something like that it happens sometimes but most of the time it looks like this this is our campus you can probably see in the upper right-hand corner a little body of water that's actually part of the Mississippi River and st. Paul is actually to the back out of the picture Minneapolis a little farther in the distance we actually have campus both on st. Paul and the Minneapolis side and I live on the the st. Paul side you might also think of this when you think of Minnesota I mean you've seen this movie beware though where is Fargo North Dakota so people associate this movie with Minnesota quite a bit but interestingly the title comes from a different state so I go Google at some time you can get into all that the exciting thing though is the Coen brothers who directed and produced this movie and many others are from the suburbs of the of the Twin Cities I might use a few strange words foreign words in the minnesotan vocabulary so i listed a couple of these so i want to point out holy buckets so when my wife swears as a native minnesota that's what she says you might have all seen that there were there was a infographic that made its way around on doing an analysis kind of a tf-idf analysis of dating profiles and the most significant word appearing in the dating profile of Minnesotans but that's also rare across other states so highly frequent in the state but rare across the collection of states is the word cabin and I find when people hear the word cabin they think like of Abraham Lincoln Log Cabin no indoor plumbing the most the rest of the world calls cabins vacation homes or second homes and the reason people can afford a second home is because the housing cost is about one hundred and twenty dollars per square foot so I actually checked for San Francisco what the equivalent was it's like seven or eight times that so you could buy seven or eight homes if you can afford a home in San Francisco in Minnesota so one of the first things I want to state is that you are not normal and I mean this in a good way you are not normal in the sense that if you are at this conference it because you already know something about spark and Scala at least enough to know that it's important to be here and you are likely either familiar with these technologies or deeply involved with these technologies or already on the path to do so but I am interested in a little different problem i'm interested in the problem of how do you take the masses to scala as a language how do you take the masses to spark especially if they come from a mapreduce background i also have to address the question my classes are increasingly asking which is well if the spark scala stuff is so hot why are we doing the Java MapReduce stuff and my answer is usually well when the technology changes you we invalidate your master's degree and just have to come back for another one right so what I've got to do is tell them that the concepts we are dealing with are really the ones predominant in our Twin Cities market where there isn't as much spark many as there is around here and also to take comfort in a lot of those concepts will transfer to spark an to scala going forward so I'm interested in groups of people wear these might be some of their characteristics in their background no undergraduate computer science degree we see a lot of people that are changing careers coming from fields like Latin and law and medicine in some cases and are interested in software engineering maybe that's been a side project maybe that's they just decide to make a radical shift in their career we're seeing an increasingly number increasing number of international students probably upwards of forty percent now used to be we had students from India and China pretty regular regularly students from Africa students from the middle our growing population often people do not have Linux experience coming into our program they might have only recently learned the program but are very confident I've gone through a Java class but given a piece of paper and a problem have no idea where to start and if they do know computer language is likely Java it's likely only one language and not only don't know Scala they probably have not heard of Scala they might not even have backgrounds and operating systems or architecture or data structures or algorithms things that are important to think about especially in these highly scalable systems that are determining performance and scale characteristics are so complex they might be aspiring to their first job in IT or they might already be working in a company but one that's very conservative about their technology selection and forays into new technologies I have students that program in COBOL so languages that one of the conclusions here is languages never die people still need Fortran and COBOL to some extent and Scala is hard to the majority of students now there are certain students who you know we can learn all this without much direction now those aren't the ones I'm worried about the problem there is keeping them engaged in busy I'm more interested in the problems I outline here and you know why don't we just say well go away until you learn all this stuff and have this background come and go get it somewhere else and and come back when you have it and one of the things we very much believe in is the power of Education for many of our students having a master's in software engineering is literally life-changing and our focus is applied so we're not after the training side which is important there's aspects of training are important because that'll help people get their internships or or the first job but we very much also want them to have some foundation in theory because that part of education can last last them for decades so spark mania I use that term already you know where we out was far can you hear various things on here you still need it for the biggest data it's still a fact that traditional MapReduce is is powering most of today's big data but there's growing movement to the bottom right hand side where people are viewing MapReduce is dead that is is probably too strong a word here and law and loose park and there's some technical resources and reasons for this one is just a fundamental technology trend about the decreasing cost of memory and increasing capacity and we love to use fast memory especially for these iterative algorithms that have arisen in in areas like data mining and machine learning but spark is also very attractive because it's a much simpler API it's it's more fun it's easier to write those programs but spark mania is not only a technical issue one of things I'm interested in are really some of the non technical issues in programming languages in programming paradigms why do languages become popular there are over 2,500 programming languages most of them go nowhere why do some bubble up so here are some of the non-technical factors I think of first all the cool kids are doing it so you got to jump on the bandwagon if you're a startup and you need some glitter in your funding proposal things like skaaland spark can certainly provide that often startups can be free or at least free or from a lot of legacy constraints so free or to use the latest technology spark certainly looks great on a resume even better if you know something about it and it's important for recruiting and retention these are hot and interesting technologies there is a lot of industry investment in this area and certainly a lot of momentum so but most of the existing knowledge in big data is still on MapReduce and still Java most of the existing practitioners were brought up on these models so the question especially in an environment where quote everyone knows java and quote no one knows scala how do we officially move people from point A to point B and one way to answer this question is just say well that's really not the right question list have everybody program in SQL and whatever the underlying engine is it'll just work so all you really need to do is teach people some SQL and and everything just magically happens under the Neath unfortunately there is not only one SQL and big data there are multiple they have all different not only some syntax differences but operational characteristics that differ and fundamentally I'm a believer in the best way to be effective whether you're a high-level language programmer whether you're a database programmer program at SQL or even if you're using higher level tools the more you know about what's going on under the covers the stronger you're going to be the better you're going to be able to deal and anticipate and problem-solve scale and performance issues so spark has an interesting profile of languages you know beginning with Scala Python and Java bindings more recently adding our and SQL to the mix there were some data published earlier this year about the languages spark programmers use and a surprisingly number number we're using Scala eighty-eight percent java was in second and python was down pretty far earlier this conference I actually talked to the person involved in a study and they said that this was not a data science community this was a software development community that was polled here so I would expect if we asked data science people Python would probably pop up to the top here but fundamentally spark is a nice API and the idea of a functional language better matches what happens in these big data applications where you rely on parallel ism problem for a university is well why don't we just start teaching Scala and one way to try to answer that question is look at where the industry is at in terms of programming language popularity so I'm going to flip through three charts you're going to see different answers on different surveys in part to make the point that this is a hard question to answer and how popular various languages are but also show you that Scala isn't in the top 10 on any of these languages so ti OBE index often used to look at programming languages especially over time scala's 33rd on the list this was a nitro police spectrum list that just came out recently Scala not on the list of the top 10 languages its 19th on the list and Cody Val put out a survey this is the highest it's been on any survey I've seen 12th on the list more interesting to me is there was a survey done of the top 39 computer science programs in the US looking at the language that was used to teach programming at an introductory level and Python showed up his number one I was a little surprised at this I was a little surprised I matlab being rated so high I wasn't so surprised at the sea in the job or the C++ the only functional language on there is scheme MIT was famous for introducing scheme believing that it's important to teach freshman computer science students how to program functionally before using an object-oriented or an imperative paradigm but they even have moved to python and they have done so because computer science departments often feed other engineering and stem programs that programming levels vary and different libraries are more interesting than others for certain domains so that can can push more toward the the python and matlab kinds of applications so ideally if somebody said you know i was just stranded on a desert island and i'd like to learn how to do this scala spark stuff i think this is the ideal if you're starting from a java-based spend years trying to learn Haskell a pure functional programming language the reason I say that is I think it's easier to be forced into a paradigm rather than it being a position with a multi-paradigm language and run door under whether you're doing it right not and scala you can say well I get my code to run but am I is a really functional or not then spend a little time weeks and months going to scala and then a few weeks going to the spark Scala API more pragmatically though going from a java-based I favor an approach where you spend days to weeks understanding the old features of Scala and then maybe weeks to months learning enough of the functional features in Scala to use spark most likely as a domain-specific language and then enjoy the multi-year feast on all the intellectual richness behind full-blown functional programming so I'm going to outline for stumbling blocks that commonly encounter with students who are trying to learn how to use Scala one is the idea of functional programming in your program doesn't run iterated line by line by line and I've found it helpful to talk about Excel spreadsheets which everybody knows how to use and how spreadsheets are a model of computation one where firing firing firing firing of functions feeding other functions conveys the functional paradigm and it also can convey laziness and short-circuiting and even introducing the notion of things like infinite streams the second building block i run into or stumbling block i run into is hey what's the difference between val and VAR and you start off by saying well just just use vowels don't worry about VARs how many of you have a three-year-old like niece or nephew or kid or have had one and it's kind of like them asking you why the sky is blue you give an explanation well why is that well why is that and you enter the infinite regress of explanations and pretty soon you're in quantum mechanics and explaining why the sky is blue although I used to at some point just say ask your mother although i've heard that that a more modern answer is what is the internet down can't you google it and there are good reasons for doing vals and more importantly using the immutable state here are the benefits of immutable state and avoiding side effects but these are not easy and obvious concepts to get across and understand and increasingly I answer a lot of these is just be a little patient and you'll see the value of doing all this stuff as time goes on you might have seen this XKCD code written and Haskell is guaranteed to have no side effects in the smart alec answer is because no one will ever run it Haskell is even less popular oval language than Scala another stumbling block what's the difference between vowels and deaths what's the difference between variables that can hold functions and methods of objects so that takes a little time to digest and get used to how many of you of you know what recursion is typically ten percent of my class knows what recursion is so spending some time talking about linear and tail recursion and getting men identified the patterns in the forms and going through exactly what's happening on the stack and the scale issues is something we need to spend a lot of time on because so much of Scala and functional programming is recursion based and also I find it helpful to start with a piece of code in the upper right which is something that might look more familiar this is a sum of squares in an imperative style in Java this is Scala eyes Java and the upper right and then doing continuous refinement to show how you can transform it and be more functional i also like talking about parallel collections because if you jump to the bottom there it's kind of like mapreduce do a map and a sum is like mapreduce so people kind of see these functional concepts relating to the MapReduce concepts they might know I also like sharing with students my language opinions be arnes to strip who created C++ so there are only two kinds of languages the ones people complain about and the ones nobody uses so I leave them with a list of things i like about scala and i leave them with some things that I don't like Java 8 is one that worries a lot of people I your students say well java 8 is out and it kind of does functional stuff does that mean we still have to use scholar can we just use Java from now it's hard to convey to them that every scholar programmer that I've met when you ask them about Java 8 and it's functional people capabilities they say yeah interesting I'm not not yet met one who is going to change say oh yeah I don't need to use call anymore I'll go back to Java I've never met one in two years of asking that question I also give students a little first cut at how you go from concepts in MapReduce to spark and notice there are some things in the column that you don't need to worry about at all and other things that are new concepts that are only available in spark so this gives them some assurance that the concepts they've learned in MapReduce do carry over to spark and an initial guideline of where to look to take concepts they might know on the left and move it to the right I also encourage a little different format of spark and Scala code that you might be used to in exposing the r dds and separating out input output from the processing in the middle and primarily I do that for its unit test capability if you've used mr unit you've probably seen the nice j unit like unit testing environment we're able to emphasize that so in conclusion here for many applications the sparks call api is preferable and it really won't take long for most java programmers to become proficient enough in scala to be able to attack spark and more importantly i truly think scala provides an intellectual feast that will last for years and i'm going to go out on a limb here i did this the last time I did this was in 1995 when there was only one java book on the shelf i said i think job is going to be the next language i'm going to i haven't had this feeling since 1995 but i think scala will will eventually become the predominant language and notice i used enough soft words in this proclamation that I'm guaranteed to be rights no matter what happens so it's fun stuff you haven't haven't used sparkin and scala together jump in questions from anybody or comments on your own experiences yes um can I bring the mic to you so you're a slide on journey to scala you kind of assume that people always start from java right and i think that's not completely right because i have actually not not started from Java I don't program in Java I've never programmed in java professionally I started from Haskell and before that in C++ so that assumption that people will always start with Java or why do you think that that is even a good starting point okay so one of my first slide says you are not normal and you are not normal I mean that in a good way the C++ and jaw is in Java backgrounds are interchangeable having some notion of a strongly typed language statically typed language in a software development environment is very common so Java and C++ are interchangeable there the Haskell experience truly is unique I can think of one student in five years that had any background in Haskell so you're right is so the ideal jury the ideal journey would be to go to Haskell and then go to scala i actually thought you were going to say you were coming from a python background and and which is very common for data science yeah and again I'm kind of addressing a software engineering body of students also I think your survey which says about functional programming languages restricted to the US universities as well only and I think if you will pull if people pull more universities in Europe you might find that Haskell is not that uncommon and I did go to school in in Europe so Haskell was actually the first language that was taught to us yes good point and I'm actually thinking about a sabbatical in a couple years in Europe to look at functional language adoption there and how it differs from the US so if you can catch me some time in the conference I'd like to learn from you other questions all right thank you