Cognifest NYC 2017: Jerome Nilmeier, Getting Started With Data Science on Spark with Scala Pt.1
[Music] well thanks for coming out and thanks to Alexi for organizing this and thanks to I li and the IBM errs for helping us with with all of the organization so we're coming from San Francisco so it was great to be able to come out to New York I got to see the sights and really got to meet a lot of great people while I'm here so this has just been a great experience so thank thanks thanks to everyone for for having me just to you know just a couple of logistic details this is the network and this is the login would you like to oh no that okay cool okay can you guys hear me okay okay great I might try to set the mic down because of because I have to do a lot of typing so let me know if if you guys can hear or if I need to speak up but yeah I kind of I if I had a thing to attach it might be a little easier to type but okay so the so so this is the the network details here this is a red squares and network Caplin and welcome the course materials are here is everybody been able to get online and download the course materials okay great and then I just want to jump I just want to jump to the getting started slide just to make sure everybody's on the page with that we've got I can do a little bit of talking while you guys get all this set up so there's no hurry but how is everybody doing with Java has everybody got Java eight working yeah okay is everybody got is anybody has everybody been able to run SPARC she'll get SPARC shell okay and if you type SC you get a spark context okay good good okay great so then you'll should be able to run the examples based on that if you download there's a directory called code examples and all of the examples should just run you just type load in the case of the scallop codes and they should they should just run so while well while we're getting all that running I'll go ahead and talk a little bit about ibm's commitment to apache spark so patch IBM got into apache spark about two years ago their big contribution was a spark SQL they brought in a lot of people from their db2 teams and big SQL teams to work on the SQL engine that is in is in spark they also did a lot of contributions in the ml lips so they were the number one contributors to spark SQL and then and then I think the number two contributors for for ml Lib and so they really made a made a huge commitment to it because of as Alexi said because of all the products that they wanted to build on top of it they quickly realized that this is going to be really the next-generation framework for Big Data in the same way you know this was modeled after IBM's Linux Technology Center and at that time Linux was not necessarily the standard operating system and it became sort of the de facto standard and so so once once we saw IBM's commitment to spark we realized that it was going to become one of these standards standard frameworks for for big data processing so it's so it's really worth learning if you have a little bit of time to to spend on learning it because it it will be useful everywhere just about me I started out as a computational biophysicist and even before that I was a chemical engineer so gradually migrated through the high-performance computing space into the informatics space and then finally into the big data space where you know people could pay their bills and so so it's been great it's been a great journey journey for me today we'll just talk about what is SPARC we're gonna talk we're gonna learn a little bit of Scala just to just to show of hands how many people are familiar with Scala okay good how many people are familiar with SPARC Apache spark okay and how many Python programmers do we have mostly Python and then how about Java okay okay so a good mix a good mix one way to think about one way I like to think about Scala is is it's somewhere between Python and Java it's all you know it works on it works on the java virtual machine but it has a lot of the elegance that python has in terms of ease of reading and writing and it actually has I think a few more features that make it they make it an even better language to write in so back to the getting started slide how's everybody doing on this good okay yeah feel free to raise your hands if you have any questions on getting this this part going again the trickiest part is getting java working so if you can get that working the rest of it should be pretty straightforward so so talk a little bit about what SPARC is and it it's basically a again how many people have heard of Apache spark I don't think I asked that question okay so this is even this is even new the new idea to a lot of you so it's basically a cluster computing framework for large-scale data analysis which gives you parallel the idea is that you spark is the go-to code for running running analyses on something that requires more than one computer so as soon as you're in that space you're dealing with clusters and you're dealing with you it would be more than likely dealing with spark if you're familiar with the Hadoop world there were some limitations to MapReduce because it was not able to do in memory processing it also had it was very disk dependent on all of its operations you would do a MapReduce job you'd have to load from disk run the MapReduce job and then write to disk and then run another job and in this whole sequence of events became very slow because she had all this disk i/o that was happening in between these jobs SPARC solve a lot of those technical issues and they're so well solved that it's almost a legacy concern at this point but it's a the other thing that's really great about it is it's it's comprehensive it's got a lot of great libraries and so you can try out a lot of things once you have it installed you can try out you know a graph library and then turn around around SQL on it and then do some other some other machine learning processing in a way that you really can't with any other big data framework so so allows you to do a lot of exploratory things that you may not be able to do because of the the barrier to entry for for getting all of your code up and running so so so that part is really great it's got some very easy-to-use api's Python Scala and Java are the main ones we strongly urge you to learn it in Scala and for those of you starting a team or starting a project we urge you really to start the project in Scala because it will as soon as you hit a difficulty in the code you can always you'll you'll have more luck rewriting it in Scala because it's the native language of SPARC so it's worth it to spend a little time learning how to write in Scala now having said that but python api's are really great it gives you access to you know you can run numpy and matplotlib and plot all of those things while you're running SPARC and it's getting easier and easier to install pi SPARC these days I think you can do it through Khanda so so PI SPARC is also a very low barrier to entry but in terms of when you start to deal with the production workloads Scala is probably the way to go even even more so than Java if you're used to coding in Java so so for the Java programmers is probably worth it's been a few days getting familiar with the syntax it's you know it's it's written in Scala and it runs on a JVM so all of the nuances of running Java can be understood when you're when you're running Scala as well so so spark is you know it's it's open source it does in memory processing which which is a huge advantage for iterative processes so when you're solving these these objective functions for machine learning you're repeatedly accessing the same objective function which requires sort of a fasting memory access and that was really the driving force for for the development of SPARC it gives you distributed data processing so so once you write it on your laptop with your small file you can write it on your your cluster for your terabytes size file and and the scale up is pretty transparent so so it's a great it's it's a great framework for for for these reasons now how many people I mentioned some of the MapReduce issues how many people are familiar with MapReduce written MapReduce code okay okay so yeah it's getting to be these days if you haven't done MapReduce it it's likely that your first exposure to big data is is SPARC at this point but this was not the case just a few years back so it's it's really it's really become the the standard but just to compare what was happening and I mentioned this before you're basically you read and you run through an iteration and you're always disk bound on all of these operations and the resilience in the Hadoop file system is you have trip you know data stored in triplicates usually in triplicate so across your across your cluster so the resilience is at the file system level but if something crashes in the middle of your iteration you have to start all the way back from zero so SPARC takes that one step further so it can store a very complex series of operations and you don't need to have this intermediate write state because what it does is it stores the Trant the series of transformations that take place I do need to walk take this with me if I'm going to walk over here huh it stores a series of transformations that take place when you go from the input all the way to to the antutu to the point where you're going to be writing the file or summarizing it and it can recover it can recover the entire series of operations if one of the nodes goes down so it stores this thing called a resilient distributed data set which is basically a series of of actions that that start it at load time and then goes through that transformation so if your process goes down it can reconstruct the entire process so that allows you to have a lot much longer chain a much longer execution plan and still have fault tolerance so so that was really the big design advance in in when-when-when Apache spark first came out so this started out at UC Berkeley in 2009 again this is you know is basically you know it was thought of as the next generation MapReduce was open sourced in 2010 IBM jumped in around 2015 now it's a it's a it's a sort of a 2.0 software release so a good portion of the code has really become hardened and optimized but then there's still new features that are coming out particularly the streaming is really taking off these days so the you know the nice thing you can do with with with spark is it gives you this sort of interactivity with with your with your open spark context you can do iterative machine learning you've got you've got access to a streaming library virtually all data sources are supported now this was a really big advance leading up to the 2.0 release it's used by it's used by data scientists and it's also used by data engineers and this is this is one of those frameworks that may actually change the nature of what of how we understand what a data engineer and a data scientist does because it all happens in one code base whereas maybe previous versions of code had you know a data scientist developing it on a laptop scale then handing it off to an engineer who would scale it up and so it was a little more stove-piped and maybe there was some loss enos in the way the algorithms were scaled up or the way they're interpreted spark can sort of all have and in one place data scientist a person with a machine learning and data science background can work with the same codebase that the data engineer is is also where he he or she is working on production quality issues so you know once you once you learn this syntax especially if you learn Scala they can all sort of work on the same codebase so I wanted to jump in and do a kind of a Scala primer and and this is you know sir Scala is a very very deep subject and really what I just want to try and cover is you know get you familiar with the syntax enough to run Scala to run SPARC code and be able to write be able to write SPARC job so I won't go into too much detail about a lot of the real fancy stuff in Scala but definitely get you familiar with expressions and lambda expressions and and how to read and write this code so you know the main reason we're advocating Scala's because because apache spark is written completely in scala so if you're gonna write a library you may as well write it in the same language that the code is written in for example the Python API you actually have to write separate connectors to the methods and actually expose these methods to the Python API so the Python even and even the Java API will lag behind the full functionality of the Apache spark release so the code the code is really pretty easy to read and it's still statically typed so this strongly typed the notion is something that you see in Java it it really it really comes it really shines in in Scala as well my personal favorite is that it just naturally has this repple environment you can open up a scholar shell and import all of your java libraries test things out do it all from the command line don't necessarily need to set up an IDE you certainly can but you don't necessarily need to so type safety will allow for compile time error handling rather than runtime error handling and we'll show I'll show you some examples of that but once you start to get a handle on the power of type safety and you're in good coding practices you're really come to appreciate using using a strongly typed language and and the other thing that that's that that Scala has is it has thread safety baked into the syntax that means as long as you write the sort of standard spark code or standard Scala's code all of the parallelization issues can be handled transparently so you don't so you don't necessarily write for loops because for loops could have you know I equals one to ten on different nodes and you could have different those good those could be happening at different speeds and it could introduce parallelization issues Scala will is written in such a way that you don't necessarily need to worry about all these iteration issues and they have a notion of the the iterable that we'll talk about a little bit you grab a glass of water excuse me so so so the origins of Scala was really to try to make a better a better version of Java and so we actually had Martin O'Dare ski son Jakob working at spark Technology Center with us for for a while so we had a lot of really deep Scala expertise for many years and you know he was originally a writing Java compiler so really he came from a Java centric world he just really wanted to make something that was much easier to read and write and so around 2011 the company called typesafe which is now light Bend was founded and today I think it's Scala 2.12 running on Java 8 tomorrow uh it's running it will run Java Java 9 and that's I think more true for spark have to double check on these versions the the entry point into Scala is this read evaluate print loop it's called a repple and this is just you know command a command prompt that allows you to explore the variables the the great thing about it has tab autocomplete so if you guys have opened a spark shell you'll be able to you know defined variables and and do tab autocomplete and that'll tell you all the methods that are available to it so we'll do our first example and the way this is structured is is all have a snippet of code that I post up here that you can you can take a peek at then I'll have the name of the file and then you should be able to go to your code examples directory and open up that file and be able to run snippets of code from it so this simple examples has a whole list of little snippets and so typically what I do is I open up a text editor and I just copy it and paste it into the shell prompt you can copy multi lines and and run it that way if you don't if you don't have a text editor that supports Collison text there's one called sublime you could try downloading that one VI I have a tar file that has bindings for Scala syntax so let me know if you want to try to get that working the so there's plenty of other text editors that supports Collison text we'll talk a little bit the first sort of basic idea for okay so just walk through the examples we'll just I'm just gonna do a couple of cut and paste some you know we don't know just gonna go scan everybody whoa okay if you see me typing and and you still see a slide up there ping me because it I have to get out of presentation mode to bring up the terminal sometimes I don't realize that I do that okay so you can define this is just to show you what immutability is okay so when you define it it'll say it'll say this is a little bit bigger it's gonna say this is a you defined you define this thing it's a string so it says okay Val hello equals and then string and it says oh you said hello : and then it gives you the type it's a string and then it just tells you what the value is the only thing that this example is supposed to convey is that you can't modify oh yeah so you can't modify it in place but you can which just means you know want you to find it it's immutable and this is a this allows the parallel construct to to hold up and so and so so so the thing that remember is that when all of the parallel constructs and sparker immutable so try to try to say mutable for for most of your code and most of the SPARC code will sort of you know hem you into - to doing - doing only immutable operations but you can you can do this var which gives allows you to let's yeah so now you can append to that string in place with if you if you do var so if you need to iterate through something but you can see why that the immutable things if you wanted to do an I plus plus as you're iterating through loop and this is happening on different machines and it's happening at different speeds you know it's it's harder for it to keep track of that so there's actually a special construct for for doing that okay so I'm just gonna go through some of these expressions this one here says so an expression is basically think of if they think of an expression is something typically you put it in curly brackets right so so you could put this in since this is one line you don't have to put it in curly brackets but an expression can it's actually a more general form because there's a there's kind of this ambiguity between what a variable and what a function is in a functional language so it really is contained in this notion of an expression so here everything is is is covered in this definition but sometimes if you have a variable that's not defined in the expression it'll return a function so you say you know this this so for example if if a hello else goodbye then it would that would return a function of a which would allow you to pass in a boolean value and then if it's true it would say hello if it was false it would say goodbye so that's actually this lambda expression here right so so you've got val coming or going and now this is the what this is a fundamental idea in Scala that makes it a functional language is that you have a variable and then this forward arrow means that it's a function so this is a lambda expression and so now what this is saying is saying it's saying this is a function of n that function of n is if N equals 1 hello else goodbye right so all you're doing is is is taking what was an expression and and not defining it completely and saying oh now it's an expression but it's a function of n right now you passing this value of n in and and you get a value and so but it still treats it as as a vowel right so this is the notion in in in in functional program where they say functions are first class citizens right you can pass in a function in the same way that you would pass in a variable right so there are other way other things you can do with expressions right if you have multi-line expressions then you start to do the curly braket thing and the other thing to remember about expressions is the last line whatever is in the last line is returned as an output so so you could actually put return here like you wouldn't Python or something like that and then it kind of then it kind of looks like a function right but again this is a this is a this is something that that we that we think of as an expression so a lot of times when you have multi-line multi-line expressions it's always good to put the curly brackets around it sometimes sometimes the command line it helps a command line editors too to see what's going on so we can we can run a was how we doing for time too we're doing okay so yeah we'll jump into the strong typing real quick the talk the talk last night I just tossed this example in last night because the talk was this was the X dot a I and what was they were they were saying one of the strengths of the Scala language is a strong typing because it partially because they have so many possibilities they have to scroll through with these with these chat BOTS right there if this then that if this then that lots of case statements and things like that and you you know you may you may you may not be clear if you've covered all the possibilities when you're writing the code so when you do strong typing it what it will do let's let's go let's go to this one because this one's kind of an interesting one this poorly typed expression will say is saying if it's true return to one but it doesn't say what to return if it's false right so it gives you this type called any vowel which is not exactly what you want you want you want an integer right as your as your return value so example what you really want to do is you can enforce this typing by saying okay I know let's say let's say I covered all my cases I I'm gonna get an integer in every case so I expect to see an integer return right so it's saying wait a minute no you you're not you didn't cover all your bases right you missed if you missed a spot you know it's it's this type that's different than what you thought you were gonna get and what's nice about that is this type checking happens before you load your terabyte data set right before you commit to running all of this massive amounts of code so so this really will save you a lot of time if you you know if you start to get the hang of how to do the typing so yeah you you don't have to you don't have to specify the type in the definition but when you do it it becomes very very helpful all this stuff happens in the IDE too so if you once you get your IDE set up all this typing will happen not only a compile time they'll just be happening while you're editing so if you're familiar with the Java typing it it works in a similar ring okay so we'll talk a bit about methods that there's not there's nothing too terribly exotic about the way methods are defined in Scala but you know they start with a def statement that way you're saying well this is a function or a method here it allows you to pass it in defaults so let's do so once you do it it will create it'll create this method now this is not a vowel this is actually you know treat treats it as a function and it allows you to just do some defaults so if you just do let's see name it'll return first and last if you don't do anything the thing I like about this too is you can you know you can you can say you can say what variable you're you're specifying so it allows you to do some it gives you a little bit of clarity when you're defining your function so this is this is the kind of things you can do you could say first equals this last equals out that way you know which which variables you're passing in makes a little bit more readable there's this notion of it this is kind of a unique construct to think about again I'm not going to deeply into the object-oriented stuff because you don't necessarily write your code you're not gonna be dealing with object-oriented issues as much as a code base gets larger clearly you will but this one will show up when you're when you actually commit to compiling your Scala code because you need to create a singleton object for in order to compile it that's like the main main object and so a singleton object is just something that has a one instance so you don't need to once you define it you could just you just refer to it that the name you can refer to it by name oh yeah and don't forget that it has the auto autocomplete so it'll tell you what-what-what-what fields are available there this is this is a this is the way you would define a class I'm just gonna talk about this one again it's just pretty compact right I mean if you look at if you look at creating a class with fields in Java it's gonna take you ten lines or something like that and and or five lines or something like that this one here is is all it's a it's a one line type of operation this one and this one here actually shows up a lot so this this one is worth thinking about a little bit in spark just so so you understand one because I think the nomenclature is a little confusing as to how it you know what what it what it conveys in terms of his intent and what it actually does at least in spark and what it what it doesn't spark is it creates a container for different variables of different types and so so you have a case class here oh and let's just go ahead and find that okay so you created your case class and now you got tests and then now the fields that are available these are all these you know kind of methods that come built in to a case class but in there is is a message so you can create you know you do a case class and you say okay the first the first the first field is message the second field is something else that's an integer or something else that's a float and you basically create a a something that can contain multiple types in a way that you can scroll through and that's useful in spark because you create something called a data set or even a data frame that you know where it's essentially a table and each column has a different type so this is how this is what you'll see in spark when you're defining these multiple types these you know these rows that have the possibility of having multiple types so so yeah just keep an eye out for this one this is a people like to use this one because it's compact an elegant way to to define you know all of your different fields but but you'll you'll see this you know even before you even get to the object oriented stage of your development you'll see case classes and your you know your your script level code not going to go into companion objects you don't seem as as much I I do it I'll talk for time I'll talk a little bit about the something called the simple build tool so maven does work for Scala how many people are familiar with maven okay okay so maven works for Scala in fact maven is now the preferred build tool for SPARC it's the recommended build tool for SPARC simple build tool is still the recommended build tool for Scala centric code so you you know you can you can use both SBT has a few nice features all the examples today have designed to run just from the repple so just didn't just in the interest of time we didn't get into how to run compiled code and do build codes but I do want to point point to a few really neat things about SBT it also has so at Build time you can invoke a console and you bring up the console and then you can do exploratory analysis of your variables through the command line and in fact what when you open up SPARC shell what you're looking at is an SBT instance where they invoke the console so they loaded all SPARC libraries for you and then they said hey bring it bring up the console and then you can do exploratory analysis on it and it also has continuous compilation so you can have this running while you're while you're editing your code don't it doesn't necessarily have to be in an IDE but while you're writing your code it will debug your code and tell you tell you what's going on and and you can modify your code that way so yeah it's the native it's the native compiler for Scala and it's it's worth it's worth checking out I actually preferred over over maven and so the concept of a collection in Scala is it's just a it's a pretty fundamental idea it's it's worth trying to get your head around because even writing the simple RDD based spark code you're you're going to be doing these operations collections and I know this is all it's kind of hard to read but the top there is a notion of a traversable and then right under that as an iterable and then and then it branches out into all these other types that are based on that but the notion of a traversable is really you know it just the the traversable and irritable really kind of tell you what the idea behind a collection in scala so this you know it's a 1 d list of items and there's some recipe for traversing that list or iterating through that list that means again it's designed with scalability in mind so if you say you don't you don't say for I equals 1 to 10 add 1 to each element you say dot map X forward arrow X plus 1 which just means that apply this function to this list of variables in this following this recipe for iterating through the collection that is you know predefined that that is already understood cz I don't care what order you add X plus 1 but you know I don't care if you do it in blocks or how you do it just you know just just iterate through it and in the way the iterable specifies so so that that allows you to go through these collections in in kind of an automatic automated way without having to worry about parallelization issues so when you write your code on a single that runs on a single thread or a single CPU and you scale it out those all those paralyzation issues are kind of handled under the covers so so that that's the main idea behind most of this this the way collections are designed in Scala and so let's see yeah so just to just to show you what some of the some of the some of the cool some of the basic operations on collections that you can do I guess the first thing that was confusing to me when I first started to learn Scala is that a tuple is the one thing that's not a collection a tuple is this basically kind of a you know you just stick a bunch of stuff together and that's your tuple so it's kind of like a case class actually right but here here's just a parenthesis and you you can access it by element but it's actually the access model looks very different it starts it but it starts at underscore one let's let's go ahead and run those so we can get some get some get a feel for how these look okay so so when you define it it figures out it figures out that it's a type figures out that the first element is this integer the second element is a string and then you just access it and if you look at the methods that are available to it it says oh I can get the underscore one part of it I can get the underscore two part of it so that's actually thought of as a method in in this in this type and it also for the two element tuple it looks like it looks like something called the key value pair and a key value pair is something again if you're coming from the Map Reduce world the key value construct is something that you see over and over again when you're doing word counts for example so so so that's that's just an example of what a tuple is but the important thing is that there's no recipe for iterating over a tuple because there's it doesn't know that the first type is going to be an integer de novo and the second type is going to be a string you know so there's no clever tricks that you can possibly apply really to iterate through that so they just say okay we'll just give you an access model for accessing each element and and that's that's how that's set up so you can do you do something like maps maps or iterable and you can do things like union operations so let's just take a quick look at what that looks like this and so you have okay so it's an immutable map oh yes thank you all right yeah let's try that again okay so define map one and then we define map to is everybody able to run these examples the idea was kind of run them and kind of interact with them to see what they see what they're doing and then you can do this Union operation so now you can look and then it just create it just puts puts those together but you can look at the methods that are available to a map they're very different than the what's available to a tuple so you've got things like zip to array collect which is interesting filter you can do all these other stuff to these collections once it gets into once it becomes an iterable that you that you can do with with tuples so i think we're yeah so let's just take a look at some of the stuff you can do with collections so and this is really when you're when you're noodling out the syntax you can end up writing a very complex iteration in one line of Scala code but that line of Scala code may take you a long time to write because you got to figure out the syntax and it still takes the amount of time that you you know you're right a 20 or 30 line double or triple for loop and and it takes it takes a while to write it it'll be very compact but it'll still take a while to write it because you have to make make sure it's right but but so that that's why having that command line to kind of noodle through the syntax because the the syntax can trip you up for for a while so the first thing that's done here is you create this range so one two five this is very similar to the Python way of building ranges and then now we're doing this lambda expression here so we're saying one to five map and then again in a function of n which is n plus one that's the that's the land expression so before I do that does I mean does anybody know what that's going to be or anybody yeah it's one to each element so that one is pretty straightforward good and it just gives you two to six right and and then there's a shorthand here there are lots of short hands in Scala I try not to cover too many of them because it's already hard enough to get your head around the basic syntax but but you'll see this one a lot where you just replace this whole lambda part here with this plus plus equals one and and so that that's essentially that will return exactly the same value now this is really cool if this lambda expression gets complicated then you just define a function and replace that with the function right so again here we're defining a new function as a foul which we can do because it's a functional language and so now we go add one one that gives you two right now we're gonna just do and this is kind of a nice a little bit cleaner way to do it right you can again you define your function inline or have a method or something and then and then just do this map and and and then as the functions get more complicated that improves the readability so yeah here's just another you know starting to get a feel for what would happen if you wanted to get a little add add you know add some some vector of strings or create a vector of strings or something like that now the reduce operation is again a you know this is a it takes two values and returns one value and and so so you so you're gonna add one to it and a reduce operation means take two values return one value and what this does is it it it assumes that it doesn't matter what order you add these in right or what if you add groups together and if those groups add and and so so you can always have this you know in in goes two values that outcomes one value and so the syntax here is this tuple of a comma B lambda spreads from a plus B and so and so basically now we're gonna take that array two to six and then we're gonna add up all those numbers so you'll see this okay it returns an integer and this is another piece of syntactic sugar here that you'll see a lot is this underscore plus underscore and this doesn't necessarily have to be an addition operation or although ninety percent of time it is it could be multiplication operation you it doesn't have to be integers that are adding you can append strings you know anything that takes two values and returns one value so here's a here's a here's one where it reduces the strings again yeah so it just turns out that the plus operation is recognized as an append for string so we'll so so actually in this one here the order does can-can matter so so so so there are places where you you want to exercise caution with that but and you there's things called fold left and fold right and all these other operations as you get deeper into it that'll become apparent and more interesting but but the basics are covered here and filter is another one that you'll see a lot and all this does is it now now this lambda expression is something that has to return a boolean so anything that meets the criterion X is greater than two or it returns true is going to be retained in the collection so that one and three four five okay so here's a just a simple example of processing a collection create a arrange one to 5,000 filter out that those that are greater than 400 take five would be take the top five elements and then print out the results and so we actually put we actually wrote this as an expression just because it's a multi line it could be it can now be a multi-line expression you know sometimes it doesn't figure out if you if you don't put the dot and this line here it will think that that maybe that's a standalone expression so if you put it in curly brackets it knows that it's an it has to go through the entire entire session so that gives you four hundred fourth out four hundred one to 405 was that sort of what you expected for that but does that make sense and I think this is the same one yes that's the same one the only thing that that I wanted to show here is that you don't have to put the the periods in between the methods so if you like this syntax some people do some people don't you you you can just put a space in between the methods and so this is exactly the same method as that one is has just got spaces there and it returns the same same result people tend to like this because you can it it looks a little bit more like a like a domain-specific language or something like that but but either either is fine so let's see I've got yeah we can we can talk a little bit about creating more complex collections figure we got just a couple more examples and we'll take a quick break get some air and then we'll jump into spark examples after the break does that sound good for everybody okay okay so I thought very nice which I might skip this one this my my one my own that be working yeah we'll jump back to this later I just talked about this one rather than noodling through it so the way you can build a if you think ever like you're writing in a procedural language you would say you would iterate you would say you know 4 1 2 4 4 X's in a 1 2 4 4 y equals 1 to 4 you're gonna do this X Y combination right and that's gonna go you know 1 1 1 2 1 3 1 1 4 2 2 2 3 right this is the procedural way of doing it now what you would you would you what's not recommended in Scala is to do this through a an array that you update in a mutable way right you create an array with all zeros and then you update the particular entries with new values as you iterate through the loop that would be like the Fortran way right and and that's because the types are immutable right so you can do it recursively by just basically giving a set a series of instructions and having to do this kind of a for each operation where you go through you're doing it recursively so you're not actually all of the instructions are being built up in the stack and then it's sending all the once once the recursion is done then it sends all those instructions builds a single immutable array one time based on all the stuff that's that's on the stack probably not the best way to do it particularly in SPARC because the recursion stack gets very very large because you're carrying all kinds of other stuff in the instruction set in addition to just the simple recursion so I can very quickly launch memory and you can you can you have to adjust your memory in order to get it to work correctly so probably the best way to do it is to is to try to think in these Maps rather than iterations and then so you can do something like this where we're basically now you're saying map X is 1 2 from X go from 1 to 4 and then do another map and then y equals now you've got the XY get the matrix entry and then flatten that into an array so really I mean the the the take home here is just to try to think of instead of for loops thinking in terms of maps and a lot of a lot of the parallelization issues will be handled transparently so let's see what's that one looks like did that one now a lot of this stuff is something's something's goofy with it so we'll come back to it yeah have to come back on take a look at that example but okay so these are oh and here's some here's some further reading there's an O'Reilly book dean Wampler wrote a great a Reilly book and then this is written by Martin O'Dare ski I would say this is this is a more over the Java centric person a lot of detail but they're both great they both cover actually the Java side of it very comprehensively as well my personal favorite scala site and the one that really got me calibrated to scala syntax was this one here it's called I don't know if you can see it or not it's called Scala 9999 Scala problems and it's just a lot of you know a lot of really cool you know reversing your lists and doing all this other kind of stuff I think he actually read a few games in it graft searches and all kinds of cool stuff and it really gives you a feel for what what you what you can do with Scala and all the syntax nuances so so that's it for Scala we're about an hour and a half into it so I can answer any questions at this time if not we can take a little break okay great yeah I feel free to come grab me if you have questions - so I will take five a ten-minute break go just kind of hang around we'll convene [Music]