data.bythebay.io: Joel Horwitz, Host Sponsor Welcome: IBM Analytics
Recording: data.bythebay.io: Joel Horwitz, Host Sponsor Welcome: IBM Analytics
I don't I don't know how fresh my face is, but I'll I'll take the compliment. Um, hey, thanks everyone. Excuse me. Thanks for coming out today. Um, and thanks Alexi for hosting IBM and and allowing us to uh to sponsor this great event. I've been following um this thriving community now for the last I want to say almost four or five years since the emergence of Hadoop and open-source uh analytics as I call it. Um it's pretty exciting to see where we're at um in the kind of in the in the uh you know in the steps to kind of reimagine what we can do with data. Um I don't know what's going on behind me so I'll just keep talking
Uh but anyways um it's it's pretty cool. So, as Alexi mentioned, we have a lot going on at IBM. Um, we have a garage here, uh, in Galvanize called the Bluemix garage. Um, so if you haven't, uh, gone over there to see it and you're around today, I encourage you to go check it out. Um, so yeah. So, so basically around this time last year, we uh announced our investment in Apache Spark. How many of you This light is really bright. How many of you have heard of Apache Spark before? Wow
Okay, good. Everyone. So internally, we've been talking like do data scientists and data engineers really use Apache Spark or is it more of like this low-level kind of engineering thing? So it's it's pleasing to see that it's it's it's being adopted. Um how many of you would consider yourself data scientists? Wow. Okay, that's a good good amount. So it seems like the community is growing. So that's that's awesome. Good work, Alexi
Um so as I mentioned, we um we announced our investment Apache Spark last year. Um, and the way I see it is, you know, early in the 60s, um, we invested in a platform called System 360. Um, it sounds pretty dated when I say it because I'm, you know, I wasn't alive when we did that. Um, but but what that did for, you know, the information community at that time was create a an operating system really for people to build um, systems and integrated systems on, right? And what's interesting is not only did that um operating system um create an easier way for people to essentially program um information systems, it all also launched the whole discipline of information science and and what's interesting about that is that's what's actually carried us essentially up until now I would argue right I mean for the last you know 50 years really and that that's a long time right and so what we're seeing you know and then and then what we saw was around the 90s you know um people looked at you know uh system 360 it was very heavy it was a heavy system and and that's when Unix came on the scene you know across the across the bay here in Berkeley um where they you know had the BSD license they open sourced this system and what was interesting is um you know Linus Torvalds went over and tried to use Unix right himself on his own machine just like all of you have your own machines and want to do analytics he looked at this you know large um operating system and said I can't use Unix So I'm going to build my own called Linux, right? Named after himself. And that became pervasive. And why did it become pervasive? You know, my hypothesis is that it became pervasive because he created a community and it was free and it was available and people started building applications. And in my mind, that's really what launched um the you know, the age of the internet. Um really it launched computer science, I would say, as a discipline
um up until that point, you know, that was around the time I was actually in college. Um and I remember, you know, as I was in electrical engineering and looking across um the street and seeing, you know, the computer science building going up at the University of Washington. So in our lifetime right I we already saw computer science kind of take hold and that's you know no one would have imagined you know I don't think Lionus would have imagined um you know Facebook or Twitter or Amazon or any of these you know companies that exist today uh that took advantage of of that operating system right um Linux. So you know my thesis now is that Apache Spark is really emerging as the you know um analytics operating system. Um I use that and I know people kind of throw throw stones at me when I say something like that because like oh well an operating system follows these characteristics. Well if you look at you know what happened with Hadoop um you know it it was exactly like Unix right it was this heavy like no one has had installed. How many of you have Hadoop installed on your laptops? A few. Okay
So it's not a pleasurable experience and I would argue that it's not necessarily the best use of Hadoop on a single you know on a laptop right. So what happened you know um essentially a bunch of you know smarter people than myself mate and others you know came up with Apache Spark because they looked at it as saying you know that's not a very easy um you know system to program on right the you know map produce was was you know pretty painful uh to use right and so so they came up with Apache Spark and I think a lot of the comparisons with Hadoop are are fair but and they and they tend to point to performance but that's not really what this is about right what it's really about is ease of use and that sounds like a very marketing you know term but I I really believe that and it's about data access and so you know last year as I mentioned we invest in Apache Spark because um we needed a way uh frankly internally um to you know essentially standardize if you will or or simplify um the way that we build our analytic capabilities and those range all the way up to Watson and what we're doing with cognitive um so you know Watson is using Spark you know, all the way down to what we're doing, you know, in the open source, right, with our, um, Bluemix services like Spark as a service and these capabilities. Um, so it's pretty exciting times at IBM. Um, what was actually pretty exciting to see, uh, was that, you know, there wasn't like a mandate that came from Jenny or that came from, you know, the board of directors that said, "Hey, you must use Spark. That's the hot thing." It was happening organically internally already. So it was actually for so what I did when I joined IBM um just over a year ago was I I observed this I said look this is organically you know growing inside of IBM and all I did was expose that to the world um I think IBM you know over the past 50 years has had um uh has had this kind of thinking of the not built here syndrome um and that's and that's changing and so I'm excited because as Alexi mentioned we're sponsoring more um community events like these um we also launched data pulooa last year. In fact, we just had um an event in China. Um we were expecting, you know, 50 people um 250 people showed up
Um and these weren't like, you know, students or people just like kicking the tires. These were, you know, also data scientists, you know, PhDs, researchers. So, it's really taking off. Um we are now I want to say at about six data pulooas um both locally or or domestically and internationally. Um the next one we're having is in Denver um at Galvanize in Denver um as well as in Tel Aviv. So you know back in 2011 when DJ Patel said you know data science is the sexiest career of our of our time. Um I believe it it's true and it is you know that what's different about the information age then in the 60s and I would call the data science era or the cognitive era today is that people are far more creative today. um the technology is a lot more elastic and a lot more fluid
Um so it's really up to all of you uh to make you know the most out of this time and not just the technology but this era where people are more open the culture is more creative. Um, and who knows, right? What's the next Facebook, you know, for data science, right? What does that application look like? Um, I don't think people really have a clear idea of what that can be yet. Uh, so so it's it's a pretty exciting time. Um, so with that, I'm going to leave you uh to this awesome event. 170 speakers. That's incredible. Uh, let's give a quick hand for Alexi for pulling this together. Nice work
And, uh, with that, I'll bring him back and, uh, we'll get started. Thanks, Alexi.