data.bythebay.io: Joel Horwitz, Host Sponsor Welcome: IBM Analytics d2
Recording: data.bythebay.io: Joel Horwitz, Host Sponsor Welcome: IBM Analytics d2
I don't I don't know how fresh my face is but I'll take the compliment hey thanks everyone excuse me thanks for coming out today and thanks to Lexi for hosting IBM and allowing us to to sponsor this great event I've been following this thriving community now for the last I want to say almost four or five years since the emergence of Hadoop and open-source analytics as I call it it's pretty exciting to see where we're at in the kind of in the in the you know in the steps to kind of reimagine what we can do with data I don't know what's going on behind me so I'll just keep talking but anyways it's it's pretty cool so as Alexei mentioned we have a lot going on in IBM we have a garage here in galvanized called the bluemix garage so if you haven't gone over there to see it and you're around today I encourage you to go check it out so yeah so so basically around this time last year we announced our investment in apache spark how many of you this light is really bright how many of you have heard of apache spark before wow ok good everyone so internally we've been talking like do data scientists and data engineers really use apache spark or is it more of like this low-level kind of engineering thing so it's it's pleasing to see that it's it's it's being adopted how many of you would consider yourself data scientists wow ok that's a good good amount so it seems like the community is growing so that's that's awesome good work Alexi so as I mentioned we we announced our investment apache spark last year and the way I see it is you know early in the 60s we invested in a platform called system/360 it sounds pretty dated when I say it because I'm you know I wasn't alive when we did that but but what that did for you know the information community at that time was create an operating system really for people to build systems and integrated systems on right and what's interesting is not only did that operating system create an easier way for people to essentially program information systems it all also launched the whole discipline of information science and and what's interesting about that is that's what's actually carried us essentially up until now I would argue right I mean for the last you know 50 years really not that's a long time right and so what we're seeing you know and then and then what we saw was around the 90s you know people looked at you know system/360 was very heavy was a heavy system and and that's when UNIX came on the scene you know across the across the bay here in Berkeley where they you know had the BSD license they open source the system and what was interesting is you know Linus Torvalds went over and tried to use unix right himself on his own machine just like all of you have your own machines and want to do analytics he looked at this you know large operating system and said I can't use unix so I'm going to build my own called Linux right named after himself and that became pervasive and why did it become pervasive you know my hypothesis is that it became pervasive because he created a community and it was free and it was available and people started building applications and in my mind that's really what launched viet you know the age of the internet really it launched computer science i would say as a discipline up until that point you know that was around the time i was actually in college and I remember you know as I was in electrical engineering and looking across the street and seeing you know the computer science building going up at the University of Washington so in our lifetime right I we already saw computer science kind of take hold and that's you know no one would have imagined you know I don't think Linus would have imagined you know Facebook or Twitter or Amazon or any of these you know companies that exist today that took advantage of that operating system right linux so you know my thesis now is that apache spark is really emerging as the you know analytics operating system I use that and I know people kind of throw throw stones at me when I say something like that because like I will an operating system follows these characteristics well if you look at you know what happened with Hadoop you know it was exactly like UNIX or I do is this heavy like no one has Hadoop and salt how many of you have had you installed on your laptop's a few okay so it's not pleasurable experience and I would argue that it's not necessarily the best use of Hadoop on a single you know on a laptop right so what happened you know essentially a bunch of you know smarter people than myself matei and others you know came up with apache spark because they looked at it as saying you know that's not a very easy you know system to program on right the you know MapReduce was was you know pretty painful to use right and so so they came up with apache spark and i think a lot of the comparisons with hadoop are our fair but in they and they tend to point to performance but that's not really what this is about right what it's really about is ease of use and that sounds like a very marketing you know term but i really believe that and it's about data access and so you know last year as I mentioned we invest in Apache spark because we needed a way frankly internally to you know essentially standardized if you will or simplify the way that we build our analytic capabilities and those range all the way up to Watson and what we're doing with cognitive so you know Watson is using spark you know all the way down to what we're doing you know in the open source right with our bluemix services like spark as a service and these capabilities so it's pretty exciting times at IBM what was actually pretty exciting to see was that you know there wasn't like a mandate that came from Ginni or that came from you know the board of directors that said hey you must use spark that's the hot thing it was happening organically internally already so it was actually for so what I did when I joined IBM just over a year ago was I observed this I said look this is organically you know growing inside of IBM and all I did was expose that to the world I think IBM you know over the past 50 years has had has had this kind of thinking of the not built here syndrome and that's then that's changing and so I'm excited because as Alexi mentioned we're sponsoring more community events like these we also launched datapalooza last year in fact we just had an event in China we were expecting you know 50 people 250 people showed up and these weren't like you know students or people just like kicking the tires these were you know also data scientists you know phd's researchers so it's really taking off we are now i want to say at about six datapalooza both locally or domestically and internationally the next one we're having is in denver at galvanize in denver as well as in tel aviv so you know back in 2011 when DJ patel said you know data science is the sexiest career of our of our time i believe it it's true and it is you know that what's different about the information age then in the 60s and i would call the data science era or the cognitive era today is that people are far more creative today the technology is a lot more elastic and a lot more fluid so it's really up to all of you to make you know the most out of this time and not just the technology but this era where people are more open the culture is more creative and who knows right what's the next Facebook you know for data science right what does that application look like I don't think people really have a clear idea of what that can be yet so so it's a pretty exciting time so with that I'm going to leave you to this awesome event 170 speakers that's incredible let's give a quick hand for Alexei for pulling this together I swear and with that I'll bring him back and we'll get started thanks Lexi