scale.bythebay.io: Fred Reiss Interview
Recording: scale.bythebay.io: Fred Reiss Interview
hi I'm Fred Ryce I work for IBM at the IBM spark Technology Center here in San Francisco where I'm the chief architect we focus broadly on the use of Apache spark in the enterprise and there are lots of very interesting issues that come up anywhere from how to get a patchy spark integrated with enterprise IT to how do you make machine learning algorithms run at scale we also have a group that's working very hard on making a data warehousing operations work at scale reliably on apache spark a good example of an issue at scale that we've just been working on the last couple weeks is making some of the transformation operations in apache spark scaled better with really wide large data sets so things like recoding string columns when you have very large number of distinct values we've been making these things scale better by better taking advantage of the capabilities of the sequel engine built into spark in the future spark isn't going to need to move forward into new areas of machine learning you're seeing that a lot with some of the add-on packages like tensor frames and tensor flow on SPARC being added to spark but I think as the integration with deep learning gets deeper into the system we're going to see some architectural changes happening to be able to handle those algorithms at scale with top efficiency functional programming is really at the core of Apache spark I don't think we would have a spark technology Center without Scala without dysfunctional programming without that ability to ship a function to every node of a cluster and operate on performed distributed computation using that function that is one of the core benefits of Apache spark it makes the api's for programmers so much easy - easier to use makes the experience of writing a distributed application feel like you're writing an application that runs on one machine my favorite part about scaler by the bay is almost called that Scala by the bay there but the fact that it's such a nice raw new conference with a lot of edgy content if you go to a more mainstream conference you'll see a lot of very polished talks but it's about stuff that was done one or two years ago here it's scaled by the day we have a mix of some of that older content a lot of newer topics that are just starting out and it's nice to be able to catch those new projects when they're at the infancy [Music] you [Music]