scala.bythbay.io: Nimbus Goehausen
Recording: scala.bythbay.io: Nimbus Goehausen
you my name is Nimbus go howson i work at bloomberg and i am a spark platform engineer so I used to work at a web startup called radius where we were doing you know kind of his large-scale business matching and we eventually moved from like Python into Hadoop native MapReduce into that kind of cascading thing and eventually when spark was around one point oh I decided to move it on to spark so I've been working with spark around the since around the time that it moved to one point oh and i started usually started using it with pi spark is python because you know was restraint forward but really that's a PI spark is a gateway drug for Scala and because just teaching you the collections API and I was finding oh to do make the use of this effectively it's Scala's the way to do it with spark so you know kind of attending conferences and kind of you know kind of talking with people in community you know met some people at Bloomberg learned that they were you know kind of trying to move some things on the spark and do is you know kind of there's a lot of people excited about it and you know wanted to get it working but it's difficult so they were looking for people who have been using it and wanted to contribute in a way that's slightly more general like you know kind of having the right sort of infrastructure and tools to support people using it in effect your place so that's that's how i came into my current role [Music] so currently i'm actually working on something that is a little slightly unrelated to what i did a talk on today the latest stuff is i'm trying to actually make that environment happen in bloomberg and its spark on cooper Nettie's basically although in specific its spark on openshift and there are other people kind of pursuing this as well that i am getting newly acquainted with including like red hat they have like a kind of a you know thing on github called rad analytics and i'm seeing if i can make use of it but i'm looking well my ideal stack right now is I want one user / spark cluster when that spark cluster is a set of containers running on Cooper Nettie's open shift and you have separate separate storage services you might have several people several users on the same team that want to share storage and I'm thinking ideally assuming it works all appropriately using a Luxio as the tiered storage so you have like memory best but then you have some kind of under storage probably SEF which has the s3 API or you could just use s3 and that way you could share like the same namespace among multiple users and not have to have like essentially duplicate things and also have people be able to turn on and off clusters without it being much of a big deal so that's that's what I'm looking to to do [Music] I the the Corsair talk that Alexander gave our I'm not sure if I'm saying its last night Alexander Ark Ark involved he that's I I really like having having better better build tools and and you know kind of SPT can be difficult to time so any anyone that's trying to make my life easier like that I very really appreciate that the Monad talks several to choose from I thought Kelly Robinson's talk was was pretty nice and it also got me thinking you know maybe I should use maybe I should use shapeless or you know Scala scalzi door for some of these things in order to kind of make the the abstractions I want I want to make [Music]