scale.bythebay.io: Pathikrit Bhowmick Interview

Recording: scale.bythebay.io: Pathikrit Bhowmick Interview

[Music] my name is Prateek Adamic I work at a hedge fund called core to management we're a hedge fund located in New York and a venture capital in Menlo Park I'm a principal engineer there working specifically on Big Data the key issues for us is how to run complex algorithms on huge datasets of multi billion rows in a reasonable amount of time and how to quickly iterate on those algorithms [Music] yeah so we use park a lot and there there are some caveats and lessons we learned on the way while using technologies like spark like you could have weird distribution issues Kunis in your data that makes it very tough to do some even basic operations like goodbye and because some nodes get overloaded and some are just empty waiting doing nothing I think going from tens of billions to hundreds of billions to to hundreds of billions we have to rethink how we can do it in a incremental fast way versus in a batch world because it's hard to iterate on those enough if we process hundred billion at a time yeah so we follow functional programming idioms as much as we can having non mutable code or state in our court helps it really easy to reason about our codes even though we write lie write large systems with a very small team but we're very confident about what we write and our bugs usually are not errors that we wrote during coding it's our errors in our certain our misunderstanding of the world versus I made a typo I made something in our code I think it's the community I hear about our interact with this with a lot of people over github or some chat room it's really good to finally meet them face to face and put a name to a face or face to a name and talk to them and hang out with them you