scale.bythebay.io: Shiva Chaitanya Interview
Recording: scale.bythebay.io: Shiva Chaitanya Interview
I'm Shiva Chaitanya I work a Netflix for depersonalization infrastructure teen I'm a senior software engineer and we help our machine learning teams accelerate a BTS [Music] one of the major challenges for us is collecting a lot of data for machine learning we have more than hundred million members using Netflix and there is a lot of user behavior that we want to capture because really the more the data that we can feed to our machine learning models we can learn better about our users and organizing all the data in a standard format and feeding into machine learning pipelines which are both open source and proprietary is a big challenge for us with scale especially that pipelines that are real-time that means not that need to finish with a pretty strict SLA let's say we have a streaming job that needs to work on micro batches of data and need to finish that within 10 minutes one of the challenges that we keep facing is because our jobs depend on so many external services even if one of them goes down that tends to back up our processing and it takes a while for us to debug what's going on and because it's real time we don't have the luxury to rerun it or backfill the data for ones that we didn't process earlier and we need to keep moving on so informing the right teams debugging it in a fast manner as possible as far as it's hard for us and especially when the scale is large the problems get compounded [Music] as we keep growing as a company we just need to make sure that things that we take for granted at a smaller scale continue to work and as we on board a lot more tests as we collect more data about our users as the demographics keep changing we expand to newer countries where the infrastructure may not be the same so ensuring the status quo for a much larger scale in low bandwidth countries making sure we collect all the data in a timely matter all that is going to be a significant challenge for us machine learning lends itself naturally to functional programming because you've taken a lot of data then you apply vector of operations and all of on that data and you typically if you look at tens of flow frameworks or spark machine learning lab library which is written in Scala they taken a bunch of data apply a lot of transforms using vector algebra and functional programming let's you think about that more naturally than say purely imperative way of writing these programs so we use scholar quite a bit in Netflix for all of machine learning processing [Music] my favorite part was learning about Scala because I'm a new adopter of Scala so going especially to the functional track and learning all the new ways of using Scala to meet my needs was eye-opening to me and I would you know given a chance I will keep coming you