scale.bythebay.io: Neville Li Interview
Recording: scale.bythebay.io: Neville Li Interview
so my name is Nev ollie and I work for Spotify as a data infrastructure engineer well probably the biggest issue is supporting a very large engineering organization so we have maybe 200 to 300 developers using Scala for data pipelines machine learning on a daily basis and be able to build libraries that's easy to use and also performance and support that many users that's the biggest challenge for us I think because we use Scala for data processing and in a fairly functional way there's some problem with the performance especially when it comes to allocating tiny objects when you do data processing with immutable collections and in-memory data structures it could potentially generate a lot of temporary objects that cause problem with the chaplet JVM GC garbage collection so this is one of the challenge that we had to face [Music] I see so that one project we're working with is called Apache beam and we have a SCADA idea for a cross shield so the idea is to be able to do both batch and streaming within the same unified model it sounds really good in paper but in reality streaming is probably more complex than batch and a lot of unknowns like how do you scandal or failures latency delays those kind of problems it could make the code of pipelines a lot more complex and that's something we have to deal with I think if the biggest benefits for us is developer productivity because most of all the applause are not probably not real functional programmers or engineers a lot of them from a data science background but functional programming for data processing it just makes a lot of sense you apply functional transformation to immutable data and then it fits the mentality of how scientists or mathematician would think so I think this is the biggest benefit I think it's it's the community the people that come here get to meet everyone in the Bay Area and from all over the world people who are doing some really advanced stuff with Scala pose in reactive and in data processing and catch a medium I think that's the greatest thing about scale by the base [Music]