scale.bythebay.io: Sergei Winitzki Interview
Recording: scale.bythebay.io: Sergei Winitzki Interview
[Music] my name is Sergei Magnitsky I'm a senior in software engineer at workday and I'm working on backhand machine learning and data pipelines [Music] mostly it is how to make reliable data pipeline we need to process sometimes large data sometimes smaller data but the most important question is correctness and reliability another concern is data security so the most important thing for us is to avoid bugs that can expose customers data or bugs that can lead to data loss one issue is that we have varying data volumes sometimes volumes grow sometimes we don't expect that so we need to be elastic and that has been a challenge because we're not always using architectures that are sufficiently scalable we have some legacy code work there is not a recent startup so what I'm working on is Scala code and we're trying to do it right but there is legacy code that gives us some challenges so actually we're using Scala to overcome their architectural challenges for scaling the data [Music] so in the future we expect to grow our data volumes significantly in certain products so we need to plan an architecture today that will enable us to handle that one one thing for instance is we have a search engine that means a lot of indexing and that has become slow with growing volume so we refactored that recently into a streaming pipeline so that enables us to scale it horizontally much easier so I would say there are two key benefits that I feel right now in my work the first benefit is that we don't debug much the functional programming especially using Scala allows us to write programs such that they mostly work correctly without debugging and the errors or bugs that we find are not the kind of bugs that require us to sit all night and debug and trace these are significant problems with understanding what we want to do these are really high-level questions and not something like sitting with the debugger tracing the program for hours and hours something I have done before Biff I when I was not using functional programming I was a front-end developer so this is the first key benefit second is architectures we can use modern architectures such as Kafka akka streaming our HTTP these architectures make us much more productive so that I would I would say it would be impossible without functional programming and without Scala even to come up with these architectures effectively [Music] my favorite part is that there are people from all kinds of places that do very unusual and unexpected things with Scala I'm primarily in for Scala programming expertise and new applications and there are things that I would never have thought about that people have done and I can talk to them after the talk I mean it's not always so easy to see immediately what they want to say so I can talk to them and I can get in touch with them and learn that's my absolutely most favorite thing I've been at these conferences for three years now it has been consistently my experience