scale.bythebay.io: Alexandre Bergeron Interview
Recording: scale.bythebay.io: Alexandre Bergeron Interview
my name is Alex Bergeron I'm a senior software engineer at the Reno data lab here in San Francisco I'm responsible for developing me different that many different features and services and I can features of the terminal platform which allows data scientists to build train and productionize their models and keep track of all of their operations and all of their work in their day jobs [Music] some of these issues that we have writing a platform that allows people to run models which means arbitrary code written in Python and are in productions means that making sure that these models running on these services don't interfere too much with each other don't consume too much resources don't send too much data to old logs and Entering that all of these different services and all of our platform remains responsive under load and under stress and continues to behave well also we're having a lot of issues trying to build trying to build our team and trying to develop our platform as we move forward we were still moving from a mostly monolithic monolithic application something that is more service oriented and whilst remaining deployable to all of our customers so as an enterprise software we're in an interesting situation where instead of having many thousands people using a deployment it could be anywhere from five to a hundred which means that our scalability issues mostly comes from the models themselves we're we've often had some issues recently where recently with our logging in infrastructure and at the same time being in an enterprise space means that or issues with scalability will need to be deployable on multi at multiple customers so we do need to remain to keep our solutions simple in our particular use case which is kind of interesting you see at many of many different previous companies we've often went for more complex more full-featured software's that would guarantee that we'd be able to scale on one deployment when you need to deploy that at multiple customers and some of them will never need to scale up it becomes a lot more interesting and you become a lot more interested in having some simple solutions that will still allow you to scale to your current problems [Music] well particular to my work at thermal data lab and also what I'm seeing at many different companies there's a tendency to go with really complex solutions that are meant to scale at every at huge levels that are meant for companies like through Google Twitter LinkedIn where they need it to scale to millions millions if not billions of users I tend to be more programmatic yet pragmatic and realistic and I often feel that there's a lot of companies that invested a lot of time building some complicated infrastructure that actually does not is not really the best fit for the level of scalability that their product is focused on which ultimately leads to having an increased load and on the engineer maybe engineers being uncle and there's definitely an interesting issue that might arise where like it can become art to get some engineers that are really specialized in these scalability issues because they tend to be really in them and really focus up on the complex infrastructures that they're bringing and on making them as efficient and as performing as they can be [Music] yeah yeah yeah yeah I've definitely really benefited a lot from functional programming like it's to be honest it's been four so programming has been an interest of mine as I was still it to my programmers class I fell on an SQL tutorial trying to understand for soul programming and monads and not really being great at it I definitely got back into fights on programming when I started doing scale about around five years ago it definitely felt like it allowed me to write some simple some clean code that was declarative and not imperative that was a lot simpler to debug and analyze yet you can you'll argue that you don't really know at runtime if you're using a collections like the scale collections library what instance you are running and it can become somewhat art to reason about the performance of your operations but ultimately the code that you're written instead of having something that is very complex very long and over taught a lot about which data structure you use you just read from internal data to it a point to achieve the result that you want which ultimately in my previous license the engineers and as packaging engineer has been really practical as getting cold quickly and then you can optimize this later on if you need it I've definitely enjoyed a lot seeing lots of great technical talk about some of the leaders on the scholar community about some libraries that are really popular I'm doing about the FS to talk yesterday there's been a lot of great tutorials on tight glasses on monads applicatives and it's just being a great conference and a great occasion to see what's happening in the community at large right these days