scale.bythebay.io: Leif Wickland Interview
Recording: scale.bythebay.io: Leif Wickland Interview
[Music] hi I'm Lee footling I'm a principal engineer at the Rubicon project so I think the two most interesting engineering issues we run to work our performance issues because we deal with large data that we need to process quickly and so if we allow a PhD to walk in and write some Python it's going to be orders of magnitude too slow so the tools that we have at our disposal with Scala helped us out quite a bit and doing better than that and the other is in simply reaching disambiguation quickly so I've been working with data scientists they come up with models who need other people to be able to understand those models and quickly convert them into code that works and so that engineering issue there is more of like a process issue of ensuring that the the humans can communicate with each other clearly so an obvious place where scale hurts us is we need to do some sort of ad hoc analysis and we know that there's data that's busted somewhere in our big data Lake you know we were seeing a production problem we don't know exactly why we know that the logs similar we'll have it if we just drop onto a box and start grepping that's going to take a long time if we just start SSH into boxes that's gonna take too long so having a system that can distribute lis search for us is a whole lot more helpful that's a place that having like a spark shell pointed at our big pile of wads has been really useful for us so the industry that I work in ad tech is going through sort of its typical change very few years and our current issue is that the volume of data that we're seeing keeps doubling every few months which means that we're having to find ways to reduce the cost of processing data just to stay competitive and not lose our margins so being able to process data efficiently at ginormous scales faster every few months as a big deal for us and crucial to keeping the business alive I think I benefit tremendously from functional programming because it allows us especially in its well typed variants to introduce a degree of rigor that gives me a greater degree of confidence when I drop in to look at a pull request somebody's written that it's correct and when I start writing code against what somebody else has done I feel confident that I can compose off of that work in a way that will work without incredible amounts of testing to be confident and I don't have to worry so much that I'm gonna get paged in the morning because something was wrong I have been exceptionally impressed by how talks are answering problems that I've had in the last two weeks that I didn't even know that there were solutions for that I'm finding so the Vegas library that gives me plotting akin to pythons MATLAB is really helpful in exactly what I needed the ML library that we talked about earlier today is really similar to something I need so I need to go back and look at those you