scale.bythebay.io: John A. De Goes Interview
Recording: scale.bythebay.io: John A. De Goes Interview
I'm John Digos and i'm CTO of SLAM data and i'm also co-founder one of the big things for us is correctness because our software is an advanced analytics compiler and they can take workflows and it can deploy them on any target infrastructure whether that's MongoDB or MarkLogic or Couchbase or spark or Postgres or whatever it is and the queries that we take as input into our system have to be translated down into something that has a precise meaning and it needs to give back the exact correct results a hundred percent of the time and so we we can't have off-by-one errors we can't have those types of things in our system and so having correct software that also runs performant ly enough to satisfy our customers needs is extremely important to us [Music] so I'd say that we run on the JVM and one of the issues with scaling that's come up is that it can be hard especially in a programming language like Scala but even Java to some extent to minimize heap churn there's just so much junk constantly created and so much garbage collection and getting efficient code that is also correct and typesafe and ideally functional and composable getting all that to be really fast and and to not consume a lot of memory has been a challenge so we'll have challenges around distribution so as we go from a single node solution to a multi node solution we have to figure out how we're gonna deploy this and that's gonna depend on how our customers are deploying other types of infrastructure are they using meso so these in something that yarn on what are they using and so we're gonna have to figure out the package and distribution story and then along with that you know error recovery and redundancy and load balancing and all the messy stuff that comes when you take that leap from single node solution to clustered solution [Music] so functional programming I think more than anything else it helps us reason about the software that we write we're able to understand it because in functional programming a equals B so you know the meaning of a simply by substituting B whenever you see an A and this property called equational reasoning it allows you to know the mean of any expression merely by looking at it and seeing what that equation is defined as and then doing substitution also I think it it helps our developers code without fear when they see a type signature that says this is a function that takes an N or turns a B they know for a fact they can feed it any a and it's always going to return and it's always going to return a B it's not gonna return null it's not gonna throw an exception it's always gonna return a B and not only that but for the same a it's always going to return the same B because all functions are you know total and deterministic and they're only effect is computing their value so when our programmers jump into a code base they can look at the types and they can know with confidence what it does they don't have to program fearfully and they don't have to dig into the implementation of a function to understand what it does because all that stuff is reflected and it's tight and it's type classes and its laws and so forth so it helps our programmers manage the complexity of what is a very very large open-source scholar project it's hundreds of thousands of lines of Scala code and our team is able to be very productive in there and we can add more people to the team and our velocity doesn't slow down because these are very strong properties that help you build maintainable software but also we have we have no runtime errors so a fact in which I'm quite proud we've never like had a customer say oh no nullpointerexception or some runtime exception thrown it doesn't happen our software just runs and yeah occasionally you'll run out of memory and cross the JV in that way or something like that which you can't guarantee from the types but like it's just very very solid software that just does this job and keeps on running and that has a big effect on our customer satisfaction and not just that but large or a developer satisfaction because they spend less of their time stepping through code and debuggers and so forth and more of it just Goldeen the features needed to our business and make our customers happy [Music] so I really love the networking I think I I love some of the talks as well but it's just great to see so many people that I know who work in the functional programming community who came here and gave really great talks and just get a chance to catch up and see what the industry is doing it's it's wonderful [Music]