scale.bythebay.io: Kuassi Mensah Interview
Recording: scale.bythebay.io: Kuassi Mensah Interview
[Music] yeah I'm a product manager I I mean I I got a ng I have an engineering degree no but I do mostly product management I don't quite good you know I do small demo but I don't write the software you know I'm a product manager I'm a curator for that management where I can talk about the scalability issues I mean that it's not affecting us well we take care of that you know we mind scalability okay so for example in my talk I was talking about how to access relational table from spark one scalability concern is how to make sure that we can have parallel access to a single table from spark if you have 100 nodes how do you access a single table we've run 100 sparknotes accessing that same table so with the way to deal with that is to have a logical partitioning of the table and then each spark node will access only its logical partition [Music] right so my I will probably speak on behalf of my development team because as I said I'm product manager but I'm involved with the design discussion you know the issues we face and one of the issue we faced was how to avoid the double conversion double type conversion and also you go from sickle type to Java type to Scala tie well how to avoid that and the solution was to inject in the driver the possibility of avoiding one type conversion so we will go from sickle type and produce the raw bytes directly to Scala for example well that's a great question I think personal programming is reducing our QA I mean stuff you know for example it's easier to to to it to detect and to fix you know when you do functional programming you can address individual module or micro service and that way you reduce the keyway workload you reduce the time to QA to debug and fix a problem [Music] yeah it so first of all I was happily surprised by the attendance you know it's a great great conference and I can see a lot of people here so I was first impressed by that I liked the venue and I look forward to learning more about you know everything happening in the reactive space in the the discolor the spark and also learned stuff in the functional space so I look forward to learning a lot of things okay [Music]