scale.bythebay.io: Vladimir Bacvanski Interview
Recording: scale.bythebay.io: Vladimir Bacvanski Interview
Wladimir back landscape principal architect with strategic architecture at PayPal [Music] one of the big engineering issues is the spread of functionality across multiple machines the processing of data at our scale can be accomplished only if we distribute the processing to a large number of machines in the cluster and that brings in various technical challenges with regard to performance with regard to distribution of traceability of the work and also with regard to accessing data one of the particularly issues with scale is that with the move from analytic to distributed systems particularly these days to micro services we have a number of services or components that are communicating with each other this communication is now happening across the network and with that it is bringing additional latencies so suddenly we have ability to process much more by spreading our processing on many machines but on the other hand the communication between parts are not in the same address space anymore but we have to go through the network and that is introducing latencies and also in some ways you can think that this is reducing the efficiency of our computing in general which is then causing increased cost and pure utilization of our software so I think what we have experienced in the engineering community is this thread and going from analytic to highly distributed systems but then for the optimization the next thing that we will need to do is to go from this highly distributed and start grouping things together to be located on the same machine for better efficiency [Music] so one of the natural extensions with processing at scale is that our datasets are also growing so so far we have seen at the area of big data being relegated to specialized systems like spark playing traditional Hadoop and similar what we see now is that the big data is becoming part of normal operations when the separation of normal processing and big data is getting blurred because often in order to do some real-time processing even for 10 sections that we have today we need to do some processing on the big data side decide of what we need to do and based on that input perform some operations real time one of the interesting aspects of this type of technologies is the growing role of streaming architectures which are able to process events in near real-time or through a real-time and this is a very nice blend of conventional processing where we expect results very fast and also Big Data where we are dealing with huge volumes of data [Music] so functional programming is very attractive for the reason that it is reducing the impedance mismatch between our ideas and how we express them in the code with functional languages we typically tend to work at the higher level of abstraction so we can focus better on the intent that we have in writing the program one of the interesting things in a multi paradigm language like Scala is that we can express ourselves in a functional way but sometimes and that way is efficient for the developer but not necessarily for the machine and in such cases we can always revert to a more conventional procedural approach and do it as an optimization so we can combine the high expressiveness and where needed we can also do the low-level optimization my favorite part of scale by the bay is the community and the people that come to this event first the event is organized it really really well and it brings together the community in a very spontaneous way if you want where the communications and interactions with various really interesting people in software development and data communities are easy to make and one gets exposed to so many different ideas and one can brainstorm with people and see how these ideas actually applied to one's work or area of research so very beneficial conference also great fun and definitely an event that I don't miss any year you