SF Scala: Alexy Khrabrov interviews Konrad Malawski

SF Scala: Alexy Khrabrov interviews Konrad Malawski

Recording: SF Scala: Alexy Khrabrov interviews Konrad Malawski

hello everybody I'm Alexi kov the organizer of SF Scola and here we on location at typ safe and with us today we have Conor Bowski who is a software engineer on the AA team hello everyone H yeah so uh Conor is going to talk about ractive dreams he's a well-known contributor scholar community and he resides in in kov in Poland yeah and uh it's you know not often we get Con in this parts of the world so we're going to ask him all kinds of questions about schola and open source so um first of all I noticed you know through many channels like I see very strong uh open source community in Poland specifically a lot of strong Scala centc companies uh what is it about polish open source Community which kind of create this strong uh culture yeah so that's actually a funny story so when I first started Scara four or five years ago I guess um there wasn't many scholar developers so in kco it was basically me and when I would go to a Meetup and say I'm really interested in scholar people would you know laugh a lot about me like what is this funny language you're you're interested in and nowadays yeah it's really crazy like everybody's both into it and actually lots of companies really centering around it and support and just helping other companies to go in into it and deliver you know an entire team basically to help your company start with it so that's the usual setup I see nowadays uh with the new company starting out with Scola and helping out others so Poland is Al also very popular for what do we call I guess near Shoring so it's not Outsourcing it's just near Shoring so people realize we really want a strong team and i' rather pay for a strong team and two free guys and have them either visit me very often maybe in England maybe in Poland maybe in Germany MH so that's a usual settlement that's how people got into Scola I think because there's both a very strong need in Europe for strong experts in the area and these polish companies kind of recognized the need and now are benefiting from it that why are they kind of such such a you know good amount of strong Engineers was it because of Education Community like what are the driving forces uh both things that you actually mentioned so both education very very strong like universities in Poland are really topnotch uh technically um I mean well technically has a wrong meaning in English um in technological Fields that's what I meant to say yes nonh humanitarian yeah correct so in kco we have yeah five universities and two of which are really like topnotch and top tens of worldwi yski and the aggh which is the technology University that's where I graduated from and in warsa we have another one which is also top 10 of in the worldwide rankings so that's one part of it but that's doesn't that doesn't make a good software developer in the real world but may make a good algorithmic like person right but then you need to do the brid kind of to what those act businesses need right there's lots of Trends and lots of Technology that's somewhat related to algorithmic stuff but not directly right so we have a very strong community in terms of um conferences and user groups so there for all kinds of Technologies user groups in many cities so it's not very centered in only one city but in all the cities you basically have some kind of user group for Eva Scara or random technology name whatnot right and then we have a bunch of community-driven conferences one of which I'm also running it's the geekon conference so geek conference uh and it's been running uh 6 years now we got 1,200 people each time so pretty big we rent a cinema basically nice when do you run it usually it's in May every May in CCO and what dates uh second week of May always yeah if you want to come down we usually help out um if you're from the US for example and you want to speak in Europe we usually try to help out speakers to come over not always can we cover all the costs but sometimes we split it up so we pay the flight and you pay the hotel or something like that because it's a Community Driven thing that's good to know may every year correct okay cool keep it in mind and because it's Community Driven and the community nowadays is very inter scholar so there's been a lot of scholar talks on that one even though the focus of the conference is very anything around the jvm basically yes and nowadays Scola is very much on top of that and with the fast data things like spark and all the other things around it there's really lots of scolar happening in those areas as well so this again feeds into interest which feeds into interest into the meetups which feeds into good developers so yes interesting so here's kind of question I don't have an answer right I have theories but so you know for many years we have all these Niche languages and FP F programming was pretty Niche and we had you know Hill and a camel now there's a list before that and scheme and airling and right all kind of stuff and it basically stays about the same size so you know uh M of some of these languages you know in San Francisco basically 20 30 people right like in in the bakery uh you know when we do meet up like tonight we have 250 people on the weight list right and so um what is it about Scala which made it basically the main kind of function program language object function which broke through to mainstream I mean it's not as big as Java but it's fairly big right so what kind of made it I would say the main IND support function program langage what what do you think why did this happen right so I I think the main thing about it can be described in one word which is pragmatism MH because um like some of the languages you said um like hll very nice pure functional Concepts but not as much real world application of them right and sometimes when we took Concepts that's a a Side Story maybe but such pure functional Concepts taken from H directly into scolar one of them being enumerates and ites back there a few years ago in play turns out well one they're too complex really it's a very nice pure model but is really too complex and two well not not that fast right so it's very pure it's very nice and we strive to go into the functional direction of languages but sometimes we are pragmatic enough to say well it's not worth the tradeoff and I think that pragmatism has driven Scara into a good spot where we do get all the benefits of functional programming and thinking in that style however sometimes we do acknowledge well sorting an array just that's the fastest way so I'm going to sort an array yes right makes sense and so with people uh we mention complexity right and so that's a typical objection we get right that kind of scull is too complex so when companies are avalia in this right how do you discuss this kind of what what's kind of your way to discuss you know this argument the SCH complex yeah um the best counter example to that claim I have is um so before joining types I was working at eBay mhm so maybe not that known for adapting the latest and greatest however um so we were basically one of the first teams in the London offices MH and of course starting out with the existing stack which was all Java based and spring based and whatnot however I managed and a few other guys managed to convince the rest of teams to Tri out Scara mhm and well one of the things these teams were saying then that well actually this is so simple and what I'm getting at is well the entirety of Scala yeah it's pretty complex because it's marrying a lot of Concepts however it's doing so in a um consistent way right so even though you learn maybe more Concepts but they fit together so I think the term complex is overloaded it's a broad language as I like to put it and you don't need to learn or even use all of the breath of it yes and even in our teams we would decide well let's not go into the Super advanced stuff because we're just learning right because we're a new team we're you know starting out and when you actually restrain yourself to the very simple things the good parts as Martin likes to call them and actually the code is both much simpler where is less code for one thing and it's more types safe in terms of when you compare it to a um Java I don't know um annotation driven application you lose all all of the safety guarantees because it's now all annotation driven which is not type checked at all yes so you get more safety even while working on the code base so I do think it's a sometimes misunderstood term in the um from people who don't know Scola yet yes yes makes sense yeah I I remember you know Mar basically has a talk where you know he explains that SC is actually simple and kind of he actually I think shows how you know object orientation function program work together yeah and to me the way like the easiest proof how simple it is that success of spark right and Mark has this talk called you know spark the ultimate scull collections at Big dat skull in August and uh where he actually said that spark is a sculla DSL right scull makes it very easy to write something like spark right the collection interface is very intuitive yeah right so I think that's a testimony right in my mind the reason for spark adoption one of the key reasons was the Simplicity of the collection yeah exactly very good point and it was both simple to write such a DSL in scolar and then it's simple to use such a DSL in scolar for people who have actually no idea that they're using scolar which is a pretty typical use of scolar I see in the field and even in eBay you would get people who are very familiar with let's say r or matlb or Python and they get a spark script program however you want to call it and they just understand it yes they don't even realize it's scar because it just makes sense oh it's a filter it's a group etc etc ex exactly so I think you know if if you look at subset of Scholars it's us for spark yeah that's a very natural way to write data programs right so but you're on the AR team and kind of like let's kind of make a connection here right because uh obviously you know now more people are familiar with spark and maybe with play than with AA as a part of of of play uh so and this kind of a little bit different domains right so we have kind of you cannot write an API supporting a million iPhones in spark yeah uh right because of this kind of widespread ad optional spark some people actually don't realize this I got this question you know why just use park right and so are different use cases and so uh What uh use cases do you see for AA uh right now being a t save kind of how are customers using AA what are the best situations where AR shines right so at the core of it it's a really a toolkit that you can use for building either highly concurrent or highly scalable in the sense of multiple nodes or well if you want to stay local multiple threats and the core thing about it is the concept that you use to think about it stays the same right because you only think about messaging and once you Wroten something using messaging it it feels native that it's concurrent and it can be distributed easily so it's the opposite of people what people did in previous attempts to do distributed computing where people would hide the complexities and make it look like a method which we believe is a well the wrong thing to go about it because well it doesn't behave like a method because there's latencies involved however when you turn it around and you make messaging a first class citizen now suddenly you all the time realize well there's going to be some latency and when I'm distributed there may be message loss and now it's it feels right it makes sense and it's not surprising anymore MH so what we see people use AA for is um when they need High throughput applications or they need really to react uh very dynamically to to incoming load like today we have a talk by the guil uh no the zelando guys um but they have a very similar examp use case to what guilt has right they do flash sales right yes so I want to scale up to a few thousand nodes but only basically for an hour and then I want to scale down again that's why I want to be on Amazon so I can yeah boom you just add the noes they spread the load and then you collapse again and these are exactly the use cases AO Rel shines because of the fast messaging the fast connection of the cluster I mean the last time we benchmarked how fast a cluster can connect multi more notes into it we did a benchmark on Google's compute engine and I think we scaled up to 5,000 noes in 2 minutes which is including starting the nodes W so that's pretty impressive the latency of joining nodes to the cluster and they're ready to serve the requests is really really low interesting so uh so in in in scard We have basically two major Frameworks for RPC we have AR we have finagle I don't know how much experience we have with finagle a little bit yeah and so you know in the summer we did actually the first finagle conference as a schar by the way and kind of it's very interesting I'm curious uh kind of this just two different uh ways of writing uh programs you know how would one approach kind of picking one versus the other kind of like not really seen this communities overlap much because I think it finagle in Twitter develops separately but and now it gets more into mainam there are some customers finagle how would you compare the two things and kind of like when one should use what I think now uh now especially it will be a little bit closer in terms of usage patterns especially because we have AKA streams and AKA HTTP based on AA streams and both of these based on reactive streams which I'm talking about today M so the core about these thing these apis is one they're type safe as in you have the actual types of messages you can handle which was the usual complaint against yes so we do have addressed that one in the new apis is it now the recommended API the mainstream yes and no so for certain patterns if you have a a communication pattern that looks more or less like a pipeline so stuff coming from this end to the other end then yeah definitely this is the thing we do want people to use for these kinds of things however AKA still really shines if you have a mesh like thing right for example if you're modeling uh devices talking to one another or you're doing a simulation that you know you're bring up completely unpredictable device correct yeah and it's you know you cannot set it up up front because you don't know what it is up front yes and in these situations AKA really shines because it's so Dynamic nature and other Frameworks um including AA streams and finagle and stuff like that they're really not meant for Pure top stuff it's I take in finagle case a future and I transform a future and eventually it's going to be a response yes so we take it a little bit further with a htdp so we don't say it's a future we say it's a continuous stream of something an example being an HTTP server is a stream of incoming requests and they stream of outgoing responses M and this really allows for a lot of interesting implementation kind of details which affect the usage of these tools so basically what we say well HTTP is such a flow of incoming requests to in outcoming responses yes so far so easy however that's basically just a wrapper on top of something which is TCP which is incoming bytes outgoing bytes yes so we can basically exchange the underlying engine at will because it's just this this flow and we have all the infrastructure that we can dynamically compose these so in when you want to write a test you don't need to start a server because it's just the stream and the most most interesting bit about it is the back pressure thing so that's actually something many of AA support tickets from customers or complaints would be about so sometimes uh customers don't really say that AA is too slow that rarely happens actually and if it happens it's usually some silly mistake what more often happens is AKA being too fast and let me explain what I mean by that sounds like a good problem to have but actually isn't um so what we mean by that and what customers see is that well for example it's very simple to produce loads of data and I'm just sending this data to this other note for it to work on it but this other note can't keep up so what ends up happening is eventually one or the other note will start buffering the data and eventually either start GC like crazy or it needs to drop the data or it needs well or it blows up up with an out of memory error yes and this happens in all kinds of systems but in distributed systems or in asynchronous systems in general it's really all over the place so reactive streams and our implementations basically address this need of communicating back pressure so if the downstream can't keep up I will know about that and I can do something about it so I can use the signal to from the pipeline maybe add add more resources or slow down the generating of the data and stuff like that so this is actually a lesson that we've learned the hard way um yeah exactly from our experience with uh customers and production systems and also we've used that experience both to build reactive streams and our implementations but also to help spark actually so we have a team at typ safe who which is basically dedicated to help spark be better MH and in the in the latest uh spark release these how many is it now three4 guys basically spent a few months working on spark back pressure mechanisms which was addressing the problem of well if if I have a you know huge spike in my load spark is you know failing right people would say that people would not realize what's happening what was happening is exactly that that one note would be sending too much to another Noe in the spark cluster and they would you know oscillate and start failing with out of memory so lessons that we've learned on our side we're bringing it also into other valuable projects in this scar ecosystem like spark right so that's already merged and there's going to be more strategies coming right so in spark since you're using it for for example some analytics right so a valuable strategy may be um to start sampling the data right to not collect all of it because I can't keep up so I can collect 80 90% whatever until a new note comes in and then we can go full speed again exactly so well this is great so this is actually nice connection kind of back to sparkk from from right so which is probably kind of good point to to wrap up and make a prediction uh as we like to do so so you're on thead team obviously there is a lot of new uh uh kind of progress and kind of reactive systems is kind of uh uh spreading not I would say not everybody majority of the world does not know what we know yet right like it's our job to to uh to push the of the world right and and kind of uh what do you think where we're going to be in the year you know let's make like a prediction so next year next jav one right systems AA how many like what what's what would be the adoption of reactive systems just you know what what should we see you know what kind of customers can benefit from this like any wild guesses are fine right happen wild guesses so my wild guess is we're already seeing that but I think it's going to pick up even more now um so we're seeing lot of how call them dinosaurs they don't like to be called like that and maybe I shouldn't but you know who I mean when I say that so they're very slowly adopting companies which like stuff that has proven over the years and you know are slow to adopt stuff they are getting onto our platform and yeah it's stuff like Banks and hotelier systems and Airlines these big players right and when they make decisions it's usually both they make it very slowly and then they need to be really convinced about it and they don't just watch a presentation and this idea let's use that they really want hard numbers and now we're pretty much in a position to show hard numbers about you know there's been so many actual migrations to our Technologies and this style of development reactive reactive systems and it has worked out for these companies so it's finally convincing the big players to move along as well all right and so hopeful it means that the they will have to adopt Scola and learn it along the way I hope so yeah all right so let's hope that reactive will be as big a driver for skull as spark and play before it exactly all right thanks con we're looking forward to your talk and thanks for coming thank you very much thanks so [Music]