Scala 1 26 16 paul kinsky interview

Scala 1 26 16 paul kinsky interview

Recording: Scala 1 26 16 paul kinsky interview

hello everybody I'm Alexa crabber the organizer office of scholar and here we are on location for our first no top of the year the theme of the meetup is akka and it will have several acts on Doc's here we have with us one of the speakers Bachynski hi mom do I introduced myself now yeah right i'm a paul Kinski giving a talk in a little bit about using a cast reams as sort of the lego to connect different stream processing applications right like you want to stream the output of a process to a file to a web socket or like upload it to s3 whatever it's there for like just gluing different things together with stream processing in this case or in this presentation I'm going to be showing how to break right a really quick simple server that uses akka streams to serve a bunch of connections with WebSockets collect events and real time via those web sockets publish them to Kafka by a single stream and then you can use that with spark you can use that with whatever your preferred machine learning system is just to do any kind of data analysis but that's kind of the missing piece and a lot of things right like you've got your cool new popular app it's instrumented you've got your back-end data processing system how do you glue the two together but this is just one possibility but I think it's pretty neat one it's really exciting we tried applications and I won't kind of you know step back a bit and kind of retrace just you know your path to scholar anak alright so it's almost exactly a year since we've met at Northeast column in bandar during the snowpocalypse right and now have another smoke ellipse in another part of east coast but on here fortunately so can tell us a little bit you know what transpired during this year I so you got workman's all this technology is can we join San Francisco tech scene can it just a little bit what you see you know in the Scala seen wearing I see a lot of diversity of application right people aren't just using Scala for any single use case right it's not just like Twitter and a few other small companies like I see people like nitro using scholar for document processing people using it for real time and just tons of companies using the same set of tools for different applications and all like contributing back to open source projects like for example like I work at nitro I was able to open source something I've been working on there avro codegen just that we've made internal there that i put together internally for generating scala case classes from Avro schemas it's similar to like proto buffer thrift right and we're just using that internally and we were able to give back to the community and some small amount and it's a very nonzero-sum situation because everyone's contributing technology everyone was able to use that pool of technology and it helped us out compete people who use worse languages yes and that's actually a very exciting right because we needed our for civilization and and kind of you know we've seen a scallop or debuff used kind of paralyzed features and you know once you start this and you know publish that we suddenly got folks just contributing and improving it right this is oh yeah this uh this one guy he begs her i think is his name he had contributed like a whole new Avro parser because there was some features he was interested in implementing and he'd just like filed a request to do it and then just over a weekend at some point just drop this huge update that would have taken like a few days of work time if we could have gotten the time to implement so we would really appreciate the contributors to this to this project and I think it's really exciting to see kind of everybody else jumping in and using this took a little bit I think that's kind of very a good example for for the kind of data pipelines wish you know our customs is a part of so why you know for folks who don't know what's mike's tag or active systems are you know like why do we send this messages in ever and what happens to them when kind of their life cycle you know whether they come from whether do they go to how does like a stream help I'm gonna move along all right well it isn't anything really specific to using Avro but we get these messages they're created as response to like various imports by our back-end service they're published to Kafka it's a distributed log a very easy way to run like event queues or with multiple subscribers and we just use Avro because that's apparently or that's very widely used data science and that allows our machine learning people to use the tools that they've ready become accustomed to mm-hmm and so kind of when you see lacus news and you mentioned Kafka so what are the kind of typical data pipelines end-to-end can adjust describe some of the components you know we use a natural like some of the components you see people use in other places all right one of the big things that I think is going to become a widely used pattern is well pipelines like you said something that both reads commands from a Kafka topic does some processing right perhaps like downloads files acts on them uploads files does or whatever and then publishes some result for that command with like some concept of failure in the result right so that way you can just have some complex piece of technology out there on its own with the only real part of your system it talks to being Kafka so accepting commands from CAF ka-shing's results to CAFTA you can very easily like paralyzed for example you just run more instances haven't talked to Kathy with the same group by day mm-hmm but yeah basically the thing that both consumes messages from Kafka and then produces messages when it's done processing I think that's going to be a widely used pattern okay and so you know ARCA stream basically the you know versions through processing and a lot of folks have not seen actually three process and yet you know there may be heard the term and kind of there are different kind of incarnations right there is spark streaming there are streams in Java 8 and right so so I'm most people in this space is still do Nikhil and sometimes we kind of have this term no etl because we don't want people get up to to stop and do heal but we want them to kind of do things wrote are gonna can talk a little bit you know how kind of extremes us different from my TL and what's kind of makes them different mode of working with your data versus in originals you know in the TL typicality I'll flows like regions out from us three and then do so I wouldn't contrast contrast and so much with etl is just batch processing in general where you have like one input and one output yes with streams it's more that you have a steady flow of inputs arriving it point some time and then you produce a steady flow of outputs with no required correlation forever between every input and output right they could have just like the sim at the simplest like stream processing stage that groups messages like in two batches and it might receive like a thousand messages and only produce like one sequence of messages hmmm something like that right and the place i see fracas streams is as sort of the glue between different things that really enables composition because you can just take anything really that uses the reactive stream standard and combine it with anything else right like it's a very widely used standard people from netflix were involved people from twitter people from various other companies and sort of the dream right is that you can take one open source project using one implementation of reactive streams combined it with another using a different implementation and it just works and that'll let you let us have reused even more components and avoid writing even more code cool and I'm sure that you'll cover more territory in your technical talk you were looking forward to so I'll probably just wrap up with kind of a general question so basically you know about a year you've been just call it nitro kind of going to be the ops and kind of connecting with the bear community what's your personal experience you know uses Colin in industry and what would you what we would be advised to nubes corner in Scala or the companies who are basically trying to introduce color to their code base what's your suggestions how to make it easier for individuals and teams what you find works best was it's harder kind of any any advice in this area well definitely ask questions if you have them the community is really welcoming really great but before you do that if you have a chance just open up the scholar repple right i just type scala on the command line like try something out see if you have like a hypothesis try and test it like just it's very very easy just to type a few lines of code and like throw something together just to prove out a concept or like prove that it cut some design or concept won't work and it's much quicker you'll really understand what you're doing better if you actually test out hypothesis as opposed to just like coffee tasting stuff together yeah just open up the repple and try out whatever you're thinking of see if it works or not cool I think this is the really some of the best advice you know I believe you can give to an ube you know skulls interactive lets you know open up a ripple and try things Thanks bowling I'm looking forward to it up absolutely thanks