Stu Hood, Q&A with Alexy Khrabrov of SF Scala @Twitter 20150217

Stu Hood, Q&A with Alexy Khrabrov of SF Scala @Twitter 20150217

Recording: Stu Hood, Q&A with Alexy Khrabrov of SF Scala @Twitter 20150217

hello everybody i'm alexis krabrov the organizer of sf scala and here we are on location at twitter headquarters where scala is used to power all the tweets you guys can send about this event and here we are with stu hood who's a staff software engineer at the developer productivity group at twitter who's going to talk about the history of scala at twitter and today is a very special method because we talk about the past and the future hi stu it's great to have you with us so uh twitter is a well-known success story uh for scala uh but uh when you started uh i think it was not yet clear how skull is gonna fit what's gonna happen so can you kind of shed the light a little bit uh why did you join with her what brought you to scala and what made it such a formidable weapon in the arsenal of social media absolutely so uh i had been a java developer for a little while before joining twitter and actually at least partially because of twitter i started looking at scala and kind of wrote a few toy projects initially slightly after that my my path aligned in terms of i was working on an open open source project called cassandra and uh twitter was using cassandra it ended up being more significant that twitter was using scala in the longer term because that's where i ended up staying involved longer so it was fairly early in twitter's scholar days in 2010 where we finally we first crossed paths and so i guess why scala right now is i think it's uh we know that twitter is a success story for it but in 2010 probably was not such an obvious choice right um i think there's been plenty written on on on the fact that they wanted a language that uh was as expressive as ruby while providing vastly more type safety and performance than they were getting with with dynamic languages and that appealed to me as somebody who enjoyed writing performant code um data structures and algorithms mostly just worked in scala uh the focus on immutable data structures was huge um yeah that's all great uh it's actually that's very interesting to to know about cassandra because cassandra is now big with spark and so spark runs on top of cassandra so there is a website there is a second coming of scala to cassandra which serves this important uh role uh in persistence uh but uh i i've seen a lot of projects actually coming out of twitter in terms of storage and a lot of different so there is finagle which is it's all on our pc stack so can you talk a little bit about how the whole stack evolved and kind of what are your favorite pieces about the stack where are your contributors kind of which is your kind of best playground sure um well the stack of twitter evolved uh again kind of out of the service service oriented architecture needing to needing to bloom behind the scenes behind the front ends um over time and actually during the talk you'll see there's there's a particular inflection point where it starts to move closer to the front end and be more of a general tool uh there were a few questions in there what are their questions yeah so what are your favorite pieces favorite pieces um so i worked a little bit on the social graph database for a while i wouldn't call it my favorite piece of the stack i think it attempted to be a little bit too reusable um too generic uh didn't include enough of the api but finagle is kind of the the classic example of a library that i that i absolutely love um something bird came out a little more recently and is also just a great great composition of a few different uh twitter libraries um i think there's lots of other oh stitch i don't know how much of a splash stitch has made but stitch i think is uh a very very important part of twitter's features we had the talk on it uh at the same scholar i think a few months ago here and also it's called by the band it really sounds like a like yes we got jake to speak and it's it's available in functional.tv and it's a great great piece of software it's huge um because i think more and more if you go back in database papers you see discussions of how you build queries in this in an environment where you don't have a single database anymore you've got multiple databases um and you want to be able to deploy them independently and you want them to have their own management um and i think stitch is the beginning of an answer to that for twitter um we'll see do you see adoption happening inside inside twitter yeah yeah i measured some of that it's still early days you can't you can't quite see a trajectory but the the adoption is is very good and in some cases it's it's it cannibalizes uh usage of futures um in cases where well i was doing something eagerly but i might as well do it lazily um and so but to some degree it's kind of cannibalizing some of that usage but i get i get numbers in the deck it's at one or two percent of uh scala files import stitch interesting interesting so uh so you're one of the early uh skull people at twitter right and kind of uh i mean in internet years right this is ancient history and like there is this whole success story which twitter kind of travel through so i wonder and you see uh scala engineers joining it's probably one of the biggest uh groups of scholar developers in the world and uh i wonder how how the dynamics works because you know we have this so-called skull tribes uh which we kind of had we know there is a extreme fp tribe and there is java like tribe and i think twitter has this efficient scholar tribe you know which mario has described as the middle uh grab but obviously stitches a little bit i think to the fp side of this uh so how do you see engineers uh of all kinds kind of uh absorbed into this big bird organism and kind of finding that tribe how does the dynamic work inside twitter uh so i think that that was a problem particularly in the early days and that because there wasn't a huge established code base core libraries weren't as ingrained into the code that you were going to be reading um but i think having core libraries that that display these uh appropriate conventions um has been huge so we definitely still hear people join and say you know i'm familiar with doing something in java but they can find tons and tons and tons of examples um and many of them are in our open source code as well but you know very large code base internally that they can see anecdotes in um so i think over time it's been less of an issue that that people are totally disoriented and unsure of which direction to go in um yeah i talked about it in the talk but there are uh plenty of examples where there's a choice between doing something in an oh or functional paradigm and it's not always obvious it's very context specific which i think is part of the the strength of scala it's it's not clear that you should always be functional um so yeah still plenty of debates about that but it's mostly mostly faded how do you guys solve this actually uh in real life let's say somebody wants to use object oriented idiom and somebody else wants to use fp is it decided by seniority do you have to convince people uh how it plays out i think one thing that uh ends up guiding the decision to a huge degree is that we all pretty much agree on immutability there's a little bit of there's a little bit of usage of mutable variables but always in closed states and that kind of ends up limiting some of the patterns you can use so you'll only use certain object oriented patterns like the builder pattern is actually an object-oriented pattern but it it is immutable and so it works fairly well other things don't work so well so kind of on a pattern by pattern basis uh it's fairly clear interesting so twitter also spends i think now a huge variety of applications so you have finagle you have scalding and something bird how do these pieces of twitter stack uh coordinate right so do you have a lot of cross-pollination between different groups do you have kind of a lot of shared libraries or is it mostly silos or how does it work between the pieces of organization we have a huge number of shared libraries and i think the glue of that is still finagle um in the sense that there are lots of services deployed and even if you're an analytics job you may end up using finagle to talk back to online services and i think finagle has has has sprouted its own teams essentially for coordination um there used to be a coordination team now they're calling it traffic which i think is an excellent name um but it's all about routing and uh kind of replacing components of the routing um i hope that marius will give we'll give a great talk on that sometime soon yes definitely yeah all those capitals yeah yeah but but i don't think any of it is uh closed source that any of the stuff managed by the traffic team it's uh it's all in and and also we are uh planning to have a financial workshop a scholar by the way in august so folks uh watch scala dot by the bay dot io website we hopefully will have a lot of training and talks about finagle and other pieces of twitter stack i wonder it's i think it's always this perennial question how finagle compares to akka and its own stack because i think we have kind of this dichotomy where rest of the world is using akka and people are actually using fenagyo because i think twitter offspring spreads around uh the community so folks like pinterest are using finagle but not necessarily all of them are used mazes and financial combination so can you talk a little bit about like this whole package do you have to have messages to run with your stack is it fine just to use pieces of this uh and what's your view on kind of uh finagle versus soccer and kind of these two different stables of code i mean you definitely don't need to use mesos in aurora too to use finagle i think and in particular you don't have to use you do have to use twitter util to use finagle but you can just embed that into as a library into almost anything you build the only other thing we'd recommend using with it is a twitter server and twitter server is just cli args um and then a tiny tiny harness kind of in the vein of uh coda hale's um what was it called shoot do you have the code hell wrote the it's a java it's a java helper library um great library and it's metrics right and it fits into his other drop wizard yes so i think twitter server is uh twitter server plus finagle kind of gets you most of the way to something like dropblizzard um but we don't always kind of advertise it as a complete package um if you use it as a complete package it is out of the boxes is fairly good um all of them still work together right yeah zipkin um yes zipkin is still it's kind of the fabric it it an overlay an overlay of all these libraries uh and is useful anytime you're using finagle uh and zip can use to write a cassandra uh zipkin will write to all kinds of stuff it uh loggers it'll log is really what it'll do yes um i don't know if we're writing directly to cassandra right now so you basically get instrumentation for free which actually makes uh twitter sex so appealing because you can basically take all this stuff and you have all your metrics and instrumentation in the same package and i think i'm sorry to interject but uh also i think the the work that the traffic team is doing goes into the same vein of zipkin um it's more it's rather than like post facto figuring out what happened uh you know before we begin the request let's route it appropriately and then later on we can tear it down but uh more power while planning your requests um and whether to route it to a particularly an entirely different service in order to test some code over there and then return to the production path um is so sorry no that's great that's great that actually that makes it you know i think very interesting about the twitter section because you you know one dream of uh many you know uh startup founders is to get twitter in the box for the backend right so i think with uh most of uh back end in open source you probably can do right uh i actually i'm very curious about mess i'm a big fan of masses and ben kidman also was an early advisor uh and you know i kind of followed mesosphere with great interest and i kind of be proponent of this uh so how close is uh this dream of you know developers not asking devops people for things right like can you just schedule a scholar job on message cluster it will connect to other services uh and kind of just work is this really what happens at wither or do you have from time to time to have kind of manual intervention to see what's going on that really is what happens at twitter now um i can say i can say that definitively i would say that the thing that's missing um is that it it is still a human that's going to launch that job so we haven't quite reached the meta level of um i would like to spin up a unique instance of a service that's already running and not clobber the one that's running currently so sit over here ensure that all routing avoids it um only the ones that i actually route there should go there so not participating in the zookeeper ensemble or whatever or whatever it might be because it's not all the zookeeper anymore so i think yes absolutely you can spin up an application uh but multiple in the plurality of applications is still where the the uh work is happening interesting that sounds great so i'll probably close with this question you know when acm celebrated first 50 years of computing they actually had this huge hundred years of computing celebration because they were looking the next 50 years right so we're on the startup time here so it's five years of scholar twitter and so we'll look at next in the years of skeletons so what do you think is gonna happen most importantly in the next five years and where do you want it to be in ten years when we meet here again and see you know what's going on um so i i want both simplification and uh and advancement within the language one of the things that definitely brought me to it initially was the research qualities um twitter twitter pushed back on uh on development within the language because it caused it caused churn for us for a long time but i think we're finally approaching uh the point where we can say yeah i think we're definitely interested in pushing the boundaries again because we're able to keep up finally um so we we're very interested in whatever the community can can can bring thanks to you this is great and we're looking forward to the next era of scholar twitter excellent thank you thanks you