Brian Degenhardt, Q&A with Alexy Khrabrov of SF Scala @Twitter 20150217
Recording: Brian Degenhardt, Q&A with Alexy Khrabrov of SF Scala @Twitter 20150217
i'm alexi krabrov the organizer of asf scala and we're here on location at twitter for a meet up about the past and the future of scala and here with us we have brian degenorth the senior software engineer at developer productivity group at twitter and brian did a lot of interesting things he's going to give us a talk today which will also be available functional tv but i just want to ask a few personal questions uh to get to know some of the skull developers at twitter so first what brought you to twitter in the first place yeah the scala actually brought me to twitter i became a big fan of scala and was writing some stuff in my private time and i was like i need to go do this full-time and work for a company that was using scala and so i looked around and the obvious choice of a big company that was doing amazing things with scala was twitter so here i am that's great so around the same time i think a lot of startups you know tiny and small medium size also started to use scala so what was your uh thinking in terms of going to a tiniest startup or bigger company like twitter i i've done a lot of tiny startups i previously founded a company and um the the thing with tiny startups is you spend a lot of time recreating things that larger companies have already dealt with and the the really interesting appeal at twitter was the move from into scala was sort of late in its life cycle so here's this company that has all the customers has all the scalability problems but then uh is sort of forced by the by their massive popularity to redesign systems to keep up with the scalability problems and so even though it's sort of late in the stage of as far as success it was still early as far as the engineering challenges and so that's just super appealing is to go work on these really tough problems you know like but uh that still haven't been solved destruction uh so how did this so this was the idea going in how did it really work out what how did you start uh you know what piece of project uh did you start working on and how did it evolve from there how did it really work out what how did you start uh you know what piece of project uh did you start working on and how did it evolve from there yeah it's been amazing i i started and uh i was put on on the zipkin project immediately um i was actually a one-man maintainer of it which was really amazing to like take take on this whole full stack project and get to see the whole thing and the amazing part about zipkin as well is uh it's a distributed tracing system that shows the performance of all the components in your in your uh in your network and so because of that i was able to use zipkin to learn how twitter was built wholly and one of the pieces i wrote for zipkin was a distributed or an aggregate dependency graph of all the services at twitter and so by working on it i got to see how everything interrelates and it was very it was pretty fascinating actually well so you basically had a you know front row seat to all the systems because they all have to report right to zipkin yeah yeah exactly interesting so uh what was like one of the most like interesting systems you found exists which you kind of didn't expect existing or you know something like that uh well one of the things you think about is that uh there's all these different systems that do all these different work but the the authentication server we have for verifying your password gets invoked with every request because you have to always the first thing we do is check if you're logged in so it was sort of interesting to notice that all the variety of operations you do you're always checking your password as well and so the amount of requests that that system gets is so you know huge just the frequency of doing that check is so ridiculous that it's amazing to see the request load on that so there's some high performance computing going on there yeah there's lots of stuff yeah it's good to know all right like we're always curious about how high performance is done in scholar so probably they will we'll ask those folks to talk about it uh so but i think this is a great advantage of of the two intersect that it comes with this uniform instrumentation uh so i wonder how did this help make twitter so fast right because now you know you're tweeting on the iphone i have bad connection right like have a few bars 4g i'm tweeting and somebody retweets or replies i immediately see the notification pushed back i'm actually constantly amazed how how reliable and immediate these notifications are so can you speak a little bit about what goes into making this whole thing happen well yeah if if you're using uh if you're looking at your twitter webpage and it's slow we have a firefox plugin that enables it to integrate with zipkin and so if you look at firebug you can see each individual http requests and how long they take but with the zipkin plugin you can also see of each individual http request which components of the distributed system were involved in serving that response so now you can load your homepage your timeline and not only see every single http request but all the different systems that went involved and see the whole breakdown of what took the longest so uh that's all open source by the way you can get it now and additionally we have that integrated into our mobile debug builds and so we can see we can see zip code traces through every single mobile request too so if the app is slow we can we can have visibility into the performance of all the systems and just look for the long bar on the chart and that's the thing that's slowing you down this is great i mean i clearly see the effect on this in practice can you speak a little bit about uh your own um learning experience at twitter what's about the learning culture and you you obviously had a lot of veterans around you who have been doing this for a while and what are the most important things you've learned and how this learning actually happened i think the thing i've learned the most here is i've never worked anywhere where people have done such a good job at following research but also being pragmatic at the same time i i know a lot of people who've read a lot of papers would be very academic and i've worked with a lot of hacker guys i'll put lots of code but people who do both are are so prevalent here and and the the discipline of of paying attention to research and then going and implementing those things in production is is such a a common thing here for example zipkin itself was a paper that google wrote about their tracing system called dapper and it described it in pretty brief detail and then twitter engineers went and implemented it from the paper and open sourced it and then that sort of uh ability to go research cutting edge things and then implement them and use them in production is is a is a skill that i've really worked to develop here great uh i i wonder what is it about twitter culture which you know going back to your uh paper reference which makes all this kind of people who are very you know well versed in both theory and practice thrive here like marius for instance he gives talks about the right ways to do concurrency or for treating data center as a computer and reason about very systematically at the same time he is one of the most productive callers with a lot of open source and stu has a lot of contributions to open source how uh what is is it about twitter open source culture which kind of supports this uh this kind of thinking capacity and at the same time you know allows folks to crank out great open source well the the scalability problems we have uh put a very strong constraint on performance on your system so doing bad design doesn't fly because if you do anything incorrectly designed it's not going to scale and you'll see it in production and so because of that constraint you have to be versed in the theory you can't not know all the theory of everything you're doing because you'll get burnt very quickly but as we're a business and we actually have you know goals to produce things that constraint post constraints on being productive and actually getting things done and shipping it and so the combination of the two is really the factor that has developed this culture where uh you know mastery of the systems is important but also getting it out the door is is as important that's great and i guess one question a lot of developers have right now is they want to take you know the open source twitter puts out and they want to basically build pieces uh of their businesses resembling twitter right because they they may have a user facing api which they want to be a web scale or they want to have data processing you know a twitter scale so what advice do you have for folks who want to look at twitter open source you know and basically have all these great number of projects and they they basically want to replicate twitter success and reuse this open source and join the community where should they start what's a good trajectory to dive into twitter open source i think the the first thing is to understand that to target audience for our open source projects we develop open source projects for us and so we are a very large website that gets lots of requests and we build things with scalability as like a very primary component of it and if you're just getting a startup going and you you know want to download zipkin and use it i've seen a lot of people on the zipkin mailing list for example that don't have a distributed system they've got one server running ruby on rails and they want to integrate zipkin in it because twitter uses that and that's not an appropriate usage of a distributed tracing system that's used to manage you know follow you know tens and hundreds of services um so the recruiting pitch i would do about the twitter open source uh stack is to the other larger companies and we see a lot of those usages that there you know foursquare for example is using a lot of the twitter stack and the larger companies once they hit these scalability problems uh that's where i would say you should look at the twitter stack because most of the problems that people have we've already dealt with them and we've solved them in in elegant ways sometimes the elegance is lost if you're just getting started up and you just want to build a prototype of something that being said we're we're always working to get more people to adopt our our stuff because that breathes life into the system when they do grow so we sort of fight this balance of like you know inducting in the new users and the smaller users but as well as uh you know our day job is to make the stuff run fast and run performant at the same time right so i guess in order to really understand the value you have to experience the scale right so and hopefully you know the startups get to the scale and then they get to enjoy this well here's the thing is if usually the scale means success and you don't want to have to stop mid-course and rewrite everything because that's really really troublesome and and so uh if the the the deal with what we've we put out is that it's uh it's ready to roll so if you do get popular you'll you'll be able to handle it you'll be able to grow with your with your company well that's great i hope we actually uh help more folks learn twitter stack and we're going to have a financial workshop that's called by the way in august so please look at the calendar and watch scholar by the way website we'll have all the program and all the trainings published there in your course and thanks brian very much it was really helpful you