Scale By The Bay 2018: Ivan Corneillet, How Twitter teaches Scala
Recording: Scale By The Bay 2018: Ivan Corneillet, How Twitter teaches Scala
good afternoon welcome to by the way on the welcome to Twitter my name is Yvonne Connie on the next 20 minutes we're going to talk about how Twitter teaches Scala internally a litte bit about myself I started my career in the semiconductor industry designing high-performance computing Hardware along with software to design on program it after completing my MBA I switches industry into machine learning and along the way at the opportunity to teach data science program in a couple of boot camps here in San Francisco today and part of Twitter University and my main responsibilities as instructor is to level up or engineering talent so a Twitter teaching scholar is centered around three parts first I'll talk about engineering frag schools which is part of ongoing process then I'll talk about or ongoing equations effort and finally about or instructors ok let's start with enjoying Fri schools it is the technical portion of the new hire orientation program on last for four days in the first two days we teach more technical concept or on the platforms on the last two days on where variants are quite technical as we do a deep dive into a stack building a real micro service in Scala so Scala is not the only problem in language we use a Twitter I'm not going to talk about what we do on the front end but as you can see we use quite extensively Java on Python in the backend as well but still occurred is mostly Scala code so now let's talk about what Scala means to a student in aging FRA schools and I'll start with your software engineers so here I list Skylar's ranking in three major foreign language indexes sound scallop owns two positions 29 in the table index on 2014 on 2012 in the PI PL on the red manga Nexus so with lot of variants in today's numbers and I don't really know what Scala to ranking is but it's certainly not in the top 5 unlikely not in the top 10 either in contrast Java on Titan are constantly in the top five if not in the top three of his rankings so Scala is not so common into industry concretely it means that we have a hard time a Twitter hiring software engineers already proficient in in Scala so being pragmatic we agreed not to overemphasize programming experience in Scala and rather to teach it once we decide to join us so the other things to keep in mind is that we don't just hire software engineers in fact we hire for a diverse set of holes site reliability engineers on machine learning engineers for example these engineers are involved less likely to know Scala however if we take the example of machine learning engineers we will probably use or scalling frameworks to process the big data so we will need to write the code in Scala finally we expect all engineering managers to be technical on state technical so we will have to run a pub in Scala to some degree so now that we are done with those two not let's talk about what we should teach so here a compiled a surety for features that scale out first Wow as you can see with something for everyone save type value classes extractor annotations variants context bound and so on you name it you can find pretty much anything you like in Scala so we take years to master forming language and I think you'll agree with me that we Scalla being such a rich language it's not exceptions to that whole but I think it's also really hard to use a feature list to teach which part of Scala we should teach so instead let's have a look at the code that we use internally so I'm going to talk about the software stack init with more details but has a primer or micro services are built on top of the Finnegan infinitary framework both of them are a twitter open source project please check them out so Finnegan is a remote positive code system on Finn I build on top of vinegar so you can construct high concurrency servers here in the infinite ha in the trade controller let's ever look into the and all methods so that method which is the response called F given a request called method the request is really an Apache swift message on the response ease of functions that we process on the response to that message here in terms of scale a features the and emitted is carried it except one argument method and yield of functions that itself take one argument F and so on if we consider the implicit argument the color equal colon equal is a generalized type constraints it allows you from within the type parameter method indoors to constraint its type parameters so here we're going to have arcs success on service of arcs on success on the equal color equal arcs of metal arcs means that the type parameter arcs must be exactly the class arcs of the object methods the two of us are very similar on all suite constraints are stole our evidence parameters which are there themself explicit ok so the rest of the code is pretty straightforward I'm going to map the relationships between the requests on the response but oops there's a lot of advanced feature pray here to be honest with you my first reactions when I service code was trying to find the engineers who wrote that code on a spam what the hell they were smoking that must have been good so seriously I would like to do a shoulder hands so raise your hands if you think that they want a bit overboard in writing that kind of code with always scary features ok no not many okay so on the other hand wiser and if you think that the type of code you think when you have having breakfast in the mall okay not that much either right a lot of people in decided okay so before I give you my take on it let's see how some of someone who's writing a macro service using fenetre would use that and/or method in the controller so here let's suppose that we were writing a macro service that is a calculator it's given the add novel request which was called method in the previous slides the service would associates the animals functions on its write or F in the previous slide so here the request which again is going to be a sweet message would be made of two fields a and B the resulting value since we're doing additions of a on Darby would then be wrapped into a future so I talked about the future a little bit later but that's basically a value that is going to be resolved in the future so the same thing is happening for the increment method or value here right we are going to have a swift message with owner argument you going to add one on what eat into a future to send it back so here I would argue that the code in the macro service was relatively easy to write and it was quite clean and maintainable so I focus it's easy to go overboard on just cool use coefficients just because you can but please don't so in a previous example my take is that the more complex code in the framework let to simply a code in the macro service so now because we have just a few framework so in a sense the more complex code has a smaller footprint into a code base but many macro services on there for larger footprint of simpler code in a code base I think this was the right call so now let's talk about scalability classes that we teach in enjoying frac schools so Twitter stack teach new hires how to develop on deploy Twitter services on in this class we shall fold a new service in Scala on customize its tunic so here the class is now to teach Scala but because we spend a lot of time looking on tweeting Scala code I get many opportunities to talk more about Scala what's caleb is Keegan do force and I kind of see that as a top-down approach to learning Scala start with some existing code and analyze that so in this class you don't spend about 12 hours spread over two days in this class it looks like a lot but in fact we are just scratching the surface of the Twitter stack and we are in fact many more modules to teach in years how to test on product on either service how they can build analytics and so on again all of that using Scala so this approach is usually a big Lib for people new to Scala and in fact maybe too much of philip so to help them prepare for that twitter stack class they also at an scala primer we teach them the bare minimum to in fact survive twitter stack so in this class we start with a report or go over different construct and also since many people are familiar with overpowering languages like Python for site reliability engineers on machine learning engineers on Java for new graduate of interns and I think it's very beneficial to draw parallels between various languages on Scala and this is more like a bottom-up approach to teach Scala on students but about sui I was in that class ok and now let's have a last look and see what we do in Twitter stack so a big focus on the onboarding week is to do a synchronous communications using futures and wiping your header on futures texts sometimes so let's suppose that let's have a look into our get method let's suppose that we want to retrieve something specific to your user for example he or her mood from the database so since we are building a service we don't blocked here so we'll return a future the database itself is a service on these we are for going to be non-blocking on is also going to return the future so once the database future is going to be resolved dynamic functions pass to the map method will transform that resolved value and is going to be the output of the result value from the future so really an app on the future we need hacks the map as a container take a list for example you do a map on the list you are going to transform each and every element inside it are you going to put things back on a list and options works the same way it's like a container with no elements or one element a future works exactly the same way so if I were to take an example since the holidays are coming up the futures is like a gift that somebody is going to give you you open the box you know inside you take your gift you do something inside it you put back in the back in the box and we gift it if you didn't like the gift which usually happen to me quite a bit okay so another set method is slightly more complex so here we're going to have to a synchronous core the first one we're going to write something to a database on the database is going to return you need it's not going to give us much more than that it's going to resin exceptions if we are an issue but basically beside that if we needed something from that database was not for some kind of timestamp we'll have to read it so here we're going to have to assign Cano scores on the right will have to follow the read will have to follow the right so we don't have time to go too much in details on that but as enough not exercise really have two questions for you so first you could basically take this code year on map it here but we don't do exactly the same so we do a value options that get versus add up map question would be another one why do we do this on question number two we are not doing a map for first call but we're doing a flat map so I will con that and we can discuss offline why we do this kind of interesting problem okay this is pretty much all I wanted to go over regarding engineering facts schools upon graduations the new hires continue learning scholar on the jobs were to help them we provide them a very opinionated start guides and also able will code that we ship here Twitter for productions needs to be reviewed so that also happened for vets but where is more what's more is or ongoing education problem so we have a strong culture here to Twitter for learning and we are frequently offering me different different type of classes some of those classes are about machine learning analytics some are going to be about infrastructure and so on and of course there is something for SCADA so most of these classes are offered during business hours we don't ask them to do that on evenings and usually lasts about two hours so for Scala we are offering a learning path as a series of classes that follow these correlations so Scala essentials is going to follow the skylight primer it's also one of the longest class that we have also 12 I was surprised around two days so it's very much instructor late type of tutorial so Wally it isn't based on Martinez key problem in Scala book we pretty much cover in these two days the first half of the book scholar collection is only two classes also - I was not the rationale for this class is that software engineers write code algorithms algorithms need data structure on collections are great building blocks for vase data structures so personally I think that knowing the syntax of a language is not enough you also know to know about the API forest core libraries on the advanced class dive into how collections are implemented and also focus on performance so the skylight social class gives to not a broad overview of scala but doesn't focus on the specific parlament paradigm so functional forming in Scala the Nemean price is only the first class focusing on functional plumbing and here we dive into functional engrave details and also how engineers should use that yet twitter so student out there encouraged to attack the advanced class Mona functor France that class goes into a bit more advanced forming concept so finally as I mentioned earlier futures present whole world Twitter tonight enable high concurrency servers so in this class didn't spend with me sometimes or getting more familiar with them so we have more classes but that's pretty much the state of the mastering Scalla learning path the past is not static and we are really updating it based on what we see happening in the industry okay so now in yes so that's quite a few classes all right so my question now you know is who teaching them now let's talk about our instructors now this is going to be the last topic I wanted to talk about so in additions to a strong culture of running we also have a strong culture of teaching so my team Twitter University is quite small the ratio of full-time instructor to engineers is about one to a thousand right so we don't have that many are in fact we are recruiting instructors so if you are interesting let me know so another question for me is how do i scale because i'm can teach you know a thousand years i would not last you know very long well the good news is that i do not have the monopoly of teaching a twitter we expect all twitter engineers to teach to some degree so as instructor or full-time instructor so in addition to teach i motivate on coach my co-workers to teach or make sure that they are rewarded one video so besides calendar the reasons to either teach is that because we have a large custom built infrastructure you want you want to learn from your peers who happens to be experts at the thing they do the things that you need to learn and it's also ensure that your coding practices as recording parties change you teach you latest or best idioms what's in it for them well in teaching gave them an opportunity resolving a little bit different than visuals it also helped them to become better Pete speakers if you are in very interesting to that we do have a Toastmasters club but it's a great way to kind of compliment that finally it given greater visibility and that's always a good thing if you want to be considered for promotions okay so now let's wrap things up so first the majority of the code should use a reasonable subset of Scala that could then become idiomatic code for companies of course they should have exceptions but for exceptional cases second no two people done exactly the same way maximize the learning opportunities by presenting things different ways it's also one reasons why we have different instructor finally it's only by teaching the other teaching to others that really know if you get it in the first place so inside your peers to teach and more instructor teaching will also ensure that you are teaching the right things so thank you on now let's open for questions [Applause] don't be shy yeah [Music] yeah so four so the onboarding process for all technical people is exactly the same same thing alright so on as part of the onboarding I read the ongoing indications it's up to them to sign up for classes where now we are we can only recommend what we should do it's basically up to them to decide what we want to do you know this great question we plan to customize things as we go along more you know on the onboarding side but that's something that is improvised yep yes so that's a great questions so we don't so on the reasons we are offering classes internal to Twitter is because our engineers are interested in to how we do it the Twitter way so yeah we don't we don't really use kind of immaterial from outside we really customize its tuning so remember that you know we still have to work right so we offer that during business hours so we only we already have a class you know for two hours so we can have compress it I don't really you know give what we its sense is wholly in the Twitter way that's a good question yeah so it's totally voluntary so when I said that we expect all 2000 years to teach it doesn't mean that they need to be in front of classroom right they can basically coach a coach you know peers and so on but I would say that the hardest part is for event to kind of do it the first time after that it's hard to you know keep them you know not teach classes anymore because we get you know the taste of it so it's really it's difficult the first time all right but so what we do is when we want to teach a class so we can teach a class for which we already have materials we can also come up with new material for class but so if they want to sign up for class that we already teach what we ask them to do is to shadow a couple of classes kind of act as teaching assistant then the at some point they feel you know that they're already on yeah we usually do a good job yeah so yeah we so we don't we don't have anything or pencils I know that we do have some material you know Twitter github repo I think it's quite old let's chat online I mean offline on the see what we can do about that okay Julian you have a question for me we also wanted to share no it is true that yeah we definitely have a list of resources that basically point outside for what we have a Twitter yeah for people who are interested okay any other questions okay well thank you guys [Applause]