Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217

Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217

Recording: Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217

all right well thank you everyone welcome um i'm going to take a page out of oscar boykins playbook and uh relax a little bit so that's great um i'm stu hood uh you're twitter this is a scala twitter talk um i don't think i need to explain twitter uh but a little bit about myself i've been at twitter since about 2010 joined due to other open source projects and was already a little bit interested in scala but twitter's scala open source efforts attracted me initially um during this talk please ask answer ask any questions that you may have because otherwise i'm terrified that it'll be very dry so throw a hand up um it's a history lesson uh but i think it's really interesting or it was interesting for me at least kind of going back and and learning uh we're analyzing a little more deeply uh the history of of scala twitter um and these five themes i i apologize if anyone's colorblind uh did not think to also do this in additional fonts so theme one is kind of the history of scala core and how it weaves into twitter's history theme two in purple red scholar red so you'll remember that probably the rest you won't remember tooling theme two because we've kind of found that the history of scarlet twitter is intimately related to the tooling that was building the scala for you libraries because we would be nowhere without them are in yellow conventions because we don't yet have tooling that enforces absolutely everything we're thinking so we have to have some conventions and then just general usage and how it's expanding within twitter and elsewhere so first just some very fuzzy numbers because com because com's team a few million lines of code which is in the five to ten range about forty percent scala at about 25 java there are many thousands of packages because we build small packet packages woo all right that wasn't the projector going off so that's good we have many thousands of packages and we'll get into why that is we don't while our open source efforts are project-centric we try to be more package-centric internally um and that's the thing that is in the distant future in terms of resolving um average of 10 files per package i think that's low i'm unsure uh so in the very beginning there was scholar one in january 2004 don't have a screenshot sadly but there wasn't really much going on with regard to twitter at that point um twitter was not even i think odo was somewhere out there at that point in time uh where things start to get interesting is around the scala 2 0 era interesting coincidence twitter was founded and scala 20 were released in the same month so interesting um we began to have some alignment but things really start to heat up around the scala 27 era between 2009 and 2010. so by this time twitter had been building ruby web services for a while they needed back end web services i don't even know if they bothered to build ruby web services or whether they immediately knew that that was going to be brain dead and so scala 2 7 was the first version to really take a hold of twitter so scholar 27 was released internally we have a mailing list called scala cafe this is a little late uh to be frank i don't the history between 2009 and 2010 is a bit a bit sketchy um maybe they were maybe it's all in the tweets and i just haven't looked at them closely enough from that era you know don't use an immutable hash map in a in a highly concurrent uh you know domain because it's going to fall over on you i didn't read all the way through but it was probably something related to visibility of volatile fields or non-volatile fields fields that should be volatile um easiest way to shuffle a list buffer you know stuff that ended up in the standard library later but primarily because the collections library um was generic enough that you could put a shuffle method on random and have it actually apply to all the collections which might not have been possible before so then we're immediately going to take a little sag into a fairly significant talk that alex payne gave at twitter's first developer conference chirp um and i'm going to bring up brian to to talk a little bit about that but alex payne was the api lead for twitter and i believe the talk that brian's going to transition to explains that it's kind of hard to do a scala talk following or going before stu because this guy is a monster and uh it's sort of intimidating so i went out and i was looking well we've talked a lot about scala maybe i can go figure something out from one of the other talks maybe i crib some notes off them and i found this talk and it was i was like oh great this is somebody's already done the talk for me i can just borrow from this one and i started paging through it and what what this doesn't make any sense uh so uh and then i looked at the date and it's five years old uh and this is fascinating because uh five years ago alex payne gets up at front of a developer conference and is pitching scala to the whole world and um and so i'm gonna give you his talk but a little bit i'm gonna filled with commentary so uh yeah informed by five years in the future in five years in the future you could look at this talk and see how ridiculous i was and though i'll deserve it so uh let's talk about this guy alex payne he amazingly intelligent guy like one of the architects of twitter was around way way way in the very early days built some of the the foundation of the company um went on to to work on a bunch of different startups and and and now as a like a divisory role in a bunch of places and does a lot of vc and just like huge luminary thought provider wrote the o'reilly scala book like i i can only hope to be like half as smart as this guy um and i'm serious this guy's a monster and so here's his talk he's giving this talk five years from now let's talk about scala at twitter um so twitter was originally written in ruby and then the the first sort of foray into scala was on back-end services and said hey look we can use ruby for the front end uh and you scale it for the back end you used thrift to intercommunicate between the two systems informed by by history we don't want to use ruby anywhere now partly because the scalability problems and we've grown a tremendous amount since then and at first scholar was great to to put on the performance back-end systems and have the flexibility on the front end but now then the front end was getting too much requests and too much load and so even scaling the ruby on the front end was a problem and so uh now our answer to this would be well let's just use scala or let's say the jvm everywhere we're a little bit more flexible with that at twitter these days if you want to write it in java for certain applications there's pragmatic uses for java as well so we do that as well hey nothing wrong with this one skull it's great language you guys should try it okay so why are we using scala in 2010 because the type system's great yeah we like that flexible syntax um yeah except for if you want to like come up with standardization practices for formatting it becomes a little bit interesting like do you use dots on infix operators or not and we've argued over that internally and just given up on it i think um traits object or programming yeah those are all good here's my favorite xml literals this is why you should use scala um yeah the rest is good i i choosing between immutable immutable i don't know i just use immutable as much as you can lazy values overuse of lazy can get you into trouble too with the race conditions between them but oh well i'm not going to be too critical about that stuff um why are we doing concurrency in scaling 210 we're using the actor concurrency so this is funny because we're not talking akka right now we're talking scala actors pretty terrible i don't know maybe there wasn't any options before then this is pre-finagle so there's no futures on here uh it turns out informed by history most of the concurrency at twitter has done with futures now um very very little i mean we don't really use actors at all yeah i'll get into that a little bit yeah and maybe maybe offer broker which uses some actor like we're like early thing has has mostly faded into the background not right not too many languages are attempting to be like early that's funny though because i got into scala cause i liked erlang so much but yeah we can do better we can do better than that um scala is the future okay so now i've convinced you all to use scala we're going to talk about how twitter uses skull in 210 and and i'm going to be very critical about it if you haven't noticed um isolated components yet we're doing that we talk using thrift says maybe avro in the future hello from the future we're not using apra um yeah yep thrift isn't great but anytime you lose type safety by you know doing serialization it's you have to fight between you know compactness and uh and and expressiveness so and i would say that even apache thrift users are no longer using thrift proper so i mean finagle finagle has its own implementation of what you call the top thrift and so thrift thrift is great for making those structs but then past that we can't we can't trust you if the serialization format is good enough right the the network protocol is is ridiculous it's not a network protocol it's a socket you open and you dump a thrift across and you read a thrift back so that's not a protocol um yep yep i don't know what independently tested oh well yeah so this is a big deal on twitter like back then is like oh twitter was a big mono just a monolithic application and so if you wanted to change one part and twitter you would roll out all of twitter um because you couldn't change individual pieces and so this this was like oh this is the pitch is we can write just one part and just deploy that service instead of the whole website because we have a we have micro services and and uh you know and have custom operational properties like different gce uh rule sets for different problems let me just say that sounds terrible i don't want custom operational properties i want them all to work the same way well no but but for garbage collection for example search has a big search index that holds in a very large old gen and you know that sort of thing versus something that's really quick quick you want to eat in that you know i mean yeah yeah it flies in the face of consistency but yeah if you can if you can get those tunables without changing anything else about the system yeah um yeah but this is a this is uh this is a big deal back then because everything was monolithic um so here's some of the services of five years ago kestrel for queuing what do we want to say about kestrel do we still use it uh i'll save it in the presentation uh it is still there somewhere that's the thing about systems that put things on disk so right flock and gizzard is a sharding system and then the gizzard tried to be a sharding system independent of back-end storage and then flock was the that applied to mysql um it's incredibly performant it turns out you can't separate sharding from back in storage it's just not two things you can separate and so we've run into a lot of problems where it turns out if you're going to build a sharded data store you have to build it completely not separated between two libraries so while it was an ambitious and cool idea at the time it turns out that we've sort of not not too great about it hawkman i don't have never even heard of that thing so we don't have that anymore uh hose bird's still alive if you do the streaming api you will use hose bird um here's some thrift so this is an interesting thing that we've actually gotten worse here not worse but larger so twitter um you know we want to build reliable you know components that we can share amongst people and and we try to open source as much as possible and then we try to keep the not invented here to a minimum but we have very peculiar requirements we had very peculiar requirements in 2010 the requirements we have today are ridiculous and sometimes people who want to use parts of the twitter infrastructure get a little frustrated because we have it all sort of wraps you get pulled into the system but uh the thing is is to to build something to our scale we keep running into these problems that we have to like are forced to redesign systems that nobody else cares about so but we have scalability problems that no one else is dealing with and so it turns out that this has gotten bigger we're doing a very concerted effort to be outreaching to the open source community and get the little small ducklings to like use our our offerings as best we can but you have to realize like our peculiar requirements for 2010 are are even worse now and some of the benefits uh yeah i think wiley would be a great example to talk about i think um finagle gets a little bit of flack for being kind of obscure in the sense that it's a lot of components that you can put together in various ways and it's like why doesn't it just work out in the box well because maybe that's maybe that's the operational operational flexibility aspect we would like all of our services to be deployed approximately the same way with finagle but it hasn't quite reached the point where you can say this is the right way to deploy this service if you have more request throughput you're going to need to tune this down if you have services that are up for a long time you're going to want to tune this up and so it's more it's more instead of building blocks and even internally the building blocks are put together in ways that don't look like a monolith here's an example uh load balancers and finagle so if you look at and everybody wants to compare akka and finagle so i'll just do it but if you look at load balancing in uh akka i think you can do random and you can do round robin and they're i don't know if there's a load something based on load but in finagle we have the heat balancer which looks at the load between all the clients and picks the one that's least loaded it turns out that there are performance problems with that as far as gc of being very large clusters of machine your heap that you're balancing across gets large and then when it changes like somebody rerolls out the whole cluster you have to gc that whole list of hosts and get another one only something you really worry about if you're dealing with huge lists of clusters so then we went to a a different load balancer that did a weighted average based on time so that if certain things were up and down up and down up and down we could average them out because if if they're up just briefly then they have low load then they'll get hammered by everybody and so no we want to smooth it out so we have this weighted average algorithm that was called from various research papers to get just perfect and then that turned out that we needed something a bit more immediate and then the clusters got even larger so now we have a loaded weighted average balancer that only views particular parts of the cluster or take two which only compares two of them because statistically that's been shown in another research paper to work better so there's the the choice load balancer the ewma load balancer the heap below balancer and there's one other one we're working on these things provide huge huge cost savings to us if you've got a cluster that's just utterly massive we'll save you a lot of money and you'll love finagle but this is a purely requirement that we're writing five load balancers when you're you just probably just want one the heap one we wrote originally works great um but if you grow then you'll you'll you'll just be blown away by how performance some of our stuff we've written is so that's that's kind of how that works um ostrich uh here's another thing it's for doing stats it turns out uh ostrich has some problems and we're trying to get away from it now um the thing that ostrich has is that it snapshots all your stats every minute so on the minute boundary it locks collects all your stats and then stores them for them for consistently for the next minute that way you get even measurements even if you collect them in a little bit different times but because you stop snapshot all the stats and hold them around you have huge gc problems you're holding all of these stats for a full minute so they're absolutely going to get promoted and then right when you collect them it drops them and then they go away so the gc properties of ostrich are utterly pathological and you can't change this because it's kind of the api of ostriches it'll hold your last minutes of stats and so we had to rewrite it so we have a new library called metrics that we've rewritten but we're actually exploring other stats systems we've taken the finagle stats receiver and actually pushed it into util so you can use it separately because it turns out people are just using finagle for stats and not really a self-contained oh yeah the other question back there um excuse me how do you do uh the metrics like how do you actually look we just scrape them from a jc jason http web environment yeah yeah so we have a collector that goes and hits all the endpoints and collects all the json and then stores in this gigantic database of statistics and then we have a tool called viz that just graphs them all on the web page it is it is we would like to release it but it it's so glued into our infrastructure it's it's one of those things that like maybe someday we could we wrote a great blog post about it though where we describe all its behavior uh if you look at the twitter engineering blog you can see a very detailed description of how it works yeah the question is about the infrastructure and i think not only is that not posted on cassandra now it's hosted on manhattan which we've had blog posts about you kind of end up with this tree of dependencies that you'd have to open source before or genericize spend spend x amount of time genericizing before you want to open source them so yeah it's not that the will isn't there it's it's just some of the stuff becomes a lot of work to cut them out i would say one of our one of our goals is definitely to make things easier to open source in general and um i believe that that's something that i can talk a little bit about near the end of mine uh don't use ostrich configi so at least no you shouldn't use config either configuration is really really challenging the problem you have here is i have this application that i've written and i want to configure different properties of it because when i run on my laptop it's going to behave differently than if i run it in the dev system versus if i run a production right in production i connect to the production clusters and maybe i want some stub data if i run it locally and so you make this configuration file but the problem with configuration files like configs is they're not type safe and so now how do you prove correctness of your configuration it becomes difficult so when they originally wrote configure it was pretty cool but then realized that type safety is important um as it is in our code it also is in configuration so then the next generation was util eval which was just bring along the whole scala compiler and your configuration file is a scala file that we just eval at runtime so now you're type safe but you just get runtime errors in your configuration and that's not great either because if you can compile your configuration you can start pushing program logic into configuration what you really want the correct system is going to have the utmost the smallest amount of data in configuration as possible and all of it is in your actual code and that way it's much much more testable and you just have very very very tiny flags that you switch between a finite set of options and so our philosophy now is just that is just command line flags and and try to boil it down to the to the the essence of your application and just pass in flags when you launch it yeah and those command line flags come in via util app which is provided by twitter server which are both open source as is metrics right so i'd also like to point out we still are the biggest fans of the developers of these libraries some they're amazingly good developers but it's some there's context that we have now allows us to say this kind of yeah roby poynter wrote configure that guy's my hero uh and he wrote an amazing blog post like two years later explaining why it was incorrect and it's really really awesome to read the the original and but configure actually was type safe config is based largely on configure so it's not like these ideas are are poo pooed in the industry it's just uh it's it's just it's just a little bit tough in in if you want to be utterly consistent oh question yeah is that sort of using doctor uh no no no as far as i know um but there is probably somebody in the room who can answer that question better configi uh it's not type safe uh specs we use scala test now although you know testing frameworks are set much more backseat into uh how you write your tests um i mean if you wrote them all in scala check then you would be a superhero but you but most of us are just aren't that ambitious um we wrote our own scholar jason parser which was unfortunate because so did everybody else there was an official scholar jason yeah we have the twitter scholar jason but there was official an official one too oh like there was one it was more like an example yeah it didn't work really well so we had to rewrite it and then now there's like 500 no jackson yeah jackson jackson um yeah but i mean the problem with jason is there's no standardization and so no no no um okay other things we don't use uh nagadi has been replaced by finagle i've never heard of smile queerless is a uh it's a mysql client but it wasn't based on finagle and so it doesn't use futures and so it's single threaded so that can be a problem i've never heard of jackhammer but we do have a load testing framework that we've released it's called parrot um and there's other things we've we've rewritten it's called parrot in the open source world yeah we renamed it to confuse everybody compareless uh i think is not entirely replaced by finagle my sequel but that is it's maybe a 50 year 50 deprecation yeah you should use finagle mysql if you if you get into like really really heavy mysql hammering um the problem with all these things is like they're all really good libraries and and back to our sort of like unique requirements it's like a very very small flaw in a library we'll make it unusable when you bombard it with a ton of traffic and so it's it's one of those things that they might be even slightly more usable like configure for example is extremely easy to get started with which is sort of enticing but we get stuck here because like it has to not have bugs and so you need the type safety and then you get this whole complexity of like trying to boil down your configuration which is harder to do uh yeah yeah and i don't know what that is but great oh by the way let's talk about build tools sbt 0.7 isn't wonderful uh this i didn't like spt 0.7 when i was out um yeah it was unfortunate the thing is with sbt though has gotten much better lately so even though we've been driven to to roll our own uh i'm a big fan of modern sbt but uh man it must've been rough you were around what how was that yeah okay yeah specs is it's fine uh ides were bad then they're still kind of well we we still use intellij it's gotten much much better actually ids are great now um yeah it does work with sbt yeah so uh just in closing i think this is is a really interesting the the overall theme of looking at this talk is like how much things change in our industry like even in the scala community you've decided to use scala and you're going to use all the latest coolest stuff and you've got the hippest guy presenting and telling you what you should do and it's all changed five years later not that they're wrong it's just you outgrow it or things different things come out that are better and uh and it's kind of terrifying like like to look at this because i can't hope to be as talented as this guy and the people who originally built this stuff and and just everything gets blown away by the future progression of the technology so uh i don't decide to find that interesting um yeah thanks if you have any questions all right yeah i think yeah absolutely and mesos i think is a huge part of that right now again keyword in five years will that be the right answer i don't know but i do think that um part of what i was talking with alexi about before coming up here was that uh what we need to figure out is now we can run processes anywhere what is the standard api for processes to communicate um what is the for locating one another service discovery it's probably it's the oldest story in the book but it's being rediscovered in the new context of the the data center operating system um so watch that video because i'm probably not going to talk about it during this talk uh yeah i probably won't i probably won't get into that too much uh about any more about the data center operating system but i'll try to be interesting okay so we're resuming um that was april of 2010 still on 2 7. um creation of com twitter util um in terms of long-lived libraries and widely used libraries contour util is probably the most widely used library at twitter um and i kind of i've injected uh percentages of twitter code that are using just future i would say that something approaching 100 of twitter code is using util in some way or another but as util launched uh there was the introduction of future and try but not the scala concurrent future and try um com twitter util future and try august 2010 um and this was the symbolic birth of finagle finagle didn't really i don't think i apologize i don't have but i'm sure it's a matter of public record when finagle was initially open sourced but this was the symbolic of uh birth of finagle in that finagle is 100 all in on the idea of future and promise-based concurrency never even touched actors as a potential uh model the other thing that was introduced with util is scala's configuration which is kind of the next evolution after configure was let's just eval the whole thing it's much easier to write it's much it's type safe the issue then becomes as brian said you can put other stuff in your config evaling your config might have side effects um it might work in some languages but it does not quite yet work in scala right without without potentially firing the nukes as it were okay uh ooh let's let's go back to colors i'd have forgotten about the colors yellow is libraries and green is convention so we had libraries to do uh config as actual config files and config as code and so at this point scala's configuration was was booming unfortunately and inevitably it took people a few months in production before they realized i should compile my config i should i should have a unit test that compiles my config so that it doesn't blow up at runtime and everyone rewrites that anyway purple is tooling um and so this is about where we've we've reached this critical mass of of uh scala libraries at twitter um they're all they all have interdependencies finagle and util are very careful not to have other dependencies but if you write a useful library and you try not to re-write existing code you're going to end up with dependencies and the thing about it is you you want to encourage dependencies if you can in the sense that you should not write something in the wrong place if you can avoid it and so if you're trying to avoid some dependency you might end up writing something in the wrong place or rewriting it so we want to avoid that but at the same time so fairly early on we're figuring out we need some other solution uh an approach to composing source trees with spt anytime you see a quote like this it's it's essentially verbatim from internal mailing lists like let's figure this out um and this was a proposed solution to doing this uh standard twitter standard project and inline dependency plugins were plugins for spd that allowed you to compile things from source your dependencies from source if you were next to them um and the from source part solves the oh is it close to me or is it far away i'm kind of just tempted to rewrite it because it's too far away um so it but what it would also do is it would try to add version checks so i have the source it has a binary it has a version but i really just want to edit it and i really don't want to try and publish it in between so let's try to make sure that when i edit the source and then build and then later on publish and then depend on the artifact that i published things are kind of okay sort of but what that trends toward is i do a release for every git shop because if i don't i have kind of the sparse history thing and i'll come back to that um more tooling stuff still in 2010-ish so there's lots of back and forth because this stuff hasn't rolled out fully you have breaking api changes in libraries um and people do their best they're sending out email you know i bumped i bumped the major version or i forgot to bump the major version even if you don't forget you end up with diamonds right diamonds where i'm depending on you you're depending on something i something else i use at a different version now i need to go figure out how to publish that other thing right publishing is the bane it's a theme um we definitely had a few spt and compiler bugs in 2007. uh i would say compared to everything we were dealing with internally actual scala bugs probably were never a huge issue but you definitely did see some copy pasta between core libraries even um that just weren't weren't inlined appropriately and between configuring ostrich as an example i shouldn't have called that out here because it was it was happening everywhere so um it's a little bit too difficult to make a change over there and then publish it so i'll just edit it here i'll just inline it the bad way any questions okay we're still in 2010 i believe uh on the theme of libraries yellow there was a fair amount of internal early usage of actors and you see in in alex's deck that we were recommending actors um they kind of eventually all died out in terms of usage of com twitter util future and it happened fairly early even i there's a bullet point coming up my opinion on actors is just that the same way you don't almost do your devs using threads you definitely don't want them using actors if you need to use threads it's because you have this private state and you don't want private state anyway right so the vast majority of our mid-tier code is written to use futures and there's almost no usage of actors internally so again opinion um but it's what it's what has triumphed at least within twitter so i'm gonna make a seg uh into talking about monorepos because they go to that source dependency point um i put i put the most controversial stuff on the slide with the smallest text okay so we'll get through that fairly quickly um but the idea behind a monorepo is that you build everything from source or you have the source nearby there is no near far question it's all near you know i can build i can touch anything within twitter's code base fairly easily and it so it eliminates binary dependencies or inconsistent between inconsistencies between the source and binary dependencies from your mind additionally it lets you change multiple projects atomically in one commit so if i'm changing both finagle and util which was the very first place that twitter was using these together right these are these are definitely independent libraries you shouldn't use both of them together excuse me you shouldn't be forced to use finagle if you're using util but at the same time they're so intimately tightly related that not being able to change them together at once would be insane i have to commit over here and then i have to attempt to commit over here but using the right published version from over here and worked for a little while but really probably in this time frame like six months um so you get you can also kind of shorten or remove deprecation cycles because if i commit to both of them i don't have to release either of them um and so this this like oh anyway so top to bottom continuous cross project integration so the idea is if you're building everything from source you can build everything you don't need automation to go and bump the version manually and check whether it still builds and in general you avoid publishing so if i am fixing a bug six six libraries away i don't need to publish six times to get back to where i am um and get bisect is great so we like being able to do that across the entire history um and so obviously this is this disagrees with how most open source is done and we're still trying to reconcile that um but we think it's really important to reconcile that because we absolutely believe in open source so stay tuned any questions about mono repos okay so then scala 2 8 2011 and 2012. anyone using two eight no not quite uh no responses to that thread scholar 281 is released all right we did finally upgrade uh it was very hot to hawk we were still in this phase where projects were living all over the place they were in their own github repos it was difficult to i need a thing where do i find the thing well i'll search github for the code that like the person sitting next to me wrote so i can figure out where to get it then i figure out what you know is weird about their spt config so that i can actually publish them so various upgrade branches for each of the projects live for a while relevant stuff was was published and then you had to go figure out what was published because you didn't you know you can't just say i want the latest of a particular version because you've got diamonds to deal with uh cross publishing the question was about crop cross publishing yes um and i'm going to get into that cross publishing is kind of great but it still doesn't tell you um precisely which version to depend on right um yeah so cross publishing is a thing that we we have missed to be sure um so there's a stronger push to get projects on two eight uh and it mostly happened mostly happened we can't say that it happened at a particular time because these desperate projects there was a long tale of things upgrading um but we do know that two seven bugs were still biting people way into the summer of 20 or the spring of 2011. um so yeah i mean tooling wise there were diamonds diamonds in the core libraries is there a standard version of scala uh not really but at some point in that in that summer it was finally upgraded we did experience some minor incompatibilities but again the theme you'll notice is that we had all kinds of other to deal with it was not really scala's fault per se um so the birdcage i'm sure it's been talked about somewhere but the birdcage was twitter's first scala monorepo um first commits in march of 2011 and kind of a wider uh announcement in april 2011 there were 12 projects in the birdcage so it's a it's a monorepo containing the source for finagle util various other things that are just kind of critical the very corest of core libraries um and that's com twitter future was ut was imported at that point and that's when i began tracking history i didn't i didn't spelunk farther back um but at that point com twitter util future was used in 26 of scala files which is something like 500 files so not terribly impressive in march um so then the first major i say third party only in the sense that it was not in this source repo kestrel was initially an actors based library it migrated to futures uh it went fairly well they found a few blogs in futures but in general the code got much cleaner roby who we love switched to comic stands on the mailing lists i don't know if it's because like future's just drove him crazy or actors had or who knows what but in any case um we love him uh 2011 2012. so we opened source scholar school and that was because we had a lot of new hires at that point uh not because the ruby and scala worlds had really begun to mix we'll get into that later um additionally you know and this is kind of a theme that you'll see forever is uh somebody's asking like i need something like type classes and somebody recommends view bounds and nobody really pushes back now we know um in any case the the debate about oo versus functional continues but i think as i've said here it's mostly resolved by having good conventions i have got lots of great models i've got tons of anecdotes in terms of code that's running and has run for quite a bit of time but again it's really not clear there are a lot of things that are idiomatic in certain contexts as oo or as functional and i think the more fundamental thing in terms of just uh like dividing line stake in the sand is whether you're immutable or not being immutable rules out certain noble patterns just entirely the builder pattern is one that's still huge because it allows for immutability um which we love so but in some cases you know modeling things as glasses is still the way back on tooling what was green huh convention yes thank you okay so now we're back on tooling um in 2011 2012 uh the first suggestion of using switching to pants so this is weird where did pants even come from how are we suggesting switching to it um the issue that has plagued twitter for a very long time is that we wanted mono repos we wanted to be able to not have it near and far in terms of code but the fact of the matter was that we had a repo that was to be frank exactly modeled after google's repo internally and it was entirely java code with a small with the exception of a small amount of scala and i i apologize for not including when that scala landed there in any case pants had been born in this repo that was building the majority java code and a little bit of scala code and over in this monorepo the birdcage there was the very first suggestion of maybe switching to pants from sbt unfortunately pants wasn't quite mature enough yet and so the actual work doesn't start until january 2012 but then i'll get to that pants build is open source um and i'll talk a little bit about it at the end but it is now we are now all in on pants as our build tool so in 2011 tail call tail calls began being eliminated uh in future allowing for a kind of infinite infinitely recursive flat maps um which is fairly handy being able to loop uh i don't know yeah yeah um it's good well and and there's there's a relevant data point in here which is um well the question was how long is it how long before that was in uh scholarly current future there's a relevant data point in here which i'll get to in a second this is also around the same time that scalding was first advertised internally first commit was in may 2011 thanks to avi bryant um and oscar boykin has a great talk about that so that's there but it was open source in january 2012. i will get to more on scalding later now we're in the two nine era so here's that point i was talking about uh sip 14 is the um scala futures and promises uh sip and marius commented on it and he recommended against tail call implement our tail call elimination because it really does complicate things um at the same time the expectation you get on the other side is i want to be able to write a loop and i don't want it to blow out my stack um my my time time disconnected stack async stack um so he had some great comments on that uh i think the opinion was then and is still now that once everyone gets their things in order and apis settle on both sides it's a thing we totally desire to do to to merge i'll talk about that a little bit more at the end but at that point it was just advice for for the scholar developers on on the development of this new api um effective scala was open sourced still a great great resource hasn't changed at all i don't think it's still still a great resource i think just by by virtue of focusing on precisely what we were certain of there's plenty of stuff that's not in there because we weren't certain about it uh now in 2012 maven that came out of nowhere so we were using spt um but we were using spt with internal plug-ins that meant that we were building from source but it turns out that this is the thing that maven has supported forever i don't know all of the details of why but i do know that for a long time there it rolled out relatively easily people got tooling fairly quickly it disconnected us it forked us from the community in the sense that we were now not doing what everyone else was doing but it was a short-term gain um meanwhile also in the birdcage about tooling there was some friction about squashing and squashing history within git in order to keep things consistent so we're now committing all of our code approximately to this one repo which has this consistent view of history of all the code um and in order for that not to contain fix me fix me again to do and all kinds of stuff like drunk and last commit before you know before shipping you you squash commits you end up with a consistent history um but this idea of getting everyone consistent was was super super out of twitter's uh wheelhouse it was not something we'd done before you we had open source repos they all had their own conventions um there wasn't really any consistency so the idea that consistency might be more important than um i don't know opinion uh was was difficult to swallow in any case scholar 29 was released and i think that's what probably pushed us over the cusp we got excited about scholar 2-9 and we still had few enough core scholar developers that we got everyone in a room um and unfortunately because it had occurred shortly after the switch from svt slightly short-sighted uh there was no cross-publishing and so you did major version bumps like i'm going to do a major version bump release the library continue to release on on the previous major version um and then that major version is the next next version of the library there's a reason cross publishing exists and you probably shouldn't do that um in any case it was very organic and kind of bottom up due again due to these repo's not all being within the bird cage also during this time frame there were we contributed a bunch of time to improving maven and scholar support and intellij and it's now very good two you know two two to three years later people complain that we want to take it away from them so i don't know it's maybe it's just uh stockholm syndrome but it's on we're unsure okay again this is on the uh the usage theme so in 2012 and 2014 still on 2-9 there was a huge onboarding of twitter devs to scala um because for who's not familiar with the monorail what a monorail might be okay well i'll talk briefly very briefly about it the monorail was the monolithic ruby application that was powering the the front end of twitter but also a lot of api endpoints um so this was a a finally a top-down effort to finally to say um we are not going to write anything else there it's time to go all in on scala and before this it was mostly limited to services but now it was additionally many front end and api developers people were building custom web frameworks we have now ended up with one yes what's going on we still use ruby in places where scalability isn't a concern true statement that the statement was we still use ruby in places where scalability isn't a concern yes correct yeah and ruby and python are kind of going at it internally for internal apps at the moment um so there were a ton of aggressive internal courses it resulted in tons of questions to the internal mailing list like this isn't like you know this is a little bit too different from ruby i'm confused um big up uptick in activity on the lists uh but it all settles down people people seem to figure scala out um some more library stuff bijection was open sourced in 2013 and it caused a bit of friction because mathematical terms was fairly quickly resolved and is now terribly useful so use it um scallop two nine again 293 we switched to internally we were already on 292

um and then april 2013. we deprecated get and apply um across the entire monorepo scala code um in a kind of an in an atomic commit and this was via compiler plug-in which i need to go and research a little bit more because we're finally we're we're back in positions where we need to be using that again um but the the goal of this was to begin to align with current future it's it's glacial glacial movement but kind of for necessary reasons like we need we need apis to be in a land or else we'll waste our time on this um but by this time com twitter util future was imported in 30 of scala files that doesn't seem like a huge improvement except that by this time there were tens to 20 thousands of scala files and so the fact that it maintained its its uh usage as the company continued to grow is really relevant um also in this time frame stitch was created uh who who's familiar with stitch wow okay so wow stitch is a library for composing data requests stitch is open source yes the question is yes oh that's why that explains that apologies yeah uh soon soon it's a great library um it's been promised over and over yeah yeah yeah we are we are actually very close this time yeah travis says we're very close this time so stitch will be open sourced that's good because i think it's amazing um i think it's critical to a a service oriented architecture that things can mostly just act as if you have a single database and stitch allows for that by allowing you to compose queries um in a way that you usually can't so highly recommended um created okay yeah i didn't say open source that explains that very relevant so then now we're in the current slice of time uh 2014 through who knows when um so the 210 upgrade was initially blocked on being 100 on pants wait a second back on pants yeah so as pants matured internally living in this other repo it was finally good enough to kill maven um and pants has tons of merits it's it was designed from the ground up to be uh a build tool for building from source whereas maven is always assuming that you have jars and is not necessarily trying to optimize across library boundaries so pants was was blocking the upgrade for a short period scholar 2104 was released a common theme you'll see is that the the last stable release of scala was is released and then we upgrade um i don't think that was intentional but somehow that's how it ended up working out this is also when summing bird was open sourced another great analytics library that's absolutely open source i know that uh okay so rumblings around about around scholar 210 amplify upgrade actually starts in earnest because we're in a monorepo by this point the scala upgrade across um two to three million scholar files is accomplished by two or three people eighty eight excuse me ninety percent of the upgrade is accomplished by two or three people right and then you have some number of failing tests you ask people to come in and fix their tests and then you commit it and that's approximately what happened so brian had a huge hand in that and he didn't mention that um but it's huge and i think it's really relevant to twitter because it was kind of the birth of a framework for how you do these huge org crossing projects you got a question so yeah in my organization that would be like an incredibly scary thing because of our test coverage are you confident in offering your test coverage that like if the best pass works not necessarily but you have to separate the difference between upgrading the scala library and rolling out the service um and so if we separated it by as long as we got all the tests passed and got it committed and sourced then you as a service owner now you get to go task to have fun with that yeah the question was like how do you actually test when you when you're landing it all atomically and the answer is you live in a branch for a while right the thing about having a monorepo like this is that you can actually use git and branches to the fullest so here is here's the 2-9 universe here is the 210 universe and it's literally everything it's everything that compiles with 210. um and so if you want to validate your service and we ask people to you know deploy out of this branch and it gets it's drastically easier than the idea of you know finding all of the right 210 versions updating your updating your your uh dependency file having them conflict doing it again right until you finally converge so essentially we have the the framework now is if you want to make this massive crossover change you just create a branch and then you invite people to work in it um but but also it's important to distinguish the difference between committing it to source and rolling out that that code and we didn't roll out all of twitter on 210 in one day right oh i hope it works you know we wrote a piecemeal over over months but as long as the whole system builds and runs all the tests then your little component will build and then it's maybe there are bugs but they're not ones that you exercise you know so your thing works correctly and that's all that matters as long as your testing is good and actually we do have a really good test coverage we actually didn't have any runtime issues after the upgrade that i'm wearing no no incidents at all um you know ellen degeneres takes a picture we get an incident but this upgrade didn't do anything which was amazing and there was a couple of bugs in here that were really scary map ordering changed some people in two to nine i think map ordering if you iterated it was sequential based on insertion and then in 210 it was random and so if you trusted that map ordering on iteration was the same as when you inserted it you're going to get in big trouble with that nobody ever depended on that luckily yeah i think one one thing that's critical there too and i totally forgot to put it on the monorepo slide um we call the monorail a monolith we call a monorepo a monolith but the fundamental uh game-changing difference is that you don't deploy it monolithically it's deployed independently on a team's schedule when it's sure when they're certain that they are going to be there to babysit it um it's not you know an army of people on call for everyone else's stuff when you monolithically deploy so you monolithically build you guarantee it all builds together but deploy wise uh it's on your schedule um we've just guaranteed that in the meantime your code hasn't rotted um just to to talk more about that also we do stage deploys and as we're talking about ostrich and metrics we have just a ton of metrics so when you do a deploy you have this huge huge cluster and you pick a few couple machines on it you roll your code out and you look at the metrics as the graph is being generated and then you see if anything drops then you go over reverb revert and if it doesn't then you keep going and so we do these very careful deploys to ensure that maybe if it does have a problem you're only going to affect you know a tenth of one percent of the user is going to see it and if they do see it they go huh and then they reload they get to a different server and it works again so the problems are consistent and so it doesn't actually cause any interruption of service fingers crossed yeah any other questions sorry any other questions okay yeah okay so by this time we're in 2014 actually we're midway into 2014. um com twitter util future has kind of peaked out it's in a third literally a third of twitter's code which means that a third of twitter's code is explicitly async um some of it is library code that is math code for example it's not going to need to ever represent itself as a future it's going to be run synchronously on a thread in some number of microseconds and so we're not concerned after this after this 33 peak i think stitch actually begins to cannibalize a little bit of futures usage internally and it's currently at something like two percent two percent of code is using stitch and 32 is using futures they are partners they're not there's no no chance that stitch is going to take all uses your future anyway um scalding at this point which we talked about uh about a year and a half ago in terms of this timeline is now the default for all analytics jobs for a long time we had pig usage usage of apache pig and contributors to apache pig all the contributors are totally on board with not doing that anymore and scalding has become the default um so i don't know what the percentage of usage is but it's uh there are no new pig jobs being written so that's that's very significant and scholar 211 5 is released this has not immediately led to our scholar 2 11 upgrade why is that so continuing story um we now have this framework in place where we feel comfortable or we're very close to feeling comfortable with doing these massive upgrades i think brian brian and and ian connell uh o'connell excuse me ian o'connell's work on the 210 upgrade was a huge um kind of uh milestone in the sense that it says a framework for for future upgrades um the next one will be very easy what will be the thing that pushes us over the edge uh who knows but it's probably not nearly as far away or nearly as nebulous as the the previous upgrades have been it is worth noting that lots of our open source libraries are already available for 211 and they are being used in production outside of twitter yeah so travis was saying that a lot of our open source libraries are already cross-compiled thank you spt for 211 and yeah so use them one of the things that's sort of delaying me is i think we can get such good tooling to do these upgrades that they can be nearly automatic yeah and so doing a manual upgrade for a month even two people seems wasteful to me instead of spending two people spending a month writing better tooling that'll just go upgrade it all and fix all the bugs like refactoring code for example i'm gonna be really close to getting stuff to work like that correct so i'm holding back because i think we can write these tools that'll just make it super easy yeah yeah so we expect a few future updates to be drastically different um than the past ones have so now i'm going to just going to talk about kind of the continuing story this is this used to be titled low hanging fruit they're not all low hanging but they're things we desire um so in the skull of core we just we always have a desire for less object overhead you want to be able to use the idiomatic library or api excuse me the idiomatic api because scala does a pretty good job of creating idiomatic apis that are fairly straightforward but in cases where they have zero cost oh man it's just it's a no-brainer um so we want more zero-cost idiomatic apis if possible um we'd love improved support for maintaining consistent code bases because we now have many millions of lines that we need to keep consistent um and we think abide is potentially a great way to do that um so we're on board uh libraries we're not using the async macro despite having future in 33 of our code which is kind of nuts that's at least partially because of the 210 thing but i think there's also a little bit of a boiled frog effect people haven't noticed that that would probably make some things clearer so we'll say that we'll probably get a little bit of usage soon um this one's kind of more personal i i and other people who write uh apis that deal with data or records really want scala to be able to do something like shapeless shapeless's hlist if you could just drop that feature out and put it in the standard library it'd be amazing but we have relatively little usage of scala z and shapeless and the other uh highly functional libraries within twitter um at least partially because we've discouraged them in favor of just kind of well-documented more verbose uh abstractions um but hls is killer and so i'd love to see that usage uh scala js is kind of mind-blowing and we have a lot of javascript in in that gap of 40 scala and 25 java there's a fair amount of javascript um it would be great to start using that but i can't make any promises um also our code is is java with with regard to android um so we should definitely be using scala there as well as much as possible tooling we'd like to see so pants has a distributed build cache which means that when you're building from source it's not like you're building from source really um for this sha and these sources uh fetch me the artifact i don't need to publish it manually it's just automatic which is kind of lovely uh but that's the thing that my team uh the deadpret team um is working on improving within pants so that it's just ubiquitous you don't even notice it's being cached um everything's cached and should allow for some great great runtime performance improvements um as it stands if you do need to test absolutely all of that those many millions of lines of code it still only takes like 30 minutes so we're doing a decent job but we could be doing much better and as brian mentioned or as we kind of indicated when we did that refactor from future.get apply to a weight we really need large-scale refactoring tools we have everything in one place um and we control what tooling we're using so this this last bullet point is actually kind of an invitation to the community we're ready for things to change in the sense that you know if we don't get an automated refactoring tool we could use said and and get it done and make sure it passes the test and then commit it so that you know that's not too bad um we'd love to see continued improvements simplifications uh and expansions to the late to the library or excuse me to the standard library and to skull itself and that's not i think it's very important just to point out that that's not a place we were when everything was disparate and living uh all over the place so it's new um thank you a huge amount to all of the kind of early scholar adopters these are all twitter former twitter employees who i love um sorry they're not all former whoa excuse me there are some amazing yeah current and twitter employees in here as well who i won't call out many many great contributions from everyone who has questions of performance you mentioned you're getting that um do you see maybe a place to get proportionate here and someone's working with guys that open jdk you're doing the speed up the vm itself you see that and i mentioned about three to six months ago sometimes they'd probably yeah so the question was about performance from the jvm itself and speeding up kind of sequential code when you need to um we we have a ridiculously good jvm team many of the former oracle uh gc folks and they're doing a great job of bringing in new features and and uh maintaining very very careful patches atop the jvm um so i would say that we were positioned to take advantage of of everything that's coming down the pipe and probably to optimize some things on our own um that that example was really just like if you're doing numerical computation which is fairly rare like the idea of computing fibonacci and then returning it by finagle futures you're not going to do that very often but if you are you can put it in a thread pole excuse me future pool um or execution context that uh where the real performance on the bottom like everybody else is doing large distribution systems like if you open that up and people can access your cloud pool they'll give you performance resources i can do a web browser for javascript for for our twitter instagram so like yeah we'll make your performance better and faster yeah uh so i guess i guess i'm sorry what you were getting at was open sourcing jdk related stuff yeah i think travis could speak more to that i don't know if um there's anything like that in the pipeline but i don't see why not so yeah yeah we'll see what's going on uh yeah so the question was are we using apache spark do you know no no there's definitely been experimentation i don't think it's in production anywhere i mentioned scalding as having taken a huge proportion of our of our jobs there has been experimentation about running scalding on spark so it's more it's it's not an api that people again kind of like you know do people use hadoop they at some point they stop realizing they're using hadoop because they're using a library that's like 10 layers above and so i think that's probably what will happen i don't know who's going to be using spark directly at twitter it'll it'll be most likely it'll be just like all of a sudden okay yeah we decided and then now everyone's using spark without knowing it okay i think we have time for one more question before adrian's talk who is the best one put your hand highest okay uh who was first anyone knows i'm sorry yeah uh one question uh during uh talk you mentioned different build system miami sbt and uh finally what's uh build system so the question was about what twitter's build system looks like i love that question thank you so i currently work on pants pants is open source um it was initially developed in that repo to build java code it sprouted support for scala code it kind of always supported python so it builds javascale and python it focuses on packages not projects so the idea is if you have a scala package or a java package that's you want it to be fairly small you want it to declare its dependencies and you want to require that it actually declares its dependencies that you don't have kind of hidden cycles in your build graph so the focus is on really small reusable packages rather than even projects so it's written in python it's open source pantsbuild.github.io i believe and it's developed in collaboration with foursquare and square and some folks i'm not going to mention too many names because but i definitely know we're working with them all the time and that's worked out great in terms of open source in the sense that uh it was a rough a rough birth but we are now at this point where it's kind of it's a tripod of equal contributions from from those companies so i highly recommend usage of pants we do also provide spt builds for all of our open source solid projects and we use those sbt builds to publish the open source i just want to ask a follow-up on pants so first of all thanks to and run for a fantastic history arc and i actually have this kind of peculiar interest in pens because i asked several times uh you know i heard about it from mario's parents when he was advising my startup and i was really curious right because in the world of uh sbt and madden choices like cylinder right like can you do something else so spence was actually found in the open source but nobody wanted to talk about it and i actually tried to get bitter folks and foursquare folks to talk about it and they wouldn't talk about that and i wonder uh like what is the kind of idea of pants right now is it is it ready for big time does twitter wants it to go big time right uh can it become a real big alternative to spt what what do you guys think right so the question was essentially do we want do we want more developers of pants or use excuse me consumers of pants right now the answer is it's it's definitely still pre 10. we know precisely what we need to do to make it 1-0 but at the same time it is it is the build system at uh at three very large organizations so um it currently needs developers not users is what i would say but they're welcome thank you very much