scale.bythebay.io: The Legends of Twitter and Beyond: Real-World Architectures at Scale Panel
the first day of the conference well what's up here so welcome back you know we like to start as a one big community and then we'll fan into three different aspects of software engineering and then we reconvene here and then look have a happy hour and this time I also have a fireside chat so this is personally really exciting to me so a last year who's here last year so last year I kind of moderate all the panels because I like doing it but I kind of realized that you know I kind of reached the level of my competency and so I'm only moderate in one panel today we should still be on the level of my competency but I love history of software engineering and I think we have really you know five years is a huge you know it's like it's it's half-life of like dog life it's really tremendous a poke in in its it's a really long time right what I'm trying to explain so so you know if you've been in this industry right five years ago a lot of things which we take for granted are ubiquitous and I think personally that a lot of credit goes to this community and it goes to Twitter right and a lot of folks who basically started this they found out into the world so once we were we're a Twitter and I actually try to do similar things last year but we kind of didn't get the momentum and I think this year we succeeded in bringing you know some of the original folks who started the technologies in use everywhere let's are them at Twitter so so they're the kind of goal is to kind of do a recap and understand how this technology's fit with each other what each of them does right and what is kind of the contribution to global community where and where people can take them you know where people can make use of of the stack so so it's supposed to be a conversation so we have no everybody has a mic so this is gonna be team work right so this is for the panelists basically we're gonna do it like a distributed system where you look for the nearest mic and write and and we'll see how this goes right so we'll have a little distributed system going on here we need consensus so and this is supposed to be conversation so I'll you know ask you guys to introduce yourselves and describe kind of your original story compressed at Twitter and and how would led to the things you are doing now right and hopefully will pick up on this and maybe one thing I'd like to see if you if you can think about this how do you relate to other people in the stack so consider this kind of stack right it may be out of order so Ben should be somewhere like platform wise under everybody all right like so so so if you can kind of think you know give us some hints how you relate to other people in the second maybe tell us how historically how we were relating to each other when you were overlapping it with yourself I will start from Aris all right so are you saying this is a yapping philosopher problem like exactly yeah boy did that luck anyway so Sam Myers Erickson I was a Twitter from 2009 or so late 2009 until last year I'm now out of the online transactional processing business and working in very happy working on batch computing processing very large data sets at Twitter I spend some time doing this sort of initial geolocation stack and then as we transition into a more service-oriented architecture I helped start the product that became finagle and a bunch of other kind of downstream projects from that so this includes something we call Wiley which is this naming system and and a few other assorted projects that relate to that and my last year Twitter working something that's really really exciting I think all is a kind amenable evolution of this space but I'm not sure I can actually talk about us in here so that's my TL DR hi I'm William Morgan I'm I work at a pretty cool company called buoyant and let's see I was at Twitter 2010 I think that 2014 or so in contrast to the many illustrious figure figures on this panel I was mostly a parasite that didn't really produce very much of value and that's it ya know I was I was there as an engineer mostly primarily as a service ng soner so I built some of the early kind of media services and now what we do a buoyant is much more related to stuff that I didn't actually kind of directly I said I was a happy customer of things like like finagle but my co-founder Oliver who's not here is the one who actually knows how to program computers these days I'm so I'm mostly here to paint the rosy vision of how awesome everything is hey my name is Evan Weaver I'm at phonon DB right now I'm CEO and co-founder it's a distributed database company available in the surrealist clan on premises as well and that's a direct descendant of the work we did in Twitter on what was called the infrastructure team because I joined Twitter in May 2008 as employee number 15 at the time there were about six engineers Alex Payne brought me on who probably is something to do with many of your introductions to Scala and I ended up managing the the engineering team that built all the distributed storage for the core business objects so that was tweets timelines the social graph user storage image storage the cache probably some other storage there's a lot of storage we didn't do ads we didn't do analytics we didn't do anything that was outside of the critical customer facing facing requests fans but basically our story there and I'm I'm not sure probably to their benefit I'm not sure I directly managed anyone else who's sitting up here but uh as the team grew and we expanded into a service-oriented architecture and eventually micro-services architecture my team was frustrated that we never had the chance to build a truly reusable storage platform especially for a transactional data so essentially fauna is that platform it's it's still on my name is Mark McBride I joined Twitter in 2009 a little before Mariusz but not much and stayed through 2012 my first job there was working on the streaming API which I think was Twitter's first production Scala service so that was an interesting Road manage that for a while and then managed Twitter's front end team front-end traffic management team which resulted in TFE for a while during one of its darker periods and then went from there to looking as this Twitter got more service-oriented trying to improve productivity across the team as we got you know more services more changes of the codebase and started a team called developer productivity which fixed things about two years after I left and so I went from there to nest and ran server-side engineering and really missed a lot of the things we'd built out of Twitter and so after leaving nest started a company called turbine labs which is really trying to take a lot of the more modern Network traffic management things that are available today and use them as a way to improve overall team productivity as you scale up orgs and scale up the number of services and try and get more cloud native I guess is the term around stuff so alright so my name is Ben Heineman I joined Twitter early 2010 that's first two consultants then I was an intern and I was a consultant again an invention employee I worked at first under the chief scientist we had a research project which was to try to do cluster management resource management around the same time Twitter started to become a so uh eventually microservices and we built out a project here called Apache Mae sauce at Twitter which we used to run a lot of those services and and I started a company called mesas fair hey I have not worked at Twitter but they're recruited you have TP I probably shouldn't say this but I work at Apple and I probably can't say very much but anyway when I was at VMware I started jumped into Scala Anaka at the same time on it was the first girl team at VMware a while back and we were working on I was four senior engineers doing a new cloud orchestration service for companies like Verizon and from there I went into I actually was an early contributor to acha acha cluster when it was still experimental adding two new features after that I joined a crowd strike which is a cyber security company and that's actually where I got into analytics and I realized after I built you know a couple of pipelines for them the first ones that they had in Scala and I realized that you know who duped really sucks and this whole batch thing also is kind of lame we need to do some real-time detection and so I started to look into what those architectures could be and this is when all of the the way that we do things now no one was really doing them they weren't really alive in any systems I think and so I I found some folks that did a stacks that we're also interested in you know doing away with using Hadoop and doing things more in a streaming manner and so I joined them and started working on systems like that and with Kafka and just you know making everything much more real-time and now I might not vote thank you so I guess one question I have about software's texture so we talk about real life software architectures so one way people talk about stacks right and let's question everything so it's doing in stacks and what what's tech should look like right so I think it's very interesting for me to see that so Helana opened this morning about distributed systems I built a lot of them I build a stack so we we talked about this Mac stack right we should spark Matsusaka Cassandra and Kafka basically a lot of companies keep reinventing right so if you look at any large web scale company if they have an API maybe its user ring and the millions of users banging on iPhones right then they have and so they have an API then they have a message boss they have computes they have persistence they have operations right so this essentially smacks deck it's it gets so every letter is replaceable and every letter does one thing and does it well right but now we see very interesting things so each of you guys from Twitter kind of took the letter and went away with this right and maybe and so you kind of want to provide that that piece of the stack is a service so you expect others to integrate with that and so what's interesting to me in for instance fauna is doing stuff in the database alright so like everything can be done in the database streaming people say everything can be done in a stream right like you just stream stuff through it and we'll so you know Kafka that Kafka sequel Casey cool right so so it's kind of every letter says I'm the main letter right it's very easy to add stuff to me and I will do everything for you and so some curious you know can it all be true at the same time and and you know that's one thing so what is your kind of vision for for your letter as if you know to say that and and and so you like people Twitter we're in the same company right so it's it's easy to communicate with a common code base right as there is a whole bond a lot of boundaries are are not there right but now you kind of when you separate this concerns you basically become the service-oriented global marketplace of services right and so there is a whole bunch of problems now because when customers come in you don't bore them in to tell them how to interact with your system so I'm so I'm basically curious what do you think about this idea of a stack right and you know how how can it be viable for for individual letters alright non-focal and I guess it kind of who implemented the smack stack and opensource right maybe like what is your view basically right from the outside of the letter so I might want to shatter that idea uh-huh what I've seen is you know I used to in smaller companies see that as applicable but now I really and in in recent years in general actually I mean there are so many different use cases there are so many different sizes of projects between where a system needs to the variety of places people projects need to store data and how they need to store it and how often they need to access it and how they what's accessing it what algorithms you're using to access it based on where it's stored globally I mean I think that it's not that simple at all which i think is no surprise to anyone in the room I mean I think the smack stack is very applicable to many situations but there's so many cases at the same time where you have to interject other technologies as well that and the thing that I always like to talk about is the collaborative type of technologies where you might have that foundation but there's other ones that work in collaboration with them to to add-on so I throw and I mean is from a does a stack make sense perspective or what's important about the stack I mean I think the most important thing is if the technologies can compose well together so I you know I agree that I don't think smack is sort of to me it's a good initial perspective but there's other C's in Kay's and Q's and ours and T's and V's and whatever else that most organizations I think ultimately will probably adopt and I think what's just critical is if we can think about just building the the software in ways in which you can compose well together so you can bring in new services to help solve problems easier and replace services and I that's sort of what's going on over there they're not composing well yeah so so I'd to me that that's I think the most important part less exactly what the acronym is more that the technologies can compose so I would take that even further I would say that so um of the kinds of interesting and generic products that exists the usually either carried those abstractions down or erect new ones right and so yeah I can give some examples so so we sort of build up this set of assumptions about or the the shape of the way you solve the clear kinds of problems like this smack thing or Hadoop or whatever else and then somebody comes along and says actually you know what this would be much more elegantly solve this it's done directly in some way or we actually introduce a new kind of abstraction like Oscar Lux to put mono it's everywhere right and I feel like those two things so this is sort of like an interesting balance to strike I can give an a a recent example from my own experience we have a data processing system where I'm working now which I talked about this morning where we were actually able to by basically not reusing the kinds of traditional system section might think like kubernetes for example we're able to just dramatically simplify an entire software stack by building effectively a monolith and it ended up being you know far easier to you know reason about the whole thing to introduce new features to introduce type coupling because that's actually what the problem needed right and so I feel like ultimately the only you know the the the real answer is that you always need to solve problems nothing too much about stacks right if it happens that you know your letters are you know useful tools to use and by all means use them but don't lose sight of the problem that you're solving right did you call kubernetes a traditional the traditional stack so we had an interesting experience with linker D which is our open-source project I think is kind of blew me away we're you know we started basically with with finagle and and we wrap this thing up as a as a proxy and we're like alright this thing is gonna be really useful you know to people who are like running micro services and ok so what are we gonna call it well you know it's an RPC proxy so like we called an RPC proxy and people were like well I don't I don't need that and you know first of all I've got a CH a proxy we're like no no that's totally different from H a proxy and then they're like and we're not even using RPC of like we're using HTTP and they're like no no in our ontology C HTTP is like a subset of RPC so totally make sense and by the time you went through that conversation they were like off doing doing something else and so we had to iterate on what we were calling this thing and like eventually you know after calling it like an application router and RPC proxy we started calling it a service mesh and that was like a new term that we invented and like no one knew what a service mesh well I mean we didn't even know how the service measure was but by giving it this new name we it was like a blank space where we could write in there and we could say hey and and this is what a service misha's you know it's a dedicated layer for infrastructure managing your services service communication and so now like when you know the idea kind of caught on and like the service mesh is now part of the cloud native stack but you know that that was not a thing that happened you know that was not a that was not a thing that existed or that wasn't a part of the stack you know even even a year ago and the product hasn't changed our liquor D hasn't changed its finagle hasn't changed like this thing is still the same thing but the way that people are thinking about it the way that people are understanding like what you have to compose together has changed so I guess what I'm trying to say is like the notion of a stack I think can can can be useful clearly it's a guideline but it also changes over time in you know not just like adding and removing but also like inserting things and INRI partitioning and moving functionality around between between the border so should we call it smacks or smack so it is Twitter um as I'm sure you're well aware started on Rails which was iteration of the lamp stack and to an extreme degree - and I think the way I view it like a framework or a stack is a stepping stone it helps you get something up and running into end very quickly but obviously Twitter did not end on the rail stack and I don't think any modern consumer enterprise web property whatever you want to call it has actually conformed to the guidelines laid down by whatever framework they started with in the long run I think similar to everyone else what we've discovered is there's a huge diversity of potential problems and potential tools out there that people want to use with your own software and our goal with Fawna for example is basically to make the biggest possible hammer for a set of common problems and let you do whatever you need to do to solve your specific problem and the concept of a stack doesn't really come into that because we want to play nice with whatever other systems you need to use yeah and I think from our perspective like stack is is interesting as as frameworks but I think a lot of the things that have happened recently in compute over the last two or three years have really made it much easier to deploy a much larger number of services and people were a decade ago so you have companies like uber that have I think more than a service per engineer at last count which it still seems nutty but a decade ago that wasn't the case and now that you can do things like move to containers so you can deploy things head of granularity that's less than a single computer easily you get smaller processes that are worthwhile to run at Twitter in 2010 you like felt bad if you weren't consuming all the resources of some type on a box like it wasn't worth it to get a machine to run a process that was going to use 64 Meg's of RAM it just didn't make sense but with containers now sure you know you run 50 of those containers under box and it makes sense but as that happens I think the challenge becomes more and like what's the organizational infrastructure you need to help teams work together in that environment and so you run into cases where there's lots of different stacks in use but teams still need to work together to get a consistent experience out to customers and looking at the tools you need to make that happen is you know equally interesting to us as it is like what particular letters are using in stacks across the company there's very few places I think over time that keep a single consistent one Google is probably the exception but I think even Twitter was like we're just gonna be mezzo Center on data centers and then you acquire vine and it's like okay well you can stop doing development for a year and a half and move into mezzo san our data centers or maybe you get to keep running on AWS and we can ship features and so I think that kind of embracing organizations that use different stuff but having common abstractions that let you ship and experience out to customers is a different sort of abstraction layer that you want to work across and get standards to find in iiiiii I will say that probably one of the most valuable parts of the acronym is to just help classify a set of problems that people are thinking about solving I mean like we heard the word lamp earlier which I mean it I most organizations that I went chat would they were not exactly lamp but they were taking this patterns of lamp to try to solve their problems and I think that's a really really valuable part of these acronyms and these stacks when we think about them they're just gonna have lots of different letters but it helps us all collector they go yeah okay or the I want to solve this class of problems and people are solving it with lamp or people are solving it with smack and and and that can be really helpful from that perspective so it's interesting in the lab case the only part you actually provided yourself was to pee everything else was you know just like a template right and and sort of out of the box software and so you only iterated on the pee ever well I mean we iterated on the M yes yes but most people ever only iterate on the pee right and there are lots of people that weren't also not doing it on the hill right they're doing on something else sometimes there but but anyway I I guess I guess my point is that that seemed like a simpler time right there were sort of as a simple users less machine awesome but uh there was sort of this one variable that you kept changing but now with you know the sort of modern computing environment you're changing everything all the time and and you know like SOA for example is more like archetype than it is sort of kind of pattern and you're left to answer lots of questions on your own with lamp there was really not that much you know there wasn't there there was not that much variability between different implementations right and so I think that's something to keep in mind actually as you move forward like what you know the the beauty of some of my clamp is that it was very simple and easy being a developer and really really simple to iterate and push new things out as we've added all this complexity sometimes for good reason sometimes not we often make it much more difficult to actually operate and to deploy and practice and so on and so forth and I think that's kind of a sort of underappreciated aspect of engineering in general is sort of developer productivity aspect of this as well and I think you know it's just something to keep in mind I guess you made a really good point about it made me think about how there's literally no technology that solve a problem that you can apply to every use case and so in the framework of a stack I don't really see that as a way to see things because in every project every company you might want a few base things like Kafka for example or Cassandra or something mezzos but um there's there's all these edge cases where you need to apply other technologies with them and yeah so thank you so so here you know patterns right here that a lot and so I I'm curious can you think about so you know for you those of you who have companies serve external customers you see this patterns in the real world so what do you see you know can you generalize in terms of architecture what do you see kind of what is it you know traditional architecture as customers what do they actually have right like we hypothesize they want them to be all advanced like literal but a lot not so what do you see in the real world and where for those of you who provide the letter where do you provide the most service to those customers where you plug in in them right and for folks on the edges who kind of right not not have companies but you know in a huge company or work and a start-up right how do you kind of I mean there are probably some Potter's there as well right so so what kind of partners do you see emerging and where do you want this patterns to go if you know somewhere and I'll start so I I mean one of the big truths for a lot of organizations that are outside of the Twitter tech companies is that there's a lot of work to get those organizations to actually change their their the way that they do things to work with a lot of these modern technologies a lot of the technologies were built assuming particular expectations about the environments in which they can run like places like Twitter and and it's it's a it's a big jump so I mean I think the obvious big patterns that we see a ton is people moving to microservices which we didn't call it micro services at first Ravi had so on all his slides when he first thought about it it was just without the message bus for the most part and then eventually the term micro services came on but you know a lot of organizations trying to adopt that pattern they really struggle and and you know some organizations it actually doesn't make sense like perhaps maras this organization sounds like it didn't make sense and and and that's that's totally fine you know the other big pattern that we do see though is people basically want to I mean that we see the smack pattern a lot it sound like a broken record up here but we see basically people want to get some data in do some processing on it store it for long term serve out some information from it to other people and and they want to be able to do that you know really elastically when there's more data they want to be able to run more computation and when there's less data they want to run less that's a pretty typical pattern but again a lot of the letters letters tend to be swapped in and out yeah I mean I those are those are those probably makes biggest patterns that I see it most the organization to talk to but I know one thing that I think is interesting is just that a lot of organizations that are outside of traditional tech companies how many people have heard of the term digital trance information raised okay so who hasn't heard a digital transformation raise your hand okay that's that's I appreciate the honesty right so so the digital transformation was a term from like a decade ago and still turn today because big companies are actually trying to digitally transform themselves because it's really hard because the companies still run mainframes who runs a mainframe anyone run her mainframe here all right we got one you know the a lot of the organizations which I would they hey that's what they have they have these mainframes they have a lot of really old tech and so you know the pattern to move to micro services is tremendously difficult they need a lot of help with the tech and what Marius was talking about earlier I think I think is a really really interesting point if somebody asked me the other day what I thought was the scariest thing for you know these big enterprises looking to do something like smack and and my answer was that there's a new acronym there's a new letter for the acronym like every week so it's just like you can't is no way you can you can catch up to that like you got a you got a so you know again it gets back to what I said earlier which is the best thing is to find some technologies that compose really well together solve your problem with those technologies and and and and then move forward you're probably gonna replace some of those technologies in a couple years that's fine but you should you should pick some good technologies now that can actually actually solve solve the problems you still not really get a conversation over there between us I was just gonna say you know we we talked a lot of companies at buoyant because of the nature of where liquor D sits it's like an you know kind of deep in the in the horrible like bowels of like their systems and the one kind of constant that we found is that any company that's been successful you know it just has a mess inside it's just like it's never it's never clean it's never if you're like you know the only clean pure companies are the ones that like die you know if like life life is guys like that's really want to come fountain now the reality is like if you if you survive like you do it by I'm scrambling from thing to thing and like you you know you end up with these horrible systems and everyone feels really bad like when we talk to them everyone's like yeah I'm really sorry like you know we kind of got halfway through that thing and then you know we had to shift over here and this other thing so everyone's really apologetic about the own zone mess but everyone has this mess and you know I think I don't know if there's a lesson to be learned from that other than like if you are like I think you just have to expect that you're gonna change like them so you're gonna change your infrastructure layer you're gonna change a lot of the dependencies and assumptions over time and like you it's going to be very very rare for you to have something you know it actually lasts for a really long time unless it's super super hard to remove that and even even if something is old usually kind of surround it in layers of other stuff so yeah it's I think that's what we found it's like it's horrible out there you can you can speak for yourself I think layer which is beautiful one of the interesting things we've noticed is that there's a lot of a lot of customers especially and the bigger the enterprise and the more risk there is to change who will just skip multiple generations of what we've been considered technology progress because they're waiting for a new wave which is compelling enough to actually be worth the investment and like this happened to an extreme and very obvious degree with the first generation of nurse Eagle for example where a lot of companies got badly burned by inadequate implementations for new data systems in particular fled back to their legacy Oracle cleanse and are staying there waiting for something which can actually improve on every dimension of their experience of using this systems rather than be a strict trade-off between usually upfront productivity on one side and then operability data integrity correctness on the other and I think an another example of this is cloud operations generally which is what's driving first the containerization movement and then also the surrealist movement where people were like oh the cloud super-awesome wait you know why am I still using cfengine to like mess with my ec2 this solve nothing it's just a different way to buy hardware but it's equally difficult to develop a software and those those adopters and what okay we'll move some workloads to cloud and maybe we'll do some new stuff there but we're not going to try to constantly forward port everything to the next trend and I think I don't know if this is part of like digital transformation generally but I think I think in particular the enterprise has gotten a lot wiser in situational about when to adapt new technology which ultimately is good for vendors because you have to actually deliver high quality software you're not just chasing trends and trying to sell fads like the early days of so when XML was supposed to solve all your problems and if it didn't you used XSLT and that solved all your xml problems you know people made a lot of money on the conference circuit for that but they weren't helping customers deliver business value and we sort of saw that again with the first generation and a sequel I'll let The Container people we don't use containers we use Java right once when anywhere run anywhere the dream of the 90s is alive at fauna but the container people can talk about how you know the early days of like docker in particular some of these other configuration platforms maybe if you consider puppet and chef forerunners of that didn't really pan out in terms of business value and I think the thing that is most gratifying is that customers know it and are making much wiser engineering decisions not just will use the trendy stack I would say as another way almost another way of phrasing what Ivan just said is that you know I think ultimately a stack becomes successful when it becomes basically invisible and when it becomes boring right like it just does its job and it does get in your way and it allows you to you know it's just another tool for you to solve your problem like nobody talks about Linux system calls unless you're a Linux kernel developer right it did just work they're there and they're very useful right and I feel like ultimately the way to measure the success of any of these technologies is to observe them become boring over time and then I would say that's the the ultimate hallmark of success yeah another example I think is Hadoop like Hadoop has never become invisible for any in particular enterprise adapter and there's a big there's a big backlash because of that and people are saying you know where's my vertically scaled columnar database like let's go back to the 80s and at least you know build a product yes I think it is interesting to see to see the trends like containers have have taken off you know pretty dramatically over the last few years but the benefits containers give you aren't just that you run docker instead of having an ami or the M or whatever I think is that you think in terms of it's not a computer it's less than that and and that seems to be something that has accelerator pretty dramatically like serverless is the evolution of that and I don't even need a whole process I just need a function it's gonna be really interesting to see what that lands and to Evans enterprise point if I weren't enterprise I'd be kind of nervous about adopting docker or kubernetes right now because the server list thing seems like it's right on the tails of that and may end up being the thing that wins like like who knows in San Francisco like kubernetes is definitely a thing that maze was just a thing there's a lot of people using that it's really fascinating to get outside and talk to people like in Boston where the challenges are still like we're trying to get Jenkins set up so we have CI CD which is digital transformation yes yes they're they're on like 4-bit digital transformation right now but it's really interesting to see you like once you get outside Silicon Valley how these technologies are being adopted and it's much much slower and so I think there is that chance that the pace here Excel to a point where it's like impossible to pick what you're gonna go to next but there's always that imperative that it's never gonna stay still so one of our premises was things are always going to change and there's there's a decision point of like what's the risk for me to make this change what's the reward like what's the payoff how much does it cost me or the expected cost and what's the expected payoff and so we're really focused on reducing the risk and that change and I think that's one of the one of the really interesting things about micro-services is instead of having function calls it's kind of your integration point between teams which people tried to make programmable anybody remember aspectj anybody yeah that never really panned out but that was an attempt to make like the function call interface programmable so you could inject functionality there it didn't really work but with micro services now the network has become that and so a lot of the new proxy technologies now give you the benefit of being able to plan much more sophisticated policies at the network layer which means you can do fancy things like gradual traffic shifting and evolution of services in a way that's much much safer than kind of the state-of-the-art five six years ago so if the risk is less then you're able to move faster to make those changes and kind of keep up with with all the nonsense happening now and like functionless or whatever is next after service I'll go next um I don't know if anyone's actually used the word scale yet on the panel but as far as stacks go one thing I've seen that's really interesting is you know so I've worked at both technology and framework provider companies and users consumers of those technologies and when your provider and I think definitely not the case with like LinkedIn creating Kafka or with Twitter because you have a lot of data but oftentimes the companies that are producing technologies doing a great job but they but they don't have the data themselves to really thrash it and so you know when you actually like at some companies that I will not name you might be able to say okay let's try out this new technology might be really you know works really well you you know a lot of companies use it but then you can sit there with an espresso and watch it just completely melt down under the load that you have even even with it you know properly configured properly deployed everything like that so I think it you know it becomes really interesting when you serve the scale problem at it and that's where you have to get creative so focus I'm gonna open the floor to audience questions next right so our indefatigable Technology dev and will will hand the mic so maybe you know I'll ask one of you guys to surrender one of the mics we'll have to increase our teamwork by a factor of one third so I'll ask one more question and meanwhile you guys can prepare so we'll you know raise your hand and Devon will bring them back to you so one question I want to ask so we're open source community and you know almost everything is happening in the open source so everybody here relies on the community and so I'm really curious right how do you guys see kind of so when when open source companies bring the product to the world they need vibrant communities so so everybody is doing a lot of work in in kind of cultivating the communities given the open source right and and obviously there is a lot of tension here providing you know an open-source version and kind of fun born in people on commercial version so how do you kind of you know taking stuff from Twitter out or you know I mean I don't know like Apple users open source not much kind of you know it puts some version of GCC but it's also has some open source so how do you how do you see the community right like is it still the mode of operation that you have an OSS component and have a commercial component is this model gonna change right we see like resurgence you know of the reaction in your reaction there's the same like let's just roll back open source let's just sell commercial software I heard an executive at the conference saying like you know pretty become when I'm not gonna make saying like I'm gonna be like I'm gonna you know educate open source like the good for security as a security company we're gonna not use open source because a security company so we don't have a place for open source in our stack right so so how do you see this kind of duality of open source and openness which you know and see a Twitterer to the extreme you know finagle is a github organization of its own writers I think it said one of the leading examples versus you know the tension of building commercial companies and selling software so I always said once very surprising and interesting I guess reason and also benefit in side-effects from doing open source is that it forces you to adhere to a higher-quality right and so one of the things are at least early on at Twitter one of our sort of core reasons for open sourcing stuff was that we wanted to make sure you weren't actually embarrassed by this right is to some extent and that's everything from the code itself to the waste document to you know the fact that you have to be able to write down a coherent sentence about what it actually is right and so I feel like that's sort of an underappreciated aspect of open-source is that by putting yourself out in the open you you're you're kind of applying a higher standard to yourself there's one you know it's it's one thing to write a piece of internal software where anybody can come to your desk with any problem they have and and you can make all sort of excuses it's another thing to actually put something out and it kind of been you know it requires I think a higher standard of work and I think that was the device that we used in several Twitter projects fairly successfully I think I want to totally support that statement I think that everyone should be contributing to open source I know they're it's hard to find the time but you become a better engineer if you look at the code of your teammates or whatever that aren't contributing to open source you can really tell because people that have contributed to open source you test things better you document everything it's written well like the grammar is good the way that you code is easily readable and you you think in terms of how you're going to construct the code so that it's easily maintainable by other people around the world and also it's used in various use cases not just in your company and you have people testing it out in different deployment strategies and everything I mean it's so incredibly valuable and we definitely do at Apple contribute to open source amount of project where myself and Evan Chan who presented today were primary contributors to file ADB so we're interesting and we don't offer any open source as part of the company like we leverage open source heavily and we commit to the projects that we use we actually do open source one our CLI because that's the easiest way to get go installed but that's just really a quirk of go but you know I think what we've done like to Alana's point is that a lot of our stuff is ready to open source we've just made the commercial decision that it doesn't make sense and having that tooling in place and the rigor in place to say like if it makes sense for us tomorrow to open-source this project we make the github repo public does enforce like a lot of discipline that we would be like tempted to to abandon for the sake of expediency so I think like 90% plus of our code is like ready to open-source tomorrow if we decide it makes sense but our kind of criteria for that it's like what would be the point of open sourcing those would it be useful for other people standalone if not then don't why bother if it's going to be something that's you know going to get us reach as a business and then that's another another question if it's purely just to say like we open-source this thing then it becomes like one an API that were kind of de-facto locked into by having that project open source if you if one person uses it then we have to have like an implicit or explicit contract there and - it's it's it's a headache for both an adopter and us to say like is this thing it to be maintained going forward so we've we've chosen to keep large portions internal but go through the same discipline of it being open source ready just purely for hygienic s-- that's gonna I was gonna say so for us we have both an open source we have many open source projects and we have Enterprise projects as well it's definitely a there's definitely tension as in doing that I think this strategy for us is we learned so much from people that just take the open source project and and use it that's how you know it worked with meso here at Twitter bunch of organizations took it and gave us amazing feedback as they put it to scale as you were talking about earlier it's kind of funny you when you I mean we had companies like Netflix and we had companies like like PayPal we had companies like other companies who would you know actually push it to some serious scale and and that was trying to see helpful when you end up stirring a company like to to get the same feedback yourself the kinds of like ways of trying to run your product that way would just be prohibitive expensive to try to exercise it in that same way it that's really really tough yeah on the flip side and this is I think the part that that is the trickiest for a lot of enterprise businesses is there was one person in the room that raised their hand when I asked about mainframes which to me means that 99.99% of everyone this room probably doesn't pay for software which means when we put open-source software out that out there you're going to consume it and use it and that's great and then what happens to the enterprise business that's potentially trying to support it so like this semi is the conundrum that occurs it's a very deep conundrum for you know an engineer who just wants to build really fun things but I also recognized that when a lot of the businesses that are driving a lot of these open-source projects if they were to die the open-source projects would in many circumstances die as well so it's a it's a double-edged sword you know it's like you definitely want to work with the open source community get as much feedback as you can but you also need to figure out the best way to make the business successful and honestly I think the jury is still out for what is the best way to monetize open source open source software you know there's a handful of companies in the last decade that have tried to do it and there's not a lot of companies to point to and be like they've gone huge with a completely open source model one one more note on open source is you know we talked a bit about how working on open source makes you a better developer but I think there's also a widespread expectation that working on open source people do like after-hours on their own time which is really really a luxury for people who have after-hours time to spend doing more work for free so I'd encourage people in the room who are in a position to let people work on open source to do so like we do we encourage people to do that and if you're not in a position to do that to start demanding that your employer's let you do that on the company dime because it's kind of to be a good engineer you have to get paid to do your engineering work for 60 hours a week or whatever people demand these days plus do another 20 hours a week outside that to actually be good is is like a huge amount of for people who have you know kids or or other kind of constraints outside work it's a it's a totally unfair expectation so fun a fun on TV is not open source I think the business justifications for open source are threefold it's a way to force like others talked about your internal team to basically ship to internal customers and not just toss lumps of code over the wall in our market at least her operational databases the quality bar is already I think higher than the open source provides as a forcing function at the same time open source is a channel which is the traditional business strategy either you know a Twitter we did a lot of open source not just for quality reasons but for recruiting purposes too it was a channel to hire into the engineering team for a typical open source open core business the open source product is a channel to sell the enterprise product on the customer site excuse me on the customer side it's a way to increase customer choice increase the number of ways that the product itself is deliver and the business continuity protection that the customer receives buttoned opting open source at least for us like no one wants to own their own private operational database and maintain that thing forever so we found that having a cloud product in particular a serverless cloud which is easier to get started with an open source you know to provision hardware you don't have to operate anything at all it's at least an effective channel for onboarding people and took on a DB is I think the thing that's really undermined or cannibalized open source business models is cloud operated software whether that's open source or not no one really wants to operate their own software so they'd rather pay someone to run it and operate it then operated themselves and pay a vendor to support it oh I built my entire business purely on open source what should I do everything everything we do is open source and it's a huge investment from us because not only are we making amazing code because open sources you know makes us do that also we're amazing people but like we have to invest in the community and we got to keep people excited and engaged and like grow the community and that's a huge amount of effort on our part to do that so it's a really expensive proposition to be open source on the other hand like it gets us into all sorts of cool places that we wouldn't be without it so yeah it's just a minor question of what's the what's the cool business model on top of that which I have a great answer for come see me afterwards oh that let's open the Florida thing it's an Oscar was raising his hand and by the way Oscar is another legend of Tudor he just you know didn't like legendary enough let's go we don't have chairs so I I kind of you know I like I like open source you guys were talking about it with regard to business choices but I and I don't want to put you on the spot too much about this but I think open source is important as a kind of Commons for things that maybe aren't by themselves gonna make great products but they're still very important for engineering and I think we still have a challenge to like figure out how to fund and build these things and I think mark kind of brought up some interesting things but just to like bring it back to like the Twitter angle I would be interested to hear your take on like do you think like Twitter did this well or not because in my view of people outside of Twitter have a somewhat like a monolithic view of like what Twitter might have been like or what like like how decisions got made and I'm sure this changes over time but people inside Twitter like see it as like I mean my my take just lay down is that open source is mostly driven by engineers who are passionate about it and was somewhat reluctantly usually supported by management and sometimes even met with some hostility so like I I would be sure like like did they get it right is that maybe the right take like just like or like do you know I don't know like what's your take what what should it have been I thought you're gonna say monolithic us in repository so I I mean I felt the same way I felt those definitely and so I'll talk about the two projects so so mesas came in open source which was was good there was a open source advocate at the time Chris Anna and a check who I think did a really good job actually making getting mesas to be an Apache project before was really pulled into to Twitter so I think there was definitely advocates that were helping but I completely agree I think a lot of it was pushed by engineers so take Aurora which was that project took forever to open source from from inside Twitter so long that people outside Twitter started to build many people outside Twitter built basically replacements for it Netflix built their own replacements uber built their own replacements other companies built their own basements and eventually even even even us you know when we started a company leaning upon a replacement which is I think horrible for the open sores can make certainly being so so so many but I think there other things that we did well I mean the IPA stuff that Chris pushed through which was the individual patent agreement stuff I think that was a very very Pro open source and proprietary I I didn't see many companies actually doing that I mean I think that made engineers feel like they could work on really really cool stuff and feel like they could actually own it so I think we did send some stuff really well but yeah I think there was a lot of it was still a lot of uphill battle so I think Evan talked about the three reasons you would open source from like a pure commercial capitalists perspective but I think it's and this may be me mythologizing the generation of Twitter engineers before I got there but Twitter was kind of rare and that they were like serious anarchists that started the company with like deeply held ethical beliefs on software in a lot of cases and so I think a lot of that kind of bled into the later generations of Engineers like like me it was just like we should definitely open sources and make it free just because that's the right thing to do and so I think a lot of the open source kind of followed out of that there may also like I didn't work at Google before I came to Twitter but I know Google was notoriously hard to get stuff open-source to out of and so I don't know if there was backlash from the people who'd left Google to come in as like holy now I can actually open source a thing without having to open source all of the Google 3 code system to do it so I think aside from the like business justification of Twitter open sourcing things there was definitely like you know a core ethical angle of open sourcing software and then just a freedom that this is a small enough company that we can actually get this done and it's gonna be great Facebook for whatever reason like about the same time like coming up had kind of a different experience with open sores and there's some big projects that came out of there but it didn't seem like they were nearly I guess is voluminous and their contribution to open source I'm kind of curious just like a sociological perspective light was but I mean one of the one of the earliest reasons we contribute into open source was we just needed help like we we were hey we were a very small team especially in 2008 struggling with scalability issues the extensible topic of this panel and I think we got involved in the Kassandra community very early it's the first adopter outside of Facebook hosted the first meetup fixed the build wrote the first tutorial even before it was called rip tonneau at the time now data stacks was founded just because we wanted a system like that to exist and we didn't believe that we had we need the engineering resources to build it ourselves internally so in the in that sense like there was a lot of idealism about you know that's all like we're all in this together we collaborated with Facebook on that and other projects even though from a business side we were sometimes friends sometimes competitors but just trying to build something that we thought would be great and get whoever could help us to help us do so I would also say that like open-source is really one of the only means to actually influence technology generally speaking right and I think pretty much every engineer has this kind of instinct to to try to push their own ideas and everybody else cuz you're doing it you know you're building it the right way right and so I feel like there's a kind of strong foundation just in the basic personality of people who tend to work in places like that to actually pursue open-source so for example a lot of project that I worked on at Twitter it was not going to be open source but we always had it in mind and you know we're careful to make sure that the right interfaces were in the right places so that it could be cleaved off and things like that and it actually enhanced the quality of the code as well right but I feel like yes it's ultimately driven by by engineers but actually good engineering it's in itself as an enabler of open source as well yeah I just want I just want to fall I feel like a lot of a lot of us as engineers weird we're like scientists and like we do like a lot of the work we do are like science experiments and we want to like share those with the world and to me open sources like this ongoing science experiment of engineering it's like how you influence and how you actually actually can contribute new things back and some of us have a lot of science experiments and are constantly going to the next project in the next project but you know some people stick for the same science experiment for a really long time just wanted to say I think marty at this point was a really good one which is that it feels you know it open source is a really effective way of changing the world ok maybe like the world of Engineers and it feels it feels really good to do that like you know one of the one of the things that I feel very good about at buoyant even if even if point instead of our current rocket ship ascension and like sings under the water I'm still gonna feel very good about the contributions that we've made because we've actually you know we we've had an opportunity to really affect people's lives like not in like a super sappy way and like making their job easier or whatever but that's still like that feels really I think that feels really empowering and it's one of the things that can really motivate you getting out you know out of bed in the morning like and going to do you're going to do your crappy job if you're like okay this is actually going to make a difference to someone out there and for us like the the link Rd being open source I think was a huge Avenue to that you know to that actually being possible so raise your hand if you've committed to an open-source project that's pretty good thanks guys so I want to shift gears for for a moment and we have a guest of honor here with us from you know last year like I really was wondering if founders of Twitter are aware of or the structure of Technology which exists here and drives you know a lot of projects which went into the world and so when I've seen that bit stone is coming back to Twitter I've seen a medium post about this and I think my interpretation of the mandate was to bring the world to Twitter and Twitter to the world and so that's actually cultural is what we were trying to do is color by the bay right because essentially to reproduce all this tremendous software but it's not actively popularizing it outside because it's not the business model so I thought okay like if tutors are coming to the world we'll just bring the world into Twitter and inject it right here right and so you know if there's some new stuff to learn about finagle and and you know other technologists will basically you know bring people in and mix them and match with Twitter community and I think we will have more and more Twitter folks attending this conference or as we have it get more business days so most people can can attend but I'm really curious right about historical evolution of kind of technology a Twitter and you know if the company's whole knows what kind of value is here what's it plans to do going forward so I'd like to welcome Bastogne to address scale by the way welcome is thank you I think I'm supposed to make a few remarks and then we're all gonna drink some beer it's good to see you familiar face actually just to just to give you a little fun fact jack and i and early 2008 and i think you might have been there Evan we very seriously considered open sourcing Twitter itself but EV didn't wonder that anyway is this being recorded anyway I just wanted to say I'm business town and I work here with the wonderful remedy who is our open source there is open source program manager and I don't usually share this because it's sort of sickeningly hallucinogenic lis optimistic but you guys were like that too so I'll uh I'll share this I have a vision for Humanity in 1,000 years and it kind of in a nutshell it's comes down to global collaboration so just imagine the achievements we could make if we stopped dropping bombs and started dropping knowledge and that if if if sharing our breakthroughs was just the norm I think we would we would we would sort of we would we would do in one year what it normally takes a hundred years to do and and in the process of this grand collaboration I think we were also just simultaneously almost even as a side effect solve the world's biggest problems and that's what open source is like a peek into that like it gives us a peek into a non dystopian future you know we're all actually working together and we're building off each other's work and to me open source has always been about you know helping other people make progress and and that's that's kind of what Twitter is about - that's what we've always believed Twitter was about and that's what we think and that's why we think open-source is so incredibly incredibly important you know it's about giving back it's about helping people make progress and that's what we tried to do or we try to do with Twitter and we said that from the beginning when we formed the company I gave a presentation I said if we're gonna have a company this is how we're gonna do it we're gonna try to give back the community to the world and we're gonna try to be successful as a company - and it's sort of this the open-source ethos just matches really well with Twitter culture ethos so anyway I just I also wanted to say welcome to Twitter and I'm very proud that we've been able to host the scale by the bay conference for two years running now I hope maybe it can become a regular thing that would be really great and with a lot more participation from from Twitter so I that's really all I have to say I think I just wanted to thank everybody for coming and for being here and for letting me say some stuff and for REME for inviting me and yeah we can if you want to do questions you can I don't know that much text of what we're doing right now but I'm fully supportive and there's also beer out there and cheese beer and cheese so thank you everybody I really appreciate it go scale by the bay love it [Music] think of this so I think