SBTB FinagleCon 2015: Phil Calçado, Finagle at SoundCloud

Recording: SBTB FinagleCon 2015: Phil Calçado, Finagle at SoundCloud

name is Phil I work at SoundCloud in our billing office has offices in Berlin the solar headquarters we also have a fairly big office in New York these days and a very small office here in San Francisco so what I'm going to talk to you today is really about how yes it's about how we use vinegar bath soundcloud butts more about the motivations for us to to use it away we do and even to come up with the idea of using finna go to begin with so first because you know it's it's I I kind of know some of you we talked about but I don't know if if you all know what soundcloud is some cloud is the largest repository of audio online and I know what you thinking I was like what I mean largest you know this there's all these other streaming services in various different things don't they all have the same catalog maybe not Taylor Swift with pay you know about the same stuff and yeah this is true but some calls a bit different because soundcloud doesn't have just the major catalog we have all sort of all sort of different content and because we let users upload content as well so have user generated content so a sound card this is just a random assorted pic of my own likes and people i follow you're going to find Neil deGrasse Tyson Foo Fighters some Brazilian singers that we liked NASA and you know our offspring also are different things in some called we have literally from Skrillex to the sound of bats having sex and to me the sound absolutely same also some people some people accuse us of inventing or at least broadcasting dubstep I hope that's not true but yeah so we have to have all this interesting kind of different content and what that means that we have about 11 hours of audio uploaded every minute and we get visited by about 300 million three hundred million people every month I'm really nervous with a microphone like this because I'm Brazilian and I use my hands are up so I feel like it sorry from shaky a bit don't worry anyway so it when interesting thing and one interesting constrain about this is that to us every time a piece of art is uploaded it's not just making it online we actually need to transcode validate clear for copyright infringement and all kind of stuff it's a lot of work that needs to be done in under one minute so it's quite complicated and in you know I don't I don't want to tell the same story everybody tells but just like everybody else some crowd started as a radius monolith very much like Twitter I don't know how many times we actually had to start being told here today anyway long story short we kind of went like this from a sacrificial architecture you know code written by the founders in a dark room in building I'm not sure they were in a dark room in the link but they were in building two microservices this lots that happen in this migration there's lots of talks and articles published about this if you're interested just come talk to me later anyway so we have services now and to be honest to meet the slave microservices it's really complicated because it doesn't mean anything you know this books and articles but i am not too sure what that means and so the way i like to think about is that we have services and some of these services are micro some of them is small some of them are bigger some of them are really big ass like massive things but the most important things that we have services so one thing that it's quite common even though I don't necessarily understand microservices me means the one thing that's quite common every time people keep talking about microservices every time I hear somebody talking about microservices is that often these companies these organizations have lots and lots and lots of services in our case we have I actually just checked last night we have about 100 services to 130 engineers more or less so this 130 engineers this number is really interesting and I'll keep coming back to it because this is by design and we we have we had our hyper growth phase where we're hiring 20 people a month but that's not what we want to be let's not want to do we want to keep the team as small as much as you can we want to keep our culture we want to you know do all the things with just hire people to make sure we can deliver things obviously this also comes with their own constraints and that's we're going to talk a bit more about this so 100 services 130 engineers and there's one thing that I've learned in previous lives in a service-oriented architecture but also in these micro services whatever means life which is each service you bring up has a fixed cost so every time you come up with something new every time you create a new service you have the course of setting an appt monitoring making a production-ready whatever you you name it so if if if you have 300 microservices going sorry 100 micro services like this each one of those is urine Korean disgust for each one of those and this this course is a bit less boys it's harder to perceive will have less services or maybe just one big monolith like we used to have because then you start treating these things like pets right so you have all this craftsmanship going around you create the best build system as you create the best pipelines I actually think our our mana left which to exists we call it the mothership it's it probably has the most stable deployment infrastructure and the fastest and best best really testing pipeline why because we've been developed these things for seven years and people people pay a lot of attention in trying not to throw up these things because they know it's really hard to fix it after the fact so going back to the microcell side i am but like i like to talk about three different costs of facets of the cost of a new micro service and finagle comes in in the last one but there's also influence our decision to go with finagle the first one is just cause of black city I don't I don't have a better name for it but the main parts that somebody mentioned earlier today that when you have it's like when you have microservices you just get out of the complexity that it could have inside maan good and kind of split all over here your company maybe your data center is whatever is and you need to you need to be really diligent and pay lots of attention on what you're doing because otherwise you end up with something like this one of the reasons right up with something like this especially now occasion that has happened in some parts so many times is that companies organizer on micro services tend to be a bit more hand to end with the teams so say instead of having a web team and a back-end team you may have a sign-up thing or growth team or in our case a playback team recommendations thing and what happens odds that teammate depends on something by Team B but they don't actually talk so much in be you know maybe they are not living the same duty timbi then depends on something team C and they don't realize that team see I could depend something onto my back again so this graph obviously is an illustration but there's some circular dependencies here which is the kind of stuff that actually happens a lot in Microsoft's architecture there's various interesting technologies in the try to make sure you do end up in loops like this but it's really hard to avoid when you have in our case I don't know man teams have probably have more than 10 things working on on their own backlogs one of the ways we we found to deal with the problem at soundcloud was trying to lay a Down architecture a little bit so what we've created that actually I just saw the diagram by marys is money in there was a very interesting variation of this kind of stuff and was that going down instead of going this way and the names are different but in the end is very similar what we have is at some club right now is three layers the first one called egg services which the name i think we stole from netflix then we have a core value added and foundation and what what's this like the all requests coming through here and they go to something here and probably here as well the interesting thing about Microsoft's architecture is that is very unlikely that you're going to be able to do anything hitting just one of this box maybe not even hitting just two of them right you really need to go deep at least a little bit in a tree and the way we think about this is services here they are required they're absolutely required for every single request they usually validate the request they do rate-limiting I know the authentication and authorization they must be up the decks la for these guys is insane and it must be real real fast services in the bottom here I think I think was called persistent or something in my slide those are like services that usually have the interface that takes an ID and returns data takes a list of ID's in return a list of data it's really like wrappers around database and these things are very stable they don't change as much but they they must be they also must have massive availability because there's no fallback so if these things down you don't have the data you need there's no there's no there's no you know work around you can make you don't have the data you can have it later these guys in the bureau they are the vast majority of to all the services those are systems that receive your first request from these guys aggregate data probably from some of the said their peers and something from from bottom layers and return a response oftentimes these guys can actually have some form of fall back for example soundcloud the classic one is what happens if i'm trying to display the play button for track and the play button has a little box that says how many likes the track has had what if the like service is down wow it sucks for the likes service but this guy he can just return 0 and that's okay or maybe depending on what kind of a kind of representation returning you just omit that data gray it out don't show that box there's various different things that we can do work and I saw a system like this it is able to manage the situation a bit more without having to introduce a lot of bureaucracy and just as an example he's like completely made up but we could have like various different api's they call a user profile service which then goes to other fundamental service to to render enough data to return the full user profile experience this is pretty common these days in micro service architecture Spotify has very similar textured netflix addressing architecture but it took as a very long time to get here and and this doesn't solve all the complexity hasn't solve all the complexity problem for us it made everything better because now we can reason about these things this architecture just the same as you would like to layer your software but it didn't solve everything another thing we've done recently was introduced what we call services directory services reactor is actually based in a in a concept called humane registry if you're familiar with pass away from I don't know the 90s sorry the nineties you will remember things like uddi and crazy registry with a lot of XML the idea behind our main registry is a bit different it focusing information that's accessible to humans I had to I actually couldn't sanitize it so I had to put a lot of red bars everywhere to hide some host names all my security team will get really upsetting me but this is when you go through this system you can there's a list of services and if you go to a specific services like hundred of those right you can see stuff like repulse rename the URL the description of service who owns it with an email address you can send basically incidents to these things and our odds kind of stuff as you can see which totally me I was created by developer a back-end developer because it sucks we are really really trying to get some front-end people to help us here at least thanks Twitter for bootstrap everything looks better bootstrap right but yeah it's really a trait on it I clear would like to make this open-source at some point it's a bit too tight or infrastructure right now but at least idea itself in a lot and that's one third step that has helped us a lot which is actually creating some policy around what technology is what tax tax while languages we support at first Sun cloud didn't really have any kind of guidelines around which language one could use or should use I know felt like using we had everything you know in our stack and most recently after realising that again with 130 people if somebody decides to be something Haskell this person is the only person who knows has gone the whole company somebody's mentioned that leave what holidays policy in France in France is seven weeks in Germany's 22 days and but this this one interesting thing in the age group we have our engineers at which is parental leave so parental leave in Germany can go to nine months paid so you know every now and then somebody disappears for nine months and that was the only guy who knew Haskell so what do you do after various different instances of these problems we kind of decided to nail down what we actually expected from people and what we would support as a company so the theory is that Skylar and Skylar was chosen because of finagle so it is kind of putting the cart front of the oxen or whatever the expressions like I'm translating my head from Portuguese but we chose skull because of an ego but the idea is that everything built infinite scholar should be a no-brainer there will be support we have lots of tooling have lots of documentation everything for scour things all the JVM should be easy this these things are really wish you were shooting no brainer easy what does even mean but I think they they they illustrate what we're trying to say so they will be easy and one of the one of the ways to make them easier than what it could be is even though we write all of our libraries and our power tools for the jvm languages in color we write all the tests in Java so this is a a hack really or a technique we've developed recently and it helps us identify when there's something real bad we introduced in the system you know if if you have an API that depends on implicit what something that's completely impossible to use from Java we know straight away because our of our unit tests are written in Java so I have to go through the pain these other people would have to go through and building something else should be possible this is the even the most wishy-washy of them all what's possible pretty much what that means that everything we build is based on standards the other can be implemented in any programming language we think of like HTTP are chasing or there will be support through sidecars sidecars an interesting concept Netflix also pushed a lot for which having a small application deployed with your application so for example if you have something Julia and Julia doesn't have support to distributed racing so instead of trying to bake that in in your Julia codebase you can actually deploy a sidecar an application using finagle that then proxies requests to and from the jeweler base we seldom do this but you know it's possible so this is this is three these are three different ways where we tackle the complexity cost and then another one when it comes to micro services and again think about I think Paul microservices the only conclusion I can get to is that there will be a lot of services and maybe in one month we have tenure services and then next month five of them would die be merged be splitting to anymore so there's a lot of setup every time somebody starts a new service they have to set up something and then this is interesting thing there is the u.s. BTW file so my teary about svt is that press we all know that S&S petition for the same as as in soap right this should tell up but I think only there's only like one Beautyrest beautify I've ever written and we've been copying pasting and changing and I mean it's yes it's a joke but at some cloud this is pretty much it true we created one of these guys that looked exactly like this and remember the day that it was written and since then everybody copies and pastes the same stuff sometimes this is check here for java version and the variable says in Java 8 inch x which are eight sometimes you see if you say is java 7 and here's two checks which I've eight it's it's insane and to be honest after working for y ou a sweetie I cannot blame those people it's it's not easy and just remembering to put spaces here's like drives you nuts anyway um but come again great I didn't even know that I'm so afraid of it that I just press a hell and advice so one one of the things we've done about this is actually instead of fighting with you I mean to be honest I know some people use all the things to force color but I I've tried and I don't think I still think a sweetie is the best we can get you unfortunately but as things you can do right for example our case because it creates so many of this project it was worth it to invest in tributing and plugging so we actually here on the top I'm really sorry can't read but the slides are going to be up we have the same I don't know if it's the same project exactly but this projects are very similar using our SBT plugging we call it we call it our Java library or JVM lavage FM kit so going to see jeff m kit everywhere and that that reduce these big piece of stuff to this but that's not actually true there's something here in the bottom that actually became very important to us recently we call it a manifest files a manifesto JSON file the lives in the root of every single repository and what this file has is basic stuff what the name of the application the runtime so jdk 8 this is well there is a docker image in our and our registry in our internal dr. registry with this tag with his name some description who owns it the state is if its production development Lola the type if it's a library or an internal service in this case data sensitivity we operate on the European privacy law which is really really interesting and appendices so all the systems you depend on as you can imagine the services director I showed just before consumes from this guy so services director itself it has a database were just cashing because our information actually shows you comes from these and other systems so this small ways BTW file and this becomes a bill dots color which i really prefer because then i can use my ID properly and this manifest file form everything we need to actually create a new application this is also the also saying the manifest file is really important to us because not only is used for this where we found so many different use cases for it right now these file plus some tooling lets us export matrix two parameters we're going to talk about bit more about primitives later let's set up rotation schedules and whatever they call skeleton policies on patient early Jenkins pipelines how to create Debian packages docker images I think if you don't know what I think is if you hate nagios if you hate Nagas raise your hand hence check icinga if you haven't yet it's not the solution to your problem but we make a life a little bit better set up in femoral environments and amazon and now it's kind of stuff coming from this file so this rio plus a bit of Thule has saved our life so many times so okay these are true interesting well iguess one really won't want an interesting solution to the set up problem after we adopted this getting a product getting a project up and running being deployed to production through a beautiful I so a Jenkins pipeline it went from like a day to its two is one our and a lot of it is because even with docker and how this stuff we still have the problem of you know my my biggest challenge right now is how to reduce the size of IV to that directory is my main problem my nemesis right now anyway it takes a while but still much better than what it was before but I now we come into we coming close to finagle so once you know what I was saying how you have all this different you have to create all these different services you have to set up them you have to make sure your architecture kind of made sense is understandable but giving you have multiple services you also need to make them talk to each other and we had lots and lots and lots of different presentations on this topic today but going back to the sunwell case back in the day we had a little bit of a problem when we had multiple different programming languages environments which was we just had everything other saying like this is not too far away from reality Michael had all these languages at some point reyes I don't think we're always going to get there ever going to get rid of phrase Twitty still has the ray of stingray yes no no so there's four ratings for boxes left that's pretty good I still have a couple hundred but we are getting there I you know it's being only only three years so we actually we started our JVM development with closure we went way for I'm kind of all this guy like converted to scour so I was pushing for closure we had Java we have we had to go we have lots of people who really really really like go a SoundCloud if we also had people who when the corner doing Haskell and it didn't really notice we had we had all sort of stuff and the worst thing about this I mean I I'm an engineer and I'm I'm like a big nerd I love technology and I love all these things but when you actually have to get stuff to production it's a bit complicated and one of the problems these things introduce is that it's always the minimum common denominator what's the what technology what protocol what stacker can use that all these things can talk they all can talk to each other and in this case it was pretty clear was a cheap people as Jason and so since forever will be using HP play Jason without these things we've developed our own internal standards with headers you use to broadcast user ID authentication information and whatnot but the reality that we cannot get rid of it to be honest when we first started doing these things I didn't really mind I spent a lot of time in a company called tour works that some of you may have heard of where people who wrote the restful in practice book used to work as well and I spent way too much time with those people so like yeah rest let's go all the way rest rest the solution to everything anyway so we're doing our services everything was good but even evenly even if you have satirized everything in java or closure escala there was one thing that was to non great to us what this is kind of the typical flow for so many of our services first you go through somewhere say the search server no hang on I get here is like get ideas of all tracks foreign artist so I go through the Skrillex oh why I want to check out tracks by Skrillex I go to the tracks system sending this relax ID receive back a list of ID's typical right and then for each of those ID I get the track metadata because one thing you start learning working at sound club is that each music has from seven to ten owners each piece of music and depending on the geographic region they change so we need to check out what's the permissions for this track in this territory today which is a really complicated logic so we go get this and because your social platform you might as well just get all comments for the track I mean people comment a lot on different tracks so we do this racetrack return a list to client that who does something like this in that hands up so this is n plus one problem right like we did it just computers that's what they do but the main everything especially coming from Reyes I Reyes being not very good at doing more than one thing at the same time we always done this very linearly and takes forever to process a list and what we've identified that there is obvious not problems because I'm a manager now so i have saved challenges obvious challenges opportunities for improvement here in the sense that once you have this list you can send each of if you have enough processors being threads or CPUs or whatever units of processing I can send each element of this list of ID's to be processed in in parallel once they are into this processor I actually have three different tasks here that I can use some concurrency to take advantage of and if you disagree of my usage of concurrent and parallel this track and you think that's wrong please tell me what's the right one because those concepts confuse the hell out of me all the time but what I mean is that this can be made much faster so okay we we understand that that can be made my trust but what should we use what's the what's the industry standard run that remember we are back then we're even less than 130 people but we have at least one fourth of this type of other competitors we do not want to be build infrastructure cells so we go to market and we check out what's what's in there the first thing that you know we we hit was the netflix stack and one of the reasons because you know media company which a company maybe i'm sure a lot of what we do they do very similarly to various different things and what we stumble upon unfortunately with something like this which is not really this because these are copied from the documentation page two or three days ago it was something very different but very similar at the same time which is shitload of callbacks this is i don't know if this is reactive because reactive is like microservices i don't know exactly what it means but every time I have a code I have some peace of God with so much callbacks and reactions I feel like it's when you're having a fight with your partner and they start throwing things at you and they throw the cat and like all the cat I'll get a cat over to the plate not a plate that dodge it's like it's too much stuff happen at the same time I don't know how to pay attention to this and obviously being Java without lambdas I don't even know this part lamb just what they do but this is like from this week so I don't know it was just like not sure about this don't you love ones public void did like the rough type phone calls anyway that works for lots of people but it didn't work for us and then we finally some people who had what we had who had been to in my team when different teams had tried finagle before like probably i love the first versions didn't really like it but we decided to you know take a look anyway we found this kind of stuff on the on the documentation page and i have to say that whoever wrote this you're in the wrong team you should be marketing because this looks clean and nice and understandable and there's lots of ellipses here and dad and like but you know i can relate to this it's like yeah cool i'm gonna use this and that's kind of what happened to it so we first started building our first api out of the monolith in finna go we had we had actually a great time it was it was interesting challenge because was our first API in seven years that was not the rails API well no it was fun but obviously he had a lot of problems with some well some people just not happy for example this email that's almost verbatim from somebody who wasn't a team like that said my overall critique of finagle is that a lot of publishing plantation exceptionally complicated for the problem domain entail a bunch of onerous transitive dependencies I'm not a native English speaker and I always had a problem with this person the like big words complicated anyway this onerous language means that you know it's too complicated too many dependencies I have to agree with the too many dependencies with I don't know I don't know if you guys share this feeling but yes there's lots of dependencies well you know we had lots of conversations and email threads going forever but one interesting thing about how we adopted or why we adopted finagle is that remember the number of programming languages and different things we had we actually trying to convince all those people to adopt a single programming language and a single to set this wasn't easy people come from different backgrounds some of them hadn't seen the JVM in 10 10 years since they left school it was really complicated and but ultimately one thing that helped us understand and actually create some commitment for the whole company to go ahead with the decision areas finagle was the vinegar model actually think the paper came after we had adopted finagle I don't want to remember but it doesn't matter because what the important thing here is that just like Mario saying this morning as well there's only three things you actually need to understand to understand finagle future service filter everything else kind of derives red yes there's lots of details yes surface discovery can be really funny yes you know there's all these different things but in the end of today if you forget these three concepts you know how to model systems you know how to build systems if you one thing that we've done is that we put aside a team of actually you have two people which saw a library steam and this dissimilar tries to encapsulate as much as the complexity as we can because a lot of our people you know it's an iOS developer who wants to do the API because she wants to access that data but she doesn't want to spend any more time on back and land and checking needs to because you know the value of her work is I can on creating the iOS interface or whatever other interfaces so if you get these three concepts and you start building your systems like the paper keeps talking about you these actually shapes your mind is it's a really interest seemed really functional if you will our way of thinking about things and turns out that it's not that hard to implement at least is in our experience for example we have this thing called the request file data it's not really called a request for later but let's call it the request relate which is every request that hits the system needs to need some checks right we need to make sure that loops we need to make sure that we have all dedicated a request we again I was saying just just before how complicated music is in each geography has a different owner so we need to know where this request coming from a knee to nose and the rate-limiting which features are available under all the circumstances and out stuff we had this kind of logic forever we always had it and as you might imagine in rails is implemented as one very very very long if and it's the one class you really don't want to touch and you have to have in our den for example only recently winter is rate-limiting so you have to go open a class first thing that happens in max crashes like open a class again maybe try him go for it find that one piece of code to add another condition to Dave and you know run integration test yes pretty sure will work and oh my god such great limitée how do you even test rate-limiting anyway eventually you get there but takes forever it may take this height down twice that's not great and you can do even though I keep bashing radios it can do exactly the same in any programming language ray and that's what a lot of people do in fact we're following the model it gave us a different different way to think about this problem which is just a pipeline rights like the UNIX way is people keep saying all the time its pipes and filters if you are into design patterns we just implemented these things as separate filters that compose in one way another and he take you to the actual actual feature actually some of this food these filters are also used in different parts of the system so we got some rules ability as well and it's great in fact lots and lots and lots of what we do this is it just implemented suit is this is just command and my IntelliJ and you know this dispute is for everything from business rules to a break secret records cause there's lots and lots and lots of stuff in here so that was good like it was an interesting side effect of choosing finagle wasn't really the reason which is infinitely but having all these plans pipes and in services spin it's been great but and then we came here because of performance and we don't have the luxury of having a lot of people with experience in back-end development or trying to we are talking there was a session earlier today where we are talking about tuning of jvm and what happened some clouds that somebody to the JVM everybody else copies what efforts that is it was irrespective watcher system does if it's the same a not just keep copying keep copy you know this person knows more than I do this happens a lot so we needed to find we needed this framework whatever the framework would be to provide good enough performance for people who were to be honest not experts and it expert some different things are not expect in you know server-side concurrency or whatnot and two guys finagle totally deliver them that this is a recent span this is totally not optimized exactly I think you can shave off at least thirty percent of what this is doing but it's a screenshot of our zip king and you can see these things talking to you ten different services has folly to spend so how can it scrub it spin furniture bars Ambrose three levels deep we try not to go too deep in our micro service act a cute three to four four steps is kind of what it is n kans to deliver 92 milliseconds written by somebody who barely knows any scholar you know who keeps growing between IntelliJ and this color school or whatever whatever skull of documentation they use so that's great but also saying there is to some some challenges challenges on this and one thing that I've learned is that I came to I came to work with SoundCloud after five years spent in the other company thinking rest I was good enough screw you like seriously rest always good enough you're doing it wrong you doing i'll get i'll get your book very good but all the PhD thesis go for it and turns out that after a while there I was like yeah not sure the first thing is I had this like this deep belief I really really there was something that was my religion and if you had a good enough api you need no client you need no client like specific to the application all you needs a HP client you follow the links lalala you done well after a while what we realized that across all over multiple different microservices we are writing the same code over and over and over you get a JSON representation from this URL we were chance on me this kind of class then we would move there and then we would send it back to the services this way we'll get these in these in this attribute because you know people don't want to do the same stuff over and over and over again things just got together created another project on github and dumped all the client calls they had so these like clients for every single service we have pretty much and some of them updated 4 months ago some of them are there you go and keep this thing still evolving so effectively we were writing client code like over and over and over so much the people got cigarette and just create a library just for that so they were maintaining a library just to run just with client code something that could be totally automated but speaking of not using HP plus reason eventually we decided to go for thrift max which we actually always here I was talking to travis and the other guys from the college team on we were still debating if we should try to implement protocol buffers on max or should go for thrift marks in the end we decided to go with reflux because we didn't have time i had this project to deliver the same excuse everybody gives and but anyway we recently migrated to to drift marks and I was saying that there was you know those 92 92 92 milliseconds could be shaved off we have some really interesting examples of this happening as we migrated services to drift and this is one of them from last week I think where are we these are two instances of the same service at the same codebase exactly same code base both of you both scholar Botha finagle same version I just like the red one is responding on HP the the green one response on first max this is CPU usage and that's when we shift from H to achieve declines from using the HP interface to the thrifty interface so you can see like massive massive massive drop in cpu usage and we had to estimate that twenty percent of our CPU time across the whole Sun cloud was actually being spent in Jason and HP and I mean we are not Facebook where if you save zero dot one percent to save I don't know how many million dollars but seriously this is a lot and actually after looking at this graph I think would be optimistic I think it's more than that I think we're wasting way more severe than we thought we were going to or we were so also saying we use we try to use as much as possible whatever is default we don't want to build infrastructure in often if restructure business we're not big enough we have a really really tough problem to tackle selves we don't need to get new problems to solve but there are some places where we don't use the defaults some places where we do something a little bit different from some other people using finagle one is service discovery where we use DNS so use sfe records so if you don't know the difference this is just a dick dump from from a bind server so this is a mem cached is right you see the a records here so this is probably the nest most people know where if you ask to resolve memcache the true dot ephemeral you get this back just AP but with SRV records like this one's if Wescott memcache d dot ephemeral you get some well it resolves to a name but it could be an NPA gas but also returns a port so with these two things obviously you have service discovery right I can find the server can I know which part I'm talking to him the decision to use sfv records were exactly made by our den infrastructure team actually was the same person who wrote the email just thought about and this so it was already there and there was no point in fighting with without infrastructure teams for you know to get anything different everybody was extremely scared of zookeeper for some reason i asked the no didn't have never run zookeeper but there is to scale so we we actually wrote some code and we are still using SIV records for indian ass for service covered actually works ok what we do is that we don't use a javeon client we had used an external client we may make this code open source at some point if helps people I don't know but we are probably going to keep doing this for at least a while we find enough we do have zookeeper now but we use it for future flags and rate-limiting not for not for service discovery another thing that we do a bit difference that soundcloud has decided to make an investment in a monitoring to none yeah telemetry to called primitives its own primitive I oh I recommend you check out is it's a building piece of technology but one thing we've done is that given the manifest that I show you before we can actually generate something like this a dashboard automatically for you for for your application and this has like a number of states code of each kind returned open connections a cheap proxy response time level the lots of interesting metrics on a dashboard like you know available to you without much effort when interesting things that we're trying to standardize these dashboards so you can have you can actually it doesn't show here but you can feel the with these dashboards and change the date range or resolution whatever you want but we try to make it so that in a very known URLs like from dash slash the name of your system there's always a dashboard like this exactly the same as for every single service this is extremely important for us to compare two different services when something Bad's happening it can get really confusing and the kind of last thing we'll do differently is we don't use Twitter server we developed in parallel I think we fought with the service to the server was made open source something very similar in terms of admin endpoints so every time you start your service he binds to a second point he binds to one port to respond to requests it binds to a separate part two for admin functions and this is very similar to an extent Watson the Twitter server the difference you have we use secret breakers heavily so there's a hour s aureus who have really really really strong about having something to control the circuit breakers like open them or close them Oh another one because we used in s and because then has details are not the nicest thing on earth another point to refresh their nests which like the emergency button you press when things go to drop or open connections this is the real emergency button is like just just forget about everything and to control feature flags in various different things / matrix is a primitive thing you can configure from meters to go to a different URL but we decided to go with vanilla configuration so we have a slash matrix thing yeah so um this another thing that we do differently but that's that's because we we don't own the life cycle of vinegar right as you we had also had conversation this morning talking about life cycle and roadmaps in different things every other two to three weeks so I don't actually know how often a new version comes up and we were having a lot of problem keeping up with the new versions even even if they're not so frequent even it's not once once a week there's a new version we had no way to actually test the thief diversion was going to break something at least like really really bad break something we had cases where applications would not start I don't know what r 0 depending on after what so the usual procedure would be we would then get to one of our applications get that application to run to FM kit for sorry run the new version of a Negra for a while it's really complicated being in Europe and having some so much of our user base in the u.s. because you know the time you're leaving the office this time you guys are waking up so we need to let it go for a bit our rule of thumb that let it run for one week see if something bad happens but we are trying to reduce this business obviously very wasteful and requires a lot of minimum work so what we are doing now we invest you more and more in just integration pipelines so this is a gem get our our internal library which depends on finagle we just keep adding more and more integration test for own stuff like our own service discovery thing choose Dennis integration primitives I want to get from em casually these things like this is probably outdated that's probably I know three or two more of those but this this is exactly paying off the investment on this now we we spending much less time worrying oh my god something going to break in this version you know somebody's application is not going to react well because we will at least covering the integration points next step weekly want tribute all of our services with this snapshot that we push here this is not that far away for us to invest in we're making kind of an interesting vestment in continuous delivery and could use deployment tools right now but I also it's totally paying off it's insane that there's not enough of good tools out of the box around this area but the more we invest on these more we get out so yeah in the end of the day not Oh not using using flag which is like using Linux right you don't control the linux kernel but for example you should be using debian and if we use in dr for a bit you probably have had a lot of problems than to you finally managed to migrate everything to jesse it's the same problem we have every time we depend on open source software going to have its problems you just have to find your way around it and find make sure you're part of the community and participate take part of it all right so i think i am secondly much quicker than i thought it was going to be so anybody has any questions